Well with "AI" all we're talking about is theory-free frequentist modelling.
We can already do that for the weather: just take the mode of the last 10 years of the weather on this day, and some sort of weighted mean for temp/humidity/etc.
All "AI" is doing here is overfitting this process. On the overfit range, yes, you'll get better peformance. But the baseline performance here, from taking averages, is already pretty good.
What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.
This is simply not true and an uninformed opinion of what modeling physical events with AI looks like.
You seem to assume it’s a purely data-driven approach, it is not. You could use a physics informed neural network (PINN) that utilizes both approaches. That is, it uses both historical data (and likely synthetic data from physics models), as well as physical equations in the loss function (in this case atmospheric as fluid equations) as part of the training. It can truly be the best of both worlds if approached correctly.
That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems.
Additionally, you can generate synthetic data with the physical models of “edge” and “tail” events to train the model on. This by itself allows the ML model to be able to model almost all events that we can physically model, so at its base it’s at least as useful as the big O order models we use while being orders of magnitude faster. This doesn’t even account for using the physical equations to assist in training the model directly (through architecture tricks or in the loss function).
Source: I work on AI models that merge data and physics for dynamical physical systems
> That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems.
I think one surprising outcome of the recent wave of ML-based weather forecasting models is just how accurate the "dumb" approaches are. Pangu-Weather and a couple of its peers are essentially vision transformers without much explicit physics-informed regularization.
If you have explanatory models constraining the space of possible function fits, etc. etc. then I concede the point -- though, I rather regard it as my point.
The comment I replied to used "AI" in its generic sense which I take to name the theory-free frequentist stats currently in vogue. I don't regard theories as AI -- so adding physics to a NN is, in large part, computational physics. You can call it "AI", but then so-goes any use of a computer model of any kind.
Well, the difference is the data-driven aspect of parts of the model. While its constrained by physics during the learning process it isn't just running a forward physics model to get the solution. The upfront computational load and extremely fast inference times through parameterization IS what makes it AI, and what makes it useful versus a normal numerical computer model.
Physics has used "empirical/phenomenological models" where curve-fitting to data has served to preclude the need for simulation, or if it's computationally intractable, etc. I'd agree that it had been underused, since I'd say such modelling is held somewhat in contempt as giving up on doing physics.
Do you have a paper that discusses any of this work in these terms? I'm presently writing a larger survey on XAI towards a theory-informed approach, and it seems these mixed models might have some novel explanatory upside/needs. At the moment i'm inclined to partition the world into theory-based and theory-free.
Although it's obviously difficult to crack open an ML model, they do perform enough computation to have potentially learned something like the dynamical equations for the atmosphere.
At the same time, some ML models are surprisingly parsimonious. Graphcast has about 37 million trainable parameters, but its output is a forecast (increment) of six variables on a 37-level, quarter-degree lat/lon grid. That's about 235 million outputs for a single forecast date, so it's safe to conclude that Graphcast cannot memorize its training set.
Researchers are also explicitly interested in probing the out-of-sample behaviour of ML models for plausibility. A paper last year by Hakim and Masanam (https://arxiv.org/abs/2309.10867) put Pangu-Weather through some simplified but out-of-sample test cases and saw physically plausible outputs, so the ML models have at least not fallen at the first hurdle.
Meanwhile, it's also not quite correct to give traditional models an automatic pass for out-of-sample behaviour. The large-scale dyanmical equations of the atmosphere are well-understood, but so much of the chaos comes from poorly-resolved, poorly-modeled, or poorly-constrained processes near and below the grid scale. The microstructure of clouds, for example, is completely invisible to models that must necessarily run at kilometer or tens-of-kilometer scales. Operational weather models rely on parameterizations to close the system and statistical correlations to assimilate observational data.
As far as I'm aware, all of the operational weather models missed the rapid intensification of hurricane Otis last year, an out-of-sample event with deadly consequences.
There really isn't anything to crack open. The models are curves fit to data, the units of the weights are whatever the units of the data are... so, eg., if fit to temp data, then temp.
If you draw a line through measurement data of one kind, you arent getting a function of another: a function is just a map within this space.
Why it should be that drawing a line around shadows is a good prediction for future shadows isn't very mysterious -- no more and no less regardless of the complexity of the object. There isn't anything in the model here which explains why this process works: it works because the light casts shadows in the future the same way it does in the past. If the objects changed, or the light, the whole thing falls over.
Likewise, "generalization" as used in the ML literature is pretty meaningless. It has never hitherto been important that a model 'generalizes' to the same distribution. In science it would be regarded as ridiculous that it could even fail to.
The science sense of generalisation was concerned with whether the model generalizes across scenarios where the relevant essential properties of the target system generated novel distributions in the measure domain. Ie., the purpose of generalization was explanation -- not some weird BS about models remembering data. It's a given that we can always replay-with-variation some measurement data. The point is to learn the DGP>
No explanatory model can "remember" data, since if it could, it would be unfalsifable. Ie., any model build from fitting to historical cases can never fail to model the data, and hence can never express a theory about its generation.
> weather models missed the rapid intensification of hurricane Otis last year
Which happened because there was very little data to feed into the models. AI isn't going to help with this. The Atlantic Ocean and Gulf of Mexico have tons of data-collecting bouys and the Hurricane Hunter aircraft fly from the eastern US. Hurricane Hunters that go to the Pacific fly out of Mississippi, which adds quite a lot of latency to the data collection probes.
We should be adding more bouys to the Pacific, and need to add a Hurricane Hunter crew in San Diego (or perhaps the government of Mexico would like to host and pay for them).
Then we can start seeing what the models and AI will do.
I'm not up to date on the latest literature re: the Otis miss. Is the conventional thought that the ocean was in fact warmer than the models supposed, either at the surface or with a warmer upper-mixed layer?
If the problem was lack of constraint from data, this is still fixable in a probabilistic sense: we'd "just" (noting it's not that simple) need to assume more variability in ocean conditions in data-poor regions.
> What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.
Yes, it seems like weather forecasting is a simulation problem not a low-shot prediction problem. I assume it's one of the computationally irreducible problems Stephen Wolfram talks about
We can already do that for the weather: just take the mode of the last 10 years of the weather on this day, and some sort of weighted mean for temp/humidity/etc.
All "AI" is doing here is overfitting this process. On the overfit range, yes, you'll get better peformance. But the baseline performance here, from taking averages, is already pretty good.
What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.