Imagine I write a computer program to simulate a horse race. Then I write another computer program to predict which simulated horse will win.
My second program does wonderfully. Predicts the winner, the finishing time, the whole enchilada.
Have I figured out how to make money at the racetrack?
Well … before you mortgage the house and head to the Off-Track Betting site, there’s a small matter we might want to investigate.
Does my simulated horse race have anything to do with actual horses?
Keep that question in mind.
In my chaotic peregrinations of the Intarwebz, I came across an article with the following headline and subhead.
Caltech-Led Team Advances AI Models to Forecast Irreversible Shifts
Caltech-led researchers take key steps in developing deep-learning models that can predict irreversible shifts in climate and other systems
It discusses a recent paper in the Proceedings of the National Academy of Sciences that bears the somewhat chewy title “Tipping point forecasting in nonstationary dynamics on function spaces.”
Here’s the TL;DR version: The authors have developed an artificial intelligence method for anticipating “tipping points”, which are abrupt changes in systems that lead to a totally new regime. Among their examples are cloud disappearance in a simplified atmospheric model and aerodynamic transitions in simulated airflow over an airfoil.
From the first section of the paper:
“While most prior works predict tipping points in simpler systems, we demonstrate that our approach works on large-scale complex problems, including forecasting a climate tipping point in cloud cover and aerodynamic instability in airfoils.”
They can forecast “a climate tipping point in cloud cover”?, sez I. Amazing if true!
And on page 3, we get this:
“Our proposed method can also forecast real-world tipping points, such as the aforementioned cloud cover transition, using only partial knowledge of the underlying physics…”
That caught my eye. “Real-world tipping points”?, sez I. Good stuff! How do they do it?
So, I read the paper and its thirty-eight pages of supplementary information, and I ended up with questions instead of answers.
Where were the real-world tests? Where were the cloud forecasts checked against measured atmospheric transitions? Or, for that matter, an airfoil in a wind tunnel?
Nowhere in the reported experiments.
Their claims of real-world forecasts turn out to be vaporware. In fact, they are using a computer program to predict what a different computer program will do. Which is not only a world away from predicting what the actual climate will do—it’s a real world away.
Now, I’m not saying this is useless. The authors’ method is clever enough to deserve an explanation.
They train a “recurrent neural operator,” which I’ll call the AI to spare both of us some syllables. It learns how a simulated system behaves before a tipping point. The AI then predicts the system’s future evolution.
How do they decide that a tipping point is approaching, without knowing the future?
They check the AI’s predictions against selected physical constraints. In the cloud example, they use a mass-conservation constraint. For the airflow example, they check whether the predicted flow satisfies the incompressible flow equations. When the discrepancy becomes sufficiently large, an alarm goes off.
In plain English, the AI has learned the old regime. When asked to forecast into an unfamiliar regime, its predictions can start coming apart at the seams. The authors use that coming apart of the AI predictions as their warning signal.
There’s an obvious complication. An AI can make a bad prediction without a physical tipping point being anywhere in sight.
The authors acknowledge this:
“…this approach makes mistakes, confusing large errors on pre-tipping regimes and actual tipping incidence.”
Indeed.
They deal with the problem by calibrating an alarm threshold using the distribution of prediction errors before tipping. That gives the threshold a statistical interpretation. Useful work. But we’ll return to the alarms shortly, because the supplement contains a rather revealing discussion of them.
First, let’s look at the supposed climate test. The cloud experiment uses five coupled ordinary differential equations. They represent boundary-layer depth, liquid-water static energy, total-water specific humidity, cloud fraction, and sea-surface temperature.
Five variables. A compact model of a cloud-topped atmospheric boundary layer.
Nothing wrong with that. Simple models can help us work out how things might operate.
But a five-variable model is not the ocean and atmosphere I can see from my kitchen window, and it doesn’t become one because we teach an AI to imitate it.
The paper itself is quite explicit:
“We use the ODE model developed in refs. 3 and 4 as an idealization of the physical climate system; this model exhibits a tipping point.”
Notice the last six words.
The model exhibits a tipping point.
The AI’s task is to anticipate that model’s tipping point, its transition into a new regime.
The authors generate 150 simulated twenty-year trajectories. Forty-five go into training, one hundred into calibration, and five into testing. All are generated by numerically integrating the cloud-model equations.
The test set contains cloud-collapse events generated by the computer model—not cloud-collapse events measured in the actual atmosphere. Its five test trajectories are simulations.
Now, someone will reasonably object that the cloud model contains observational information. Quite right. The supplement says that parameters in its empirical formulations are based on satellite observations or high-resolution simulations. Some cloud-fraction parameters are fitted to earlier data.
That deserves acknowledgment. This isn’t a model assembled entirely from moonbeams.
But fitting parts of a model to observations is not the same thing as testing the AI’s tipping forecasts against independent observations. The first can help construct a plausible simulator. The second tells us whether the forecasting method works outside the simulator.
That second test is missing. All they are testing is whether a model can predict a model.
The paper advertises a cloud-transition forecast nearly two and a half years ahead, with an error of “only 10 d.” Figure 5 gives an error of 0.03 years, roughly eleven days, at a forecast horizon of 2.45 years.
Sounds impressive. And within the experiment, it is.
But eleven days from what?
From the transition date in the computer model.
Not from the date on which an actual atmospheric cloud deck disappeared under rising CO₂. Not from a measured event independently withheld from the researchers.
The authors describe Figure 5 as showing results on “a given test trajectory.” The supplement adds comparisons using different alarm settings and another example using the full set of cloud equations as the physical constraint. Those are additional numerical tests, not additional observations of the Earth.
To circle back to where I started, my AI horse-race predictor might get the finishing time right to a hundredth of a second. That tells you how closely it agrees with my horse-race simulator.
Whether you should take it to the racetrack and bet real money on its forecasts is a very different question.
The eleven-day figure leaves out uncertainty about the cloud model itself. Does the proposed transition occur in nature as represented? At what CO₂ concentration? How does its timing change when you alter the model’s assumptions?
The AI’s agreement with the simulator does not answer those questions.
Here’s one experimental choice worth knowing.
The prescribed CO₂ trajectory starts at 400 ppm and grows at a steady 10% per year, with an added random perturbation. Ten percent annually is the chosen input to this computer experiment. So the much-advertised “2.45 years ahead” belongs to a system driven along that particular trajectory.
Does the AI retain its eleven-day accuracy under substantially different CO₂ growth rates? The supplement examines changing forcing rates in the airfoil experiments. It does not report an equivalent sweep of CO₂ growth rates for the cloud forecasts.
There’s a further problem with this scenario. It specifies that the CO₂ is increasing at 10% per year. Having worked as an accountant in my checkered past, I’m familiar with what’s called “The Rule Of 70”. It states that doubling time in years for compound interest is roughly equal to seventy divided by the interest rate.
So if CO₂ in modelworld is increasing by ten percent per year, doubling time is about seven years … but out here in the real world, CO₂ in 1850 was about 285 ppmv … and now, it’s about 430 ppmv. So in the 175 years since 1850, CO₂ still hasn’t doubled.
Figure 1. CO₂ levels from nice ice cores and from the Mauna Loa Observatory data, 1850 – 2023
Their model starts at 400 ppmv, and the test runs last for 20 years. So that means when the test ends, the model has its atmospheric CO₂ level at 2,691 ppmv … far beyond what we’d have if we burned every scrap of fossil fuels in the ground.
“Real-world” forecasts, my okole …
Then there’s the ocean.
This model has a slab ocean one metre deep. The authors call the depth arbitrary and say it changes the equilibration time but not the equilibrium results. They justify the choice because the forcing is slow relative to their approximately fifty-day equilibration timescale. Ocean heat uptake is set to obtain a sea-surface temperature of 290 K at 400 ppm CO₂ and is then held constant.
That may be a perfectly serviceable arrangement for studying the equations.
But hang on. If the advertised achievement is predicting when something happens, a choice that affects adjustment time is worth investigating.
I’m not claiming that a deeper slab ocean would necessarily wreck their forecast. They haven’t shown that, and neither have I.
I’m saying that eleven-day agreement within this setup should not be mistaken for eleven-day accuracy in the real climate. The timing has been tested with these assumptions in place. We do not get to erase the assumptions when announcing the result.
But that was all just for openers. This was the passage in the supplement that made me stop and scratch my head:
“We note that by construction, it is difficult to avoid predicting a tipping point for any significance level α larger than the minimum allowable value when the evaluation trajectory is sufficiently long.”
Read that again.
For a sufficiently long time horizon, the system generates a tipping-point prediction except at the most conservative allowable setting.
That’s quite a qualification for an early-warning system.
A smoke detector that correctly announces a fire is useful. Before installing it, though, I’d also like to know how often it will wake me at three in the morning because it has developed philosophical doubts about the toaster.
The statistical terminology does not make that practical problem disappear. A limit on an individual threshold-exceedance probability won’t stop the system from giving a false alarm somewhere during years of repeated operation. The supplement’s own warning makes clear that we should not read the paper’s “rigorous measure of uncertainty” as an all-purpose warranty.
The supplement offers this advice:
“When representative trajectories with known tipping events are available, practitioners may therefore select alpha empirically to achieve the desired operating characteristics prior to deployment.”
Fair enough. If you have known events, use them to help choose a useful alarm setting.
Except that the main selling point of this method is its claimed ability to train without actual tipping data from real-world observations. The motivation for developing the method is the lack of any available historical record of a cloud-collapse tipping point.
There’s no contradiction in the narrow technical sense. Training the AI and choosing an operational alarm threshold are different jobs.
But the practical difficulty remains. We have never observed a climate tipping point of the type simulated in their model. If you don’t have representative physical events, how do you know which setting gives useful advance warning without an unacceptable procession of false alarms?
Next, there’s a further point worth keeping straight.
The alarm comes from a forecast violating a selected physical constraint. The actual atmosphere does not stop conserving mass when clouds change. Nor does incompressible airflow abandon its divergence-free condition when an airfoil stalls.
The AI’s forecast is the thing coming unglued.
That can be informative. It isn’t a unique fingerprint of a tipping point.
The authors test some ways their method might go wrong. Useful work. But those computer experiments don’t establish that it can distinguish an approaching real-world tipping point from a bad forecast caused by missing physics, inaccurate observations, or clouds that the model handles poorly.
Nor does obeying one physical rule make a forecast correct. You can account for every last gram of water and still get the cloud cover hopelessly wrong. Passing the bookkeeping test doesn’t mean you’ve understood the weather.
So yes, the method gives useful warning signals in these computer experiments. Whether those signals reliably warn of tipping points in the actual climate still needs testing.
So … what would persuade me that this approach has some value?
Actual validation.
Train the method on measurements from a physical system before a transition. Fix the forecasting procedure and alarm settings. Then test it on independent physical cases.
Tell us how early the warnings arrive. Show the misses. Count false alarms over long non-tipping periods. Report the timing errors across cases, rather than letting one attractive trajectory do the advertising.
For the climate application, test sensitivity to the forcing history, ocean representation, cloud parameterizations, and observation limitations. Training on one model formulation and testing on another would also be informative—although even that would still be a model-to-model test.
The authors themselves acknowledge that sparse or low-resolution data can prevent the AI from learning the underlying evolution properly. They warn that unresolved physics can make the learning problem ill-posed.
Yes. Exactly. Those difficulties are part of the real-world problem, not inconveniences we can waive because the simulator supplies cleaner data.
And there is an even deeper problem.
We have very few examples of climate “tipping points”, times when some threshold is exceeded that leads to an entirely new system. For example, the low-cloud “tipping point” they investigate in this study has never been observed. This makes it extraordinarily difficult to even test such an AI method for predicting climate “tipping points”.
It’s true that laboratory systems and physical aerodynamic experiments offer opportunities to test the proposed method against some aspects of nature. But until such tests are done, the claims need to stay within the evidence.
Unfortunately, regarding the real climate, lab systems and aerodynamic experiments don’t help. We simply don’t have the examples we need to test the system.
To be clear, this work may prove useful. It may become a fast way to investigate transitions in expensive simulations. It may even eventually work with observations. The authors have supplied code, data, architectural comparisons, and a supplement that admits several limitations. All of that is welcome.
But “may eventually” is doing work that “can forecast real-world tipping points” has quietly dismissed.
They have taught one computer program to anticipate another computer program’s behavior.
Before we use that result to tell us what the actual climate will do, there’s one more participant we need to consult.
The real-world Earth.
[CODA]: After I wrote all of the above and was preparing to post it to WUWT, I realized a further untouched problem with their method. The AI analyzes a simple climate model on computer-generated test data, and when the AI becomes unglued from physical reality, it says it’s because there was a “tipping point” in the data.
The problem is that actual modern climate models get unglued from physical reality all the time. Here’s an excerpt from my post “Meandering Through A Climate Muddle“. In that post, I looked at the code for one of the more prominent climate models, the Goddard Institute for Space Studies (GISS) Model E. In the code comments, one or more of the programmers spell out exactly where it becomes unglued, and how that is handled. Spoiler: It doesn’t assume it’s a tipping point … here are some examples, starting with the issue of how to determine the albedo of sea ice:
What this means is that there is code to calculate the albedo of the sea ice … but sometimes that code comes up with unrealistic values. But rather than figure out why the code is coming up with bad values and fixing it, the climate model just replaces the bad value with the corresponding maximum or minimum values. Science at its finest.
Here’s a comment describing another bit of melt pond fun:
C**** safety valve to ensure that melt ponds eventually disappear (Ti<-10) if (Ti1 .lt.-10.) pond_melt(i,j)=0. ! refreeze Without this bit of code, some of the melt ponds might never refreeze, no matter how cold it got … gotta love that kind of physics, water that doesn’t freeze.
This is what the climate modelers mean when they say that their model is “physics-based“. They mean it in the same sense as when the producers say a Hollywood movie is “based on a true story” …
Here, for example, is a great comment from the ModelE code (the “c” or the “!” in a line indicates a comment):
!@sum tcheck checks for reasonable temperatures
!@auth Ye Cheng/G. Hartke
!@ver 1.0
c ----------------------------------------------------------------------
c This routine makes sure that the temperature remains within
c reasonable bounds during the initialization process. (Sometimes the
c the computed temperature iterated out in left field someplace,
c *way* outside any reasonable range.) This routine keeps the temp
c between the maximum and minimum of the boundary temperatures.
c ----------------------------------------------------------------------In other words, when the temperature goes off the rails … don’t investigate why and fix it. Just set it to a reasonable temperature and keep rolling.
And what is a reasonable temperature? Turns out they just set it to the temperature of the previous timestep and keep on keeping on … physics, you know.
Here’s another:
c ucheck makes sure that the winds remain within reasonable
c bounds during the initialization process. (Sometimes the computed
c wind speed iterated out in left field someplace, *way* outside
c any reasonable range.) Tests and corrects both direction and
c magnitude of the wind rotation with altitude. Tests the total
c wind speed via comparison to similarity theory. Note that it
c works from the top down so that it can assume that at level (i),
c level (i+1) displays reasonable behavior.… when the climate model goes off the rails, and the wind is blowing five hundred miles per hour, don’t look for the reason why. Just prop it up, put it back on the rails, and keep going …
As you can see, the fact that the model comes unglued from physical reality is solved by pushing it back onto the rails, not by declaring a “tipping point”.
I also realized I do have a great graphic of tipping points. Here are a couple thousand runs for the climateprediction.net climate model. The historical phases use actual post-1850 historical data as inputs to the model.
Figure 2. Calibration, model years 1–15: calculate the ocean heat-transport corrections for the simplified ocean. Control, model years 16–30: run with preindustrial CO₂ fixed at 280 ppm. Doubled CO₂, model years 31–45: run with CO₂ fixed at 560 ppm following an instantaneous doubling.
What is of interest is that despite stable control run CO2, a very large number of them went through a “tipping point” somewhere in the control runs and apparently in all cases tipped into some imaginary ice age rather than Thermageddon™ … and that clearly never happened in recent times.
In any case … it’s models all the way down, and Tinkertoy™ models at that …
Best to all on a sunny October afternoon,
w.
