Guest essay by Eric Worrall
h/t Janice – This video dates back to July, so it might not be news for some viewers. But the video elucidates in clear and simple terms why climate model error is actually far worse than those pretty spreads provided by the IPCC. I thoroughly recommend watching the video to anyone keen to understand why climate models are so bad at prediction.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
I’m sympathetic to this sort of analysis, but I think the error bar chart could be misleading. If the error due to cloud cover in any year is random within plus or minus 4 watts, then the probability of an error of + 4 watts every year is extremely small. Smaller net errors over the years will be more likely than large ones, as negative errors in some years cancel out (to some extent) positive errors in others. It would be helpful to see a chart with the probability of a cumulative error (in degrees centigrade) as a function of the size of the error.
David, the ±4 W/m^2 statistic represents a systematic error not a random error. It’s the annual average of error for the tested CMIP5 models. The propagated uncertainties are representative. An uncertainty statistic is not error, which is a physical magnitude.
Physical errors combine as their sign. However, physical errors are not known in a futures projection, so all one has to gauge predictive reliability is the uncertainty from propagated known errors.
“David, the ±4 W/m^2 statistic represents a systematic error not a random error. It’s the annual average of error for the tested CMIP5 models.”
That is the basic confusion here. If it’s systematic, then why is it ± ?
In fact, I can’t see such an error quoted by Lauer and Hamilton, and I don’t think they could, since the uncertainty of the observations is ±5-10 W m⁻². What they say is:
“For CMIP5, the correlation of the multimodel mean LCF is 0.93 (rmse ± 4 W m⁻²) and ranges between 0.70 and 0.92 (rmse ± 4–11 W m⁻²) for the individual models. “
It seems to be the rmse (root mean square error) of a correlation, not a systematic error.
Nick, “That is the basic confusion here. If it’s systematic, then why is it ± ?”
Because it’s the root-mean-square statistic of the annual average of individual model systematic error.
“It seems to be the rmse (root mean square error) of a correlation, not a systematic error.”
See their equation (1), page 3831, where they describe the error calculation: “A measure of the performance of the CMIP model ensemble in reproducing observed mean cloud properties is obtained by calculating the differences in modeled (x_mod) and observed (x_obs) 20-yr means. These differences are then averaged over all N models in the CMIP3 or CMIP5 ensemble to calculate the multimodel ensemble mean bias Δmm, which is defined at each grid point as Δmm = (1/N)[sum over N of (x_mod minus x_obs)] (1).”
With regard to the Taylor diagrams of error, they write that, “the linear distance between the observations and each model is proportional to the root-mean-square error (rmse)…”
Lauer and Hamilton also note that long- and shortwave cloud forcing errors are much smaller than the errors in total cloud amount or in liquid water path (unit area mass of liquid water droplets in the atmosphere). They ascribe the lower error to modelers focusing their model tuning to minimize the cloud forcing errors, so as to attain top of the atmosphere energy balance.
That rather makes the ±4 Wm^-2 a lower limit of a lower limit of model tropospheric thermal flux error.
“See their equation (1), page 3831”
That is an equation for bias. And it is simply added. No RMS.
“With regard to the Taylor diagrams of error, they write that, “the linear distance between the observations and each model is proportional to the root-mean-square error (rmse)…””
That quote refers to ‘the standard deviation and linear correlation with satellite observations of the total spatial variability calculated from 20-yr annual means. ‘
They are talking about the (spatial) sd of each model wrt its own mean, and correlation (not difference) with satellite. There is no “systematic error” there, and no time sequence as you have it. And that is what the 4 W m⁻² comes from. It is a measure of spatial variability.
Nick, calling your attention to eqn. (1) was just meant to show that Lauer and Hamilton calculated the difference between observed and modeled cloud properties. Not between model and model. The multi-model annual error statistic is calculated as the rms error of all the individual model errors relative to observations.
It is the inter-model cloud error correlation matrix I showed that demonstrated that the cloud error is systematic. Systematic error propagates through a serial calculation as the root-sum-square of the errors in each step.
I wrote, ““With regard to the Taylor diagrams of error, they write that, “the linear distance between the observations and each model is proportional to the root-mean-square error (rmse)…”
To which you replied, “That quote refers to ‘the standard deviation and linear correlation with satellite observations of the total spatial variability calculated from 20-yr annual means.”
No, it does not. Here’s what L&H say about the diagrams (p. 3833): “The overall comparisons of the annual mean cloud properties with observations are summarized for individual models and for the ensemble means by the Taylor diagrams for CA, LWP, SCF, and LCF shown in Fig. 3. (bold added)”
That makes it pretty clear that the Taylor diagrams represent model error.
They go on to write, “ These give the standard deviation and linear correlation with satellite observations of the total spatial variability calculated from 20-yr annual means.” indicating that the correlations are between model simulation and satellite observations; not correlations between models, nor individual models vs. model mean.
You wrote, “They are talking about the (spatial) sd of each model wrt its own mean, and correlation (not difference) with satellite.”
Given the quote from L&H above, they are doing no such thing.
You wrote, “There is no “systematic error” there, and no time sequence as you have it. And that is what the 4 W m⁻² comes from. It is a measure of spatial variability.”
Here’s what Lauer and Hamilton say on page 3831 about how they calculate cloud forcing (CF), “The CF is defined as the difference between ToA all-sky and clear-sky outgoing radiation in the solar spectral range (SCF) [shortwave cloud forcing] and in the thermal spectral range (LCF) [longwave cloud forcing]. A negative CF corresponds to an energy loss and a cooling effect, and a positive CF corresponds to an energy gain and a warming effect.”
From that, it is very clear that the ±4 Wm^-2 is not a measure of spatial variability.
The LCF is a thermal energy flux, and the ±4 Wm^-2 rmse of the simulated LCF is the mean model simulation error statistic in that thermal energy flux. It represents the mean annual simulation uncertainty in the tropospheric thermal energy flux, of which energy flux CO2 forcing is a part. That was how I used it in my error propagation.
Dr. Frank does an excellent job of presenting the statistical situation, without losing the audience with the fine detail that only the well-versed statistician would understand. In other words, the perfect level for the “informed” layman. I would really like to see more analyses at this sort of level, of all the impressive sounding stats that are bandied about by the climate elite. The 97% consensus has been well covered, but how about all these claims of 95% certain etc. that spew forth from the IPCC and get reported ad nauseum by our ignorant activist politicians and MSM as carrying some sort of great weight, simply because they sound near enough to 100%.
A major problem is that across the board (almost) government research money has gone exclusively to “scientists” dedicated to the mission of proving that AGW exists and is a major threat to mankind. Any deviation from that conclusion results in a cut off of funds and blackballing by the “peer review” process.
An apt analogy would be the Joyce Foundations history of funding gun safety “studies” with over 50 million $ provided exclusively to researchers committed in advance to anti-gun conclusions.
Another would be the cosmology community with everyone arguing that 99.4 % of the universe consists of “dark matter” which cannot be seen, measured or even proven to exist – now or literally ever. The “proof” is a hundred layer deep accumulation of mathematical scab patches to an originally empirically false theory.
If the theory doesn’t fit reality the proper next step is to adjust the theory, not adjust reality. Just a layman’s view, but the official government agencies “correction” of past temperature records is proof that at least the upper management of the agencies are nothing more than political hacks.
Dr. Frank – your cloud error analysis is based on an annual cloud error. Since the climate models are attempting to predict global temperatures out dozens of years would it make sense to do a cloud error budget based on a 5 or 10 year moving average of the cloud errors? If so how would that impact your conclusions?
Mike, the annual cloud error is the average from 20 years of calibration runs of 26 CMIP5 models. So, it really is the annual mean of a multi-year, multi-model set of results. Is that pretty much what you had in mind?
Yes thanks for clarifying
Pat Frank provided a very educational blog post and supported the discussion of it throughout the comment period so far. I am very appreciative of those commentators that do that give the amount of their time it requires. During the comment period on WUWT is where I learn the most about topics and usually make up my mind about the blog post. Although I have some background in statistics, error and reliability engineering, I have not used the background very much since college. This discussion brought back memories of long forgotten classes. I also took the time to look at references and read more on the topic of error propagation. My opinion after doing that is Dr. Frank won the argument with Nick Stokes. For what it is worth, I believe Dr. Frank is correct. The IPCC climate models are not useful in predicting future climate.
“Ensemble Average” : if I have two models, one totally wrong and the other fairly good, but I don’t know which is which. It does not seem to make any sense to calculate an Ensemble Average of this set of models. An it makes even less sense to use this average as a “consensus predictor”.
Hansen predictions of 1988 were fairly well within the ±14 °C error margin; this doesn’t validate his model.
Willie Soon alerted a number of people about the November 9 Carbon Brief (Clear on Climate) website post, presenting a series of interview videos and comment of climate scientists, bemoaning the election of Donald Trump: “US election: Climate scientists react to Donald Trump’s victory. There’s a lot of upset, anger, and anguish.
I posted a short comment and a link to the head-post presentation. Christopher Monckton quickly became embroiled in debate there, and, well, so did I.
But I discovered the comments were closed before the debate was resolved. So, if the moderator allows, I’d like to post a couple of replies here. Perhaps the pingback will bring the debaters here.
BBD you wrote, “You have jumped from the topic (global warming trend) to a non-topic (regional climate effects) invalidating your (but not my) argument. I never made any claims about the regional predictive skill of the models as that was not the focus of discussion. Don’t play crude rhetorical games, please.”
Apparently you don’t realize that localized precipitation is the mechanism for global heat transfer across the top of the atmosphere; a critical control element of the tropospheric thermal content driving air temperature. “crude rhetorical games” indeed. Merely central to your knowledge claim concerning AGW.
But let’s cut to the chase, shall we?
You claim your “greenhouse effect theory” explains the effect of CO2 emissions on climate.
For example, “As I keep telling you, this is about greenhouse effect theory not some ‘theory of climate’.” Except that the CO2 greenhouse effect on climate requires knowing whether there are any negative feedbacks. I.e., requires a theory of climate.
You do seem to have a problem distinguishing scientific knowledge from thus spake BBD.
You say your theory consists of radiation physics. Perhaps you consciously include the assumption of constant relative humidity. But that’s just the explicit part.
You apparently don’t realize there’s an implicit part of your vaunted “greenhouse effect theory,” that you’ve left unstated.
You assume no compensatory changes (negative feedbacks) in cloud cover or precipitation or IR radiative egress.
So, let’s itemize. Your “greenhouse effect theory” consists of radiation physics plus three assumptions left unstated.
That’s your “greenhouse effect theory.” It is a cryptic theory of climate, with hidden claims about the behavior of clouds, of precipitation, and about the rate of thermal energy flux through the top of the atmosphere.
The fact of your cryptic theory proves that a theory of climate is necessary to the full meaning of CO2 emissions on the climate; something you’ve repeatedly denied all the while implicitly hewing to it.
That you deploy a theory of climate, but do not yourself realize it, tells us all we need to know about your grasp of science.
Maybe seeing your implicit claims itemized will finally convince you of the scientific fatuity of your idea that radiation physics alone is a valid theory of the greenhouse effect; as though all other other parts of the climate were stationary.
Your position is an obvious crock, BBD. I suggest you follow up your own determination to “not be [it] saying again.” Such an ignorant display is embarrassing, even in a debate adversary.
You wrote, “Since sensitivity is an emergent property of model physics and not parameterised…” in the face of F.A.-M. Bender (2008) A note on the effect of GCM tuning on climate sensitivity Environmental Research Letters 3(1), 014001, from the abstract: “[This] study illustrates that the climate sensitivity is a product of choices of parameter values that are not well restricted by observations…” and R. Knutti, et al., (2008) Why are climate models reproducing the observed global surface warming so well? Geophys. Res. Lett. 35, L18704, p. 3 “Changes in model parameters (e.g., cloud microphysics) may result in compensating effects in climate sensitivity…”
After which you wrote, “You are a way outside your field of expertise…” One hopes your inadvertent irony is apparent even to you, BBD.
You’ve never grasped, or perhaps avoided, that my argument is about physical error analysis not climate physics.
You wrote, “Your entire argument collapses on the logical fallacy of appeal to your own (non-existent) authority and that is where it stops.” Wrong again, BBD. I’ve rested my argument on the scientific merits. Merits, let’s note, that you have conspicuously failed to address. You have yet to mount one single analytical point of objection. You’ve offered nothing but vacuous dismissals, denials, and derogations.
You wrote, “There’s no evidence that there was anything even approximating to a formal review process [to my Skeptic article]. Are you prepared to post all reviewers’ comments here?”
You’re welcome to contact Michael Shermer and ask about his process. You’ll find a direct debate about the article with Gavin Schmidt of NASA GISS here. Search my name. If you read to the end, you’ll find that I carried the debate. Gavin was reduced to accusing me falsely of a log(0) error that does not exist in my work.
I have posted on the quality of my reviewers’ comments here. Help yourself. None of them have indicated any understanding of the meaning of uncertainty derived from physical error. One of the Skeptic article reviewers distinguished himself by accusing me of scientific misconduct. You can read all about that false charge in the article SI (892 kb pdf).
You wrote, “Your (false) claim was that models alone were the source of our knowledge of the effects of CO2 forcing on climate.” And so they are, deploying as they do the relevant climate physical theory.
The above analysis shows that you, too, ‘claim that models alone [are] the source of our knowledge of the effects of CO2 forcing on climate,’ because you yourself deployed an implicit climate model. Except that you didn’t know it. By now you should have figured that out, though I doubt you’ll ever permit that understanding.
You wrote, “To be specific (because you have since tried to obfuscate the point) [the effects of CO2 forcing on climate] meant the effect of CO2 on GAT on centennial timescales. I pointed out that this was evident from palaeoclimate behaviour and that you were wrong.”
On the contrary, BBD. You never met the challenge that the resolution of the PETM data cannot support your claim. You merely assert it. Bald assertion is all you’ve done. Bald assertion is no proof. It’s just more thus spake BBD.
You also wrote, “You still are and now you are lying about it.”
I’ve lied about nothing. Once again you assert baldly and without evidence. And let everyone see your resort to character assassination when you cannot argue the evidence.
You wrote, “You can’t tell that WUWT is bullshit and most of CA is wrong? You are beyond help then.”
Yet another claim for which you’ve provided zero evidence. Just yet more of your thus spake BBD positivity.
Lionel Smith, your dismissal has no substantive content. You’ve merely accepted the mistaken claims of contradiction at Tamino’s “Frankly Not” at face value, with no evidence that you understand the argument. Or that there was no contradiction, as I showed.
You have every right to accept an infallible AGW priesthood, but don’t expect to get very far with it in a debate about science.
No one at Tamino’s “Open Mind” (irony alert), ever figured out that the cosine analysis of the global air temperature record was supported by an observed cosine residual in (T_land-surface minus T_sea-surface). Neither have you.
You “suggested reading Bradley wherein is described the varied methods of dendrochronology data collection and research [which] should answer your issue with my, supposedly, not providing the ‘physical theory Bradley uses to get temperature’”
No supposedly about it. You provided no physical theory linking tree rings and temperature. Neither has Raymond Bradley.
I fully understand that Bradley’s paleo-temperature reconstructions strictly employ statistics alone. They are based on no physical theory. The “temperature” numbers he elicits therefore have no physical meaning.
You reject that obvious conclusion. Faith in your priesthood again.
You wrote, “The comment thread at Real Climate ‘What the IPCC models really say’ is replete with similar criticisms of lack of coherence in your arguments.”
Criticisms I showed were incorrect. But you apparently passed over my demonstrations.
Gavin finally supposed I made log(0) error. That was his only remaining criticism. However, the regression stopped at log(1) = 0, something Gavin apparently never figured out. You merely quoting a series of replies supporting Gavin’s incorrect claim is no rejoinder.
The rest of your post supposes that AGW is causal to the current turmoil in the middle east. There’s as much evidence for that as there is of human-caused global warming itself.
Almost forgot — thanks, very much Mod, for allowing those responses. 🙂 I truly do appreciate it and am grateful.
Pat
Thank you Pat Frank.
Thanks right back, Roy. Your positive interest is appreciated.