Guest essay by Pat Frank
For going on two years now, I’ve been trying to publish a manuscript that critically assesses the reliability of climate model projections. The manuscript has been submitted twice and rejected twice from two leading climate journals, for a total of four rejections. All on the advice of nine of ten reviewers. More on that below.
The analysis propagates climate model error through global air temperature projections, using a formalized version of the “passive warming model” (PWM) GCM emulator reported in my 2008 Skeptic article. Propagation of error through a GCM temperature projection reveals its predictive reliability.
Those interested can consult the invited poster (2.9 MB pdf) I presented at the 2013 AGU Fall Meeting in San Francisco. Error propagation is a standard way to assess the reliability of an experimental result or a model prediction. However, climate models are never assessed this way.
Here’s an illustration: the Figure below shows what happens when the average ±4 Wm-2 long-wave cloud forcing error of CMIP5 climate models [1], is propagated through a couple of Community Climate System Model 4 (CCSM4) global air temperature projections.
CCSM4 is a CMIP5-level climate model from NCAR, where Kevin Trenberth works, and was used in the IPCC AR5 of 2013. Judy Curry wrote about it here.
In panel a, the points show the CCSM4 anomaly projections of the AR5 Representative Concentration Pathways (RCP) 6.0 (green) and 8.5 (blue). The lines are the PWM emulations of the CCSM4 projections, made using the standard RCP forcings from Meinshausen. [2] The CCSM4 RCP forcings may not be identical to the Meinhausen RCP forcings. The shaded areas are the range of projections across all AR5 models (see AR5 Figure TS.15). The CCSM4 projections are in the upper range.
In panel b, the lines are the same two CCSM4 RCP projections. But now the shaded areas are the uncertainty envelopes resulting when ±4 Wm-2 CMIP5 long wave cloud forcing error is propagated through the projections in annual steps.
The uncertainty is so large because ±4 W m-2 of annual long wave cloud forcing error is ±114´ larger than the annual average 0.035 Wm-2 forcing increase of GHG emissions since 1979. Typical error bars for CMIP5 climate model projections are about ±14 C after 100 years and ±18 C after 150 years.
It’s immediately clear that climate models are unable to resolve any thermal effect of greenhouse gas emissions or tell us anything about future air temperatures. It’s impossible that climate models can ever have resolved an anthropogenic greenhouse signal; not now nor at any time in the past.
Propagation of errors through a calculation is a simple idea. It’s logically obvious. It’s critically important. It gets pounded into every single freshman physics, chemistry, and engineering student.
And it has escaped the grasp of every single Ph.D. climate modeler I have encountered, in conversation or in review.
That brings me to the reason I’m writing here. My manuscript has been rejected four times; twice each from two high-ranking climate journals. I have responded to a total of ten reviews.
Nine of the ten reviews were clearly written by climate modelers, were uniformly negative, and recommended rejection. One reviewer was clearly not a climate modeler. That one recommended publication.
I’ve had my share of scientific debates. A couple of them not entirely amiable. My research (with colleagues) has over-thrown four ‘ruling paradigms,’ and so I’m familiar with how scientists behave when they’re challenged. None of that prepared me for the standards at play in climate science.
I’ll start with the conclusion, and follow on with the supporting evidence: never, in all my experience with peer-reviewed publishing, have I ever encountered such incompetence in a reviewer. Much less incompetence evidently common to a class of reviewers.
The shocking lack of competence I encountered made public exposure a civic corrective good.
Physical error analysis is critical to all of science, especially experimental physical science. It is not too much to call it central.
Result ± error tells what one knows. If the error is larger than the result, one doesn’t know anything. Geoff Sherrington has been eloquent about the hazards and trickiness of experimental error.
All of the physical sciences hew to these standards. Physical scientists are bound by them.
Climate modelers do not and by their lights are not.
I will give examples of all of the following concerning climate modelers:
- They neither respect nor understand the distinction between accuracy and precision.
- They understand nothing of the meaning or method of propagated error.
- They think physical error bars mean the model itself is oscillating between the uncertainty extremes. (I kid you not.)
- They don’t understand the meaning of physical error.
- They don’t understand the importance of a unique result.
Bottom line? Climate modelers are not scientists. Climate modeling is not a branch of physical science. Climate modelers are unequipped to evaluate the physical reliability of their own models.
The incredibleness that follows is verbatim reviewer transcript; quoted in italics. Every idea below is presented as the reviewer meant it. No quotes are contextually deprived, and none has been truncated into something different than the reviewer meant.
And keep in mind that these are arguments that certain editors of certain high-ranking climate journals found persuasive.
1. Accuracy vs. Precision
The distinction between accuracy and precision is central to the argument presented in the manuscript, and is defined right in the Introduction.
The accuracy of a model is the difference between its predictions and the corresponding observations.
The precision of a model is the variance of its predictions, without reference to observations.
Physical evaluation of a model requires an accuracy metric.
There is nothing more basic to science itself than the critical distinction of accuracy from precision.
Here’s what climate modelers say:
“Too much of this paper consists of philosophical rants (e.g., accuracy vs. precision) …”
“[T]he author thinks that a probability distribution function (pdf) only provides information about precision and it cannot give any information about accuracy. This is wrong, and if this were true, the statisticians could resign.”
“The best way to test the errors of the GCMs is to run numerical experiments to sample the predicted effects of different parameters…”
“The author is simply asserting that uncertainties in published estimates [i.e., model precision – P] are not ‘physically valid’ [i.e., not accuracy – P]- an opinion that is not widely shared.”
Not widely shared among climate modelers, anyway.
The first reviewer actually scorned the distinction between accuracy and precision. This, from a supposed scientist.
The remainder are alternative declarations that model variance, i.e., precision, = physical accuracy.
The accuracy-precision difference was extensively documented to relevant literature in the manuscript, e.g., [3, 4].
The reviewers ignored that literature. The final reviewer dismissed it as mere assertion.
Every climate modeler reviewer who addressed the precision-accuracy question similarly failed to grasp it. I have yet to encounter one who understands it.
2. No understanding of propagated error
“The authors claim that published projections do not include ‘propagated errors’ is fundamentally flawed. It is clearly the case that the model ensemble may have structural errors that bias the projections.”
I.e., the reviewer supposes that model precision = propagated error.
“The repeated statement that no prior papers have discussed propagated error in GCM projections is simply wrong (Rogelj (2013), Murphy (2007), Rowlands (2012)).”
Let’s take the reviewer examples in order:
Rogelj (2013) concerns the economic costs of mitigation. Their Figure 1b includes a global temperature projection plus uncertainty ranges. The uncertainties, “are based on a 600-member ensemble of temperature projections for each scenario…” [5]
I.e., the reviewer supposes that model precision = propagated error.
Murphy (2007) write, “In order to sample the effects of model error, it is necessary to construct ensembles which sample plausible alternative representations of earth system processes.” [6]
I.e., the reviewer supposes that model precision = propagated error.
Rowlands (2012) write, “Here we present results from a multi-thousand-member perturbed-physics ensemble of transient coupled atmosphere–ocean general circulation model simulations. “ and go on to state that, “Perturbed-physics ensembles offer a systematic approach to quantify uncertainty in models of the climate system response to external forcing, albeit within a given model structure.” [7]
I.e., the reviewer supposes that model precision = propagated error.
Not one of this reviewer’s examples of propagated error includes any propagated error, or even mentions propagated error.
Not only that, but not one of the examples discusses physical error at all. It’s all model precision.
This reviewer doesn’t know what propagated error is, what it means, or how to identify it. This reviewer also evidently does not know how to recognize physical error itself.
Another reviewer:
“Examples of uncertainty propagation: Stainforth, D. et al., 2005: Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature 433, 403-406.
“M. Collins, R. E. Chandler, P. M. Cox, J. M. Huthnance, J. Rougier and D. B. Stephenson, 2012: Quantifying future climate change. Nature Climate Change, 2, 403-409.”
Let’s find out: Stainforth (2005) includes three Figures; Every single one of them presents error as projection variation. [8]
Here’s their Figure 1:
Original Figure Legend: “Figure 1 Frequency distributions of T g (colours indicate density of trajectories per 0.1 K interval) through the three phases of the simulation. a, Frequency distribution of the 2,017 distinct independent simulations. b, Frequency distribution of the 414 model versions. In b, T g is shown relative to the value at the end of the calibration phase and where initial condition ensemble members exist, their mean has been taken for each time point.”
Here’s what they say about uncertainty: “[W]e have carried out a grand ensemble (an ensemble of ensembles) exploring uncertainty in a state-of-the-art model. Uncertainty in model response is investigated using a perturbed physics ensemble in which model parameters are set to alternative values considered plausible by experts in the relevant parameterization schemes.”
There it is: uncertainty is directly represented as model variability (density of trajectories; perturbed physics ensemble).
The remaining figures in Stainforth (2005) derive from this one. Propagated error appears nowhere and is nowhere mentioned.
Reviewer supposition: model precision = propagated error.
Collins (2012) state that adjusting model parameters so that projections approach observations is enough to “hope” that a model has physical validity. Propagation of error is never mentioned. Collins Figure 3 shows physical uncertainty as model variability about an ensemble mean. [9] Here it is:
Original Legend: “Figure 3 | Global temperature anomalies. a, Global mean temperature anomalies produced using an EBM forced by historical changes in well-mixed greenhouse gases and future increases based on the A1B scenario from the Intergovernmental Panel on Climate Change’s Special Report on Emission Scenarios. The different curves are generated by varying the feedback parameter (climate sensitivity) in the EBM. b, Changes in global mean temperature at 2050 versus global mean temperature at the year 2000, … The histogram on the x axis represents an estimate of the twentieth-century warming attributable to greenhouse gases. The histogram on the y axis uses the relationship between the past and the future to obtain a projection of future changes.”
Collins 2012, part a: model variability itself; part b: model variability (precision) represented as physical uncertainty (accuracy). Propagated error? Nowhere to be found.
So, once again, not one of this reviewer’s examples of propagated error actually includes any propagated error, or even mentions propagated error.
It’s safe to conclude that these climate modelers have no concept at all of propagated error. They apparently have no concept whatever of physical error.
Every single time any of the reviewers addressed propagated error, they revealed a complete ignorance of it.
3. Error bars mean model oscillation – wherein climate modelers reveal a fatal case of naive-freshman-itis.
“To say that this error indicates that temperatures could hugely cool in response to CO2 shows that their model is unphysical.”
“[T]his analysis would predict that the models will swing ever more wildly between snowball and runaway greenhouse states.”
“Indeed if we carry such error propagation out for millennia we find that the uncertainty will eventually be larger than the absolute temperature of the Earth, a clear absurdity.”
“An entirely equivalent argument [to the error bars] would be to say (accurately) that there is a 2K range of pre-industrial absolute temperatures in GCMs, and therefore the global mean temperature is liable to jump 2K at any time – which is clearly nonsense…”
Got that? These climate modelers think that “±” error bars imply the model itself is oscillating (liable to jump) between the error bar extremes.
Or that the bars from propagated error represent physical temperature itself.
No sophomore in physics, chemistry, or engineering would make such an ignorant mistake.
But Ph.D. climate modelers have invariably done. One climate modeler audience member did so verbally, during Q&A after my seminar on this analysis.
The worst of it is that both the manuscript and the supporting information document explained that error bars represent an ignorance width. Not one of these Ph.D. reviewers gave any evidence of having read any of it.
5. Unique Result – a concept unknown among climate modelers.
Do climate modelers understand the meaning and importance of a unique result?
“[L]ooking the last glacial maximum, the same models produce global mean changes of between 4 and 6 degrees colder than the pre-industrial. If the conclusions of this paper were correct, this spread (being so much smaller than the estimated errors of +/- 15 deg C) would be nothing short of miraculous.”
“In reality climate models have been tested on multicentennial time scales against paleoclimate data (see the most recent PMIP intercomparisons) and do reasonably well at simulating small Holocene climate variations, and even glacial-interglacial transitions. This is completely incompatible with the claimed results.”
“The most obvious indication that the error framework and the emulation framework
presented in this manuscript is wrong is that the different GCMs with well-known different cloudiness biases (IPCC) produce quite similar results, albeit a spread in the
climate sensitivities.”
Let’s look at where these reviewers get such confidence. Here’s an example from Rowlands, (2012) of what models produce. [7]
Original Legend: “Figure 1 | Evolution of uncertainties in reconstructed global-mean temperature projections under SRES A1B in the HadCM3L ensemble.” [7]
The variable black line in the middle of the group represents the observed air temperature. I added the horizontal black lines at 1 K and 3 K, and the vertical red line at year 2055. Part of the red line is in the original figure, as the precision uncertainty bar.
This Figure displays thousands of perturbed physics simulations of global air temperatures. “Perturbed physics” means that model parameters are varied across their range of physical uncertainty. Each member of the ensemble is of equivalent weight. None of them are known to be physically more correct than any of the others.
The physical energy-state of the simulated climate varies systematically across the years. The horizontal black lines show that multiple physical energy states produce the same simulated 1 K or 3 K anomaly temperature.
The vertical red line at year 2055 shows that the identical physical energy-state (the year 2055 state) produces multiple simulated air temperatures.
These wandering projections do not represent natural variability. They represent how parameter magnitudes varied across their uncertainty ranges affect the temperature simulations of the HadCM3L model itself.
The Figure fully demonstrates that climate models are incapable of producing a unique solution to any climate energy-state.
That means simulations close to observations are not known to accurately represent the true physical energy-state of the climate. They just happen to have opportunistically wonderful off-setting errors.
That means, in turn, the projections have no informational value. They tell us nothing about possible future air temperatures.
There is no way to know which of the simulations actually represents the correct underlying physics. Or whether any of them do. And even if one of them happens to conform to the future behavior of the climate, there’s no way to know it wasn’t a fortuitous accident.
Models with large parameter uncertainties can not produce a unique prediction. The reviewers’ confident statements show they have no understanding of that, or of why it’s important.
Now suppose Rowlands, et al., tuned the parameters of the HADCM3L model so that it precisely reproduced the observed air temperature line.
Would it mean the HADCM3L had suddenly attained the ability to produce a unique solution to the climate energy-state?
Would it mean the HADCM3L was suddenly able to reproduce the correct underlying physics?
Obviously not.
Tuned parameters merely obscure uncertainty. They hide the unreliability of the model. It is no measure of accuracy that tuned models produce similar projections. Or that their projections are close to observations. Tuning parameter sets merely off-sets errors and produces a false and tendentious precision.
Every single recent, Holocene, or Glacial-era temperature hindcast is likewise non-unique. Not one of them validate the accuracy of a climate model. Not one of them tell us anything about any physically real global climate state. Not one single climate modeler reviewer evidenced any understanding of that basic standard of science.
Any physical scientist would (should) know this. The climate modeler reviewers uniformly do not.
6. An especially egregious example in which the petard self-hoister is unaware of the air underfoot.
Finally, I’d like to present one last example. The essay is already long, and yet another instance may be overkill.
But I finally decided it is better to risk reader fatigue than to not make a public record of what passes for analytical thinking among climate modelers. Apologies if it’s all become tedious.
This last truly demonstrates the abysmal understanding of error analysis at large in the ranks of climate modelers. Here we go:
“I will give (again) one simple example of why this whole exercise is a waste of time. Take a simple energy balance model, solar in, long wave out, single layer atmosphere, albedo and greenhouse effect. i.e. sigma Ts^4 = S (1-a) /(1 -lambda/2) where lambda is the atmospheric emissivity, a is the albedo (0.7), S the incident solar flux (340 W/m^2), sigma is the SB coefficient and Ts is the surface temperature (288K).
“The sensitivity of this model to an increase in lambda of 0.02 (which gives a 4 W/m2 forcing) is 1.19 deg C (assuming no feedbacks on lambda or a). The sensitivity of an erroneous model with an error in the albedo of 0.012 (which gives a 4 W/m^2 SW TOA flux error) to exactly the same forcing is 1.18 deg C.
“This the difference that a systematic bias makes to the sensitivity is two orders of magnitude less than the effect of the perturbation. The author’s equating of the response error to the bias error even in such a simple model is orders of magnitude wrong. It is exactly the same with his GCM emulator.”
The “difference” the reviewer is talking about is 1.19 C – 1.18 C = 0.01 C. The reviewer supposes that this 0.01 C is the entire uncertainty produced by the model due to a 4 Wm-2 offset error in either albedo or emissivity.
But it’s not.
First reviewer mistake: If 1.19 C or 1.18 C are produced by a 4 Wm-2 offset forcing error, then 1.19 C or 1.18 C are offset temperature errors. Not sensitivities. Their tiny difference, if anything, confirms the error magnitude.
Second mistake: The reviewer doesn’t know the difference between an offset error (a statistic) and temperature (a thermodynamic magnitude). The reviewer’s “sensitivity” is actually “error.”
Third mistake: The reviewer equates a 4 W/m2 energetic perturbation to a ±4 W/m2 physical error statistic.
This mistake, by the way, again shows that the reviewer doesn’t know to make a distinction between a physical magnitude and an error statistic.
Fourth mistake: The reviewer compares a single step “sensitivity” calculation to multi-step propagated error.
Fifth mistake: The reviewer is apparently unfamiliar with the generality that physical uncertainties express a bounded range of ignorance; i.e., “±” about some value. Uncertainties are never constant offsets.
Lemma to five: the reviewer apparently also does not know the correct way to express the uncertainties is ±lambda or ±albedo.
But then, inconveniently for the reviewer, if the uncertainties are correctly expressed, the prescribed uncertainty is ±4 W/m2 in forcing. The uncertainty is then obviously an error statistic and not an energetic malapropism.
For those confused by this distinction, no energetic perturbation can be simultaneously positive and negative. Earth to modelers, over. . .
When the reviewer’s example is expressed using the correct ± statistical notation, 1.19 C and 1.18 C become ±1.19 C and ±1.18 C.
And these are uncertainties for a single step calculation. They are in the same ballpark as the single-step uncertainties presented in the manuscript.
As soon as the reviewer’s forcing uncertainty enters into a multi-step linear extrapolation, i.e., a GCM projection, the ±1.19 C and ±1.18 C uncertainties would appear in every step, and must then propagate through the steps as the root-sum-square. [3, 10]
After 100 steps (a centennial projection) ±1.18 C per step propagates to ±11.8 C.
So, correctly done, the reviewer’s own analysis validates the very manuscript that the reviewer called a “waste of time.” Good job, that.
This reviewer:
- doesn’t know the meaning of physical uncertainty.
- doesn’t distinguish between model response (sensitivity) and model error. This mistake amounts to not knowing to distinguish between an energetic perturbation and a physical error statistic.
- doesn’t know how to express a physical uncertainty.
- and doesn’t know the difference between single step error and propagated error.
So, once again, climate modelers:
- neither respect nor understand the distinction between accuracy and precision.
- are entirely ignorant of propagated error.
- think the ± bars of propagated error mean the model itself is oscillating.
- have no understanding of physical error.
- have no understanding of the importance or meaning of a unique result.
No working physical scientist would fall for any one of those mistakes, much less all of them. But climate modelers do.
And this long essay does not exhaust the multitude of really basic mistakes in scientific thinking these reviewers made.
Apparently, such thinking is critically convincing to certain journal editors.
Given all this, one can understand why climate science has fallen into such a sorry state. Without the constraint of observational physics, it’s open season on finding significations wherever one likes and granting indulgence in science to the loopy academic theorizing so rife in the humanities. [11]
When mere internal precision and fuzzy axiomatics rule a field, terms like consistent with, implies, might, could, possible, likely, carry definitive weight. All are freely available and attachable to pretty much whatever strikes one’s fancy. Just construct your argument to be consistent with the consensus. This is known to happen regularly in climate studies, with special mentions here, here, and here.
One detects an explanation for why political sentimentalists like Naomi Oreskes and Naomi Klein find climate alarm so homey. It is so very opportune to polemics and mindless righteousness. (What is it about people named Naomi, anyway? Are there any tough-minded skeptical Naomis out there? Post here. Let us know.)
In their rejection of accuracy and fixation on precision, climate modelers have sealed their field away from the ruthless indifference of physical evidence, thereby short-circuiting the critical judgment of science.
Climate modeling has left science. It has become a liberal art expressed in mathematics. Call it equationized loopiness.
The inescapable conclusion is that climate modelers are not scientists. They don’t think like scientists, they are not doing science. They have no idea how to evaluate the physical validity of their own models.
They should be nowhere near important discussions or decisions concerning science-based social or civil policies.
References:
1. Lauer, A. and K. Hamilton, Simulating Clouds with Global Climate Models: A Comparison of CMIP5 Results with CMIP3 and Satellite Data. J. Climate, 2013. 26(11): p. 3823-3845.
2. Meinshausen, M., et al., The RCP greenhouse gas concentrations and their extensions from 1765 to 2300. Climatic Change, 2011. 109(1-2): p. 213-241.
The PWM coefficients for the CCSM4 emulations were: RCP 6.0 fCO₂ = 0.644, a = 22.76 C; RCP 8.5, fCO₂ = 0.651, a = 23.10 C.
3. JCGM, Evaluation of measurement data — Guide to the expression of uncertainty in measurement. 100:2008, Bureau International des Poids et Mesures: Sevres, France.
4. Roy, C.J. and W.L. Oberkampf, A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Comput. Methods Appl. Mech. Engineer., 2011. 200(25-28): p. 2131-2144.
5. Rogelj, J., et al., Probabilistic cost estimates for climate change mitigation. Nature, 2013. 493(7430): p. 79-83.
6. Murphy, J.M., et al., A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2007. 365(1857): p. 1993-2028.
7. Rowlands, D.J., et al., Broad range of 2050 warming from an observationally constrained large climate model ensemble. Nature Geosci, 2012. 5(4): p. 256-260.
8. Stainforth, D.A., et al., Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature, 2005. 433(7024): p. 403-406.
9. Collins, M., et al., Quantifying future climate change. Nature Clim. Change, 2012. 2(6): p. 403-409.
10. Bevington, P.R. and D.K. Robinson, Data Reduction and Error Analysis for the Physical Sciences. 3rd ed. 2003, Boston: McGraw-Hill. 320.
11. Gross, P.R. and N. Levitt, Higher Superstition: The Academic Left and its Quarrels with Science. 1994, Baltimore, MD: Johns Hopkins University. May be the most intellectually enjoyable book, ever.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Type B errors ignored
Thanks Pat, RGB and Kevin
The IPCC/”Climate scientists” are further oblivious to the international Metrology community’s standardization of guidelines for evaluating uncertainties. See:
Evaluation of measurement data – Guide to the expression of uncertainty in measurement. JCGM 100: 2008 BIPM (GUM 1995 with minor corrections) Corrected version 2010
This details two categories of uncertainty:
Type A. those which are evaluated by statistical methods,
Type B. those which are evaluated by other means.
See the diagram on p53 D-2 Graphical illustration of values, error, and uncertainty.
Type B errors are most often overlooked. E.g.
See also Barry N. Taylor and Chris E. Kuyatt, Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST TN1297 PDF
David, all of those references are cited in my manuscript, including the NIST. I also quote from the JCGM, showing they recommend exactly the analysis I carried out. It apparently made no impression.
Excellent
Readers call your Legislators. Ask them to include this line in a funding bill:
National reviews of climate shall incorporate international guidelines for expressing uncertainty.
WOW, do I agree with this post.
Two comments out of the many I could add:
“Bottom line? Climate modelers are not scientists.”
I’ve experienced somewhat the same in regards to archaeology. Yes, they follow solid methodology when laying out a grid on a dig site, and they carefully log things, giving the impression of accuracy and scientific rigor. But then 99% of what they write about is about qualitative things and about religious ceremonies and the meaning of artifacts – all of which is pulled out of their collective agreed-upon interpretations of the past societies, not on scientific quantification. YES, they can tell you relatively well enough about WHEN something was laid in the dirt – but when it comes to WHAT it means they pull out the interpretations of 1850. For those who say that they use C14 dating and OSL and such, you have to remember that they are sending all those samples off to LABS to do the actual science. It is those labs which do the science, not the archaeologists. In analogy, lawyers and police send off samples for lab testing, but no one would assert that a cop or lawyer is a scientist, merely because he used the OUTPUT from labs.
The next time you hear an archaeologist talk about ritual artifacts or ceremonial plazas or temples, ask yourself exactly what scientific basis there is for using those ideas. What you will find is an accepted paradigm that has been around for a LONG time and which is not allowed to be challenged from within. No matter HOW reasonable the terms “ritual” and “ceremonial” and “temple” SOUND to us as laymen, please be aware that there were many very practical people living in those old societies – carpenters, farmers, metal workers, bricklayers, architects, etc. – and the archaeological view that the societies were run predominantly by priest castes is only one point of view. The portion of people who live close to the land or who work out architectural plans and design buildings that last for thousands of years cannot have been a bunch of mumbo-jumbo. Mumbo-jumbo doesn’t work in OUR practical-minded society, so why should it work in societies in the past?
Sorry if this went a bit O/T.
“The accuracy-precision difference was extensively documented to relevant literature in the manuscript, e.g., [3, 4].
The reviewers ignored that literature. The final reviewer dismissed it as mere assertion.”
I have recently run across the views of historians about assertions. If what I’ve found is typical (and I have no reason to think it is not), then basically they have the attitude that, “Everyone has an opinion and all opinions are equally valid.” This is so far from science that it shocked me. They LITERALLY think that in quantified science anyone’s opinion is worth listening to.
I have yet to ask them about 2+2=4, about the acceleration due to gravity, etc. Or if someone who thinks that 2+2=13.5 is worth listening to. BUT I WILL.
You should read ‘Forbidden Archaeology’
How about submitting it to a statistical journal?
I thought about it, Jake, but decided against because the ms is not about statistics. That is, it doesn’t advance that field.
NO
That’s a blurt-out worthy of Mosher.
To answer the question as put, if a climate model designer believes his results reflect reality then no, he is not a scientist. If the modeler believes his results are data then he is not even a modeler. Written and shall be accepted as a gender and sexual orientation-neutral statement.
No, they are engineers.
a SOLAR ENERGY CAR made by amateurs have crossed Australia. Why CAR´S FACTORIES do not want ro know that?. Petroleum´s economic interests. Shame politicians
Well maybe when they get it to go more than 250 miles, take only 10 minutes to recharge, not require $$$$ of taxpayer subsidy, operate in conditions less intense than outback Australia, and oh yea, look half-way decent, give us a call.
Just saying.
Pat,
Thank You (!) for this tour de force on all of the ways ‘climate modelers’ do not adhere to scientific principles. There is just toooooo much info here to enjoy on my lunch break today. I’ll give it a thorough perusal this evening!
Best regards,
Mac
Thanks for the encouraging words, Mac.
rgbatduke: * The Multi-Model-Mean is an abomination, and all by itself proves utter ignorance about statistics in climate modeling.
Strictly speaking, actual data have never been obtained from processes that satisfy the assumptions of mathematical statistics, including the assumptions that the data have all been sampled from the same defined population and that they have been sampled independently. The multimodel mean is not intrinsically worse in this regard than the mean of 5 realizations of a diffusion process, the mean time lapse of 4 atomic clocks (as was used in an experiment testing a prediction of general relativity), or successive measures on an industrial process.
A model with its parameters and their imprecisions defines a population, the population of possible realizations. If there are variations on the model, then the model, its variations, and all their parameter estimates and precisions defines the population. If the imprecisions are represented by normal distributions (and most other distributions, notably excluding the Cauchy distribution), then that population has a mean and standard deviation, and the mean and standard deviation can be estimated from the sample of realizations computed via randomly sampling and resampling values of parameters from their corresponding imprecision distributions. The point of doing this is that neither the model mean nor the model sampling distribution can be calculated analytically. Without doing the simulations to obtain a good estimate of the sampling distribution, you can not infer much from the misfit between one or a few of the realizations and the actual temperature (or “mean temperature” trend) — any such misfit might be due to a single poorly estimated parameter (and it is highly unlikely that all of them will have been estimated with the requisite accuracy). When, as now, almost all of the realizations are above the mean temperature trajectory, and the CI on the mean model trajectory clearly excludes the data, then you can have a lot of confidence that the population defined by the model does not include the “true” model.
The argument between Pat Frank and the reviewers seems to be that the reviewers are not interested in the results that come from sampling as well from the distribution of the “forcing” change of 4 W/m^2. In effect, they are regarding the value of 4 W/m^2 as reliable as any of the famous physical constants such as the Boltzmann constant or universal gas constant. Or it is the only value of the change in forcing that they want to consider for a while.
The modelers could consider varying the change i n forcing in a long series of simulations/samples from the model population (4, 3.5, 3, 2.5 etc) until they got some means consistently below the mean temperature. The could pick the value (my guess now is that it would be close to 1.5) that produced the ensemble mean closest to the observed trajectory; assuming the model to be correct, as a bunch of modelers do, that would then be the best estimate of the actual forcing change produced by the CO2 change. It’s equivalent to estimating the TCS for the model from the temperature trend.
“A model with its parameters and their imprecisions defines a population, the population of possible realizations. If there are variations on the model, then the model, its variations, and all their parameter estimates and precisions defines the population.”
The contingency resolved by running a model is not the same as that resolved by a physical experiment. The only thing a model can teach us (i.e. its information gain) is about the model itself! All model are question beggars and remain so until the questions are answered, by observing REAL data.
Jeff Patterson: The only thing a model can teach us (i.e. its information gain) is about the model itself! All model are question beggars and remain so until the questions are answered, by observing REAL data.
No disagreement here: that is why model outputs are always compared to data.
Except in climate science.
“…The multimodel mean is not intrinsically worse in this regard than the mean of 5 realizations of a diffusion process, the mean time lapse of 4 atomic clocks (as was used in an experiment testing a prediction of general relativity), or successive measures on an industrial process….”
Sorry. This is untrue.
jorgeKafkazar: Sorry. This is untrue.
Details please.
The mean of the sample of realizations is an unbiased estimate of the mean of the population of possible realizations of the model. The population is well defined when the model is defined and the uncertainties of the parameter estimates are specified. What part of that do you think is in error? Are you asserting that the sample of atomic clock readings was a random sample of the possible atomic clock readings that might have been taken and weren’t? The population of possible model realizations is better defined than the population of possible atomic clock readings.
Matthew, please see my reply to you above. It’s (+/-)4 W/m^2 average annual CMIP5 global cloud forcing error. It’s a physical error statistic, not a forcing.
…global warming (“good” no longer spend cold)… pouring pollutants greenhouse effect into the air is melting the Poles. Besides on melting permafrost will free into the Atmosphere million Tm of methane with big greenhouse effect. This large amount of freshwater to the ocean could stop vertical deep sea currents which depend on a starting from surface downwards on a delicate balance between fresh and salty water and temperatures. Heat from the Sun reaches the equator and currents distribute it throughout the Planet, then…goodbye to our warm climate. The horizontal oceanic currents produced by winds and some others by the rotation of the Earth, rotating all by the Coriolis effect, will continue…but the vertical currents produced by the sinking of horizontal currents of dense salty water that reaches the Poles where the water is sweeter, less salty, and form deep currents would stop (why are the Grand Banks fishing in cold latitudes?…because over there is the POLAR ICE, freshwater, different sweet/salty density, salty dense water arriving and sinks in a little salty water environment…nutrients that are removed from the bottom and rise to the surface, phytoplankton that feed on nutrients, zooplankton that feed on zooplankton, fish that feed on zooplankton)… No polar ice over there will be no vertical currents…could reduce the rise of nutrients to the surface and therefore PHYTOPLANKTON SHORTAGE MAY DECREASING ITS VITAL CONTRIBUTION WITH OXYGEN TO THE ATMOSPHERE (90 %)…fish…winds in some places of more warm latitudes carry out the surface hot water permitting the outcropping of water and plankton (the upwelling) from the bottom cold current coming from the Pole, forming other Banks fishing… Without polar ice the sea it could almost stratified into horizontal layers with little energetic movement of water masses in vertical which is what removes fertilizer nutrients from the sea bottom… Besides lowering salinity of the sea, for that contribution with freshwater to melt the Poles, will increase evaporation (ebullioscopy: the less salt has, more evaporates) producing gigantic storm clouds as have never seen, that together with altering of the ocean currents, could cool areas of the Planet causing a new ice age… Warming invasion of tropical diseases carried by their transfer agents, already without the “general winter” containing them would fall upon the World like a plague…can produce a cooling, a new ice age, like living at the North Pole…and less oxygen in the Atmosphere… Is not known to be worse… Go choosing.
You watch too much Jack Van Impe
The fundamental law of information theory show definitively that computer models by themselves are incapable of increasing our knowledge about physical reality beyond that already known to those who programmed the model. Information gain can only occur when a range of possible outcomes (contingency) is resolved by observing the true outcome. Models can create contingency (i.e. a hypothesis) but cannot by themselves resolve it. Climateers seem to think they’ve found a loophole. Just run more simulations (which they laughingly call experiments) and average the results! They really are an ignorant lot. They reject the very notion accuracy because they claim the models make no predictions (which can be tested against reality), only projections which can never be falsified. Good work if you can get it.
Jeff Patterson: The fundamental law of information theory show definitively that computer models by themselves are incapable of increasing our knowledge about physical reality beyond that already known to those who programmed the model.
I think that you overstate the case. Much depends on how thoroughly the models have been tested and how accurate they have been shown to be. Three examples:
1. A lab tech runs a sample of your blood through a measuring device and reports your measured cholesterol as 327 or something. That number is the output of a model, computed from the area under a curve put out in the device, and an equation relating such areas to actual (accurately but not perfectly known) test concentrations. Have you acquired information about your cholesterol level? What you wrote says no.
2. Above the Earth surface are interplanetary probes and orbiting satellites, whose locations 3 months from now have been computed from models, in this case really well-tested and accurate models. Do those calculations provide information about their actual future positions, information that might be used in a course correction for example? What you wrote says no.
3. Your GPS device says that you are 3.2 miles away from an unmarked street where you have to turn left to get to the next gas station. The message has been computed from a model, again it a well-tested and accurate model. Is that information about where the next gas station is? What you wrote says no.
The point of making a model is to use it in a situation where the calculated output tells you something about the reality that you could not tell without the model.
Why not add the example of calculating the the length of a hypotenuse of a right angle using measurements of it’s sides?
Simple counts as per example 1 or simple geometry with known, well defined inputs as example 2 and 3 are horrendously simplistic views of what climate models are poorly attempting. Not to mention that these examples have been proven reliable, while climate models have repeatedly been shown as unreliable.
I think you’re going down the semantic rabbit hole. Those things that you are calling “models”, most of us would call “filters”. When the climate bunch use “models”, they don’t process data, they create it. It’s one thing for Hansen to use a “statistical model” to adjust data sets. It’s something entirely different to use something that purports to be physics to “simulate” a physical thing. That’s what most people mean by “modeling”.
I think we need to stop calling statistical models “models”, and start calling them “filters” to eliminate the confusion.
You are confusing the transfer of information with the creation of information. A calculation, any calculation no matter how complex can only transfer information. 2+2=4 whether you were aware of that fact or not. When you do the calculation by hand or by computer, _you_ may learn something but knowledge (the sum total of what is known (or thought) to be true, not by you necessarily, but by “us” corporately) is not increased. A model simulation is just a series of calculations, each of which can only transfer information. The die was cast, the outcome certain, before you pressed run.
To your examples:
1) Data analysis of physical data is not a simulation but here again, knowledge is not increased. The knowledge (about how to transfer the information contained in a blood sample to some human readable form) is encoded in the algorithm. If the algorithm has been tested and found to give reliable results AS COMPARED TO SOME INDEPENDENT REFERENCE, the test may prove useful as a time effective (or cost effective) substitute for the reference test.
2) The equations for planetary motion are well known. Doing the calculation can not result in information gain.
3) Your GPS device has transferred static information (the street map) to you and calculated your distance from a specified point. If you are surprised by the result (a necessary condition for information gain) it is only because _you_ didn’t have the information. The information was in existences before the GPS gave it to you and it certainly did not create it :>)
Jeff Patterson: 2) The equations for planetary motion are well known. Doing the calculation can not result in information gain.
Really? You can not now calculate the proper thrusts for the course correction, whereas you could not do so before the calculation? Or the knowledge of the proper thrusts is not “information”?
But the models must be falsifiable, give unique predictions, and of a known calibrated accuracy. Climate models fail all of those tests.
Alx: Not to mention that these examples have been proven reliable, while climate models have repeatedly been shown as unreliable.
I agree that the useful models are those that have been well-tested and shown to be accurate. That was my main point. However, that was not what Jeff Patterson wrote.
Why not the hypotenuse of a right angle? Everybody already knows that one.
Matthew R Marler February 25, 2015 at 12:20 pm
Jeff Patterson: 2) The equations for planetary motion are well known. Doing the calculation can not result in information gain.
Really? You can not now calculate the proper thrusts for the course correction, whereas you could not do so before the calculation? Or the knowledge of the proper thrusts is not “information”?
Your answer is in the circularity of your reply. “You can not now calculate the proper thrusts for the course correction, whereas you could not do so before the calculation?” You could always do the course correction calculation, before or after the calculation. perhaps not fast enough to do you any good but the speed at which you arrive at the answer has nothing whatsoever to do with the information gain. Knowledge can only come from surprise and arithmetic should never surprise us.
In defense of climate modelers, these concepts are difficult to understand for most people. Math people have no problem but word people struggle. I suspect that people who reviewed Pat’s manuscript went through two filters. First, people who got into climate science did so because they thought they could get by without as much math as the other sciences. Second, it is the climate scientists who lean more towards words versus math that end up reviewing manuscripts for journals. My opinion anyway.
There is a lot of truth to that, I could see it myself back in my college days. Sciency people who could not do math stayed away from physics and chemistry and gravitated to things like biology, geography, geology, climatology, and other science fields that have a lot more qualitative aspects. However, at the end of the day, if you’re gonna start doing things like modelling, then you need the proper math and the statistics with the proper treatment of errors, as this article so clearly shows.
In answer to your question, “Are Climate Modelers Scientists?” the answer is No. They are just highly skilled technicians assigned to a specific task that is supposed to verify the prejudices of their superiors. Well paying job if you don’t contradict your supervisor.
I really hope you get your paper published somewhere, it’s important that things like this make their way into the literature somehow no matter how many gatekeepers try to stop it.
Oh, my. Well this one has been conditioned to respond to “models can only be hotter.” It is when he gets access to the food bowl. This person thinks they are looking at a climate model rather than the uncertainty input to the model propagated throughout to the result.
Kevin, “This person thinks they are looking at a climate model rather than the uncertainty input to the model propagated throughout to the result.”
Exactly right, Kevin. And I saw that mistake made over and over again, in these reviews.
Thanks Pat… for the excellent article.
From the comments of the reviewers it looks like the climate research field continues to grow more incestuous all the time. They refuse to look outside their own area for any relevant knowledge and apparently prefer to reinvent the wheel, jumping into the mud puddle with both feet more often than arriving at the desired ‘new and innovative solution’. They have done this consistently in everything from statistics to control theory, and now, as you have pointed out, computer modeling theory. They have made some pretty egregious errors in each, and, when these are pointed out, refuse to even admit such.
It’s pretty obvious why you paper has been rejected. If correct (it certainly appears so to me) it will nullify something like 80% to 90% of the climate papers published in the last 10 or 15 years. At least that many have been based either partially or entirely on results from the current crop of GCMs.
Please keep trying on the paper. It should be required reading for all scientists and engineer that currently accept the GCMs as proof or prophets of global warming.
Thanks, Joe. I hope to keep on trucking until it makes it through.
Pat,
As a last resort you could always try the Chinese Science Bulletin. That’s the journal that published ‘Why models run hot: results from an irreducibly simple climate model’ for Monckton, Soon, & Legates. They apparently don’t have the CAGW baggage that infects most of the U.S. journals.
It may come to that, Joe. I hope not.
I agree with the Reviewers comments. This paper should not see the light of day in a reputable Science Journal. Its unfortunate that the Author attributes his rejection to ‘bias’ instead of the paper’s own failings, which are considerable.
so you should have no issues in outlining it faults , so why not try ?
warrenlb
Why? Why are you qualified to make that decision? “Your” opinion does not matter, because – according to your answers, YOU have no ability to make a judgement. Now, “you” can repeat somebody’s else’s judgement – and have over 300 times here! – but then again, that particular trait also means you have no business voting.
Why does your opinion matter?
You weren’t one of the “reviewers” were you ? You either have first hand knowledge or you’re talking through the wrong orifice.
You agree with the reviewers comments?
Because you understand the comments or do not understand the comments but feel comfortable as a bobble head?
My guess is not the former.
Hey! What happened to warrenlb?
All hat, no cattle?
warrenlb: A hit-and-run troll comment. Evidence: Nothing to back up the “which are considerable” remark. The failing of the Climate Models are far more considerable, and will one day be written about in science history as the following paragraph to Ptolemaic geocentrism and the Vatican circa 1615 AD.
“The failing of the Climate Models are far more considerable, and will one day be written about….etc.”
A hit-and run-troll comment with nothing to back up the “far more considerable” remark”
warrenlb, can you be specific? If not, then your opinion is objectively valueless.
I can’t claim I understand it all 100% but if I understand it correctly, running similar analysis on our current relatively reliable weather models would lead to predictions such as “in three days the temperature will be thirty, plus or minus fifty degrees”. Any model of chaotic system is going to propagate its uncertainity over the whole phase space very fast, that’s not talking butterfly effect, that’s considering errors piling up in the most uncomfortable way for extended periods of time. But that does not tell us much about accuracy of the model.
Referring to the b pane of the first image, it seems to me the error bounds presented there are highly unrealistic. I really don’t think falling into ice age within 50 years is physically possible without global nuclear war or large asteroid hitting Earth.
At the point where the blue area starts in the image, the climate model is already running for ~150 years. Hitting the target result is test of accuracy of the model using known forcings. When switching from past to future, of course known forcings are replaced by estimates. But considering maximum deviation in these estimates from the very beginning does not sound like realistic assumption to me.
Kashua, your understanding needs a little work. Currently, weather modelling in SE Australia is 70% accurate at 7 days away from the date of forecasting. I find that impressive. Further out, the forecasting quickly loses accuracy, closer is much better, though not by much due to chaotic effects (mountain waves for example).
I find that the best way of viewing this is to think of it as a picture on a computer screen, say 1024 x 768 pixels. Let’s imagine that you are looking at a picture of the pope waving at you from his balcony. Zoom back far enough and the pope and his balcony are now described by 4 pixels instead of 786,432 pixels. Now you can’t tell whether it’s the pope or Catherine the Great doing something unspeakable with a donkey.
Besides that, weather models are updated every few hours with fresh data. Were they not, they’d run away from reality in a day.
Kasuha, “Referring to the b pane of the first image, it seems to me the error bounds presented there are highly unrealistic. I really don’t think falling into ice age within 50 years is physically possible without global nuclear war or large asteroid hitting Earth.”
You’ve made the same mistake made by virtually all the modelers, Kasuha. You suppose that the propagated error bars indicate the behavior of the model itself. They do not.
Look closely at panel b. Do you see the lines? Those lines show the behavior of the model: discrete expectation values. Discrete, by the way, does not mean physically unique.
The uncertainty envelope indicates how much information those expectation values have, as regards the future climate. In these particular cases, that information is: none. Uncertainty bars indicate an ignorance width. They do not indicate the behavior of the model.
The blue area shows propagated error as though the simulation started at the year 2000. When I calculated the uncertainty starting at 1850, the uncertainty envelope was already huge at the year 2000. So, I decided to be merciful and include only the uncertainty in the futures simulation. Putting in the larger uncertainty envelope would only lead the the necessity of long explanations.
Methinks he doth protest too much.
If you want to get something published — respond clearly and concisely to reviewer comments. If reviewers make mistakes (certainly not uncommon), then explain in clear terms and give simple examples to make your point. Also, do not use an indignant tone and insult them.
Methinks it was the reviewers insulting Pat.
I had the experience of having a paper rejected because of one reviewer. He said (it was obvious who it was) that I hadn’t considered something. I let the editor know that the paper contained a whole section with a titled discussing it. She gave it to a third reviewer who wrote “I refer to my previous comments” that were those of the original reviewer.
The peer-review process is flawed.
A bit of an understatement as we say in the Land of Under.
Did you respond back with a clear explanation of what they missed?
warrenlb – It was hard to write it politely but I refrained from using the f word when I pointed out to the editor that there was a section discussing it with a title, along with many other inane objections. I did write that the reviewer had clearly dismissed the paper before even reading through it.
It did get it published in a minor journal but the same person spread lies about it being mathematically incompetent. It probably could have been done better but the only actual fault I found was a subscript that was an i instead of a j and, knowing him, that was enough to slam the paper with a straight face.
I did all that Barry. Polite, reasonable, explicit, analytical, thorough. It didn’t help.
My wife was telling me of one of her colleagues that had a manuscript reviewed for publication, made all of the required changes … and then it was summarily rejected by the “journal”. To say this researcher is unhappy is understated.
I did not read all of the comments, so I suspect this has been stated. But, for what it’s worth, I feel your pain. I do uncertainty work in experimental aerodynamics and continue to struggle to get similar work out of my CFD brethren. They demand my uncertainties so that they can prove that their answer is within my error bars, but don’t realize that they also have error bars.
In any case, I suspect one of the problems is how uncertainty propagation is now being accomplished. In the good ol’ days, we did a Taylor series expansion by hand. It meant you had to do a lot of hard math and you could easily miss cross-correlated factors. But, it also meant that you could (relatively) easily find the sensitivities of the overall uncertainty to individual uncertainties (in an experimental setting, this tells you which device it’s worth buying better versions of and which ones are just not going to bring you much benefit).
With the cheapness of computing power, everyone is moving to Monte Carlo methods. To find those individual sensitivities, you have to dither (as, I assume, you did with the +/-4W/m^2) a single component, then stop and do the same with another component.
Simultaneously, many of the models use Monte Carlo methods to deal with some other types of uncertainty than the ones we are talking about here (if there’s a volcano, how many hurricanes there are, and whatnot). This gives a range of values that it could output, given a single input.
To come to your uncertainty, given current methods, you are asking them to do a Monte Carlo simulation on the results of all of the Monte Carlo simulations (since you can’t just do it on the high and low lines, because the sensitivities may vary in other ways). They probably just think “I already did a Monte Carlo simulation, therefore, I have my uncertainty.”
But this is a case that is vastly different. You’re hitting the uncertainty of the uncertainty. Because I find that very different examples sometimes help illustrate something, consider an aircraft kill-chain. If a missile has a Probability of kill (Pk) of 0.9, we can do a Monte Carlo simulation on the engagement between an airplane and an attacker firing some number of those missiles. But, to add to the difficulty, we recognize that we don’t really know the Pk. It’s 0.9 +0.05/-0.15. Which means that the whole thing has a wide band on top of the wide band. The lazy way would be to just do a simulation with a Pk of 0.75 and one with Pk of 0.95, which at least gets you closer. But you still have to do a Monte Carlo of both.
On the side of the modelers (for the sake of fairness), I would admit that the uncertainty community really has caused some confusion in its effort to reduce confusion. Eliminating the term “bias” and so forth has just confused some folks and they don’t know what is being referred to anymore.
Also, it’s always nice to remind people that No one knows the value of the systematic error!
GISS knows their systematic error. Its the residual between the RSS average anomaly and their “adjusted” anomaly.
Just skimming, I spotted what looks like a serious mistake at the core of the matter: Standard error-propagation methods are designed to help estimate the effect of a change in inputs in an output. However, the less linear the system, the worse the estimate. The climate is a system of such complex non-linearity that you cannot practically translate from results back to spaces of possible inputs. This is why standard error-propagation doesn’t work for strongly non-linear systems. More importantly, perhaps, when the sixth reviewer shifted the inputs, he actually looked directly at what the error-propagation was supposed to estimate. Using error-propagation the way it is done here shows precisely the same mistake that seems to appear in a lot of climate models, a false assumption of linearity, starting from some conditions in a system that is physically strongly non-linear and numerically chaotic.
The numerical stability is also important, but describes something other than accuracy or even precision themselves. Physically, a small change in inputs should not cause a dramatic change in outputs. However, there are approximations in the models for computational reasons which can introduce chaos. The most important of these simplifications is that the air is treated as a fluid and not an enormous collection of individual particles. The numerical stability, using the same model with slightly different initial conditions, is a measure of the impact of these approximations. That is not to say the system is non-chaotic: Hydrodynamics are always chaotic, so there are always some small changes, a few degrees in some lakes, or something like that, which will send temperatures off to crazytown, but it appears as though modelers have not stumbled onto them overwhelmingly often. However, looking at another model-output, like the (non-physical) linear sensitivity to a specific forcing, any given model could be suffering from chaos.
I really would like to see a proper analysis of physical sensitivities, but that would demand a rerun of the models many times for each parameter to map out the changes in sensitivity. Then I would want to see combinations of perturbations, requiring at absolute minimum 2^N runs of central models where N is the number of physical parameters being checked. While it may be a better use of computing resources than continued predictions of unknown reliability, the resources to do it right may not exist today.
I really would like to see a proper analysis of physical sensitivities, but that would demand a rerun of the models many times for each parameter to map out the changes in sensitivity.
Ayep. This is how you do a real test of errors in simulations: you look at how the output varies with changes in the inputs. The reviewers were right in that regard.
I find it kinda odd that almost no one here is actually discussing Pat’s work. Looking at his AGU poster from last year, we see that his calculated uncertainty grows with the square root of time. Which means that in about 10,000 years, his model of the uncertainty is supposed to include temperatures below absolute zero.
If your work includes the possibility of temperatures going below absolute zero, then it’s certainly wrong. No matter how low we set the cloud forcing in the models, they’re still not going to go below absolute zero, which is a pretty strong hint that there’s something wrong with Pat’s work. It’s “not even wrong”.
Windchasers, “Which means that in about 10,000 years, his model of the uncertainty is supposed to include temperatures below absolute zero. ”
Classic reviewer mistake, windchasers. You’re supposing, along with them, that the error propagation uncertainties are temperatures. They’re not.
A statement like yours leads me to think that you don’t know the difference between a physical error statistic and an energetic perturbation.
Classic reviewer mistake, windchasers. You’re supposing, along with them, that the error propagation uncertainties are temperatures. They’re not.
What do they represent, if not the uncertainty in the temperature anomaly in the models? It’s the right axis on the main chart. C’mon, there’s no need to be cryptic.
A statement like yours leads me to think that you don’t know the difference between a physical error statistic and an energetic perturbation.
In models, a physics peturbation is the right way to account for uncertainties in the underlying (physical) parameters. Peturb the physical parameter(s) across the true range of uncertainty, and analyze the resulting change.
Of course this is distinct from the overall subject of “error statistics”: in models, there are errors / uncertainties in both the inputs and outputs. This is necessarily so; all models are approximations.
There wasn’t a review comment in the OP that didn’t have me nodding along and saying “yep”. They pretty much nailed it: error propagation in models is best-handled via physics peturbation.
I’d be interested in reading your paper, if you have it up on Arxiv or somesuch.
windchasers, uncertainty in temperature is not physical temperature. The uncertainty in temperature is a measure of how confident one can be in the accuracy of the temperature expectation value of the model.
The ordinate axis of the plots refers to the temperatures calculated from the models. The uncertainty envelopes refer to the reliability of those temperatures. The graphic is a standard way of representing a sequence of calculational results and their uncertainty.
In the case of the CCSM4 model, as in all CMIP5-level models, the propagated uncertainty says that no confidence can be put in the simulated temperatures. They convey no information about the possible magnitude of future air temperatures.
This is not being “cryptic.” It is the direct and unadorned meaning of physical uncertainty.
You wrote, “In models, a physics peturbation is the right way to account for uncertainties in the underlying (physical) parameters. Peturb the physical parameter(s) across the true range of uncertainty, and analyze the resulting change.”
All that tells one is how the model behaves, e.g., see the fourth figure in the head-post. That exercise reveals nothing about whether the model produces physically accurate predictions.
You wrote, “There wasn’t a review comment in the OP that didn’t have me nodding along and saying “yep”. They pretty much nailed it: error propagation in models is best-handled via physics peturbation.”
Too bad. Physics perturbation has nothing to do with propagation of physical error. See Bevington and Robinson.
Agreement with those reviewer comments amounts to an admission that one understands nothing about physical error analysis, or about the meaning or method of propagated of physical error.
Pat, see my other reply here:
http://wattsupwiththat.com/2015/02/24/are-climate-modelers-scientists/#comment-1869787
I don’t think you’re actually propagating actual uncertainty.
Still, the reason we use physics peturbation as the way to do error propagation is because of the highly non-linear effects in many models. If your model is linear, then sure, a direct propagation using either analytical equations or Monte Carlo is fine.
Climate models are not linear, though, so errors in cloud forcings will have other feedback effects, whether positive or negative. The linear approach is fine for a rough guess, but it’s just a starting place.
The uncertainty envelopes refer to the reliability of those temperatures. … This is not being “cryptic.” It is the direct and unadorned meaning of physical uncertainty.
Yet, it’s plainly wrong. It doesn’t pass a sniff test: if we actually inserted this range of cloud forcing into the GCMs, will we ever get model temperatures that are below absolute zero or hotter than the Sun? No.
Most people use sniff tests to figure out whether what they’re doing makes sense. You claim that what you’re doing shows the effect of cloud forcing uncertainty in the models, but if those results are plainly not really representative of what would happen in the models, then there’s a disconnect. Your model-of-the-models is plainly off.
As for where the disconnect is? See that linked comment.
This work is wrong. Not even wrong.
Windchaser, see my reply to your reply here.
You may think I’m not actually propagating actual uncertainty, but you’re wrong. Actually I’m propagating actual model physical error into actual uncertainty.
Physics perturbation is not physical error propagation. It does not follow the mathematical form of physical error propagation. It does not propagate error at all. Physics perturbation merely shows the limits of model variability, given a range of parameter uncertainty. That is merely an exploration of model precision, because there is no way to tell which projection, if any, is physically more correct.
If you were following an error analysis protocol standard in physics, every single one of your perturbed physics projections would have it’s own uncertainty envelope derived from propagated error. The propagated error would, in each and every case, include the physical uncertainty widths of your parameter set. Uncertainty would grow as the step-wise root-sum-square of error through every step. The final uncertainty of the ensemble mean would be the root-mean-square of the uncertainties of the individual realizations.
That is the ruthless self-analysis employed within all of physics and chemistry. It is a tough standard and is singularly neglected in climate modeling. Your professional modeler progenitors have set up the system to be very easy on themselves. And your professional perception has suffered for it.
Whatever you think about climate model non-linearity, those same models linearly project air temperature. That is fully demonstrated, both in the ms and in the poster. The rest follows.
You wrote, “Yet, it’s plainly wrong. It doesn’t pass a sniff test: if we actually inserted this range of cloud forcing into the GCMs, will we ever get model temperatures that are below absolute zero or hotter than the Sun? No.”
That statement merely shows that you, like the climate modeler reviewers, have no concept whatever about the meaning of physical error.
Figure it out: physical error statistics are not energetic perturbations They do not impact model expectation values. Repeat those sentences until you grasp their meaning, Windchaser, because your argument about “model temperatures” is complete and utter nonsense.
Look at panel b of the first figure: discrete model relizations embedded within error envelopes. Error was propagated there. Nevertheless, do you see any ‘absolute zero or sun surface temperatures‘ anywhere within?
Again: physical error statistics and uncertainty bars are *not* energetic perturbations. They do *not* impact model realizations. Never. Not ever.
Your criticism is not only wrong, it reflects a profound ignorance of physical error analysis. Wherever you got your education, that institution did you a serious disservice by not including it.
WC, this might help — sure, go ahead and assume the probability of zero or Sun-surface temperatures is arbitrarily low. But the accuracy of a prediction about a totally reasonable temperature can also be arbitrarily low. Imagine monkeys throwing darts at a board with temperatures — it doesn’t matter how reasonable the temperatures on the board are, you don’t have a robust predictive model.
And that’s why we have hilarities like Lamb predicting a “definite downhill trend for the next century.”
You have to stop thinking in terms of model error bars.
Stephan, standard error propagation is in fact not “designed to help estimate the effect of a change in inputs in an output..” It is designed to estimate the reliability of a final result, given the impact of calculational and/or measurement errors in the calculational terms. One finds that in any text on physical error analysis.
You wrote, “However, the less linear the system, the worse the estimate. The climate is a system …”
However, the analysis is about climate models, not about the climate. Climate models project air temperature as a linear extrapolation of GHG forcing. That point is demonstrated in the poster linked in the head-post, and is thoroughly demonstrated to be true in the manuscript. All that business about non-linearity is irrelevant.
So, when you wrote, “the same mistake that seems to appear in a lot of climate models, a false assumption of linearity…” you in fact made the mistake; one of supposing an assumption where there is instead a demonstration.
Climate model air temperature output is demonstrated to be linear. That makes it vulnerable to a linear propagation of error.
Your discussion misses the point of my analysis, entirely.
You wrote, “I really would like to see a proper analysis of physical sensitivities, but that would demand a rerun of the models many times for each parameter to map out the changes in sensitivity.”
That exercise would tell one nothing about model physical accuracy. The effort you propose just measures model precision and is the wrong approach to learning how to model the climate. There needs to be a close collaboration with climate physicists, and a detailed interplay between prediction from theory and observation.
This needs to be done in a reductionist program, one that investigates smaller scale climate physical processes. Eventually small scale knowledge expands and can be collated into larger theoretical constructs. A global model would be the final outcome. It should never have been the first effort.
Global climate models are thoroughly premature. They have been leveraged into acceptance on the back of the pretty pictures available from realistic-like numerical constructs.
Visual realism is seductive, and convincing to the impressionable, the careless, and the fatuous-minded. But realism is not what science is about. It’s about physical reality — a much, much more difficult enterprise.
It is normal to be skeptical of one who criticizes a review board who rejects their work. Except in this case the criticisms build and build to a damning expose of peer review and those who practice it.
The first peer review quote grabbed me immediately.
Holy Batman and Robin, a philosophical position? Does that peer reviewer really see accuracy vs. precision the same as pondering how many angels can dance on the head of a pin?
Unfortunately the belief in Climate science is that the more decimal places you have the more “accurate” the model or predictions are, the more times you calculate roughly the same answer in different ways ignoring error, compound or otherwise, it is impossible to be wrong. This is beyond stupid.
So to be generous I’ll just call climate modelers childish because in a way they are right. You plug 1+2 into a calculator or you plug in 2+1 or you plug in 1+1+1 you always get 3, the calculator is always right. It is precise, but accurate only to the extent you don’t have an idiot manning the calculator who has no idea what “+” means or what 1,2, or 3 represent.
Based on the peer review comments exposed in this article, I ran a model which showed climate modelers to be children allowed to play with big computers. Then changed some parameters and re-ran the model and found that climate modelers veered into delusional psychosis. Based on the error range compounded annually, climate modelers are somewhere between childish and delusionally psychotic.
Alx, there is no evidence that philosophers ever pondered “how many angels can dance on the head of a pin”; it’s a story made up by some scientist in the 19thC and subsequently went viral on the Internet of the day (written correspondence between scientists). Concepts such as accuracy versus precision are the stuff of the philosophy of science; i.e. how do you determine whether you are looking at science or pseudo-science?
PG, don’t like to contradict, but the distinction between accuracy and precision is what separates physical science from philosophy.
Stillman Drake made this point in his excellent book, “Galileo: a very short introduction.” Galileo was the first to attach his theories to the critical test of observation. As Drake notes, attachment to observation fully, completely, and ineluctably separated science from the essences and axioms of philosophy.
Galileo was really the first truly modern physicist; a scientist in the manner we recognize. That’s what got him in so much trouble, especially with the academic philosophs of his day.
Philosophical deductions can be perfectly precise. But their accuracy is a completely nother matter. 🙂
Appreciate your understanding of the problem, Alx. As you note, it’s risky business openly criticizing one’s reviewers. I’ve never done it before, in years of peer-reviewed publication. But this experience was so abnormal, the reviews were so incompetent, and the subject is so widely important, that I just couldn’t keep quiet.
Pat. Illegitimi non carborundum.
Thanks, Harold — I’ll keep at it.
I don’t believe the gatekeepers will go without a fight: