Statistical proof of 'the pause' – Overestimated global warming over the past 20 years

Richardscourtney: Thanks for taking the time to explain the paper I brought up, its truly appreciated. This is the reason I started looking into it, post (not addressed to me) from realclimate – “Read up on Quine and the issue of auxiliary hypotheses. In practice, all theories are ‘wrong’ (as they are imperfect models of reality), and all tests involve multiple hypotheses. Judging which one (or more) are falsified by a mismatch is non-trivial. I have no problem agreeing that mismatches should be addressed, but wait for the post. – gavin]”
Hopefully this will help explain my interest.

This might be good work but it is not part-and-parcel of Quine’s work. Seems to me that it just takes Quine’s logic and adds to it. To those who pursued probabilities for various hypotheses, he remarked that he had no interest in colored marbles in an urn.
Sacrilege! Polya would be turning in his grave! Taleb, on the other hand, might not (partly because “he’s not dead yet!”:-). As his character “Joe the cab driver” (IIRC) in The Black Swan” might say to an analysis of the data above, “It’s a mugs game”. If you flip a two-sided coin 20 times and get heads every time, only an idiot would apply naive probability theory with the assumption of an unbiased coin and claim that the probability of the next flip being heads is 0.5. A Bayesian, however firmly they might have believed it was an unbiased coin initially, would systematically adjust the prior estimate of 0.5 until the maximum likelihood of the outcome coincides with the data and at this point would be deeply suspicious that the coin actually had two heads, that the coin was a magical coin, that the coin was so amazingly weighted and carefully flipped that it had p_head -> 0.999999, that it’s all a horrible dream and there is no real coin. A mugs game.
At this point, I’ll simply amplify what I said above in two ways. First, one can, with an enormous amount of effort, attempt an actual statistical analysis of a (essentially meaningless) composite hypothesis, but nothing of the sort has been attempted for CIMP5 in part because doing so would be spectacularly difficult — right up there with the difficulty of the problem that the GCMs are attempting to solve (which is already one of the most difficult computational problems humans have attempted to solve). The difficulty arises because the theories are highly multivariate and have an abundance of assumptions. Every assumption in every model is subject to Bayes theorem as a Bayesian prior! That is, when one assigns a specific functional form to the radiation profile of the atmosphere with various CO_2 levels, one is — since we cannot precisely compute this and are forced to use one of several approximations (see e.g. Petty’s book) we have to statistically weight the probability that those approximations are correct and downgrade the certainty of our results (our eventual error estimate) accordingly.
To put it more formally, the assertion is that If all the assumptions made constructing the computation are correct, then the model predicts thus and such. However, the assumptions are not certain, and the best estimate of the probability that the model prediction is correct is strictly decreased according to their uncertainty. This can be summarized in the aggregate assumption that “the internal, highly nonlinear, dynamically chaotic differential equations solved by the computer code is correct and sufficiently insensitive to the range of possible error in the Bayesian priors that the output is meaningful in all dimensions (because the code doesn’t just predict temperature, it predicts lots of other things about the future climate as well). Analyzing this precisely for a single theory is enormously difficult, which is why most of the models resort to Monte Carlo to attempt to measure it instead of theoretically predict it. But there are further assumptions built into e.g. the ranges explored by the Monte Carlo itself (more Bayesian priors), into the selection of input variables itself (hard to “Monte Carlo” the omission of a variable that in fact is important), into the granularity and geometry selected (again, difficult to Monte Carlo as the codes may not be written to be length-scale renormalizable in an unbiased way), and one cannot escape the fundamental assumption “this code will correctly predict the multidimensional future in a consistent way and within a useful precision” no matter what you do.
That is the basis for the ultimate null hypothesis, per model. In the end, the model produces an ensemble of results that supposedly span the range of model uncertainty given the priors stated and unstated that went into its construction. If reality fails to lie well within that range in any significant dimension/projection, the model should be considered suspect (weak failure) or overtly fail a hypothesis test depending on how badly reality fails to live within that range.
Imagine attempting to extend this process collectively for all the models in CIMP5! How can one rigorously assess whether approximation A or approximation B for e.g the radiative properties of CO_2 in the atmosphere is most probably correct when the answer could be both are adequate as the basis of a correct theory if everything else is done correctly or neither of them will work because a correct (predictive) theory requires an exact treatment of CO_2’s radiative properties. And then, of course, there are the rest of the greenhouse gases, the non-greenhouse atmosphere, water vapor, clouds, the ocean, aerosols, soot and other particulates, the extent and effect of the biosphere — it is difficult to even count the underlying assumptions that are built into each model and not all of them are in all of the models.
So how can one frame the null hypothesis for CIMP5? “Somewhere in the collection of contributing GCMs is a model that is a reliable predictor of the actual climate”, so that we can then assess a probability of getting the current result if that is true? No, that won’t work. The implicit null hypothesis in the figures above and used by the IPCC in the assessment reports is that “The mean of the collection of models in CIMP5 is a reliable predictor of the actual climate, and the standard deviation of the distribution of results is a measure of the probability of the actual climate arising from the common initial condition given this correct computation of its time evolution”. Which is nonsense, unsupported by statistical theory indeed (as I argue above) unsupportable by statistical theory, and at the end of the day, all that the figure above really demonstrates is that the GCMs are very likely not independent and unbiased in their underlying assumptions because they do produce a creditable Gaussian (with the wrong mean, but seriously, this is enormously unlikely in a single projective variable obtained by a collection of supposedly independent models that otherwise significantly differ in their predictions of e.g. rainfall).
IMO the one conclusion that is immediately justified by the distribution of CIMP5 results is that the GCMs are enormously incestuous, sharing whole blocks of common assumptions, and that at least one of those common assumptions is badly incorrect and completely unexplored by the Monte Carlo perturbations of initial conditions in the individual models. If one performed a similar study of the projective distribution of results in other dimensions one might even gain some insight into just what shared assumptions are most suspect, but that would require a systematic deconstruction of all of the models and code and some sort of gross partitioning of the shared and different features — an awesomely complex task.
The second amplification is a simple observation that I’ll make on the process of highly multivariate predictive modeling itself (wherein I’m moderately expert). There are two basic kinds of multivariate predictive models. An a priori model assumes that the relevant theory is correctly known and attempts to actually compute the result using that theory, using tools like Monte Carlo to assess the uncertainty in the computed outcome where one can (as noted above, one cannot assess the uncertainty linked to some of the prior assumptions implicit in the implementation by Monte Carlo or any other objective way as there is no “assumption space” to sample and nonlinear chaotic dynamics can amplify even small errors into huge end stage differences (see e.g. the “stiffness” of a system of coupled ODEs and chaos theory, although the amplification can easily be significant even for well-behaved models).
In order to set up the Monte Carlo, one has to assign values and uncertainty ranges to the many variable parameters that the model relies upon. This is typically done by training the model — using it to compute a known sequence of outcomes from a known initialization and tweaking things until there is a good correspondence between the actual data and the model-produced data. The tweaking process typically at least “can” provide a fair amount of information about the sensitivity of the model results to these assumptions and hence give one a reasonable knowledge of the expected range of errors predicting the future. One then applies the model to a trial set of data that (if one is wise) one has reserved from the training data to see if the model continues to work. This second stage “validates” the model within the prior assumption “the training and trial set are independent and span the set of important features in the underlying a priori assumed known dynamics”. Finally, of course, the validation process in science never ends.
It doesn’t matter a whit if your model perfectly captures the training data, nails the trial data square on the money, if the first time you compare it to new trial data from the real world it immediately goes to hell well outside of the probable range of outcomes you expected. If you are prospecting for oil with a predictive model, it doesn’t matter if your code can predict previously drilled wells 95% of the time if you only get one oil strike in 100 drilling attempts the first time you use it to direct oil exploration efforts. You are fired and go broke no matter how fervently you argue that your code is good and you are just unlucky. Ditto predicting the stock market, predicting pretty much anything of value. Science is even more merciless than commerce in this regard everywhere but climate science!. Classical physics was “validated” by experiment after experiment in pretty good agreement for a century after its discovery, but then an entire class of experiments could not be explained by classical a priori models. By “could not be explained”, I mean specifically that even if one built a hundred classical models, each slightly different, to e.g. try to predict electronic spectra or the outcome of an electron diffraction experiment, the mean of all of those distinct a priori models would never converge to or in the end have meaningful statistical overlap with the actual accumulating experimental data. They in fact did not, and a lot of effort was put into trying!
The problem, of course, was that a common shared assumption in the “independent” models was incorrect. Further, it was an assumption that one could never “sample” with mere Monte Carlo or any sort of presumed spanning of a space of relevant assumptions, because it was one of the a priori assumed known aspects of the computation that was incorrect, even though (at the time) it was supported by an enormous body of evidence involving objects as large as molecules or small clumps of molecules on up. We had to throw classical physics itself under the bus in order to make progress.
You’d think that we would learn from this sort of paradigm-shifting historical result not to repeat this sort of error, and in the general physics community I think the lesson has mostly been learned, as this sort of process occurs all the time in the delicate interplay between experimental physics and theoretical physics. In some sense we expect new experiments to overturn both great and small aspects of our theoretical understanding, which is why people continue to study e.g. neutrinos and look for Higgs bosons, because even the Standard Model in physics is not now and will never be settled science, at best it will continue to be consistent with observations made so far. A single (confirmed) transluminal neutrino or new heavy particle or darkon can ruin your whole theoretical day, and then it is back to the drawing board to try, try again.
In the meantime, this example shows the incredible stupidity of claiming that the centroid of the projection of a single variable from a collection of distinct priori models with numerous shared assumptions, many of which cannot be discretely tested or simulated, has any sort of statistically relevant connection to reality. Each model one at a time is subject to the usual process of falsification that all good science is based on. Collectively they do not have more weight, they have less. By stupidly including obviously failed models in the average, you without question pull the average away from the true expected behavior.
rgb

rgbatduke (Sept. 6, 2013):
To your list of shortcomings in the methodology of global warming research, you could have added that the general circulation models are insusceptible to validation because the events in the underlying statistical populations do not exist.

Gunga Din (Sept 6 at 10:48):
Thank you for giving me the opportunity to clarify. I did not mean to say “The models can’t be wrong because the models say they are right.” I did mean to say that the models are insusceptible to being validated. This has the significance that the method by which models were created was not the scientific method of investigation. A consequence is that many IPCC conclusions, including the conclusion that global warming is man-made, must be discarded. The previous sentence should not be taken to mean that we know the warming is not man-made.
The widespread view that the models were created by the scientific method is a product of successful use of the deceptive argument known as the “equivocation fallacy” on the part of the IPCC and affiliated climatologists. An equivocation fallacy is a conclusion that appears to be true but that is false or unproved. For details, please see the peer-reviewed article at http://wmbriggs.com/blog/?p=7923 .

kadaka (KD Knoebel) says:
September 5, 2013 at 2:32 am
If you have a one meter squared body of water, how much would a 1600W (1.6 kilowatt) hair dryer heat the body of water over 60 seconds, 60 minutes, from the surface?

To your list of shortcomings in the methodology of global warming research, you could have added that the general circulation models are insusceptible to validation because the events in the underlying statistical populations do not exist.
I could have waxed poetic considerably longer, for example pointing out the recently published comparison of four GCMs to a toy problem that is precisely specified and known and that should have a unique answer. All four got different answers. The probability that any of those answers/models is correct is correspondingly strictly less than 25% and falling fast even if we do NOT know what the correct answer is (the best one could say is that one of the four models got it right and the others got it wrong, but of course all four could have gotten it wrong as well, hence strictly less than). This example alone is almost sufficient to demonstrate a lack of “convergence” in any sort of “GC model space”, although 4 is too small a number to be convincing.
I could also have ranted a bit about the stupidity of training and validating hypothesized global warming models using data obtained from a single segment of climate measurements when the climate was monotonically warming, which may be what you are trying to say here (sometimes I have difficulty understanding you but I think that sometimes I actually agree with what you say:-). When training e.g. Neural Network binary classification models, it is often recommended that one use a training set with balanced number of hits and misses, yesses and noes, because if you have an actual population that is (say) 90% noes and train with it, the network quickly learns that it can achieve 90% accuracy (which is none too shabby, believe me) by always answering no!
Of course this makes the model useless for discriminating the actual yesses and noes in the population outside of the training/trial set, but hey, the model is damn accurate! And of course the solution is to build a good discriminator first and correct it with Bayes theorem afterwards, or use the net to create an ordinal list of probable yes-hood and empirically pursue it to an optimum payoff.
GCMs appear to have nearly universally made this error. Hindcasting the prior ups and downs in the climate record is not in them, to the extent that we even have accurate data to use for a hindcast validation. By making CO_2 the one significant control knob at the expense of natural variations that are clearly visible in the climate record and that are not be predicted by the GCMs as far as I know, certainly not over multicentury time scales, all the models have to do is predict monotonic warming and they will capture all of the training/trial data and there are LOTS of ways to write, tune, initialize a model to have monotonic behavior without even trying. The other symptoms of failure — getting storms, floods and drought, SLR, ice melt, and many other things wrong were ignored, or perhaps they expected that future tweaks would fix this while retaining the monotonic behavior that the creators of the models all expected to find and doubtless built into the models in many ways. Even variables that might have been important — for example, solar state — were nearly constant across the training/trial interval and hence held to be irrelevant and rejected from the models. Now that many of those omitted variables — ENSO, the PDO, solar state are radically changing, now that the physical science basis for the inclusion and functional behavior of other variables like clouds and soot is being challenged, it can hardly be that surprising that the monotonic models that all were trained and validated by the same monotonic interval and insensitive to all of these possibly important drivers continue to show monotonic behavior while the real climate does not as those drivers have changed state?
If the training set for a tunable model does not span the space of possible behaviors of the system being modeled, of course you’re going to be at serious risk of ending up with egg on your face, and with sufficiently nonlinear systems you will never have sufficient data to use as a training set. Nicholas Nassim Taleb’s book The Black Swan is a veritable polemic against the stupidity of believing otherwise and betting your life or fortune on it. Here we are just betting the lives of millions and the fortunes of the entire world on the plausibility that the GCMs built in a strictly bull market can survive the advent of the bear, or are bear-proof, or prove that bears have irrevocably evolved into bulls and will never be seen again. This bear is extinct. It is an ex-bear.
Until, of course, it sits up and bites you in the proverbial ass.
So yeah, Terry, I actually agree. One of many troubling aspects of GCMs is that they have assumptions built into them supported by NO body of data or observation or even any particularly believable theory. They have assumptions that contradict or are neutral to the existing observational data, such as “the PDO can safely be ignored”, or “the 1997-1998 warming that is almost all of the warming observed over the training interval was all due to an improbable ENSO event, not CO_2 per se”, or “solar variability is nearly irrelevant to the climate”. And every one of them is an implicit Bayesian prior, and to the extent that the assumptions are not certain, they weaken the probable reliability of the predictions generated by the models that incorporate them, even by omission.
rgb

Your understanding of the meaning of “validation” is identical to mine. The populations underlying the general circulation models do not exist; thus, these models are insusceptible to being validated.
No scientific model can be verified in the strong sense. All scientific models can be falsified in the strong sense. So what is the point? We could have validated any given GCM in the weak sense by observing that it is “still” predicting global climate reasonably accurately (outside its training/trial where this is not surprising). No interval of observing that this is true is sufficient to verify the model in the strong sense (so that we believe that it can never be falsified, the data proves the model). But plenty of models, including GCMs, could be validated in the weak sense up to the present.
It’s just that they (mostly) aren’t. They are either strongly falsified or left in limbo, not definitively (probably) correct or incorrect, so far.
I do not understand your point about the populations underlying the GCMs, after all. You’ll have to explain in non-rabbit-hole English, with examples, if you want me to understand. Sorry.
rgb