Comparing climate models – Part One

Do you have that data as a .csv or .txt file?

I don’t see much point in this exercise. Without knowing anything about “physics” involved in this model it’s hard to expect anything but that the evolution will continue at approximately the same cosnstant rate as in the recorded period. Which of course could work unless the real model comes with some kind of effect which will kick in no sooner than at step 21. And that’s what I actually expect to happen, otherwise there would be no point in making this article.
I also don’t see any relation between this exercise and the Spencer/Christy graph and rgbduke’s post.

At first glance I would need 8 possible outcomes for the next 3 time steps of each of the 18 series. Besides giving an 8 value range for time interval 23 for each of the 18 series, this has no predictive skill. Averages or trends have no predictive ability on data generated in this manner.

and there are no steps of 8, so this appears to be a simple binary walk multiplied by 4. Why bother with the scaling?

Here is what I did:
I subtracted column 1 from column 20, divided by 20, then times by 3, to establish a rough approximation of the rate for each series (Column 23 =(((U2-B2)/20)*3)+U2). I then filled in column 21 and 22 with data greater or equal to Column 20, but not greater than Column 23, using only increments of 0 or 4. Since I generated some odd numbers in Column 23, these are rounded to the nearest even number, whilst trying to maintaining increments of 4.
The average for Column 23 is 43 (Rounded to 44)
I believe my method is a sufficient approximation of the data, was easy, and I look forward to comparing in part 2 .
Thanks

If it was possible to guess at a model with faulty and/or poor understanding of the underlying mechanisms and then project that into the future with any degree of accuracy, these guys wouldn’t be fooling around with climate models but with stock market models.
The forces behind the climate are clearly not understood. Therefore no model can possibly have any accuracy in exrapolation. These models aren’t even good at interpolating the real observed data.

It’s dangerous to project a trend when you don’t know what the numbers represent. E.g., guess the next number in the series: 46, 62, 73, 80, 47, 48, 53, 42…
These figures are the price of Bank of America stock at the beginnings of the years 2001 – 2008. The next figure in the series turned out to be 13.

@David L Hagen – Thanks for the link to Fred Singer’s paper. His 400 model year limit is interesting. I compared with http://journals.ametsoc.org/doi/pdf/10.1175/BAMS-D-11-00094.1
(top of p 491, col 2)

. If all ensemble members are considered, then nearly 2,800 yr on average were simulated per CMIP3 model, but the total years varied substantially from one model to another (500–8,400 yr with a median of 2,200 yr). The total amount of CMIP3 model data archived was
about 36 TB. In contrast for CMIP5, the long-term and near-term core experiments alone call for, at minimum, ~2,300 yr, approximately matching the number of years in CMIP3.

Based on the above, it would seem that all CMIP3 model submissions met Singer’s limit, and that all CMIP5 model submissions far exceed it.

” . . .the reason why the ensemble has more skill is simple.”
Maybe I don’t understand the technical term “skill”, but it appears that the ensemble average has _less_ skill than the individual models that are below the average and closer to the actual measured temperature observations.
http://www.drroyspencer.com/wp-content/uploads/CMIP5-73-models-vs-obs-20N-20S-MT-5-yr-means1.png

Here is my guess.
21 22 23
1 56 60 64
2 64 68 72
3 64 68 72
4 64 68 72
5 52 56 56
6 48 52 56
7 48 52 52
8 48 48 52
9 44 48 52
10 44 44 48
11 44 44 44
12 36 40 40
13 32 32 32
14 24 24 24
15 20 20 20
16 12 16 16
17 8 8 12
18 8 8 8

For Geoff Sherrington:
Thanks for your response; but a couple more questions/comments.
(1) Were the 18 experiments run simultaneously or in succession (i.e., is t = n the same clock time for each run or not)
(2) If experiments were run in succession (1) were the experimental results listed in run order (1-18);
These questions are motivated by the observation that the first 10 time steps exhibit 9 state transitions among the 18 runs, time steps 11-13 show 6 transitions, and then steps 14-20 revert to 9 transitions. Yet the frequency/probability of transition drops significant as the run number increases. What confuses me is: if all experiments are identical (except for the “pair of series” you mentioned), my engineering mind wants to reject the likelihood of the correlation between “Y” and run order as shown in your spreadsheet. Also, the 9-6-9 column totals for state transitions, makes me question the assertion that there is no algorithm involved in the construction of these 18 series. . . the pattern seem to be too good to be real!
No complaint though, . . . just figuring that if I ask enough questions and stir the pot, maybe you’ll let enough slip, so that even I can get to the bottom of this puzzle!!
Thanks for the fun.
Dan

“Energy” you say, hmm …

21 22 23
1 60 64 68
2 64 64 68
3 60 64 68
4 60 64 68
5 52 56 56
6 52 52 56
7 48 52 52
8 52 56 60
9 40 44 46
10 40 44 44
11 44 44 44
12 36 38 42
13 32 32 32
14 20 22 22
15 20 20 20
16 12 16 16
17 8 12 12
18 8 8 10


1 64,40 68
2 69,00 72
3 69,00 72
4 69,00 72
5 59,80 58
6 55,20 54
7 50,60 50
8 55,20 54
9 46,00 46
10 46,00 46
11 50,60 50
12 41,40 42
13 32,20 34
14 23,00 22
15 23,00 22
16 13,80 14
17 9,20 8
18 9,20 8
Column 2 = Best estimate
Column 3 = Iterative, using steps 0 or 2 or 4

Here are some potentially salient facts I’ve noticed about the data set presented which would have bearing on the projections.
First is as follows: while at first glance the data looks like a progression of coin flips, (as DanMet’al pointed out above) for all time steps except 10-11, 11-12, and 12-13, each time step consisted of exactly nine models incrementing by four and nine models remaining flat. For the three time steps mentioned, each time step consisted of six models incrementing by four and the remaining twelve staying flat. This is too consistent to be random chance, indicating that there is a strong anti-correlation between certain models and certain others.
Next, the anti-correlations are not pair-wise; there is one model which is still zero at time step 10, but no model which has reached 40 at that time step.
Geoff mentioned the word “energy”, which means that this growth behavior may be indicative of some sort of conservation law: that is, the total growth of the overall system of models must equal some specified input: 36 for each of the “large growth” timesteps, 24 for each of the “small growth” time steps.
Likewise, Geoff indicated that for individual models an mid-range growth value of two was possible, although not observed in his data set, indicating that it is very low probability – it is probably safe to assume that, unless the overall system has changed character after time step 20, we are unlikely to see an increment of 2 in the next three time steps.
The problem here is that the system exhibits at least two growth states – that exhibited between time steps 10-13, and that exhibited for all other time steps – and insufficient data to estimate a periodicity for that state change. There was a ten time-step period between the beginning of the model and the onset of the first state change, which naively indicates that we shouldn’t expect the onset of the next state change until at least time step 24, but there is no reason to believe that this state change has a simple periodic behavior.
Basically, as of this point, the problem is under-determined. (Of course, that’s always true; extrapolation from any series of data is impossible unless we have some outside indication as to what those data are supposed to represent and how to construct the appropriate basis elements for forming the series. The set of integer-valued functions on the space of integers is infinite.)
Given the rules you laid out (including knowledge of the final ordering), I could come up with a reasonably small parameter space of allowed final values of the system as a whole based on the continuation of the “36-energy” state or the onset of the “24-energy” state at either the 21st, 22nd, or 23rd time step. I could even estimate the likelihood of each given element within the parameter space by counting the number of paths to reach that element in comparison to the total number of allowed paths. I could do further weighting by estimating correlations between specific models, and three-model correlations, and four-model correlations, and so forth and so on – or correlations based on time-shifted “wiggle matching,” or combinations thereof, or several other possible techniques. I don’t see any particular reason to think that these further weightings are meaningful, though – cursory inspection of the data set seems to indicate that, other than the overall increment law, any further correlations between models are likely to be spurious and thus the best weighting is the naive path-number weighting.
For that matter, despite having taken the time to write up how I would go about producing an estimate, it seems like too much effort to actually do so for what amounts to a toy problem. Geoff’s point about the difficultly of forecasting is well-made, and I have explained here how I would go about the process. That seems more important than actually doing it.
As a final note, given that the system has two states, what reason is there to believe that it does not have more than two states? This goes back to the “under-determined” issue – in the absence of evidence we are forced to assume evidence of absence. Based on the utility as regards “proving a point,” however, I strongly suspect that we are in fact looking at a three state system with a third state which onsets at time-step 21. My reasoning for this is purely psychological – if I were setting up a toy problem to demonstrate the pitfalls of forecasting, that is exactly what I would do – but it reinforces my decision not to attempt to provide a forecast myself.

# 9, #9, #9……
(Beatles)

So, given that the simple model failed, new model time!
We’ll call this one the “if I’m bigger I eat more” model, which is actually a one-parameter family of models where the probability for a specific line to increment is determine by the availability of food (AKA, the total energy provided, in blocks of 4) and the size of that model in comparison to the total size of the system. The parameter is the “base” or initial size, because otherwise the probabilities are ill-defined at time step zero (and, after time step zero, any line with a size of zero will never eat).
So, schematically, the likelihood that a model increments is ((size + a)/(total size + 18*a))*(food supply), where a is our single parameter. To make sure that all of the food is consumed at each time step, we’ll work our way from largest to smallest, adjusting probabilities based on remaining food and the total size of the portion of the system that hasn’t had a chance to eat yet.
Actually, I’m not going to bother to code this model up, because I can already see where this is headed: I can control dispersion based on the parameter a (large a makes this model indistinguishable from the naive model; small a means only the big guys ever get to eat), but then I’d start testing for better fits at eat time step, or testing the dispersion in more detail than just looking at the envelope, and for each such discrepancy I found I’d need to add a parameter to fix it. With enough parameters, I could make the fit perfect – all without ever considering whether or not my initial guess for the “big eats” type of model was correct.
In other words, I’d be doing Climate Modelling! Yay!