Guest Post by Willis Eschenbach
A learned man was arguing with a rube named Nasruddin. The learned man asked “What holds up the Earth?” Nasruddin said “It sits on the back of a giant turtle.” The learned man knew he had Nasruddin then. The learned man asked “But what holds up the turtle”, expecting Nasruddin to be flustered by the question. Nasruddin simply smiled. “Sure, and as your worship must know being a learned man, it’s turtles all the way down …”
I’ve written before of the dangers of mistaking the results of the ERA-40 and other “re-analysis” computer models for observations or data. If we just compare models to models and not to data, then it’s “models all the way down,” not resting on real world data anywhere.
I was wondering where on the planet I could demonstrate the problems with ERA-40. I happened to download the list of stations used in the CRUTEM3 analysis, and the first one was Jan Mayen Island. “Perfect”, I thought. Middle of nowhere, tiny dot, no other stations for many gridcells in any direction.
Figure 1. Location of Jan Mayen Island, 70.9°N, 8.7°W. White area in the upper left is Greenland. Gridpoints for the ERA-40 analysis shown as red diamonds. Center gridpoint data used for comparisons.
How does the ERA-40 reanalysis data stack up against the Jan Mayen ground data?
Figure 2. Actual temperature data for Jan Mayen Island and ERA-40 nearest gridpoint reanalysis “data”. NCAR data from KNMI. Jan Mayen data from GISS.
It’s not pretty. The ERA-40 simulated data runs consistently warmer than the observations in both the summer and the winter. The 95% confidence intervals of the two means (averages) don’t overlap, meaning that they come from distinct populations. Often the ERA-40 data is two or more degrees warmer in the winter. But occasionally and unpredictably, ERA-40 is 3 to 5 degrees cooler in winter. Jan Mayen’s year-round average is below freezing. The average of the ERA-40 is above freezing. The annual cycle of the two, as shown in Figure 3 below, is also revealing.
Figure 3. Two annual cycles (Jan-Dec) of the ERA-40 synthetic data and Jan Mayen temperature. Photo Source
The ERA-40 synthetic data runs warmer than the observations in every single month of the year. On average, it is 1.3°C warmer . In addition, the distinctive winter signature of Jan Mayen (February averages warmer than either January or March) is not captured at all in the ERA-40 synthetic data.
So that’s why I say, don’t be fooled by people talking about “reanalysis data”. It is a reanalysis model, and from first indications not all that good a reanalysis model. If you want to understand the actual winter weather in Jan Mayen, you’d be well-advised to avoid the ERA-40, or February will bite you in the ice.
The use of “reanalysis data” has some advantages. Because the reanalysis data is gridded, it can be compared directly to model outputs. It is mathematically more challenging to compare the model outputs to point data.
But that should be a stimulus to develop better mathematical comparison methods. It shouldn’t be a reason to interpose a second model in between the first model and the data. All that can do is increase the uncertainty.
In addition, due to the fact that both models involved (various GCMs and the ERA-40) are related conceptually (being current generation climate models), we would expect the correlations to be artificially high. In other words, a model’s output is likely to have a better fit to another related model’s output than it does to observational data. Data is ugly and has sudden jumps and changes. Computer model output is smooth and continuous. Which will fit better?
My conclusion? The ERA-40 is unsuited for the purpose of validating model results. Compare model results to real data, not to the ERA-40. Comparing models to models is a non-starter.
Regards to everyone,
w.
[UPDATE] Several people have asked about the sea surface temperatures in the area. Here they are:
Figure 4. As in Figure 2, but including HadSST sea surface temperature (SST) data for the gridcell containing Jan Mayen. SST data from KNMI
Figure 5. As in Figure 3, but including HadSST sea surface temperature (SST) data for the gridcell containing Jan Mayen. SST data from KNMI
Note that SST is always higher than the Jan Mayen temperature. This is not true for the ERA-40 reconstruction model output.
The obvious…..
This is one perfect example of the concerns of why validation of data with resulting claims is so important. And this is just one example.
With a few more independent surveys of this quality done, there should be sufficient quality control to use to measure the results from the BEST in Berkeley against. We should be able to verify their verification. We must be able to find out where the turtles stop.
We need to be able to see that it’s not just one turtle in between 2 mirrors.
Jim Greig says:
March 8, 2011 at 2:41 am
“If your experiment needs statistics, you ought to have done a better
experiment.”
– Lord Ernest Rutherford
Rutherford said a lot of dumb things in his life. This is one of them.
I think Steve E raises a valid point. The air temp over the land mass is being compared to (modeled) air temp over water, so there’s some comparing of “apples to oranges” going on here.
What I’m curious about is what does the annual pattern of water temps look like around the island? Because if it’s relatively stable (being a huge heat sink) then you’d expect the air temp over the island to be warmer than the air over the ocean during summer month and then cooler in winter months. But if the ocean warms significantly beyond “just above freezing” then the heat sink argument fails. So it would be nice to have data on the water temps too.
I don’t think Steve’s point should be dismissed though. It’s appears to me to be a valid one.
Dave in Delaware says:
March 8, 2011 at 7:35 am
SteveE says: March 8, 2011 at 5:11 am
Grid cell data is not point data and so a direct comparison isn’t an easy thing as this article shows. … The grid cell is an average of the whole area and so won’t reflect the exact value for the point data.
True enough, but my observation was intended to be general, not specific to this island. Where did the ERA-40 reanalysis data come from? It was “homogenized up” from the point data, yes? So it is legitimate to estimate grid area from point data, but not to re-check climate model grid calculations back to that point data?
I am proposing that we close the loop:
1) Point data -> Area grid
2) Grid values -> Climate Model
3) Climate Model calculated value -> Point Data
—————-
I’m not sure how you could back calculate point data from an averaged grid cell. If I gave you an average of ten numbers and then asked you to tell me what the 3rd one was how could you do that?
The method I usually use is to compare the averaged distribution with the point data distribution.
I’d be interested to know if there is a better way of doing it though.
Cheers
Steve
Great work and most interesting. Model calculations that do not properly honor the nearest data points are only slightly more useful then unless, unless the precision and accuracy required is so loose as to render the thing meaningless. I only have one bone to pick. Synthetic data or numbers from any source pretending to be real measured values i.e. data, of anything are an oxymoron. It is just something that bugs me about how we science people communicate between ourselves and with others. I wrote about it in one of my essay “Thoughts About Data,” (http://retreadresources.com/blog/?p=16).
Jim Sorenson says:
March 8, 2011 at 5:54 am
Re: Richard Feynman, o-rings, and ice water
I saw a video years ago in which Dr. Feynman explained he was tipped off to the o-ring demonstration by an employee of Morton-Thiokol, a contractor for the shuttle motor. I don’t find that video on the web at the moment and may not remember it exactly. Nevertheless, I think he would think it strange to find his mental abilities so entangled with this o-ring and ice episode.
For example, the issue of the cold o-rings was known:
http://www.samizdata.net/blog/archives/2004/02/reflections_on_nasas_grim_anni.html
Engineers at Morton Thiokol, the Utah-based firm that made the solid rocket modules, had been concerned about the cold-weather performance of the seals, so much so that they took the unprecedented step of issuing a “no-launch warning” to NASA the day before the doomed flight.
“It is mathematically more challenging to compare the model outputs to point data.”
For some reason that made me think of the line about how an engineer calculates the impact of a charging bull on a matador.
1) To simplify the problem, assume the bull is a sphere…
Seconded. I agree that SteveE has a valid point that this is an “apples to oranges” comparison.
Also I’d be interested in reading SteveE’s response to my ocean gridding question of
March 8, 2011 at 7:25 am above.
To the land v. water temps question, how do the ARGO data compare to remote island temp records…and to comparable grid output?
Thanks willis.
Anthony Watts:
Thanks for the information on Jan Mayen. Your history of the island is interesting in that it shows the relative importance of weather stations. Who but a weather man would forsake home and hearth to go to such a God forsaken place?
In my Ham Radio life, I have contacted Jan Mayen, and that was in CW (Morse Code). The Jan Mayen operator almost had to be one of those weather guys.
Don’t know much about statistics.
Don’t know much about Jan Mayen isle.
But today at least for a while.
I do know that one and one are two.
And I know that it’s quite a trick.
To bend an isle into a Hockey Stick.
* * *
The real story here is boring. Arguments back and forth about statistics are silly to me since just looking at the GISS chart is enough to convince any rational person that there is no upswing in the natural warming trend. Here is a dirt simple plot of the GISS data to squish down ithe ridiculously inflated and thus noise-instead-of-signal-emphasizing Y axis that hides the obvious linearity of the trend: http://oi53.tinypic.com/1qqt6w.jpg
This island is near Iceland and Greenland, north of the UK. The UK also shows a boringly linear trend, and it’s in good company throughout Europe and even North America, going back not 90 years but over 300 (!). Those I have in a single glance here: http://i49.tinypic.com/rc93fa.jpg
Not even the global average as presented by the NOAA on their Climate.org web site shows any sign of divergence from a linear trend, as I plot here minus the usual deceptive chartsmanship tricks here: http://i49.tinypic.com/2mpg0tz.jpg
“turtles all the way down”
Surely someone should mention William James….
Mr. Wright’s comment is correct, but has been used in many other contexts.
The original seems to be Voltaire commenting on god.
The most famous (infamous?) is Adolph Hitler commenting on Jews. That
particular discussion has a long write-up by Eric Hoffer in “The True Believer”.
A good book in many respects, and one that can be read with AGW in mind.
Bob Dylan said it about himself.
Willis:
You state: “The 95% confidence intervals of the two means (averages) don’t overlap, meaning that they come from distinct populations. “
That’s an accurate statement. But, even if the CI’s did overlap, does that necessarily mean that the two means were from the same population?
On Lucia’s Blackboard I believe she had a similar discussion when she was setting up her tests for the IPCC projected temperatures. I don’t remember her conclusion, though.
Would it be better to compare the mean of the model outputs to the CI of the data?
Also, Steve (I think) made the statement that your analysis fails because you are comparing point data with gridded calculations. Is it not the purpose of the gridding process to massage the data within a single cell to make it comparable to a single point?
Thanks for the article. I apologize if my questions are too elementary
Interesting. But I would be great to see a comparison for more then just one single place. Coparison for several places both over continents, in coastal areas, on Ilands and on different latitudes would be interesting. I know it would mean more work, but the result would be more useful.
SteveE says:
March 8, 2011 at 5:33 am
…As the last paragraph says; “the sea temperature around the island may increase from just above freezing” which is what the average for the modelled data is, +0.3C…
That’s 2.1C above freezing for seawater, -0.9C would be ‘just above freezing’ for seawater.
The point data for the island that is covered in snow and ice is unlikely to be representative of the whole grid cell which is modelled composed mostly of ocean.
Your point about temperature over the ocean: I’ve just got one word for you -advection.
The 2m temperature over the ocean depends on where the air mass came from and how long it’s been in place.
SteveE says:
March 8, 2011 at 3:47 am
Good questions all. I’ve shown the centre of several representative gridcells. The one used is the centre point of the nine red points, because it contains the island.
I checked by comparing the adjacent gridcell, but there’s nothing to compare to as far as air temps, only ocean. Looking at ocean temps around Jan Mayen, ocean temperatures in the area range from about 0 to 6°C. However, there’s little correspondence between sea and air temps in the region of Jan Mayen. Finally, the ERA-40 is higher than the sea temps for five months out of the year, which convinced me that the problem is not the surrounding SSTs.
w.
I have a theory …
models are not science … they are not the observation of nature in search of an answer as to how it works … the modelers stopped observing nature a long time ago …
SteveE says:
March 8, 2011 at 5:11 am
Unfortunately, the data don’t bear you out … the ERA-40 is not an average of the Jan Mayen data and the surrounding SST data.
w.
Willis,
Out of curiosity, have you seen this paper?
http://www.scirp.org/Journal/PaperInformation.aspx?paperID=3438
SteveE says:
March 8, 2011 at 6:42 am
Smart men actually run the numbers for the things that they claim before they accuse someone of a big FAIL …
We’re still awaiting your numbers that establish what, to date, is simply handwaving. I’m not a fool, SteveE. I tried to make your theory work first before I said anything about Jan Mayen. I couldn’t find a way to do it.
So I might be wrong, but you’ve not established it.
w.
The last 20 years of temperature records show – using the Hansen scenarios of 1988 as the culmination of all these computer models – the temperature history to be significantly lower than any of the “real world” considerations of the models. If no red rocket hits the planet by 2015, the projections will be so far off the mark the case for CO2 warming as the dominant factor will be untenable. Personally, I’d say the models have already failed.
What is it that still keeps the models in fashion? The temperature rise, the lack of mid-trosopheric heating, the humidity decline, the lack of accelerated storms, and the “missing heat” of the oceans seem – to me – significantly at odds with the models’ predictions. So what keeps them healthy?
To kill the credibility of a Mann or a Jones is one thing, as all you need to do is throw doubt about, or find a failing in one area to tar the rest (not a pun, but it could be). To kill the credibility of model requires at least a lack of correlation to reality. Which we seem to have. Regardless of individual stations – and I understand that 1.5C differences are germane to a “danger” measured in points of a degree – what are the models apparently “getting right”?
So that’s why I say, don’t be fooled by people talking about “reanalysis data”. It is a reanalysis model, and from first indications not all that good a reanalysis model. If you want to understand the actual winter weather in Jan Mayen, you’d be well-advised to avoid the ERA-40, or February will bite you in the ice.
Should be borne in mind when there’s discussion here about the DMI 80ºN Arctic temperature.
Just a problem, Willis.
NCAR and ERA40 have no link.
ERA40 is an old reanalysis by ECMWF, i.e. it’s European.
NCAR and NCEP have their reanalysis and is American.