Above: map of mean temperature and departure by state for February 1936 in the USA, a 5 sigma event. Source: NCDC’s map generator at http://www.ncdc.noaa.gov/oa/climate/research/cag3/cag3.html
Steve Mosher writes in to tell me that he’s discovered an odd and interesting discrepancy in CRU’s global land temperature series. It seems that they are tossing out valid data that is 5 sigma (5σ) or greater. In this case, an anomalously cold February 1936 in the USA. As a result, CRU data was much warmer than his analysis was, almost 2C. This month being an extreme event is backed up by historical accounts and US surface data. Wikipedia says about it:
The 1936 North American cold wave ranks among the most intense cold waves of the 1930s. The states of the Midwest United States were hit the hardest. February 1936 was one of the coldest months recorded in the Midwest. The states of North Dakota, South Dakota, and Minnesota saw the their coldest month on record. What was so significant about this cold wave was that the 1930s had some of the mildest winters in the US history. 1936 was also one of the coldest years in the 1930s. And the winter was followed one of the warmest summers on record which brought on the 1936 North American heat wave.
This finding of tossing out 5 sigma data is all part of an independent global temperature program he’s designed called “MOSHTEMP” which you can read about here. He’s also found that it appears to be seasonal. The difference between CRU and Moshtemp is a seasonal matter. When they toss 5 sigma events it appears that the tossing happens November through February.
His summary and graphs follow: Steve Mosher writes:
A short update. I’m in the process of integration the Land Analysis and the SST analysis into one application. The principle task in front of me is integrating some new capability in the ‘raster’ package. As that effort proceeds I continue to check against prior work and against the accepted ‘standards’. So, I reran the Land analysis and benchmarked against CRU. Using the same database, the same anomaly period, and the same CAM criteria. That produced the following:
My approach shows a lot more noise. Something not seen in the SST analysis which matched nicely. Wondering if CRU had done anything else I reread the paper.
” Each grid-box value is the mean of all available station anomaly values, except that station outliers in excess of five standard deviations are omitted.”
I don’t do that! Curious, I looked at the monthly data:
The month where CRU and I differ THE MOST is Feb, 1936.
Let’s look at the whole year of 1936.
First CRU:
had1936
[1] -0.708 -0.303 -0.330 -0.168 -0.082 0.292 0.068 -0.095 0.009 0.032 0.128 -0.296
> anom1936
[1] “-0.328″ “-2.575″ “0.136″ ”-0.55″ ”0.612″ ”0.306″ ”1.088″ ”0.74″ “0.291″ ”-0.252″ “0.091″ ”0.667″
So Feb 1936 sticks out as a big issue.
Turning to the anomaly data for 1936, here is what we see in UNWEIGHTED Anomalies for the entire year:
summary(lg)
Min. 1st Qu. Median Mean 3rd Qu. Max. NA’s
-21.04000 -1.04100 0.22900 0.07023 1.57200 13.75000 31386.00000
The issue when you look at the detailed data is for example some record cold in the US. 5 sigma type weather.
Looking through the data you will find that in the US you have Feb anomalies beyond the 5 sigma mark with some regularity. And if you check Google, of course it was a bitter winter. Just an example below. Much more digging is required here and other places where the method of tossing out 5 sigma events appears to cause differences(in apparently both directions). So, no conclusions yet, just a curious place to look. More later as time permits. If you’re interested double check these results.
had1936
[1] -0.708 -0.303 -0.330 -0.168 -0.082 0.292 0.068 -0.095 0.009 0.032 0.128 -0.296
> anom1936
[1] “-0.328″ “-2.575″ “0.136″ ”-0.55″ ”0.612″ ”0.306″ ”1.088″ ”0.74″ “0.291″ ”-0.252″ “0.091″ ”0.667″
had1936[1] -0.708 -0.303 -0.330 -0.168 -0.082 0.292 0.068 -0.095 0.009 0.032 0.128 -0.296> anom1936[1] “-0.328″ “-2.575″ “0.136″ ”-0.55″ ”0.612″ ”0.306″ ”1.088″ ”0.74″ “0.291″ ”-0.252″ “0.091″ ”0.667″
Previous post on the issue:
CRU, it appears, trims out station data when it lies outside 5 sigma. Well, for certain years where there was actually record cold weather that leads to discrepancies between CRU and me. probably happens in warm years as well. Overall this trimming of data amounts to around .1C. ( mean of all differences)
Below, see what 1936 looked like. Average for every month, max anomaly, min anomaly, and 95% CI (orange) And note these are actual anomalies from 1961-90 baseline. So that’s a -21C departure from the average. With a standard deviation around 2.5 that means CRU is trimming departures greater than 13C or so. A simple look at the data showed bitterly cold weather in the US. Weather that gets snipped by a 5 sigma trim.
And more interesting facts: If one throws out data because of outlier status one can expect outliers to be uniformly distributed over the months. In other words bad data has no season. So, I sorted the ‘error’ between CRU and Moshtemp. Where do we differ. Uniformly over the months? Or, does the dropping of 5sigma events happen in certain seasons? First lets look at when CRU is warmer than Moshtemp. I take the top 100 months in terms of positive error. Months here are expressed as fractions 0= jan
Next, we take the top 100 months in terms of negative error. Is that uniformly distributed?
If this data holds up upon further examination it would appear that CRU processing has a seasonal bias, really cold winters and really warm winters ( 5 sigma events) get tossed. Hmm.
The “delta” between Moshtemp and CRU varies with the season. The worst months on average are Dec/Jan. The standard deviation for the winter month delta is twice that of other months. Again, if these 5 sigma events were just bad data we would not expect this. Over all Moshtemp is warmer that CRU, but when we look at TRENDS it matters where these events happen
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Rob Findlay says:
September 5, 2010 at 11:31 am
“As other commenters have noted, 5-sigma applies to normal (bell-shaped) distributions. But are these temperatures normally distributed? ”
There is a parallel in finance where the father of the efficient market hypothesis, all three version s of which assume that price changes are distirbuted normally, Egunene Fama has, as a result of observing so many five sigma events in the history of price movements has disgarded the normal distribution in favour of Mandelbrot’s staple Paretian distributions.
As someone who has spent his whole working life in an industry based on using low level statistics I have learnt that if empirical data does not fit the model one should disgard the model not the data.
This seems to be a very goodc example of a classic type of statistical mistake. The mechanics of statistic analysis are deceptively simple, particularly today when we have tools like Excel, SPSS, Matlab etc.
However to actually use these tools correctly you need to have a profound knowledge of the mathematical basis of statistics, of the characteristics of the data you are analyzing and preferably also of the quirks of the particular tool you are using. Unfortunately this is very often not the case,
In this particular case CRU seems to have made one of the most common (and least excusable) errors. They have assumed that their data has a normal distribution without bothering to verify that this is actually true.
Ben says:
September 5, 2010 at 10:20 am
Although it might be as said, just a way to get rid of most of the winter data that is out of the norm, another explanation is the lazy explanation:
they put that into the code to get rid of bad temperature readings…with the assumption that anything outside of 5 sigma was a bad reading. This is a real bad way to do this, but shrug, if you were lazy and didn’t really care, and your research was funded regardless of how well you modeled…well its “good enough for government work.”
And to assume that this is what they did is the lazy way out rather than take the trouble to read the documentation!
What was actually done was to use the 5sigma test as a screen and then examine those data for signs of problems:
“To assess outliers we have also calculated monthly standard deviations for all stations with at least 15 years of data during the 1921–90 period. All outliers in excess of five standard deviations from the 1961–90 mean were compared with neighbors and accepted, corrected, or set to the missing code. Correction was possible in many cases because the sign of the temperature was wrong or the temperature was clearly exactly 10
Maybe it is the moderate climate of the UK that lends the UK researchers towards tossing very cold temperatures. They don’t know what to do with that; they never experienced anything that cold so subconsciously they decide to ignore it and code their programs that way.
So, i’d trust the Russians more about this.
Ok on the first graph there is a black line, a blue line, and red line…
What does the black line represent?
What does the Blue line represent?
What does the Red line represent?
I obviously have similar questions for all the other graphs.
Phil. says:
September 5, 2010 at 8:57 am
“[…]Seems prudent.”
Phil., i hope you don’t work in avionics?
I hate it when they make one blanket programming assumption to attempt and filter the data in some way without actually doing the footwork necessary to determine of what they are ignoring is actually valid or not. Just because it’s 5 sigma doesn’t automatically mean it’s bad. What the heck are they doing here? These data sets are becoming nothing but little playgrounds for the manipulators.
John A says: September 5, 2010 at 8:51 am
“If this is true, then it cannot be seen as anything other than scientific misconduct, unless the CRU can justify this step on physical grounds.”
This is just nuts. The criterion is clearly stated in their published paper, which is Mosh’s source. You can argue about whether it is a good idea, but there’s nothing underhand.
Charles S. Opalek, PE says: September 5, 2010 at 9:39 am
“Or, is it only cold data like 1936, or Orland CA (1880 and 1900), that gets tossed?
The big agw lie rolls on.”
Most arguments around here are that the 30’s were warmer than what the scientists say. Omitting cool 30’s data reduces the apparent warming trend.
Rob Findlay says: September 5, 2010 at 11:31 am
” But are these temperatures normally distributed?”
Good question (although they are talking anomalies here). A lot of the argument about whether recent warming is statistically significant etc is based on that dubious assumption.
Of course, the real question no-one is looking at much here, is whether tossing a few out-of-range months for individual station data really matters either way.
Looks like lazy programming and little effort to analyze the impact of the code. It’s probably a lot more difficult to do a good quality check for bad data and as long as you’re getting the results you expect ……….
Re is the temperature data distributed normally?
This link shows a distribution chart for monthly average temperature data for Abilene, Texas, from Hadley’s CRUT3 that was posted to the web shortly after the Climategate event of November 2009. The distribution is in one-degree C increments.
http://tinypic.com/r/n4xeo3/7
Having but simple statistical skills, this seems anything but normally distributed to me. I.e, there is no typical bell-curve shape, this has one “lobe” below the mean, and another “lobe” above the mean.
Perhaps others can comment on that chart, which was generated by me the hard way, in Excel (TM) but without using their statistics pack. Any errors or other faults are mine alone.
Just to be clear. As Ron Notes these are my working notes. I’m basically working through my stuff comparing it with CRU. The SST stuff matched pretty well, and when there was an difference I wrote to Hadley the difference was reconciled. With the Land portion I’ve never tried matching them exactly, so I figure I should give it a go. The large monthly deviations, seemed odd, so I went to back to Brohan 06 and sure enough, found the step documented. a quick spot check of the worst month and I know enough to move on and get back to this issue later. As for it’s overall impact? well I have to finish a bunch of other work. THEN, one can answer the question “does it make any real difference.”
Of coure, they omit 5-sigma data. They have to make sure to cook the numbers to support the CAGW hypothesis.
nick:
“This is just nuts. The criterion is clearly stated in their published paper, which is Mosh’s source. You can argue about whether it is a good idea, but there’s nothing underhand.”
Ya, I’m glad they documented it. I had no clue why my numbers had so much variability in certain months and with prototype code testing the prospect that I had made a mistake was foremost in my mind.
DirkH says:
September 5, 2010 at 1:12 pm
Phil. says:
September 5, 2010 at 8:57 am
“[…]Seems prudent.”
Phil., i hope you don’t work in avionics?
Why Dirk, would you rather not eliminate bad data and spurious points?
Tenuc says:
September 5, 2010 at 12:10 pm
Once again the CRU have been spotted throwing away the data, what a big bunch of tossers they are!
Once again some posters here have been spotted jumping the gun because something supports their prejudices rather than check the facts!
Ron’s got a really good post on more of the reason behind the difference.
http://rhinohide.wordpress.com/2010/09/05/mosher-deviant-standards/#comment-967
He tracks it down to a different reason than tossing 5sigma events.
Richard M says:
September 5, 2010 at 1:20 pm
Looks like lazy programming and little effort to analyze the impact of the code. It’s probably a lot more difficult to do a good quality check for bad data and as long as you’re getting the results you expect ……….
Looks like prudent checking of the data to me, then again I’m talking about what they actually do not your fantasy of what they do.
Met Office Meets “my backside”.
“Australia – Global warming my backside with the coldest day in 100 years
12 Jul 2010
WIDESPREAD cloud and persistent rainfall has kept temperatures down right across Queensland.
Longreach, in the Central West, received persistent rain from Tuesday evening, dropping the temperatures by about four degrees. The temperature then barely moved yesterday, reaching a maximum of 11 degrees; 12 degrees below the long-term average and the coldest July day in 44 years of records.Isisford, further south, was even colder, getting to just 10 degrees. This was the town’s chilliest day since before records began in 1913, almost a century ago.”
http://www.meattradenewsdaily.co.uk/news/140710/australia___global_warming_my_backside_with_the_coldest_day_in__years.aspx
…-
“Temperature records to be made public
Climate scientists are to publish the largest ever collection of temperature records, dating back more than a hundred years, in an attempt to provide a more accurate picture of climate change.”
Climate scientists have come under intense pressure following the Climategate scandal at the University of East Anglia, where researchers were criticised for withholding crucial information, meaning their research could not be independently checked.
Sceptics have also attacked climate change research over the quality of the records being used as evidence for the impact mankind has had on the world’s climate since the industrial revolution.
But the Met Office, which is hosting an international workshop with members of the World Meteorological Organisation to start work on the project, is now planning to publish hourly temperature records from land-based weather stations around the world.”
http://www.telegraph.co.uk/earth/environment/climatechange/7981883/Temperature-records-to-be-made-public.html
http://www.smalldeadanimals.com/archives/014798.html
Steve
Thanks for your exploring – Interesting results.
This suggests that there may be more cold excursions from arctic weather than warm excursions from the tropics. This may be systemic in that warm events may be moderated by ocean evaporation, while cold swings in the Arctic may have nothing comparable to dampen the swing.
Steven mosher says:
September 5, 2010 at 1:54 pm
nick:
Ya, I’m glad they documented it. I had no clue why my numbers had so much variability in certain months and with prototype code testing the prospect that I had made a mistake was foremost in my mind.
Ron suggests that it your choice of data set that is responsible for the differences, when he disabled the 5-sigma algorithm it made no difference to the results.
http://rhinohide.wordpress.com/2010/09/05/mosher-deviant-standards/
Fred H. Haynie says: Five sigma from what, a thirty year average?
Only if below their desired average – never if above!
The central problem is one of credibility. Neither the CRU nor any other agency has credibility if they will not publicly archive the raw data, with an unbroken chain of custody.
Whether they post their adjusted data is immaterial. The original, raw data is what matters. In fact, it is all that really matters. Throwing out data, whether it changes the final result or not, is done for manipulation.
Publicly archive the raw data for the public that pays for it.
If I had presented an assignment at university with such sloppy and incomplete graphs, it would have been returned without marking and I would have received no credit.
The standard of presentation at this site is slipping.
Some of the critics of WUWT say there is no science evident here. This posting certainly supports that observation.
Here’s one example of how this was reported in Australia at the time:
The Argus Friday 14 February 1936
NORTH AMERICA FREEZES IN ANOTHER COLD WAVE
Never in living memory has the North American continent experienced such cold
weather as at present.
Rest at: http://newspapers.nla.gov.au/ndp/del/article/11882393
Benoit Mandelbrot has a lot to say about chaotic data sets. In his book “The (mis)Behaviour of Markets”, he demonstrates that it is a serious fallacy to assume that the stockmarket behaves as a normal or Gaussian distribution, with standard deviation. On that assumption, five or six-sigma events are (would be) extremely rare; reality contradicts the theoretical… er… model, and these pesky booms and busts come along more frequently than the… er… model says they ought to.
Same principle with….. ah, but you’re ahead of me, aren’t you!