Above: map of mean temperature and departure by state for February 1936 in the USA, a 5 sigma event. Source: NCDC’s map generator at http://www.ncdc.noaa.gov/oa/climate/research/cag3/cag3.html
Steve Mosher writes in to tell me that he’s discovered an odd and interesting discrepancy in CRU’s global land temperature series. It seems that they are tossing out valid data that is 5 sigma (5σ) or greater. In this case, an anomalously cold February 1936 in the USA. As a result, CRU data was much warmer than his analysis was, almost 2C. This month being an extreme event is backed up by historical accounts and US surface data. Wikipedia says about it:
The 1936 North American cold wave ranks among the most intense cold waves of the 1930s. The states of the Midwest United States were hit the hardest. February 1936 was one of the coldest months recorded in the Midwest. The states of North Dakota, South Dakota, and Minnesota saw the their coldest month on record. What was so significant about this cold wave was that the 1930s had some of the mildest winters in the US history. 1936 was also one of the coldest years in the 1930s. And the winter was followed one of the warmest summers on record which brought on the 1936 North American heat wave.
This finding of tossing out 5 sigma data is all part of an independent global temperature program he’s designed called “MOSHTEMP” which you can read about here. He’s also found that it appears to be seasonal. The difference between CRU and Moshtemp is a seasonal matter. When they toss 5 sigma events it appears that the tossing happens November through February.
His summary and graphs follow: Steve Mosher writes:
A short update. I’m in the process of integration the Land Analysis and the SST analysis into one application. The principle task in front of me is integrating some new capability in the ‘raster’ package. As that effort proceeds I continue to check against prior work and against the accepted ‘standards’. So, I reran the Land analysis and benchmarked against CRU. Using the same database, the same anomaly period, and the same CAM criteria. That produced the following:
My approach shows a lot more noise. Something not seen in the SST analysis which matched nicely. Wondering if CRU had done anything else I reread the paper.
” Each grid-box value is the mean of all available station anomaly values, except that station outliers in excess of five standard deviations are omitted.”
I don’t do that! Curious, I looked at the monthly data:
The month where CRU and I differ THE MOST is Feb, 1936.
Let’s look at the whole year of 1936.
First CRU:
had1936
[1] -0.708 -0.303 -0.330 -0.168 -0.082 0.292 0.068 -0.095 0.009 0.032 0.128 -0.296
> anom1936
[1] “-0.328″ “-2.575″ “0.136″ ”-0.55″ ”0.612″ ”0.306″ ”1.088″ ”0.74″ “0.291″ ”-0.252″ “0.091″ ”0.667″
So Feb 1936 sticks out as a big issue.
Turning to the anomaly data for 1936, here is what we see in UNWEIGHTED Anomalies for the entire year:
summary(lg)
Min. 1st Qu. Median Mean 3rd Qu. Max. NA’s
-21.04000 -1.04100 0.22900 0.07023 1.57200 13.75000 31386.00000
The issue when you look at the detailed data is for example some record cold in the US. 5 sigma type weather.
Looking through the data you will find that in the US you have Feb anomalies beyond the 5 sigma mark with some regularity. And if you check Google, of course it was a bitter winter. Just an example below. Much more digging is required here and other places where the method of tossing out 5 sigma events appears to cause differences(in apparently both directions). So, no conclusions yet, just a curious place to look. More later as time permits. If you’re interested double check these results.
had1936
[1] -0.708 -0.303 -0.330 -0.168 -0.082 0.292 0.068 -0.095 0.009 0.032 0.128 -0.296
> anom1936
[1] “-0.328″ “-2.575″ “0.136″ ”-0.55″ ”0.612″ ”0.306″ ”1.088″ ”0.74″ “0.291″ ”-0.252″ “0.091″ ”0.667″
had1936[1] -0.708 -0.303 -0.330 -0.168 -0.082 0.292 0.068 -0.095 0.009 0.032 0.128 -0.296> anom1936[1] “-0.328″ “-2.575″ “0.136″ ”-0.55″ ”0.612″ ”0.306″ ”1.088″ ”0.74″ “0.291″ ”-0.252″ “0.091″ ”0.667″
Previous post on the issue:
CRU, it appears, trims out station data when it lies outside 5 sigma. Well, for certain years where there was actually record cold weather that leads to discrepancies between CRU and me. probably happens in warm years as well. Overall this trimming of data amounts to around .1C. ( mean of all differences)
Below, see what 1936 looked like. Average for every month, max anomaly, min anomaly, and 95% CI (orange) And note these are actual anomalies from 1961-90 baseline. So that’s a -21C departure from the average. With a standard deviation around 2.5 that means CRU is trimming departures greater than 13C or so. A simple look at the data showed bitterly cold weather in the US. Weather that gets snipped by a 5 sigma trim.
And more interesting facts: If one throws out data because of outlier status one can expect outliers to be uniformly distributed over the months. In other words bad data has no season. So, I sorted the ‘error’ between CRU and Moshtemp. Where do we differ. Uniformly over the months? Or, does the dropping of 5sigma events happen in certain seasons? First lets look at when CRU is warmer than Moshtemp. I take the top 100 months in terms of positive error. Months here are expressed as fractions 0= jan
Next, we take the top 100 months in terms of negative error. Is that uniformly distributed?
If this data holds up upon further examination it would appear that CRU processing has a seasonal bias, really cold winters and really warm winters ( 5 sigma events) get tossed. Hmm.
The “delta” between Moshtemp and CRU varies with the season. The worst months on average are Dec/Jan. The standard deviation for the winter month delta is twice that of other months. Again, if these 5 sigma events were just bad data we would not expect this. Over all Moshtemp is warmer that CRU, but when we look at TRENDS it matters where these events happen
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
As Mosher points out, this is documented in the papers describing HadCRUT and CRUTEM.
http://www.cru.uea.ac.uk/cru/data/temperature/HadCRUT3_accepted.pdf
http://www.sfu.ca/~jkoch/jones_and_moberg_2003.pdf
Yup, alledged CO2 warming does NOT cause worse temperature extremes than what we have actually recorded before the period when man made CO2 is supposed to be causing problems!!
Records indicate that the Nile river has frozen over at least twice.
Those events were probably six sigma events.
But they were real, none the less.
As Mosher points out, this is documented in the papers describing HadCRUT and CRUTEM.
So it’s alright, then.
The decade of the 1930’s is home to 25 of the 50 US State all-time hottest day records of any decade since 1880. Doesn’t any of this warming data get tossed, too?
Or, is it only cold data like 1936, or Orland CA (1880 and 1900), that gets tossed?
The big agw lie rolls on.
Its well known that blow freezing temperatures are unstable–this is because the specific heat of water is twice as high as the specific heat of ice.
“5-sigma” refers to five standard deviations outside of the average value. In experimental science, measuring such an extremely divergent data point might be an indication that something had gone temporarily wrong in the data collection process. For example, maybe a power glitch affecting an instrument, or contamination getting into a sample. In the absence of any way to verify that the specific data point was a valid one, a scientist might simply screen it out of the data set. There is a statistical argument for doing this, based on the fact that for a normally-distributed (bell curve) population of data, a “5-sigma” value should occur only one in a million times.
However – and this is a big however – if you have independent corroboration of the 5-sigma data point, you really should not throw it out of your data set. For example, the 1936 data set shows extreme cold values all over the north-central U.S., which were independently recorded. This is not simply one thermometer experiencing a glitch. Add to this the written historical records which provide a softer verification of the extreme temperatures. (I say softer, because any narrative record would have been dependent on the thermometer recordings for its objectivity.)
The irony here, of course, is that the CRU is rejecting “extreme weather events” that occurred in the past to arrive at its projections of CAGW, at the same time that extreme weather events in the present (including extreme cold snaps) are being embraced as proof positive that CAGW is happening Today! Yes indeedy, It’s the End of the World as we Know It!!
No, no, NO! That’s not the way to make the past colder and the present warmer.
They should be keeping the 5-sigma cold events for pre-1970, and tossing them for post-1970. Then, toss the 5-sigma warm events for pre-1970, and keep them for post-1970.
See? Instant global warming.
@ur momisugly kadaka: 5-sigma refers to data that lies more than 5 standard deviations from the mean. Sigma is shorthand for standard deviation, taken from the Greek letter sigma. For data that follows a normal distribution, half will lie at less than the mean, and half will lie at greater than the mean. The probability of an event occurring within a given number of standard deviations is:
+ – 1 sigma: 68.2 percent
2 sigma: 95 percent
3 sigma: 99.7 percent
There is also a slightly different formulation, based on “Six Sigma” manufacturing, which looks at defects. Here’s a link:
http://money.howstuffworks.com/six-sigma2.htm
kadaka,
Sigma is the Greek character used to denote standard deviation. Thus a ‘5 sigma’ data point is one that lies more than five standard deviations from mean. It suggests that the data point is an outlier. Or, using the real meaning of the word, it is an anomaly.
Although it might be as said, just a way to get rid of most of the winter data that is out of the norm, another explanation is the lazy explanation:
they put that into the code to get rid of bad temperature readings…with the assumption that anything outside of 5 sigma was a bad reading. This is a real bad way to do this, but shrug, if you were lazy and didn’t really care, and your research was funded regardless of how well you modeled…well its “good enough for government work.”
Upper Amazon in drought.
http://www.reuters.com/article/idUSTRE6825EU20100903
Left unsaid in article is that Peru and Bolivia are facing another record cold year.
A breaking story.
http://www.heraldscotland.com/news/politics/holyrood-fiasco-peer-s-40k-for-chairing-climategate-review-1.1052947
It’s interesting that your graphs are very consistent with forest fire intensity/severity for the 20th century. About midway in my career (forestry) I suggested that there appeared to be a distinct synchronicity to the fire seasons that must correlate wih some Uber-climatic trend beyond what we got from the day-to-day weather predictions, and maybe even multiple trends that periodically became in-phase and led to our periodic large fire years. It was suggested that I might be better served to continue cruising timber.
If the seasonal bias is in the winter, are there an equal amount of warm 5+ sigmas and cold 5+ sigmas?
The same kind of tossing is done in the USHCN dataset creation and I would expect something similar in the creation of the GHCN. Some details on the kinds of tossing done are here:
http://chiefio.wordpress.com/2010/04/11/qa-or-tossing-data-you-decide/
with pointers to the referenced documents.
So, good catch Mosh!
Also, it is my belief that this kind of low temperature data tossing is why we get the “hair” clipped on low side excursions in the “pure self to self” anomaly graphs I made. The onset of the ‘haircut’ is the same as the onset of the “QA Procedures” in the above link…
This example simply uses a very large baseline more or less standard approach and still finds the “haircut”:
http://chiefio.wordpress.com/2010/07/31/agdataw-begins-in-1990/
Canonical world set of dT/dt graphs (a variation on “first differences):
http://chiefio.wordpress.com/2010/04/11/the-world-in-dtdt-graphs-of-temperature-anomalies/
A more “standard” version shows the same effect:
http://chiefio.wordpress.com/2010/04/22/dmtdt-an-improved-version/
So take that 1936 case, and start having it applied to the daily data that goes into making up the monthly data that goes into making things like GHCN, and suddenly it all makes sense… And it explains why when you look in detail at the trends by MONTH some months are ‘warming’ while others are not…
It’s not the CO2 causing the warming, it’s complex and confusing data “QA” that drops too much good data and selective data listening skills…
Oh, and selectivity as to when volatile stations are in, and not it, the data set. High volatility in during cold excursion baseline, then taken out at the top of a hot run, so that the following cool does not have a chance of matching the prior cold run.
http://chiefio.wordpress.com/2010/08/04/smiths-volatility-surmise/
Basically, they sucked their own exhaust and believed their own BS instead of doing a hard headed look at just what they were doing to the data. Same human failure that has brought down programmers, tech company startups, and stock system traders for generations.
It should have been a cold winter.
The previous four sunspot cycles were cooler. This allowed for cooler years in cycle boundaries.
The Sept. 1933 to Feb. 1944 was the first Global-Warming cycle of that century.
Sept. 1933 was the first year of the cycle. 1934 marked the “Dust Bowl”. The average rain for the US, was just above 23 inches, 4 inches below average. It was a warm winter of an average 36 degrees.
1936 was about the same in precipitation,
The average winter temperature for that winter per the NOAA was 28.54, 2d lowest from 1896 to 2008.
Sunspot activity for the previous year, mean of 36.10 and for 1936, a mean of 79.7.
Thus, I have to ask the question, how much glacier ice was melting? Did the sudden up swing in average US temperatures in 1926 and 1927; 1931 and 1932; and 1934 and 1935 cause major melting of the Polar Ice Caps and Northern hemisphere glaciers. Was the runoff cooling the oceans and the USA?
Is the cause of the US cooling as the planet began to warm up from a 54 year dormant sunspot cycle period?
In review of Glacier Bay maps, there was little lost in the glaciers from 1907 to 1929. That significantly changed with the warmer sunspot cycles. There doesn’t appear to be new measurements of the Bay until after WWII.
The milder sunspot cycles showed continued melting of the Glacier Bay Glaciers. There does appear to be a difference between 1907 to 1929 and from 1930 to 1949 in terms of ratio.
The Muir Glacier melted almost twice as fast from 1930 to 1948 as it did from 1907 to 1929. From 1930 to 1948 (20 years) it melted 5 miles up the Muir Inlet. From 1907 o 1929 (23 yrs) about 2.5 miles with a curve in the inlet.
From Wolf point to the 1948 melt point the inlet thins out a bit. Not sure what that would have done to the melt speed. Many factors.
In the Wachusett Inlet, the glacier melted roughly 3.5 miles, a mile and a half more for the years 1930 to 1949 in comparison to the 1907 to 1929 melt period covering 2 miles of melt. There is a two-year period difference in the two time periods.
Why mention that? Glaciers have been melting since that last Mini-Ice Age, but researchers were not watching the ice melt.
In today’s Alaska, glaciers are melting at about 60 feet a year at Seward and Juneau, Alaska.
1936 had 9 tropical storms, 7 hurricanes and one major hurricane. That was the largest hurricane season for the cycle. 1937 was the peak cycle year for 1933 to 1943 sunspot cycle.
There were No La Ninas and no El Ninos. Does that tend to reflect a cold year? I have no proof of that.
This repeated again in 1979 to 1980.
When glaciers and Polar Ice Caps are maxing at the end of a cool susnpot cycle, hurricanes pick up and La Ninas and El Ninos drop off.
Strange!
Paul
For us more ordinary people, please point out what “5 sigma” stands for. We await your elucidation.
5 sigma means 5 or more standard deviations from the average. standard deviation is a measure of variance (deviation) from the average expected in single data points given a Gaussian (random, exponential, bell curve or whatever other term you learned) distribution.
If you have a Gaussian distribution then about 1/3rd of your measurements will be within 1 standard deviation of the average, 19/20ths will be within 2 standard deviations, and only about .26% will not be within 3 standard deviations (3 sigma), and .0063% will be 4 or more standard deviations (4 sigma) from the average, while the chance of any single data point meeting the 5 sigma standard given a Gaussian distribution is about .000057% – often called one-in-a-million although one-in-two-million is closer to the truth.
Note that having a large number of 5 sigma data points does not mean anythings wrong, it merely means that the data does not follow a Gaussian distribution.
this is a quickie, if not clear enough look up “probability” “Gaussian distribution” “standard deviation” for some basic stats reading, it helps a lot in trying to understand climate science.
As other commenters have noted, 5-sigma applies to normal (bell-shaped) distributions. But are these temperatures normally distributed? If they follow a different distribution then the concept of sigma, and all the associated probabilities, do not apply.
Financial markets have often run into this confusion, and (as Taleb documents in The Black Swan) this contributed to the financial crash. He says that financial markets follow a Mandelbrot distribution instead, which has fatter tails and therefore a higher likelihood of outlier events. These outliers would be preposterously unlikely under a normal distribution, which is how traders were reporting 28-sigma events and other such nonsense.
Let alone 28-sigma: five-sigma is pretty unlikely. So is a normal distribution really applicable or not?
Some examination of the 5sd filter in the UK CRUTEM code.
http://rhinohide.wordpress.com/2010/09/05/mosher-deviant-standards/
It appears that if the sientific comunity was today to design tolet paper, the brown side would be on your finger.
Check out Chauvenet’s criterion, I posted on here earlier but for some reason it didn’t make it.
Rob Findlay says:
September 5, 2010 at 11:31 am
“As other commenters have noted, 5-sigma applies to normal (bell-shaped) distributions. But are these temperatures normally distributed? ”
Actually, CRU must be unaware that to assume a normal distribution they are agreeing that temp has a normal distribution, ie, a natural variation, and therefore there are no significant anthropogenic caused forcings. This is one of the drawbacks of having “everyman’s statistical analysis” available in XLS – it allows researchers to dispense with real statistical analysis.
Above, I cited: Jones and Moberg, 1992, Hemispheric and Large-Scale Surface Air Temperature Variations: An Extensive Revision and an Update to 2001
This is a typo. The cite should read:
Jones and Moberg, 2003, Hemispheric and Large-Scale Surface Air Temperature Variations: An Extensive Revision and an Update to 2001
Thank you Rob Findlay!
I ground all the way through this thread with the Black Swan thought banging around in the back of my head. The quickest way to encounter a Black Swan is to ignore the outliers. That Gaussian cupola “thing” that they used to evaluate risk being one of the contributors to the festivities we are experiencing now.
Rebecca C says:
September 5, 2010 at 9:59 am
“5-sigma” refers to five standard deviations outside of the average value. In experimental science, measuring such an extremely divergent data point might be an indication that something had gone temporarily wrong in the data collection process. For example, maybe a power glitch affecting an instrument, or contamination getting into a sample. In the absence of any way to verify that the specific data point was a valid one, a scientist might simply screen it out of the data set. There is a statistical argument for doing this, based on the fact that for a normally-distributed (bell curve) population of data, a “5-sigma” value should occur only one in a million times…”.
Not the correct approach with climate, where temperature is the result of deterministic chaos and linear statistical methods break down.
Once again the CRU have been spotted throwing away the data, what a big bunch of tossers they are!