Analysis: CRU tosses valid 5 sigma climate data

Am I missing something? I do not any labels for the lines in three of Mosher’s graphs.

Steve Mosher: Please add title blocks to your graphs that specify what the graphs represent and the color coding of the datasets. I have no idea which dataset is which.
Title blocks are time consuming but they give the viewers an idea of what you’re illustrating. The more info the better.

Five sigma from what, a thirty year average?
[REPLY – HadCRUt goes back to around 1850. ~ Evan]

More BS from the CRU. The “consensus” continues to crumble.

You know, it would be nice if we had some sort of profession where people gathered data, tried to ensure it was accurate, and recorded and disseminated such data along with trends and patterns. As opposed to fudging, deleting, massaging, and re-imagining data to fit an agenda.
I wonder what we would call such a profession?

You can see the code for this in the MET’s released version of CRUTEM in the station_gridder.perl
# Round anomalies to nearest 0.1C - but skip them if too far from normal
if ( defined( $data{normals}[$i] )
&& $data{normals}[$i] > -90
&& defined( $data{sds}[$i] )
&& $data{sds}[$i] > -90
&& abs( $data{temperatures}{$key} - $data{normals}[$i] ) <=
( $data{sds}[$i] * 5 ) )
{
$data{anomalies}{$key} = sprintf "%5.1f",
$data{temperatures}{$key} - $data{normals}[$i];
}

It is well documented that the winter season has the highest variability.

It is immediately obvious from Table 3 that the year-to-year variability is much greater in winter months (especially January) than in the summer.

Jones, et al, 1982, Variations in Surface Air Temperatures: Part 1. Northern Hemisphere, 1881-1980

In the Northern Hemisphere, Figure 1 shows that year-to-year variability is clearly greater in winter than in summer.

Jones, et al, 1999, Surface Air Temperature and Its Changes Over the Past 150 Years
As to documenting the standard deviation drops …

To assess outliers we have also calculated monthly standard deviations for all stations with at least 15 years of data during the 1921–90 period. All outliers in excess of five standard deviations from the 1961–90 mean were compared with neighbors and accepted, corrected, or set to the missing code. Correction was possible in many cases because the sign of the temperature was wrong or the temperature was clearly exactly 10deg or 20degC too warm or cold. After this step the 1961–90 normals and the 1921–90 standard deviations were recalculated. Exactly how many values were corrected or set to missing is impossible to determine, as we do not know how many were corrected in the component datasets. We made changes to about 500 monthly averages, with approximately 80% corrected and 20% set to missing. In terms of the total number of monthly data values in the dataset this is a very small amount (<0.01%).

Jones and Moberg, 1992, Hemispheric and Large-Scale Surface Air Temperature Variations: An Extensive
Revision and an Update to 2001
Jones notes in an earlier paper that the corrections were prevalent for Greenland. These wx record coding errors remind me of the sort of thing that Anthony himself was pointing out here.
Does Jones inappropriately drop or adjust Feb 1936 records? Someone should go look. Most of the CRU data sets are available here:
http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html
———–
Just a side note, Mosher’s stuff was posted prematurely with unlabeled graphs and abbreviated discussion (for instance – which paper is ‘the paper’ reread by Mosher). Given how careful he was with the response to McKitrick, I wonder if he personally approved posting these notes in draft form?

For the love of God, please put comprehensive labels on the axes, and include keys when there are multiple graphs in one plot. You could be reaching a MUCH larger audience with just a little more work. I’m a Ph.D. research scientist, and if I can’t discern what’s in a graph, what hope is there for the average first-time visitor?

What do the various colors signify on the graphs?

Although this paper is not adequately illustrated and summarized, I get enough to see an interesting problem with tossing 5 sig for the future! If we are to have runaway global warming, soon the temp anomalies will be more frequently over 5 sig – are they going to toss these, too? You can bet your life they wont. This is a temporary thing to ensure the trend until they don’t need it anymore to maintain the present hotter than before, then, like GISS, they will re-adjust the earlier numbers.

Sorry Bob,
The Plots are just made on the fly as I walk through the data, For the chart in question, Red is the MAXIMUM departure from normal, blue is the minium. orange is +- 1.96sd, black is the mean. “index” is month from 1-12. For me this is just a curiousity that other can go take a look at as I’m working on debugging other stuff.
The procees of throwing out 5 sigma events has a seasonal bias

That’s a lot like what’s happening now. Not quite as bad yet.
What was so significant about this cold wave was that the 1930s had some of the mildest winters in the US history. 1936 was also one of the coldest years in the 1930s. And the winter was followed one of the warmest summers on record which brought on the 1936 North American heat wave.
That’s classic weather perturbed by volcanoes, and there were a lot of them in the ’30s.
http://www.volcano.si.edu/world/find_eruptions.cfm

As Mosher points out, this is documented in the papers describing HadCRUT and CRUTEM.
http://www.cru.uea.ac.uk/cru/data/temperature/HadCRUT3_accepted.pdf
http://www.sfu.ca/~jkoch/jones_and_moberg_2003.pdf

Records indicate that the Nile river has frozen over at least twice.
Those events were probably six sigma events.
But they were real, none the less.

The decade of the 1930’s is home to 25 of the 50 US State all-time hottest day records of any decade since 1880. Doesn’t any of this warming data get tossed, too?
Or, is it only cold data like 1936, or Orland CA (1880 and 1900), that gets tossed?
The big agw lie rolls on.

“5-sigma” refers to five standard deviations outside of the average value. In experimental science, measuring such an extremely divergent data point might be an indication that something had gone temporarily wrong in the data collection process. For example, maybe a power glitch affecting an instrument, or contamination getting into a sample. In the absence of any way to verify that the specific data point was a valid one, a scientist might simply screen it out of the data set. There is a statistical argument for doing this, based on the fact that for a normally-distributed (bell curve) population of data, a “5-sigma” value should occur only one in a million times.
However – and this is a big however – if you have independent corroboration of the 5-sigma data point, you really should not throw it out of your data set. For example, the 1936 data set shows extreme cold values all over the north-central U.S., which were independently recorded. This is not simply one thermometer experiencing a glitch. Add to this the written historical records which provide a softer verification of the extreme temperatures. (I say softer, because any narrative record would have been dependent on the thermometer recordings for its objectivity.)
The irony here, of course, is that the CRU is rejecting “extreme weather events” that occurred in the past to arrive at its projections of CAGW, at the same time that extreme weather events in the present (including extreme cold snaps) are being embraced as proof positive that CAGW is happening Today! Yes indeedy, It’s the End of the World as we Know It!!

kadaka,
Sigma is the Greek character used to denote standard deviation. Thus a ‘5 sigma’ data point is one that lies more than five standard deviations from mean. It suggests that the data point is an outlier. Or, using the real meaning of the word, it is an anomaly.

Upper Amazon in drought.
http://www.reuters.com/article/idUSTRE6825EU20100903
Left unsaid in article is that Peru and Bolivia are facing another record cold year.

It’s interesting that your graphs are very consistent with forest fire intensity/severity for the 20th century. About midway in my career (forestry) I suggested that there appeared to be a distinct synchronicity to the fire seasons that must correlate wih some Uber-climatic trend beyond what we got from the day-to-day weather predictions, and maybe even multiple trends that periodically became in-phase and led to our periodic large fire years. It was suggested that I might be better served to continue cruising timber.

For us more ordinary people, please point out what “5 sigma” stands for. We await your elucidation.
5 sigma means 5 or more standard deviations from the average. standard deviation is a measure of variance (deviation) from the average expected in single data points given a Gaussian (random, exponential, bell curve or whatever other term you learned) distribution.
If you have a Gaussian distribution then about 1/3rd of your measurements will be within 1 standard deviation of the average, 19/20ths will be within 2 standard deviations, and only about .26% will not be within 3 standard deviations (3 sigma), and .0063% will be 4 or more standard deviations (4 sigma) from the average, while the chance of any single data point meeting the 5 sigma standard given a Gaussian distribution is about .000057% – often called one-in-a-million although one-in-two-million is closer to the truth.
Note that having a large number of 5 sigma data points does not mean anythings wrong, it merely means that the data does not follow a Gaussian distribution.
this is a quickie, if not clear enough look up “probability” “Gaussian distribution” “standard deviation” for some basic stats reading, it helps a lot in trying to understand climate science.

Some examination of the 5sd filter in the UK CRUTEM code.
http://rhinohide.wordpress.com/2010/09/05/mosher-deviant-standards/

Above, I cited: Jones and Moberg, 1992, Hemispheric and Large-Scale Surface Air Temperature Variations: An Extensive Revision and an Update to 2001
This is a typo. The cite should read:
Jones and Moberg, 2003, Hemispheric and Large-Scale Surface Air Temperature Variations: An Extensive Revision and an Update to 2001

Rebecca C says:
September 5, 2010 at 9:59 am
“5-sigma” refers to five standard deviations outside of the average value. In experimental science, measuring such an extremely divergent data point might be an indication that something had gone temporarily wrong in the data collection process. For example, maybe a power glitch affecting an instrument, or contamination getting into a sample. In the absence of any way to verify that the specific data point was a valid one, a scientist might simply screen it out of the data set. There is a statistical argument for doing this, based on the fact that for a normally-distributed (bell curve) population of data, a “5-sigma” value should occur only one in a million times…”.
Not the correct approach with climate, where temperature is the result of deterministic chaos and linear statistical methods break down.
Once again the CRU have been spotted throwing away the data, what a big bunch of tossers they are!

This seems to be a very goodc example of a classic type of statistical mistake. The mechanics of statistic analysis are deceptively simple, particularly today when we have tools like Excel, SPSS, Matlab etc.
However to actually use these tools correctly you need to have a profound knowledge of the mathematical basis of statistics, of the characteristics of the data you are analyzing and preferably also of the quirks of the particular tool you are using. Unfortunately this is very often not the case,
In this particular case CRU seems to have made one of the most common (and least excusable) errors. They have assumed that their data has a normal distribution without bothering to verify that this is actually true.

Maybe it is the moderate climate of the UK that lends the UK researchers towards tossing very cold temperatures. They don’t know what to do with that; they never experienced anything that cold so subconsciously they decide to ignore it and code their programs that way.
So, i’d trust the Russians more about this.

Phil. says:
September 5, 2010 at 8:57 am
“[…]Seems prudent.”
Phil., i hope you don’t work in avionics?

Re is the temperature data distributed normally?
This link shows a distribution chart for monthly average temperature data for Abilene, Texas, from Hadley’s CRUT3 that was posted to the web shortly after the Climategate event of November 2009. The distribution is in one-degree C increments.
http://tinypic.com/r/n4xeo3/7
Having but simple statistical skills, this seems anything but normally distributed to me. I.e, there is no typical bell-curve shape, this has one “lobe” below the mean, and another “lobe” above the mean.
Perhaps others can comment on that chart, which was generated by me the hard way, in Excel (TM) but without using their statistics pack. Any errors or other faults are mine alone.

Of coure, they omit 5-sigma data. They have to make sure to cook the numbers to support the CAGW hypothesis.

Ron’s got a really good post on more of the reason behind the difference.
http://rhinohide.wordpress.com/2010/09/05/mosher-deviant-standards/#comment-967
He tracks it down to a different reason than tossing 5sigma events.

Met Office Meets “my backside”.
“Australia – Global warming my backside with the coldest day in 100 years
12 Jul 2010
WIDESPREAD cloud and persistent rainfall has kept temperatures down right across Queensland.
Longreach, in the Central West, received persistent rain from Tuesday evening, dropping the temperatures by about four degrees. The temperature then barely moved yesterday, reaching a maximum of 11 degrees; 12 degrees below the long-term average and the coldest July day in 44 years of records.Isisford, further south, was even colder, getting to just 10 degrees. This was the town’s chilliest day since before records began in 1913, almost a century ago.”
http://www.meattradenewsdaily.co.uk/news/140710/australia___global_warming_my_backside_with_the_coldest_day_in__years.aspx
…-
“Temperature records to be made public
Climate scientists are to publish the largest ever collection of temperature records, dating back more than a hundred years, in an attempt to provide a more accurate picture of climate change.”
Climate scientists have come under intense pressure following the Climategate scandal at the University of East Anglia, where researchers were criticised for withholding crucial information, meaning their research could not be independently checked.
Sceptics have also attacked climate change research over the quality of the records being used as evidence for the impact mankind has had on the world’s climate since the industrial revolution.
But the Met Office, which is hosting an international workshop with members of the World Meteorological Organisation to start work on the project, is now planning to publish hourly temperature records from land-based weather stations around the world.”
http://www.telegraph.co.uk/earth/environment/climatechange/7981883/Temperature-records-to-be-made-public.html
http://www.smalldeadanimals.com/archives/014798.html

It seems to me that they could be omitting this data for two possible reasons.
1. To reduce the amount of noise in the trend. This makes any “anthropogegic” fingerprint appear more obvious and greater than it actually is.
2. To reduce the appearance of unusual extreme weather anomalies in their records so that modern extreme weather events could falsely be labelled as “unprecedented”. That word that is wrongly and massively overused by the CAGW alarmists.
These are just more tricks used to bolster the very weak CAGW theory.

Phil. says:
September 5, 2010 at 1:55 pm
”
“DirkH says:
Phil., i hope you don’t work in avionics?”
Why Dirk, would you rather not eliminate bad data and spurious points?”
Why – because you would endanger lifes.
And no, in engineering i wouldn’t eliminate “bad data” and “spurious points”. Rather, i would examine exactly these “spurious points”. Very very closely.
Think about your answer. You just said that the cold wave of 1936 was bad data; a spurious point – something that has not really happened.
If all scientists work this way, no wonder they never achieve anything.

I started tracking weather statistics for Huntley, Montana back in 2006. At that time I downloaded the historical data from: United States Historical Climatology Network, http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn.html . This January I ran across some discrepancies so I downloaded the same file again and compared it to the original file. When I graphed the result it looked like someone had gone through the entire file with a computer program decreasing all of the temperature records prior to 1992. I still have both original files. February 1936 from the original file downloaded in 2006 was -.62 for the month and February 1936 from the file downloaded in January was -2.1 degrees F. My comments at that time were “Isn’t climate science fun! Everyday you learn new ways to commit fraud. I’d love to see the programmers’ comments on this little piece of code. I can imagine that it is something like “Hallelujah Brother, 2009 is the warmest year on record!!!!!”

Looks like according to GISS this is a non-issue.
Let’s look at the numbers for “Annual and five-year running mean surface air temperature in the contiguous 48 United States (1.6% of the Earth’s surface) relative to the 1951-1980 mean,” from originating page here, using the tabular data. For 1936 the annual mean anomaly was only +.13°C.
The global average temperature anomaly is of course more important as it tracks the AGW signal. By the land-only (aka meteorological stations-only) numbers, tabular data here, the 1936 annual mean anomaly was exactly 0.00°C. With the Contiguous US (CONUS) only having 1.6% of the Earth’s surface, with the seas occupying 70% thus CONUS only being 5.3% of the land area, if there was a little snip of the 1936 US winter numbers then it still wouldn’t have meaningfully changed the more important global anomaly number.
Also, from the article:

And more interesting facts: If one throws out data because of outlier status one can expect outliers to be uniformly distributed over the months.

Why?
In the winter months, around the temperate zones, there are the temporary snowfall accumulations. Snow falls, the albedo switches from dark ground to shiny snow, snow melts, the albedo is back to dark ground. With air temperatures checked so near the ground, the heating from sunlight during the day is important. So one December the meteorological conditions give CONUS little if any snow, next year there’s enough that the more northern areas have snow on the ground for the entire month. The albedo change will yield a larger temperature difference for that month between those years than a similar change in precipitation for a summer month like July, even when other factors (ratio of cloudy to clear days, etc) would yield little or no difference. Thus the albedo change gives a potential for greater temperature variances during the winter months than the rest of the year. Amazingly enough, it looks like that’s what you’ve found.
So, someone thought they had found something that looked suspicious with a temperature dataset, made up some graphs, posted here while noting he’s not making conclusions just noticing something that looks off, then certain commentators point out there is nothing wrong, it’s in the documentation, it makes no difference even if true, his analysis is faulty, etc.
Would it be wrong to say Mosher has just pulled a Goddard?
☺
——-
Side notes: GISS now says, for CONUS, 1934 had only 1.20 Celsius anomaly units, 1998 and 2006 were clearly warmer at 1.32 and 1.30 respectively. For the global numbers, the January-July Mean Surface Temperature Anomaly (°C) graph shows 2010 the warmest of the 131 year record.
2010 is gonna be da hottest evah!

Thanks Anthony!
May be good:
FOX News: Hannity Special: The Green Swindle
Sunday, Sep 5, 9 PM EST

“When they toss 5 sigma events it appears that the tossing happens November through February.”
Of course it’s the low temperatures. Look at DMI’s arctic temperatures at http://ocean.dmi.dk/arctic/meant80n.uk.php, any year. It’s the same for all latitude’s temperature readings. Maximums don’t vary as much as minimums do. I still wonder exactly why this is so, in the physics of the atmosphere I mean.
So, there’s the hike to the temperatures. We all knew it was there somewhere. And I’m sorry but that IS called manipulation of the data. Still want to better understand our atmosphere so keep the science coming Athony, it’s thanks to you, no doubt.
Does that have to do with the MSM rarely covering cold events ?? Where they deleted too ?? ☺

So ,
How many other arbitrary data “corrections” does CRU and GISS rely upon?
Seasonal bias is a big deal. How is this acceptable to climate scientists?
How about granting entities that have spent so much money on building models that provide great graphics, but such poor prognostication.
Doesn’t anyone care that the foundation data is based upon cooked books?
Virginia is poised to answer some of these questions.

From: wayne on September 5, 2010 at 6:48 pm

Maximums don’t vary as much as minimums do. I still wonder exactly why this is so, in the physics of the atmosphere I mean.

It’s just physics. Systems like to shed heat, until they equalize with their surroundings. For maximums you’re filling faster a leaking bucket, for minimums you’re either filling it slower, not at all, or perhaps even removing heat (colder air moves in, water evaporating after rainfall). Thus for the same absolute values of energy rate changes, you’ll get larger changes in minimums than you will maximums (for sunlight, gaining an extra 5 watts per square meter yields a smaller temperature increase than the temperature decrease from losing 5).

ATTN: RACookPE1978
I downloaded some temperature data for the remote Telluride CO from the USHCN.
The monthly temperature data for Tmax and Tmin are reported to +/- 1 deg F (ca. 0.5 deg C). The computed monthly Tmean was reported to +/- 0.00001 deg F. The SD was reported to +/- 0.000001. This is nuts!
The problem is that most “climate scientists” are computer and math jocks not experimentalists. Nowadays digital data from instruments is assumed to be 100% accurate especially by begining grad studentrs.

So how would the current “5 sigma event tossing” have dealt with this event?
“…The world record for the longest sequence of days above 100° Fahrenheit (or 37.8° on the Celsius scale) is held by Marble Bar in the inland Pilbara district of Western Australia. The temperature, measured under standard exposure conditions, reached or exceeded the century mark every day from 31 October 1923 to 7 April 1924, a total of 160 days…”
Over a hundred degrees for 160 days.
In the early 20’s.
And to think, the world is much warmer now…

Phil. says:
September 5, 2010 at 9:23 pm
Assumed a quote from CRU:
“We made changes to about 500 monthly averages, with approximately 80% corrected and 20% set to missing. In terms of the total number of monthly data values in the dataset this is a very small amount (<0.01%)."
That tells me nothing. Well… no, it actually does. It tells me it takes mere change of 0.01% of the records to make a rather large error in the trend that Steven Mosher is preliminarily showing.
You seem to know right where to go and get the core information. Do you happen to know where the 600 alterations/droppings are with the from and to and why, or those dropped, what they were and why. That wouldn’t be a very big file for them to keep and make public. Would save a lot of duplicate work. Or is it up to an investigator to try to recreate what they have done to the temperature records?

Phil. says:
September 5, 2010 at 8:57 am
Seems prudent

You have got to be kidding me. Outliers are indicators of either some important behavior or bad data. Either way you have to investigate. Ad hoc application of a statistical cut-off is not good data analysis policy. You are just an apologist.

Geoff Sharp says:
The 1930′s in the USA has the lowest lows and the highest highs. I think there is good reason for this, in particular 1936. The PDO was about to flip into negative, but more important is perhaps the solar position. SC16 which peaked around 1928 was weak with a today count of around 75 SSN. Take off the Waldmeier/Wolfer inflation factor and this cycle would be close to a Dalton Minimum cycle. 1936 is near cycle minimum, so with the already weak preceding solar max the EUV values would be similar or less than today.
Missed this earlier, my money would be on local events for the cold winter weather. I’d look at Black Sunday 1935 and it’s like for a cause.

So when might we see the actual data upon which the speculation wrt global warming is based? No data= no science. All claims are opinion without the missing? data. What say you Phil Jones?