Guest post by John Goetz
The GISStemp Step 1 code combines “scribal records” (multiple temperature records collected at presumably the same station) into a single, continuous record. There are multiple detailed posts on Climate Audit (including this one) that describe the Step 1 process, known affectionately as The Bias Method.
On the surface seems like a reasonable concept, and in reading HL87 the description of the algorithm makes complete sense. In simple terms, HL87 says that:
- The longest available record is compared with the next longest record, and the period of overlap between the two records is identified.
- The average temperature during the period of overlap is calculated for each station.
- The difference between the average temperature for the longer station and shorter station is calculated, and that difference (a bias) is added to all temperatures of the shorter station to bias it – bringing it in line with the longer station.
- The two records can now be combined as one, and the process repeats for additional records.
In looking at numerous stations with multiple records, more often than not the temperatures during the period of overlap are identical, so one would expect the bias to be zero. However, we often see a slight bias existing in the GISS results for such stations, and over the course of combining multiple records, that bias can be several tenths of a degree.
This was one of Steve McIntyre’s many puzzles, and we eventually figured out why we were getting bias when two records with identical overlap periods were combined: GISStemp estimates the averages during the overlap period.
GISStemp does not take the monthly data during the overlap period and simply average it. Instead, it calculates seasonal averages from monthly averages (for example, winter is Dec-Jan-Feb), and then it calculates annual averages from the four seasonal averages. If a single monthly average is missing, the seasonal average is estimated. This estimate is based on historical data found in the individual scribal record. If two records are missing the same data point (say, March 1989), but one record covers 1900 – 1990 and the other 1987 – 2009, they will each produce a different estimate for March, 1989. All other data points might match during the period of overlap, but a bias will be introduced nonetheless.
The GISS algorithm forces at least one estimation to always occur. The records used begin with January data, but the winter season includes the previous December. That December datapoint is always missing from the first year of a scribal record, which means the first winter season and first annual temperature in each scribal record is estimated. Thus, if two stations overlap from January 1987 through December 1990 (a common occurance), and all overlapping temperatures are identical, a bias will be applied because the 1987 annual temperature for the newer record will be estimated.
Obviously, the bias could go either way: it could warm or cool the older records. With a large enough sample size, one would expect the average bias to be near zero. So what does the average bias really look like? Using the GISStemp logs from June, 2009, the average bias on a yearly basis across 7006 scribal records was:
Since the temperatures in the past were colder than they are today, this all makes sense.(?)
Reply: No … the temperatures in the past for stations with multiple records have been cooled by an average additional 0.08C. – John
The net effect is a tenth of degree in more than 100 years, that’s not much. The “bias” goes to about zero after ~1990. Are you sure the curve isn’t simply due to rounding effects? That’s the kind of curve I’d expect if we had gained an extra digit (going from half a tenth accuracy to half an hundredth).
Not that I don’t find strange the other aspect you talk about, that need to “always estimate.” Weird, we’re talking about really bad programming skills there, that is the kind of thing that could be easily avoided.
Perhaps you can also explain the relevance to the data series, given that the average adjustment is < 0.1⁰C ?
Reply: I forget what the increase in global temperatures is purported to be since 1880, but I believe it is somewhere in the neighborhood of 0.8 C. Roughly 0.08 C – or 10% – seems to be due to the process of combining scribal records, and nothing more. You decide the significance of this single process step (one of many). – John
I guess I’m confused because I have not studied more than a couple examples of original scribal data sheets.
Why would there frequently be two or more overlapping scribal records from the same station. Were two or more people reading the temperatures at different times on the same days and writing them down on separate lists?
John, you’ve explained this well, and it certainly seems like a flawed method, but how much of the trend since 1880 does it actually account for? Just using some rough numbers, since 1880, the trend line rise in temps has been about 0.75° C. Looking at your chart, it looks like the trend line rise accounted for by this bias is about 0.10° C. Am I right (about the 0.10° C)? That would make the bias account for about 13% of the total supposed warming.
Does that sound like an in the ballpark estimate of the significance of this?
I can understand why a researcher (data technician) might start down this path focusing on the desire for long-running series of temperature data. At some point that person or group should have stopped and asked a few questions of the sort: “What are we doing to the data?” or “What are the alternatives?” or “How do different time periods regarding a lengthily warming or cooling influence the outcomes?”
If they did these things it seems they chose a method that gave them a warming bias. If they did not do these things, than shame on them!
Are there no “facts” to work with in climate research?
It should be easy enough to test if this method alway produces a positive bias and that would be to create a set of artificial records in which the values were drawn at random from 3 separate distributions: one where the mean trend was decreasing, another in which there was no trend and one where the mean trend was upward. This would be a simple enough program to setup and run a few thousand iterations for each scenario. If the method used to combine records is unbiased then the final dataset should not differ significantly from the original dataset for which we have the advantage of knowing what the generating function is.
Anthony, this is a bit OT. But don’t you know these guys?
http://www.ametsoc.org/policy/2009geoengineeringclimate_amsstatement.html
I know these guys too. They had a secret island full of hot compliant babes in “Our Man Flint” where they CONTROLLED THE WEATHER!!!!
Perhaps more importantly, since the adjustment was essentially flat until 1940, then went almost straight up, all of the bias occurred between the previous warm period of the mid-30s and the mid-90s.
The net result is that the recent warming thru 1998 looks more dramatic relative to the 30s by the 0.07⁰C introduced via the adjustment process.
10% increase just from the process? If a business was fudging it’s books by 10% they would be dragged into a courtroom and prosecuted.
Off topic: http://scienceandpublicpolicy.org/images/stories/papers/originals/climate_money.pdf
tracked this back from drudge. Basically its a report on all the money spent by the gov’t on hyping and researching climate change. Anthony Watts and Steve McIntyre are mentioned several times for their volunteer work. I wonder how the graph on page 3 compares to the recent rise in temperature.
The net effect is a tenth of degree in more than 100 years, that’s not much.
====
Ah but Filipe, the entire temperature increase over the past 100 years was less than 4/10 of one degree.
Are you telling me that an artificial bias in an artificial database deliberately inserted into the record by GISS has created 1/3 of the entire “global warming” that the most extreme of the AGW extremists can actually find?
Ten percent is significant.
I’ll take the +0.01 Year 2020 leaps for a buck each.
GISS: Cooling the past to bring you a warmer and fuzzier future.
The important thing to note is that the creators of this scribal data averaging/estimating technique are either unaware of this bias, or aware of it.
If they are unaware of it, are they making efforts to correct their system?
If they are aware of it, why haven’t they already corrected their system.
Either way, it does not look good.
Why do all these biases seem to accrue to the older-colder, newer-hotter side of the ledger?
Interesting. It might be interesting to pass this on to the guys doing the
clear climate code project. Step one could be rewritten to change the method
of station combining and we could get a little bit closer to something that is accurate. As for the size of the bias, every little bit of improved accuracy counts. Kudos for your continued hard work on this wretched piece of code
Nice work, John. As you say, 10% of the claimed warming may just be a result of poor programming. Another reason not to trust GISS temperatures, in my view.
Ignoring the red trend line, there seem to be three distinct steps, at -0.07, -0.03 and just below zero. It would be interesting to drill down and see what happened at the step changes around 1940-52 and again at 1994-ish.
A few steps with bias. Drop all the rural records. Only use urban weathers stations and airports, pretend UHI is accounted for and voila … fabricated warming.
I don’t know about GissTemp whatever, but something is up with the weather in the San Francisco Bay Area.
We are having very cold nights and cool days in the middle of summer.
“Reply: I forget what the increase in global temperatures is purported to be since 1880, but I believe it is somewhere in the neighborhood of 0.8 C. Roughly 0.08 C – or 10% – seems to be due to the process of combining scribal records, and nothing more. You decide the significance of this single process step (one of many). – John”
Every little bit helps, like the money in the church collection.
I presume these numbers are further corrected for UHI and then used for corrections over 1000kms? Once on the slippery road it keeps slipping :).
Just to clarify my point. Consider you have a large set of points randomly distributed according to a uniform distribution between 0 and 10. The “true” average of these points is 5. Consider now that all the points are truncated to integers. The average of the truncated points is 4.5. If the points are instead truncated to the first decimal place, then the average is 4.95 and so on.
In a system with truncation, and with accuracy increasing with time, even with a true flat slope, one would get a positive slope just from the truncation. But I’m not sure this applies here, are these measures considered as true rounding or simple truncation?
The GISS record is highly suspect. I said so on a warmists blog and have now been banned from there. They do not tolerate dissent.
Why is there such a big trend difference beween the Land based temperatures of GISS and Hadley compared with the satellite data over the same time period? Does anyone know?
Snowman if you come here and read this – hi. We could chat here. I’ve been banned at the other place
Felipe – “The net effect is a tenth of degree in more than 100 years, that’s not much.” The total warming over this period is 6 tenths of a degree, so a bias of one tenths of a degree would be significant?
Oh, my. A review can’t even get through step 1 without finding an error. That slightly changes my opinion of the chances of other errors being present.
I think the obvious question to ask is: “How many times has this procedure been reviewed?”
These people have been using this procedure for years, they should have been examining it often. Why didn’t everyone examining the procedure find this problem when they started looking at the process?
I’ve read of scientists who hired an outsider, trained them to use a copy of their equipment, and had them study their original material to see if the outsider made the same discovery they did. Why aren’t these scientists having people examine their equipment regularly?
Filipe, you mention the possibility of rounding effects… I live in the Arizona desert, and every day the temperature is rounded up at the end of the day. The result is that certain temperatures are shown for the high and low until the evening when it begins to cool, then at the high rises at least one degree later at night. I figure they justify that as the actual temperature has to be at least a fraction higher, and therefore is rounded up, always. If you don’t believe me, watch the daily noaa temperatures for Tucson.
I wouldn’t be surprised to find this kind of bias in the GISS too.
When I was younger, people said the weather service reported lower than actual temperatures during the summer so as to not scare off tourists, though I can’t vouch for it, and it may have been an urban myth.
Amazing!!!!!!!!!!
no snip’s needed for shortness…. LOL
” it’s better to be snipped than band for life! ”
I like this place.
A.W. keep moving on that publishing of the stations.
10% added to 20% parking lot error could be a bunch
nite nite
.pickens (20:17:21) :
Why do all these biases seem to accrue to the older-colder, newer-hotter side of the ledger?
Remember the blinking global temp graphs posted in other articles, where the left half is lowered and right half raised? Near the end of the Left half, it lowers again. The right hand hockey sticks.
They started in peak and valleys.
You start in the valley after the 30’s to lower it. When it can’t lower any more, you start with the output graph and pick the next valley and head left, etc.
Do the same but opposite for the right to raise it.
This is their 2nd go at it, and they have operated this time on previously altered data.
The difference graph which we see has austere step function sto it.
The whole idea of thiers is to slowly alter the global temp report in stages, hoping that the masses won’t notice the sleight of hand.
Think of it as a calibration flat.
Add it to their latest global temp graph and you have the true graph before the double butchery.
My pick for quote of the week !!!!! ~~~~~~~~~” Why aren’t these scientists having people examine their equipment regularly?” :^]
Thank you AnonyMoose. I believe the point you were making was that the scientific method has been abandoned , if I am not mistaken.
AnonyMoose (21:15:48) : Oh, my. A review can’t even get through step 1 without finding an error. That slightly changes my opinion of the chances of other errors being present.
It’s even worse than that… The first step in GIStemp is actually STEP0, and that step does a couple of suspect things all by itself. (Cherry picks 1880 and deletes any data older than that. Takes an “offset” between USHCN and GHCN for up to 10 years from the present back to no earlier than 1980 then “adjusts” ALL past data by subtracting that “offset” – supposedly to remove the NOAA adjustments for things like TOBS, UHI, Equipment, etc., …)
The fact is that at EVERY step in GIStemp, there are odd decisions, introduction of demonstrable errors, questionable techniques used to fabricate data, etc. And yes, having read every line of the code a couple of times now, I’m comfortable with the term “fabricate” for the process of creating “data” where there are none… This posting gives one example from one of the steps, but in fact there are several steps where “odd things” are done to “fill in missing bits”. I can think of no better term for this than “fabrication”. The deliberate construction of a temperature series where none exists.
I think the obvious question to ask is: “How many times has this procedure been reviewed?”
These people have been using this procedure for years, they should have been examining it often. Why didn’t everyone examining the procedure find this problem when they started looking at the process?
Frankly, the code will send anyone screaming from the room. I’ve done a “data flow listing” of what files go in, and come out, from each bit of the STEP0 process. It is at:
http://chiefio.wordpress.com/2009/07/21/gistemp_data_go_round/
I intend to add STEP1, 2, 3, 4, 5 over time; then put a bit of “what happens with each change” description in it. (It’s a work in progress…).
But take a look at it. It’s just a list of program, input, output, next program. All for just STEP0. The spaghetti is horrid. So before any “code review” could even begin to ask the question “Is this process valid” it gets stymied with the question “Just what the heck IS the process and just where the heck DOES the data go?!? ”
I’m working to take some of the worst of the Data-Go-Round behaviour and simplify it, specifically so that I can more rationally say just what is going on where in the processing.
One example: At the end of STEP0, the output file has the name of v2.mean. This is then moved to STEP0/to_next_step/v2.mean_comb and the script then advises you to, by hand, move it to STEP1/to_next_step/v2.mean_comb.
So with no processing at all done to the file it goes through three names… And ends up in STEP1/to_next_step where a rational person might expect to find the output of STEP1, to be handed to STEP2, but instead finds the output of STEP0 being handed to STEP1.
Those kinds of Logical Landmines are scattered through the whole thing. I can only work on it about 4 hours at a shot before I have to take a “sanity break” to keep a tidy mind…
Part of what I’m doing now is fixing that kind of silly endless Data-Go-Round behaviour. Have one file name for a thing, in one place. Not three. And have the name reflect reality… like maybe finding input files for a STEP in input_files rather than in to_next_step…
FWIW, I’ve come to figure out that “input_files” directory actually means “external_site_data_files” while “to_next_step” actually means “inter_step_datasets”. The code is full of that kind of thing. You can only spend so long trying to answer “what is is” before you either give up, or need a sanity break…
So I can’t imagine anyone doing a decent code review on this without re-writing it first. At least the worst bits of it.
I’ve read of scientists who hired an outsider, trained them to use a copy of their equipment, and had them study their original material to see if the outsider made the same discovery they did. Why aren’t these scientists having people examine their equipment regularly?
IMHO, this code is more of a glorified “hand tool”. Something that someone cooked up to let them play with the data. There are lots of places where you can insert “plug numbers” to see what happens. Places where a parameter is passed in, rather than set in the code; and output files left laying about where you can compare them to other runs.
So you can try different radius choices for “anomaly boxes” to get “references stations” data for adjusting their temperatures. Some steps set it to 1000 km, others to 1500 km, one to 1200 km (and in the code it chooses to execute another program on the data ONLY in the case where the parameter value is 1200; so one is left to wonder why… what makes 1200 km “special”, and if it is special, why is it passed in as a parameter that can be changed at run time by the operator?) It’s structure says that the intent is to play with settings by hand and cherry pick.
As near as I can tell, the “review” only consists of looking at the papers written by GISS, not at the actual code.
What happened in 1994 and forward?
AnonyMoose (21:15:48) : Oh, my. A review can’t even get through step 1 without finding an error. That slightly changes my opinion of the chances of other errors being present.
It’s even worse than that… The first step in GIStemp is actually STEP0, and that step does a couple of suspect things all by itself. (Cherry picks 1880 and deletes any data older than that. Takes an “offset” between USHCN and GHCN for up to 10 years from the present back to no earlier than 1980 then “adjusts” ALL past data by subtracting that “offset” – supposedly to remove the NOAA adjustments for things like TOBS, UHI, Equipment, etc., …)
The fact is that at EVERY step in GIStemp, there are odd decisions, introduction of demonstrable errors, questionable techniques used to fabricate data, etc. And yes, having read every line of the code a couple of times now, I’m comfortable with the term “fabricate” for the process of creating “data” where there are none… This posting gives one example from one of the steps, but in fact there are several steps where “odd things” are done to “fill in missing bits”. I can think of no better term for this than “fabrication”. The deliberate construction of a temperature series where none exists.
I think the obvious question to ask is: “How many times has this procedure been reviewed?”
These people have been using this procedure for years, they should have been examining it often. Why didn’t everyone examining the procedure find this problem when they started looking at the process?
Frankly, the code will send anyone screaming from the room. I’ve done a “data flow listing” of what files go in, and come out, from each bit of the STEP0 process. It is at:
http://chiefio.wordpress.com/2009/07/21/gistemp_data_go_round/
I intend to add STEP1, 2, 3, 4, 5 over time; then put a bit of “what happens with each change” description in it. (It’s a work in progress…).
But take a look at it. It’s just a list of program, input, output, next program. All for just STEP0. The spaghetti is horrid. So before any “code review” could even begin to ask the question “Is this process valid” it gets stymied with the question “Just what the heck IS the process and just where the heck DOES the data go?!? ”
I’m working to take some of the worst of the Data-Go-Round behaviour and simplify it, specifically so that I can more rationally say just what is going on where in the processing.
One example: At the end of STEP0, the output file has the name of v2.mean. This is then moved to STEP0/to_next_step/v2.mean_comb and the script then advises you to, by hand, move it to STEP1/to_next_step/v2.mean_comb.
So with no processing at all done to the file it goes through three names… And ends up in STEP1/to_next_step where a rational person might expect to find the output of STEP1, to be handed to STEP2, but instead finds the output of STEP0 being handed to STEP1.
Those kinds of Logical Landmines are scattered through the whole thing. I can only work on it about 4 hours at a shot before I have to take a “sanity break” to keep a tidy mind…
Part of what I’m doing now is fixing that kind of silly endless Data-Go-Round behaviour. Have one file name for a thing, in one place. Not three. And have the name reflect reality… like maybe finding input files for a STEP in input_files rather than in to_next_step…
FWIW, I’ve come to figure out that “input_files” directory actually means “external_site_data_files” while “to_next_step” actually means “inter_step_datasets”. The code is full of that kind of thing. You can only spend so long trying to answer “what is is” before you either give up, or need a sanity break…
So I can’t imagine anyone doing a decent code review on this without re-writing it first. At least the worst bits of it.
I’ve read of scientists who hired an outsider, trained them to use a copy of their equipment, and had them study their original material to see if the outsider made the same discovery they did. Why aren’t these scientists having people examine their equipment regularly?
IMHO, this code is more of a glorified “hand tool”. Something that someone cooked up to let them play with the data. There are lots of places where you can insert “plug numbers” to see what happens. Places where a parameter is passed in, rather than set in the code; and output files left laying about where you can compare them to other runs.
So you can try different radius choices for “anomaly boxes” to get “references stations” data for adjusting their temperatures. Some steps set it to 1000 km, others to 1500 km, one to 1200 km (and in the code it chooses to execute another program on the data ONLY in the case where the parameter value is 1200; so one is left to wonder why… what makes 1200 km “special”, and if it is special, why is it passed in as a parameter that can be changed at run time by the operator?) It’s structure says that the intent is to play with settings by hand and cherry pick.
As near as I can tell, the “review” only consists of looking at the papers written by GISS, not at the actual code.
Is there a way to get the raw temps vs. the adjusted temps graphed/animated/whatever, and delivered to a news agency or reporter? The blink charts of what was vs. what “is” are fairly damning.
I think this would be a compelling “Here is what we saw, and here is what they’re saying” story. Then ask Hansen, Mann, Schmidt, et al why the temperature observations from 1900-2008 are/need to be adJusted monthly, and in retrospect. I have yet to hear a valid explanation of why it makes sense to adjust temp observations from 10/20/50/100 years ago every month when new data comes in.
Just a thought/question.
Per the 1200km custom bit, it’s in STEP3. This is a bit from the top of the script that controls how STEP3 runs. I’ve elided the housekeeping bits of the script.
The bold bit at the top sets the radius to 1200 by default, but if the script is started with another value, the 1200 gets changed. So something like:
do_comb_step3 2000
would cause the rad value to be assigned that of the first parameter $1 (in this case, 2000).
do_comb_step3:
label=’GHCN.CL.PA’ ; rad=1200
if [[ $# -gt 0 ]] ; then rad=$1 ; fi
[…]
echo “Doing toSBBXgrid 1880 $rad > to.SBBXgrid.1880.$label.$rad.log ”
toSBBXgrid 1880 $rad > to.SBBXgrid.1880.$label.$rad.log
[…]
if [[ $rad -eq 1200 ]] ; then ../src/zonav $label ; fi
So ONLY in the case where a radius of 1200 km was used, the script calls the zonav script for further processing. One wonders why, say, 1100 km ought not to be zone averaged…
Maybe it’s a way to cause failure if someone overrides the value? To prevent a hand test from making it into production? But if that’s the case, why was 1200 chosen, what makes it the “right” choice? Or maybe it’s something more. There is no way to know…
So when evaluating STEP3, I’ll need to spend a while thinking about what zonav does and what the effect would be of NOT running it with a radius of 1199 vs running it with 1200; and wonder “Why?”…
(“Why? Don’t ask why. Down that path lies insanity and ruin. -e.m.smith”)
John Goetz, thankyou so much or these excellent and important findings.
It must have been a HUGE work to do, But you did it 🙂
Question:
1) The 0,08 K warming trend after 1940 from no less than 7006 scribal records, does this mean that we in average has 0.08 K of the GISS-temperature global warming from this error? Or what does it mean?
0,08K would be around 20% of the global warming since 1940.
2) This consequent tendensy, that the errors favor global warming, is it possible to give a statistical estimat: How likely is it, that we get this warming trend out of 7006 records?
Should not these overlapping records yield a ZERO trend?
Is the likelyness of such a warming trend occuring from 7000 records something like 1:7000.000.000.000.000.000 ?
If so, how close is this to be a proof that data by human influence is favouring a global warming signal?
The final decision (slow in developing but it will) are the actual temperatures that are occuring at this time especially in the US and Europe. Most of the highly populated States in the USA have been experience below “anomalies” and people are experiencing this.. subsequently the surveys are shifting to show more and more skepticism. It doesn’t matter how much GISS et al try to increase temps because they just ain’t rising LOL (BTW AMSU temps have jumped dramatically during the past two weeks proving that there is NO AGW (unless warmistas would contend that it’s all started suddenly) hahaha.
http://cdiac.ornl.gov/epubs/ndp/ushcn/ndp019.html
Comments?
Anthony: another developing story. I think you could safely state now that snow is falling in Buenos Aires (at least Provincia). maybe we should wait until tomorrow…
http://momento24.com/en/2009/07/22/buenos-aires-province-snow-falls-in-the-south/
This backs up the previous recent posting about snow in BA 2008. There is an intense pool of cold air around Paraguay, Uruguay and Argentina (see COLA). BBC weather is not showing though any significant Through/Front going through unless its a stationery one slightly to the north
“Norm (22:30:23) : What happened in 1994 and forward?”
– A tipping point has been reached!
Totally O/T, but there are 2 articles in yesterday’s and today’s Financial Times. Today’s is headed ‘Atkins puts moral case for climate change’. !!!!!! (my excalamation marks). You can find the stories at ft.com and search for ws atkins. The FT is a hugely influential, usually objective, newspaper read by businessmen and moneymen all over the world. Can I suggest that WUWT visitors write to the Editor of the FT to comment on these stories? The editor’s address is letters.editor@ft.com Please include your (physical) address and telephone number.
Regards
S
Nelson (19:46:17) :
Perhaps more importantly, since the adjustment was essentially flat until 1940, then went almost straight up, all of the bias occurred between the previous warm period of the mid-30s and the mid-90s.
The net result is that the recent warming thru 1998 looks more dramatic relative to the 30s by the 0.07⁰C introduced via the adjustment process.
This is a good point. There actually looks to be a slight cooling bias between ~1910 and ~1940 which might not be much – but it could be enough to ensure the 1910-1940 warming trend matches 1975-2005 warming trend. Particularly as there is a warming bias after 1975.
I hope I’m reading this right. If I am the ‘ocean effect’ must be increased and the CO2 signal must be reduced.
Before jumping to any conclusions it might be worth checking out the Hadley record.
Filipe (19:11:45) :
The net effect is a tenth of degree in more than 100 years, that’s not much.
Not sure I agree with this. The overall trend might not be affected too much but looking at the pattern of biases, the peaks and troughs in the temperature record could change quite a bit.
The “bias” goes to about zero after ~1990. Are you sure the curve isn’t simply due to rounding effects? That’s the kind of curve I’d expect if we had gained an extra digit (going from half a tenth accuracy to half an hundredth).
Why? The average of +0.1 and -0.1 is zero as is the average of -0.01 and +0.01.
I might have completely misinterpreted this post as I read it very quickly so I’m happy to be corrected on anything.
Here’s a theory to examine in your surfacestations research: that the temperature record warming trend is a bias of the growth in airline flights in the latter 20th century, and the congested air travel system: more jets sitting on the tarmac, engines running waiting to take off, is going to put a lot more hot exhaust gasses in the area of the airport weather station than planes starting up and taking off. If you correct temperature data for changes in airport congestion and flight delay times, what would the result be?
Great work John Goetz. The Team had better get the wagons circled, before all the wheels have come off.
Well its now snowing in Ezeiza airport Buenos aires so much for AGW
http://www.perfil.com/contenidos/2009/07/22/noticia_0033.html spanish
Intuitively I feel that this method would spread UHI affects through the data. Just my initial reaction – not founded on anything hard. The reason?
The longer serires is most likely to be the oldest series. Older sites are more prone to have been subject to urban growth around them. Such sites will show positive temperature bias compared with less effected sites (i.e. the shorter series).
So by adding the “bias” from the longer series to the shorter one you would be adding a UHI component from one series to the next, propogating it through the data.
I would be interested in seeing a comparison of the two groups of data – the longer versus the shorter – and some analysis of comaprable UHI exposure between the two.
Actually the bias starts before step 1. When I was in LA, it was well-known that the valleys were always warmer than downtown. The choice of proxy stations will always introduce bias, no matter how small the distance between stations. In Richmond Park London, which is several times the size of Central Park New York, but where traffic is confined to a single perimeter road, for many years there was a basin which was noticeably warmer than adjoining areas only a few hundred yards away. This discrepancy seems to have disappeared recently, for no obvious reason.
The merging of any temperature records even from closely neighbouring stations must always be suspect. Averaging is not an option. The discrepancies should be consistent from reading to reading for there to be any confidence in the splicing.
If however splicing is essential then before any attempt to splice the two sets of data, over the whole overlap period the old-fashioned, elementary technique of analysis of differences should be applied to the raw data to determine whether at least the first and second differences are randomly distributed or not.
CNN Biaha Blanca Argetina, winterconditions, snow, temp -18 degree Celsius?
So, AGW is indeed a computer made problem.
Let’s move from virtual to real world again and start solving real world problems.
Thanks for this clear and significant posting John.
The time has come to provide the US Senators with a quick course “How climate data is framed and what to do about it”.
As Cap & Trade and the current Climate Bill are under severe attack, politicians who reject this bill come up with disturbing alternatives.
We want them to decide to do NOTHING, NOTHING AT ALL!
Would anyone like their bank account or 401K handled in this manner? I don’t think so.
Does HadCRUT (2009/06 0.503) use any of the same adjustments as GISS? — John M Reynolds
Re: Filipe (20:45:34) :
They don’t truncate, they round. This means your random set always averages out to 5, not 4.5 or 4.95 or 4.995 etc.
“Ron de Haan (03:28:29) :
CNN Biaha Blanca Argetina, winterconditions, snow, temp -18 degree Celsius?”
And yet, yesterday here in Syndey, Australia, it was the warmest July day in 19 years, about 7C above “average”, it was quite balmy TBH. According to the BoM however, 1990 was the highest max on record for Sydney in July.
These “hot/cold” swings are quite intriguing indeed. Mind you when Lt Cook did his observation of Venus crossing the Sun in Tahiti in 1769, temperatures rose to on the day to a max of 119F.
Curiousgeorge hits nail on head.
This is not how responsible, accountable people would handle data.
Would it be naive to suggest that a policy think tank, like Hoover or Heritage, start an intensive program to systematically review all the temperature-station data and their manipulation over the past 150 years?
It’s wonderful that volunteers like Anthony and his colleagues devote countless hours to this effort, but it’s clearly such a huge task that it cries out for someone with the bucks to hire a team of analysts who can undertake a thorough ‘stem to stern’ review of the US and world data underlying the claims of AGW.
These claims are now the basis of major public-policy legislation (e.g. ‘cap-and-trade’), so it is way past time for them to be assessed by professionals without a policy ax to grind. You could argue that a conservative organization like Heritage would have such an ax, but a temperature-data-review program could be insulated from biases, or established at an institution without an ax in the ‘AGW’ dispute—Rand?
Or maybe a non-political foundation could be approached with the aim of funding an independent review effort, called perhaps “The World Temperature Audit.”
One thing for sure: it should not take any government money.
/Mr Lynn
@Curiousgeorge (03:35:41) :
“Would anyone like their bank account or 401K handled in this manner? I don’t think so.”
Yes, if they have the same upward bias :o))
No, if the earlier balances are adjusted downwards :o(
“The GISS record is highly suspect. I said so on a warmists blog and have now been banned from there. They do not tolerate dissent.”
Is there any warmist blog which tolerates dissenting views? I haven’t found one yet.
“liars, D@mn liars and statistics” needs to be changed to “liars, D@mn liars, IPCC liars and statistics” IPCC makes used car salesmen look honest and a lot less dangerous to our pocketbooks.
OFF TOPIC:
I have not seen anyone addressing the effects of irrigation on “Global Warming” and sea level rise. IPCC climate models ignore water as a greenhouse gas so the effect of “Modern Farming Techniques” characterizing “the Green Revolution” would not be studied. The Green Revolution refers to the transformation of agriculture beginning in 1945. It packaged specially bred High Yield Varieties with fertilizers, pesticides, and irrigation.
http://www.crystalinks.com/greenrevolution.html
http://www.copperwiki.org/index.php/Green_Revolution
http://www.i-sis.org.uk/announcingSIS36.php
Irrigation is a significant use of water world wide. “…Agriculture, especially irrigated industrial farming and
livestock production, typically put the single largest demand on surface and
aquifer resources. In water-scarce regions, irrigation can consume well over
three-quarters of total water withdrawals….” http://unfccc.int/resource/docs/napa/com01e.pdf
http://www.whirledbank.org/development/sap.html
http://www.stwr.org/imf-world-bank-trade/creating-poverty-world-banks-latest-passion.html
“….Under the USAID program, farmers set aside a small
amount of their land for fish farming and either dig a pond
or build one inside earthen walls….In a typical situation, farmers produce 1,500 kilograms of fish
per hectare a year, providing food for their families as well as
a new cash crop. The ponds help farmers cope with drought
and enable them to raise crops like cabbage and tomatoes that
require irrigation during the dry season….” http://www.worldwatch.org/press/prerelease/wwp172.pdf
African irrigation projects: http://africanagriculture.blogspot.com/search/label/irrigation
The World Bank/IMF SAP has intentionally bankrupted third world farmers, the land has then been bought up and turned into plantations using modern farming techniques. “…According to a study by Jose Romero and Alicia Puyana carried out for the federal government of Mexico, between 1992 and 2002, the number of agricultural households fell an astounding 75% – from 2.3 million to 575, 000….The vacuum created by retreat of the Mexican state from agriculture was filled by large US and Mexican agribusiness…..Transnational agri-business tends to have much closer links with larger farmers and producers, who have better access to land, irrigation and credit, all of which are scarce commodities for small farmers…” http://www.countercurrents.org/mohanty230608.htm
Europe has also been the target for modernization of farming. According to an article, 2001 Polish entry into the European Union: “…a meeting with the Brussels-based committee responsible for negotiating Poland’s agricultural terms of entry into the EU…the chair-lady said: “I don’t think you understand what EU policy is. Our objective is to ensure that farmers receive the same salary parity as white collar workers in the cities. The only way to achieve this is by restructuring and modernising old fashioned Polish farms to enable them to compete with other countries agricultural economies and the global market. To do this it will be necessary to shift around one million farmers off the land …a lady from Portugal, who rather quietly remarked that since Portugal joined the European Union, 60 percent of small farmers had already left the land. “The European Union is simply not interested in small farms,” she said….” http://www.i-sis.org.uk/savePolishCountryside.php
Here in the USA all we have to do is look at all the irrigated lawns and golf courses, not to mention farms in Arizona, New Mexico and California to see the major effects of irrigation. Not only is water brought in direct contact with the air, but plants transpire also putting water in the air. The “lake effect” seen around bodies of water show that water DOES have a climate effect.
Have there been any studies done on the effects of irrigation on climate? If 50 to 100 ppm of CO2 is supposedly cause for alarm what has all those millions of gallons of irrigation water done? Since much of the water is “fossil water” what is its effect on the sea level? “Modern farming” also cause a major loss of soil that then ends up in the sea. On my farm alone we’ve lost over 2 feet of topsoil as well as an unknown amount of subsoil. I have seen 4 inches plus my seed disappear in one rainstorm before I learned not to smooth the soil when I planted grass for my pastures.
Speaking of sea level, they now estimate there are 3 MILLION volcanoes under the seas!
“..The team estimates that in total there could be about 3 million submarine volcanoes, 39,000 of which rise more than 1000 metres over the sea bed….” http://www.newscientist.com/article/dn12218
How come no one ever points to volcanoes as a major source of CO2? “…Volcano Outgasing of CO2. The primary source of carbon/CO2 is outgassing from the Earth’s interior at midocean ridges, hotspot volcanoes, and subduction-related volcanic arcs….” http://www.columbia.edu/~vjd1/carbon.htm
Basil (19:22:24)
Very, very few North American stations have multiple scribal records that are combined. The bulk of the stations come from Asia, primarily Russia, although there are a number from Europe, Africa, and the Middle East. So the biasing really affects them and is not a world-wide phenomenon. Furthermore, without running the GISStemp process through to the end, it is impossible to know how the cell gridding impacts the result. It might increase or lessen the effect.
It seems the bias and just plain bad science has been noted by the several university geology departments.
“…It has also been established in the literature that biases, inaccuracies, and
imprecision have been introduced to the climate monitoring systems because of
meteorological station moves, instrument changes, improper exposure of instruments, and
changes in observation practices …It has become clear from various studies (e.g., Pielke et al. 2007a) that data used
in existing long-term climate assessments including the U.S. Historical Climatology
Network (USHCN) have undocumented biases that have not been corrected using data
analysis and data adjustment techniques….”
Impacts of Land Use Land Cover Change on Climate
and Future Research Priorities
: http://climate.agry.purdue.edu/climate/dev/publications/J91.pdf
John Goetz
One quick question. How many stations (what proportion) does this affect. I assume if a complete (e.g. 1880-2009) station record exists then the “Step 1” procedure is not used.
John F. Hultquist (19:24:08)
I don’t believe a method was selected to create a warming bias. As I said, the bias method in and of itself seems like a reasonable approach. Its when the estimation process is thrown into the mix that the whole process starts to fall apart.
The insistence on using the monthly / seasonal / annual averaging and estimation process prior to calculating bias is the real problem. Would it not make far more sense to simply take the monthly data in common between the two records and simply average those values without introducing a convoluted estimation process? Manual inspection of the records tells me that far more often than not there is no difference between the periods of overlap, so no bias would be introduced.
Often times where there is a difference it points to an error in an upstream process that can be corrected. For example, I have seen the occasional records where, say, January 20 is recorded in one place as being 5 C and in another as being -5 C. When monthly averages are calculated for each of the two scribal records, they differ slightly – and that will introduce bias when they are combined.
Furthermore, I have seen instances where an entire month will be dropped from the GHCN record if a single day in that record is suspicious. In the above example, 5C and -5C might be plausible January temperatures and not raise suspicion. But -24C in June is suspicions. The GHCN quality-control process assumes the entire month might be bad and so they drop the value, forcing GISS to later estimate it. However, manual inspection might show that the day before had a temp of 21C and the day following 25C. You think there might have been an error with the sign that could easily be fixed, allowing the month to be “saved”?
John Goetz “multiple temperature records collected at presumably the same station”
How can there be any overlap if the temp records are from one station? There could be gaps, but no overlap. WUWT?
More blatent Climate fraud:
ul 23, 2009
Pacific Northwest Snow Pack – the True Story
By George Taylor
Washington Governor Gregoire recently sent a letter to the Washington House delegation in which she stated that the snow pack has declined 20% over the past 30 years: “Last month, a study released by the University of Washington shows we’ve already lost 20% of our snow pack over the last 30 years.”
Actual snow pack numbers show a 22% INCREASE in snow pack over the past 33 years across the Washington and Oregon Cascade Mountains:
image
Larger image here. See post here.
ICECAP NOTE: In this story on Sustainable Oregon, George shows how choosing start and end dates makes all the difference in trend analysis. This is true because precipitation trends in the northwest are linked to the PDO cycle of 60 or so years. In the cold phase, La Ninas and heavy snowpacks are common (like the last two years) and in the warm phase, El Ninos and drier winters (as was the case from the 1970s to late 1990s). By cherry picking his start data as 1950 at the very snowy start of the cold PDO pahse from 1947 to 1977 and ending in 1997 at the end of the drier warm PDO phase from 1979 to 1998, Mote was able to extract a false signal which he attributed to man made global warming.
Arguing this point made George Taylor, state climatologist for decades in Oregon a target (he took early retirement) and cost the assistant state climatologist in Washington, Mark Albright, his job. Phil Mote, the alarmist professor and author of a discredited work on the western snowpack for the Bulletin of the American Meteorological Society doesn’t accept criticism lightly. He ironically was appointed to the state climatologist position George Taylor held in Oregion. It was Phil who fired Mark for challenging his findings. That is the way it is in the university climate world today, real data doesn’t matter so don’t bother to look and if you need to pick and choose carefully. Anyone who disagrees publically and risks funding need look elsewhere for employment.
George shows the 1950 to 1997 trend and the longer term trend analysis for several stations with good records showing no discernible long term trends.
The story doesn’t end there as this post by Jeff ID called SNOWMEN tells, another climate schiester, Eric Steig who made the headline last year when he worked with Michael Mann, the king of data fraud to eliminate the antarctic cooling of the last several decades. Eric chimed in against Taylor and Albright defending Mote and making false or at least uninformed claims about trends. It is clear from Steig’s Real Climate post never even looked at the whole data trends. Jeff correctly notes “These plots are of specific stations, however they demonstrate that at least for the above locations the 1950-1997 trend is a cherry pick, nothing more.”
Unfortunately this bad analysis has gotten people promoted and been used by state governments to make unwise decisions like supporting the flawed and costly and totally unnecessary WCI (Western Climate Initiative), which Paul Chesser writes about in this American Spectator story here. Climate frauds like Mann, Mote and Steig have a lot to answer for, if the governments measures inflict major pain on the citizens and the globe continues to cool in its natural rythym.
http://www.icecap.us
Climate Fraud gets a face:
Jul 22, 2009
Science Czar, John Holdren’s Goldman Sach’s Connection
The Liberty Journal
As I was doing some research in some non-profit’s literature, appeared before me was a 2006 picture of John Holdren, Bill Clinton, and this other guy (name not mentioned). So what, you say. Well the caption indicates, John Holdren’s Woods Hole Research Center Director excepts $1mil check from Goldman Sachs Center for Environmental Markets (CEM).
image
Woods Hole Research Center describes themselves: “The Woods Hole Research Center is an independent, non-profit institute engaged in fundamental environmental science, applied policy analysis, local and regional capacity building, and public and policy-maker education aimed at clarifying the interacting functions of the Earth’s vegetation, soils, water, and climate in support of human well-being and promoting practical approaches to their sustainable management in the human interest.”
In other words, they’re another rich environmental think tank 501(c) non-profit with rich members, well connected to the corporate world, who use their income to influence public policy to further increase their wealth. The Woods Hole Research describes this venture with Goldman Sachs and Bill Clinton:
“A new partnership between the Woods Hole Research Center and The Goldman Sachs Center for Environmental Markets (CEM) announced yesterday at the Clinton Global Initiative will develop new market-based approaches to value the sustainable uses of forests for marketable products and ecosystem services.”
John Holdren is obviously excited as stated:
“It’s particularly gratifying that we developed this project with Goldman Sachs as part of the Clinton Global Initiative – a farsighted effort of the former President to stimulate new partnerships among businesses, researchers, and public-interest organizations to address the great challenges in global health, environment, and economic development. This is not only a grant but also a partnership, in which insights from the Woods Hole Research Center about how forests work and what is needed to keep them working will be linked with expertise at Goldman Sachs about the economic forces and incentives that affect how forests are used and managed.”
Maybe Holdren and Goldman Sachs share ideas while they are at the Council on Foreign Relations meeting. Maybe it’s that John Holdren speaks at Goldman Sach’s conferences, like the “Energy, Environment and the Financial Markets: The Global Opportunity” in London.
Well, they want to make sure they know the value of every last tree and forestland on the earth. Ok, Goldman Sachs is your company if you want to figure how to equate everything with some monetary value as to create an investment from it.
Now to enforce that idea. From their website, here is how Goldman Sachs describes the CEM: “The Environmental Markets Group manages the Goldman Sachs Center for Environmental Markets. The Center works with independent partners in the academic and non-government organization communities to examine market-based solutions to environmental challenges. Two of their primary goals are : (1) Market-making in carbon emissions and other climate related commodity markets and (2) Launching GS SUSTAIN, a global equity strategy that incorporates environmental, social and governance issues into fundamental analysis of companies
Well, it’s merely the rich using tax free big bucks through non-profits to grease palms and divy up the spoils of their pillaging of tax-payer coffers. Only in American politics. See post here.
See SPPI’s new paper on climate money here by Joanne Nova. It starts out: “The US government has spent over $77 billion since 1989 on policies related to climate change, including science and technology research, administration, education campaigns, foreign aid, and tax breaks. Despite the billions: �audits� of the science are left to unpaid volunteers. A dedicated but largely uncoordinated grassroots movement of scientists has sprung up around the globe to test the integrity of the theory and compete with a well funded highly organized climate monopoly. They have exposed major errors.”
**********************
The Morality of Climate Change: (Uploaded 18 July 2009)
One has to know all the facts to determine the morality of an issue. John Christy on CO2Science.org, below and enlarged here.
See this town hall attack on Mike Castle, one of the Republicans who voted FOR Cap-and-Trade below.
“Well its now snowing in Ezeiza airport Buenos aires so much for AGW
http://www.perfil.com/contenidos/2009/07/22/noticia_0033.html spanish”
For those of you that do not read Spanish, the article speaks of an intense cold wave covering a large region of Argentina, and accompanied by below zero (Celsius) temperatures, strong winds, downed electrical lines, road closures, and damage to structures.
Continued falling temperatures are forecast for Buenos Aires.
This is pretty significant, and I would expect it to be picked up by the International news media pretty soon.
There are a few comments posted; One mocks global warming, and says that it is not happening, and one expresses concern that thousands of indigenous people may die of the cold.
Thank you, Mr. Goetz, for your analysis. So far, it appears to be very relevant; and again — this is only one step that introduces a suspicious result. There are many others. I am intrigued by your observation that multiple scribal records is overwhelmingly a non-U.S. issue; many GW pessimists point out that the U.S., representing only 2% of the earth’s surface, may not be experiencing warming but the rest of the world is.
John Goetz (05:46:56) wrote to John F. Hultquist (19:24:08)
I don’t believe a method was selected to create a warming bias.
However E.M. Smith writes (22:25:21) :
“It’s even worse than that… The first step in GIStemp is actually STEP0, and that step does a couple of suspect things all by itself. (Cherry picks 1880 and deletes any data older than that. Takes an “offset” between USHCN and GHCN for up to 10 years from the present back to no earlier than 1980 then “adjusts” ALL past data by subtracting that “offset” – supposedly to remove the NOAA adjustments for things like TOBS, UHI, Equipment, etc., …)”
And: “Those kinds of Logical Landmines are scattered through the whole thing.”
And: “IMHO, this code is more of a glorified “hand tool”. Something that someone cooked up to let them play with the data. There are lots of places where you can insert “plug numbers” to see what happens.”
IMHO the entire global temperature record MUST be reexamined by real committed scientists. Ever since Anthony’s invaluable expose on what kind of temperatures our surface stations are measuring we have known that a reevaluation is required.
Thanks to John Goetz for these great efforts and to E.M. Smith for going into more depth. And Steve McIntyre deserves kudos for being among the first, as far as I know, to take on the “evil geniuses” (stupidheads or manipulatormaniacs).
Mac at 19:48 has given us the article that follows the money.
http://scienceandpublicpolicy.org/images/stories/papers/originals/climate_money.pdf
It’s time to throw the [self-snips] out. No disrespect to mothers and their great efforts at discipline, but real science and real representative democracy also requires the “no” of the fathers. This bunch is on the take and is trying to take us to the cleaners.
This is a surprising result-a seemingly innocuous if strange methodology actually leading to a warm bias.
Some commenters wanted to test if the method ALWAYS leads to warm bias but I don’t think that’s the right question. The question should be, is the result leading to a warm bias in REALITY?
It probably is pure chance but man, what a ringer!
Correction: Steve McIntyre was among the first to expose the “evil geniuses'” many computer programming tricks. I will never be able to believe that they began in innocence, their mistakes just ran away with them, and now they must save face. Too much money is floating around for the climate scam.
“Richard Sharpe (20:44:13) :
I don’t know about GissTemp whatever, but something is up with the weather in the San Francisco Bay Area.
We are having very cold nights and cool days in the middle of summer.”
I know it’s called weather….but Ky just had it’s first July ever without a single day in the 90’s….or at least that’s the forecast. Going down on record, if it holds, the coldest July since records were started.
I’m no statics expert but the end of the graph seems very different than everything up until 1982. The early changes seem to be much more gradual and follow a general trend. Have recent temps really been that different that the extreme changes up and down have not occurred prior? It looks to me like either temps have been fluctuating much more wildly or the data is being treated differently. Can someone explain this for me?
Thanks, John
This off topic but may be of interest. Dr. David Evans points out a new trick that the Alarmists are trotting out: that the atmosphere may be cooling but the oceans are warming.
“Senator Steve Fielding recently asked the [Australian] Climate Change Minister Penny Wong why human emissions can be blamed for global warming, given that air temperatures peaked in 1998 and began a cooling trend in 2002, while carbon dioxide levels have risen five per cent since 1998. I was one of the four independent scientists Fielding chose to accompany him to visit the Minister.
The Minister’s advisor essentially told us that short term trends in air temperatures are irrelevant, and to instead focus on the rapidly rising ocean heat content…”
http://www.globalresearch.ca/index.php?context=va&aid=14504
“Sanity is not statistical.”
– George Orwell, 1984, Book 1, Chapter 9
The net effect is a tenth of degree in more than 100 years, that’s not much.
It’s a fair percentage, though. USHCN1 raw data from NOAA stations shows a 0.14C warming average per station (equally weighted) over that period. Fully adjusted USHCN1 is +0.59C per station. TOBS-only is +0.31. (Fully adjusted and gridded USHCN2 is c. 0.72C.)
Consider the following possibility (my own musings) in the average global temp. Warm oceans produce a narrower band of lows and highs without extremes but the band is higher up on the thermometer. Colder oceans produce a broader band of highs and lows with extremes on both ends. Now consider that warm and cold conditions occur regularly along with neutral conditions, with warm occuring more frequently during El Nino oscillation, and cold occuring more frequently during La Nina oscillations. What would the average turn out to be in these two cases? It would be interesting to look at the range of temps as well as record events and then correlate with ENSO to see if this idea of an average global temp could be misleading.
John Goetz, I’m sorry but I couldn’t resist when I got to the bottom of AnonyMoose’s post. I know that what you are doing takes a huge effort, and I applaud that effort more than anyone. I did not intend to detract from the posting in any way. Your work is very important, and appreciated.
This is an interesting piece, another reason not to trust anything coming out of GISS. I wonder if John Goetz is prepared to do a full posting on all of the other steps of temperature analysis, or have I missed out on these other steps in previous posts?
Well I have a couple of questions. (1) Why do such multiple records covering the same time frame even exist. Do these measuring stations take temperature readings for each day that become a solid permanent records of a part of the universe; or don’t they; how do multiple versions of the sme station data come into existence; and why do they exist for so many stations that a special algorithm has to be dreamed up to handle those cases.
My second question; How does planet earth know about our 12 month system of calendar notation; so that it can react to that, and do the correct different things for Spring, Summer, Autumn, and Winter seasons. does the planet know that not all months are the same length ?
Why not simply report the data, as one continuous stream of infromation updated daily. Do you know of any newspaper media that print news information that is classified as being spring, summer, autumn or winter news, and handled differently quarter by quarter.
The whole methodology sounds to me like a three dollar bill; meaningless pigeon holing of groups of data, to create an illusion that more information can be extratced than there is in the original daily records, that were actually noted down from the thermometers for each day.
In a normal experimental methodology, where multiple readings of some variable are made within a small time frame, the separate readings are simply averaged to obtain a presumably more likely value for the variable.
This “Bias” hocus pocus, sounds like witchcraft to me.
Ron de Haan (06:17:26) : You wrote about Pacific Northwest Snow Pack – the True Story
Gregoire’s Washington is run by her and her Democratic colleagues from the wet side of the State. There are occasional calls to separate the dry side and form a new state. Meanwhile, official State policy is completely C-AGW. Snow pack and runoff records are as easily manipulated as temperature. Although T-max this week in Eastern Washington is near 100 F (38 C) this is not unusual for late July, and irrigators have their full allotment from the reservoirs and snowmelt on the east slopes of the Cascade Mountains.
I wonder how they will correct for this:
FOR ROCHESTER…AVERAGE JULY TEMP 70.7.
COOLEST JULYS (BACK TO 1871)…
2009… 64.3 (THRU 7/22)
1884… 65.4
1992… 66.6
1891… 67.1
2000… 67.1
FOR BUFFALO…AVERAGE JULY TEMP 70.8.
COOLEST JULYS (AIRPORT DATA BACK TO 1943)…
2009… 65.4 (THRU 7/22)
1992… 66.8
1956… 67.6
2000… 67.6
1976… 67.8
COOLEST JULYS (INCLUDING DOWNTOWN DATA BACK TO 1871)…
1884… 65.2
1891… 65.3
2009… 65.4 (THRU 7/22)
1920… 66.1
1883… 66.8
1992… 66.8
— End Changed Discussion —
The increase circa 1994-5 is obviously a step change, implying either equipment or data processing failure.
An Inquirer (06:37:18) :
I am intrigued by your observation that multiple scribal records is overwhelmingly a non-U.S. issue;
As am I.
Keep up the good work, John and EM.
George E. Smith (08:47:45) :
“The whole methodology sounds to me like a three dollar bill; meaningless pigeon holing of groups of data, to create an illusion that more information can be extratced than there is in the original daily records, that were actually noted down from the thermometers for each day.”
I get the distinct impression that this could be a form of, “Now keep your eye on the cup with the ball under it.”
In other words, a lot of shuffling to make certain that we get the results we want while hiding mostly pointless processing complexities that add enough non-transparant bias to prove our point.
George E. Smith (08:47:45) : Obviously you miss the point. Warmists are statistical experts who have taken the branch of mathematics we call Statistics into the parallel universes. Their quantum-like techniques can only be understood by the Inner Circle of the Hockey Team – or by anyone who uses LSD. A simple average is so far beneath them that they have forgotten what it is.
Have a look at the latest low temperature records updated for the week ending 21 July.
http://globalfreeze.wordpress.com/2009/07/22/usa-chills-1077-lowest-max-temps-and-856-low-temps-for-week-ending-tue-21-july/
SEE
http://cdiac.ornl.gov/epubs/ndp/ushcn/ts.ushcn_anom25_diffs_urb-raw_pg.gif
FROM
http://cdiac.ornl.gov/epubs/ndp/ushcn/ndp019.html
What’s the old line about not attributing to malice what can more easily be explained by incompetence?
I’m no scientist – as a coder I’m a craftsman, a technician – but when I look around at GIStemp – the design of the whole thing, as well as some of the code (I haven’t yet gone through it exhaustively) what I see is the product of someone who was learning as he went, the coding as well as probably the analysis.
I’m no Hansen fan, and none of this excuses Hansen’s failure to recognize the limitations and inconsistencies of the end result, but it explains why it is what it is.
Given the current state of the art one could staple together the hardware for a database of all the available time-series data sets, tagged with each model’s or researcher’s adjustments, fills, averages, etc. for about the cost of a decent flat-screen TV. Instead we get models that recrunch and revise history every time they’re run.
Ron de Haan (11:08:17) : Then we can say “it” has begun. The new LIA is on.
Ron de Haan, notice the dot spread on the Hamweather map? That is more likely due to fewer stations, not because of fewer records.
JEM (13:44:47) : It’s called Hanlon’s razor and I always try to remind people to apply it. Good to see someone else do it!
Just eyeballing, but maybe the sudden increase late in the record is related to the massive drop in the number of stations toward the end of the record? Lot’s of people asked about that. It doesn’t look totally correlated to number of stations but I imagine that the effect would be very non linear and not constant anyway. I also see a possible effect of the more steady ramp up of coverage by percent area:
http://data.giss.nasa.gov/gistemp/station_data/stations.gif
The obvious situation is when equipment is replaced. Run the new and old together for a while.
Goetz and Smith – I thank you for what you’re doing, and I’m fully familiar which what you’re wrestling with. I’ve had to untangle business logic which was spread through odd corners and buried in routine procedures. It’s fun watching the lightbulbs go on when you explain how their own business is working, so they can see what can be adjusted. Of course, it’s different when all the audience and participants are friendly and helpful, rather than part of it wanting to be left alone.
john (07:21:37) :
A lot of GHCN records end in 1990 and are replaced by MCDW records that usually start in 1987, so it is only to be expected that very strange things happen around that time.
For those who have asked “how can there be two records from one station?”:
I believe that this is an artifact of the “odd choice” GIStemp makes to merge the GHCN and USHCN data rather than just “picking on”. So some of the same station raw data reading is processed one way into GHCN and another way into USHCN, then GIStemp “unadjusts” some of the records, throws some away entirely, and merges the resultant data. This can give two different records for the same station with different histories of “adjustment”.
What’s worse is that USHCN data may be kept unchanged, or GHCN data may be kept unchanged, or one may be used to “un-adjust” the other if both series exist for a site. Then the whole thing is mushed together into “one” dataset. This is what gets handed to STEP1 that then “blends the records” together. So you may well have disjoint periods of time with disjoint adjustment histories, all glued together (and with “gaps” filled in by simply making up missing data by guessing via the “reference station method”). So at the end of this, you really have at least 4 different things blended together and called “one data series” (and “smoothed”):
GHCN, USHCN, “Un-adjusted” hybrid, fabricated via reference station method.
Yes, that’s what it does…
Is that a valid technique? Who knows…
Comes back to the “accurate but corrupted” data (er, corrected data) that is used to “calibrate” the circulation models to the GIS “as-analyzed” temperatures between 1970 and 1998, doesn’t it?
Well, at long last I have a contribution based on the work porting GIStemp. I can now run it up to the “add sea surface anomaly maps” stage, and this means I can inspect the intermediate data for interesting trends. (The STEP4_5 part will take a bit longer. I’ve figured out that SBBX.HadR2 is in “bigendian” format and PCs are littleendian, so I have a data conversion to work out…).
Ok, what have I found in steps 0, 1, 2, …? Plenty. First off, though, I needed a “benchmark” to measure against. I decided to just use the canonical GHCN data set. This is what all the other bits get glued onto, so I wondered, what happens, step by step, as bits get blended into the sausage? I also wondered about the odd “seasonal” anomaly design, and wanted a simple year by year measure.
So my benchmark is just the GHCN monthly averages, summed for each month of the year, cross footed to an annual “Global Average Temperature”, and then a final GAT for ALL TIME is calculated by averaging those yearly GATs.
Now, there are a couple of caveats, not the least of which is that this is Beta code. I’ve cobbled together these tools on 5 hours sleep a night for the last few days (It’s called a “coding frenzy” in the biz… programmers know what I’m talking about… you don’t dare stop till it’s done…) So I’ve done nearly NO Quality Control and have not had a Code Review yet (though I’ve lined up a friend with 30+ years of high end work, currently doing robotics, to review my stuff. He started tonight.) I’m fairly certain that some of these numbers will change a bit as I find little edge cases where some record was left out of the addition…
Second is that I don’t try to answer the question “Is this change to the data valid?” I’m just asking “What is the degree of change?” These may be valid changes.
And third, I have not fully vetted the input data sets. Some of them came with the source code, some from the GIS web site, etc. There is a small possibility that I might not have the newest or best input data. I think this is valid data, but final results may be a smidgeon different if a newer data set shows up.
Ok enough tush cover: What did I find already?!
First up, the “GLOBAL” temperature shows a pronounced seasonal trend. This is a record from after STEP1, just before the zonalizing:
GAT in year : 1971 3.60 6.20 8.20 12.90 16.50 19.30 20.90 20.70 17.90 13.90 9.50 5.60 14.10
The first number is the year, then 12 monthly averages, then the final number is the global average. The fact that the 100ths place is always is a 0 is a direct result of their using C in tenths at this stage. It is “False Precision” in my print format.
It seems a bit “odd” to me that the “Globe” would be 17C colder in January than it is in July. Does it not have hemispheres that balance each other out? In fairness, the sea temps are added in in STEP4_5 and the SH is mostly sea. But it’s pretty clear that the “Global” record is not very global at the half way point in GIStemp.
Next is from GHCH, to GHCN with added (Antarctic, Hohenp…., etc.) and with the pre 1880’s tossed out and the first round of the Reference Station Method. The third record is as the data leaves STEP1 with it’s magic sauce. These are the total of all years in the data set. (The individual year trends are still being analyzed – i.e. I need to get some sleep 😉
2.6 3.9 7.3 11.8 15.8 18.9 20.7 20.3 17.4 13.1 7.9 3.9 11.97
2.6 3.8 7.3 11.7 15.6 18.7 20.5 20.0 17.2 13.0 7.9 3.9 11.85
3.2 4.5 7.9 12.1 15.9 19.0 20.9 20.5 17.7 13.5 8.5 4.5 12.35
It is pretty clear from inspection of these three that the temperature is raised by GIStemp. It’s also pretty clear that STEP0 does not do much of it (in fact, some data points go down – Adding the Antarctic can do that!). The “cooking” only really starts with STEP1.
The big surprise for me was not the 0.38 C rise in the Total GAT (far right) but the way that winters get warmed up! July and August hardly change (0.2 and 0.3 respectively) yet January has a full 0.6 C rise as do November, December, Febrary, and March.
So GIStemp thinks it’s getting warmer, but only in the winter! I can live with that! At this point I think it’s mostly in the data, but further dredging around is needed to confirm that. The code as written seems to have a small bias spread over all months, at least as I read it, so I’m at a loss for the asymmetry of winter. Perhaps it’s buried in the Python of Step1 that I’m still learning to read…
Finally, a brief word on trends over the years. The GIStemp numbers are, er, odd. I have to do more work on them, but there are some trends that I just do not find credible. For example, the 1776 record (that is very representative of that block of time) in GHCN is:
GAT/year: 1776 -1.40 2.30 4.20 7.20 12.10 18.20 19.70 19.30 15.60 9.50 3.00 -0.40 9.89
The 2008 record is:
GAT/year: 2008 8.30 8.30 11.10 14.60 17.60 19.90 20.90 20.90 18.80 15.50 11.00 8.80 15.90
Notice that last, whole year global number? We’re already 6 C warmer!
Now look at the post step1 record for 1881:
GAT in year : 1881 3.50 4.10 6.40 10.90 15.30 18.20 20.20 19.80 17.20 11.80 6.40 3.40 11.43
According to this, we’ve warmed up 4.5C since 1881 and the 1971 record above was a full 2.7C warmer than 1881. But I thought we were freezing in 1971 and a new ice age was forecast?!
Now take a look at January. No change from 1881 to 1971 (well, 0.1c) but February was up 2.1C, March 1.8C, December 2.2C. And the delta to 2008 is a wopping 4.8C in January and 5.4C in December, but July is almost identical. By definition, picking one year to compare to another is a bit of a cherry pick, even though these were modestly randomly picked. (There are “better” and “worse”: 1894 was MINUS 2.4c in January). But even with that, the “globe” seems to have gotten much much warmer during the Northern Hemisphere winters.
Somehow I suspect were seeing a mix of: Exit from LIA in the record that is mostly focused on N. America and Europe; any AGW being substantially in winter in the N.H. and not really doing much for summer heat (if anything), and potentially some kind of bias in the code or temperature recording system that has been warming winter thermometers (heated buildings nearby, car exhausts, huge UHI from massive winter fuel today vs a few wood fires 100+ years ago).
I’ve seen nothing in the AGW thesis that would explain these patterns in the data. Certainly not any “runaway greenhouse” effect. The summers are just fine…
So I’m going to dredge through the buckets of “stuff” my new toy is spitting out, and spend a while thinking about what would be a good article to make from this… and do a bit of a code review to make sure I’ve got it right. In the mean time, enjoy your balmy winters 😉
(And if Anthony would like a copy of the ported GIStemp to play with, well, “Will Program for Beer!” 😉
Hmmm…. A bit further pondering….
Does anyone have a graph of S.H. thermometer growth over time? It would be a bit of a “hoot” if the “Global Warming” all came down to more thermometers being put in The Empire in Africa, Australia, et. al. then to Soviet Union dropping SIberia out in large part…
Could GW all just be where in the world is Carmen Sandiego’s Thermometer?
😎