I’ve been following this issue a few days and looking at a number of stations and had planned to make a detailed post about my findings, but WUWT commenter Steven Douglas posted in comments about this curious change in GISS data recently, and it got picked up by Kate at SDA, which necessitated me commenting on it now. This goes back to the beginning days of surfacestations.org in June 2007 and the second station I surveyed.
Remember Orland? That nicely sited station with a long record?
Note the graph I put in place in June 2007 on that image.
Now look at the graph in a blink comparator showing Orland GISS data plotted in June 2007 and today:
NOTE: on some browsers, the blink may not start automatically – if so, click on the image above to see it
The blink comparator was originally by Steven Douglas. However he made a mistake in the “after” image which I have now corrected.What you see above is a graphical fit via bitmap alignment and scaling of the images to fit. This is why the dots and lines appear slightly smaller in the “after” image. I don’t have the GISS Orland data handy at the moment from 2007, but I did have the GISS station plots from Orland from that time and from the present, downloaded from the GISS website today. If I locate the prior Orland data, I’ll redo the blink comparator.
I believe this blink comparator representation accurately reflects the change in the Orland data, even is the dots and lines aren’t exactly the same thickness.
Douglas writes in his notice to me:
It appears that RAW station plots are no longer available, although NASA GISS (Hansen et al) do not say it in this way. Here is the notice on their site:
Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).
I don’t know about the “renamed” option, but the RAW data appears to be NO LONGER AVAILABLE.
Here’s a detailed blink comparison of Orland. All their options now give you an “adjusted” plot of some kind. The “AFTER” in this graph show the “adjustments” to Orland.
Here is what the GISS data selector looks like now, yellow highlight mine, click to enlarge:
Above clip from: http://data.giss.nasa.gov/gistemp/station_data/
Here is the “raw” GISS data plot of Orland I saved back in 2007:
And here is another blink comparator of Orland raw -vs- homogenized data posted by surfacestations.org volunteer Mike McMillan on 12/29/2008:
And here is the “raw” GISS data for Orland today, please note the vertical scale is now different since the pre-1900 data has been removed, the GISS plotting software autoscales to the most appropriate range:
Source:
And it is not just Orland, I’m seeing this issue at other stations too.
For example Fairmont, CA another well sited station well isolated, and with a long record:
Here is Fairmont “raw” from 11/17/2007:
And here is Fairmont from GISS today:
Source:
This raises a number of questions. for example: Why is data truncated pre-1900? Why did the slope change? The change appears to have been fairly recent, within the last month. I tried to pinpoint it using the “wayback machine” but apparently because this page:
http://data.giss.nasa.gov/gistemp/station_data/
is forms based, the change in this phrase:
Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).
Appears to span the entire “wayback machine” archive, even prior to 2007. If anyone has a screen cap of this page prior to the change or can help pinpoint the date of the change, please let me know.
It is important to note that the issue may not be with GISS, but upstream at GHCN data managed by NCDC/NOAA. Further investigation is needed to found out where the main change has occurred. It appears this is a system wide change.
The timing could not be worse for public confidence in climate data.
I’ll have more on this as we learn more about this data change.
UPDATE1 from comments:
GISS also just started using USHCN_V2 last month. See under “What’s New”:
http://data.giss.nasa.gov/gistemp/graphs/
“Nov. 14, 2009: USHCN_V2 is now used rather than the older version 1. The only visible effect is a slight increase of the US trend after year 2000 due to the fact that NOAA extended the TOBS and other adjustment to those years.
Sep. 11, 2009: NOAA NCDC provided an updated file on Sept. 9 of the GHCN data used in our analysis. The new file has increased data quality checks in the tropics. Beginning Sept. 11 the GISS analysis uses the new NOAA data set. ”
Sponsored IT training links:
Worried about N10-004 exam? Our 640-802 dumps and 70-680 tutorials can provide you real success on time.
Another new study – this time on Antarctica data adjustments:
http://savecapitalism.wordpress.com/2009/12/11/ghcn-antarctica-careful-selection-of-data/
I can find raw and adjusted Orland here which match your original raw plot, by eye:
http://www.appinsys.com/GlobalWarming/climgraph.aspx?pltparms=GHCNT100AJanDecI188020080900111AR42572591004x
Control panel page:
http://www.appinsys.com/GlobalWarming/climate.aspx
They’re just adjusting the these poorly sited stations to homogenize with the ones in the middle of parking lots since their measurements are obviously flawed
Nothing to see here, move along…
/sarcasm off
Finding the UK GISS records were seriously tampered with in similar ways to all this, I prepared a page… it has been sitting in the backwaters but I think it belongs here with everything now coming out about the GISS records. Here it is. Here too, in the British Isles, GISS have been adjusting to INCREASE the trend… consistently.
Randy (14:12:09) :
Anthony
…How many climate scientists rely on this ‘adjusted’ data believing it to be a solid foundation upon which they then do their thing?…
Randy
REPLY: The answer is, almost all of them. There are very few papers questioning the data integrity. Most take the data at face value, never questioning the measurement environment and the data procedures. I didn’t start questioning it myself until Spring of 2007. – Anthony
This lack of questioning data integrity extends to a lot of other geophysical data as well. The “Palmdale Bulge” is a pertinent example. And the use of borehole temperature records to reconstruct past climate is rife with such problems. This is why I am now interested in the status of the UHI project that Anthony proposes on the “projects” page of this site. It looks like an interesting approach to this pesky issue, which will still plague the credibility of the data even if all the other issues of station maintenance and data adjustment are resolved.
Richard
…
These teams each provide 95% confidence limits for their results. However, the results of the teams differ by more than double those limits in several years, and the data sets provided by the teams have different trends….
Now why wouldn’t anyone notice allegedly independent estimates of the same quantity that differ by two times their respective 95% confidence intervals? 95% means something specific and to differ by two times such an interval is highly improbable. If Richard is right about this, and I have interpreted what he says correctly, why didn’t more alarm bells go off? This is exactly the type of data consistency issue that eventually deflated the “Palmdale Bulge”.
Gosh, another 15 minutes of fame.
If you’d like to see Old Raw vs the improved New Raw, I have a page that has all the stations in Illinois blinking away. You can click on each one to see the station chart full size.
http://www.rockyhigh66.org/stuff/USHCN_revisions.htm
The extent of the alterations is breathtaking when you see them all together, and keep in mind, this is USHCN Raw data, not GISS’s Homogenized versions. Whoever did this is making Dr Hansen look like a piker.
These alterations have also affected the few GISS homogenized charts I’ve examined.
After recalling an old television commercial, here’s a suggestion on how to simplify the terms of the “Corrupted temperature data discussion” as follows:
Here is your raw temperature data
Here is CRU temperature data on drugs.
Just say no to data on drugs!
This is getting serious (no it’s been serious for some time, but this is a wake up call). The AGW juggernaut just keeps rolling. If the data are really being fraudulently ‘adjusted’ and there is no control or monitoring of the fixing, then I agree that we need a top lawyer and a class action. Otherwise they will continue to get away with one of the greatest scientific frauds in history. Truth and honesty will – must – ultimately prevail.
RE CMT @ur momisugly 14:37:22-
Thanks that does work, but is one method any more valid that the other? I mead is the average of daily averages “better” than averaging TMAX and TMIN separately for the whole month and then averaging these monthly outcomes? Obviously the two techniques produce significantly different results, but is one more meaningful than the other in terms of representing the mean temp of a station?
Thanks
In 1888, America Weather by Gen. A. W. Greeley was published. On page 253 of that book, the following account is given:
“The most remarkable case, that of June 17th, 1859, was at that time said to be the most wonderful visitation of this character for thirty years. At San Francisco on that date, the thermometer is said to have registered a rise in the temperature from 77F to 133F, with a burning northeast wind, which fortunately lasted for a few hours only, the thermometer registering 77F at 7 p.m. At Santa Barbara, on the same day, (in the afternoon), a strong easterly wind set in, during which the burning air was filled with dense clouds of dust, which caused intense suffering, and drove everyone to the nearest shelter. The fruit was all destroyed, and although the burning blast lasted but a few hours, yet animals, such as calves, rabbits, and birds, died from the effects. The temperature was said to have reached 133F in Santa Barbara, 102F at San Diego, and 117F at Fort Yuma.”
He goes on to say that such heat waves have occurred less frequently as time has passed. So perhaps the early Orland record has some merit to it.
The listed record highs for June 17 for these 3 cities is 96F in 1957, 93F in 1957, and 115F in 1981 for Yuma AZ (is that Fort Yuma?).
“REPLY: No it is still intact at NCDC, on paper forms, with transcription data also available – Anthony”
I have some questions.
How does the process of transcription work?
Is it in-house or contracted out?
How do they do QA on the transcription process and on the transcripted data?
What is the traceability between the paper and the transcripted data?
Is the original data paper or microfiche?
How many pieces of paper are we talking about?
What is the condition of the paper records?
Are they handwritten or computer printouts?
How do we access either the paper or the transcription records?
Reloading this data from source with complete traceability is the only way to fix this.
I design and install very high volume machine-human hybrid data entry systems and associated state of the art OLTP VLDB systems.
IMHO, its technically trivial to re-digitize this data. The only thing “new” would be the QA process and this would depend on the nature of the data and the source documents.
The systems costs I cannot cover would be the high volume scanners, systems hosting, and the bandwidth to serve the documents to volunteers. I’ll bet we can get the scanners donated for the duration. I can also design and build the VLDB and image storage systems as well as do all the back end data processing and interfaces. I would need help with the web front ends.
Anyone game?
There are so many pro AGW scientists now being interviewed in the news and claiming that the climategate scandal doesn’t matter because the USA data still shows the warming, that they are obviously now working over time to make sure that this miss-information is valid and that the adjusted raw data will now support them.
These pathetic, deluded souls are either ignoring, or have no clue that the stations are in poor shape and the data has been manipulated as much, or more so, than the CRU data; and that the freedom of information act in the USA is also being obstructed!
I hope I have missed some, but in all of the news stories I have watched so far, no one has brought up the facts of the USA data. This crucial information is being completely overlooked in the main media, so far! When that information gets out, then watch the public backlash, and interest in cap and trade, etc.! When the people finally realize they have been duped, heaven help the dupers.
Stephen
David (12:58:50) :
And while I am on this, is there somewhere that one can acquire the station data that includes Tmin and Tmax? All I have found so far is the monthly means. I would like to look at data on a daily basis, but I can’t find it. Does it exist?
Apologies if someone else has posted this. These are scans of the B-91 forms the observers keep. Some are a little rough.
http://www7.ncdc.noaa.gov/IPS/coop/coop.html
Mike McMillan (14:58:55),
Excellent job! Makes it very easy to see that GISS took the raw data from around 1900 onward, and artificially massaged it to show much more warming.
I wonder how they can explain it, since the data from 1900 was recorded by hand directly from mercury thermometer readings.
You NEVER mess with the “raw”data. Whatever you do, the “raw”data should remain acessible, unaltered. It is a sin to modify “raw”data and present them as being raw. Value added data are worthless without the “raw”data. It is better to discard value added data in any dispute, and rely on the “raw”data. And then I mean really raw data.
http://biggovernment.com/2009/12/11/un-security-stops-journalists-questions-about-climategate/#more-44722
We really must start calling this pattern :Global Climate Data Change, or CDC. The data seems to be the only thing changing.
And we really should refer to climate scientists as Dupes for using this data as the foundation of their work. That may be unfair; who would ever suspect the data was ALL fudged?
There is another name for the producers, adjusters and other guardians of the raw data itself…
Speaking of getting back to the paper records, perhaps instead of using the data as actually written down, maybe they’ve switched to the far better trick of deriving past temperatures by oxygen isotope analysis of the paper used by the weather stations.
It would be nice if someone go on television with a copy of modern data on a particuclar site and an old copy of a newspaper from the same place, showing the high and low for the previous day, arguing that “they’re lying about past temperatures, as this newspaper illustrates, because to make a bogus warming trend they have to either lie about the present or the past, and they figure that nobody from a hundred years ago is going to show up to demand a retraction.”
At http://cdiac.ornl.gov/epubs/ndp/ushcn/access.html I read
“Please note: for users already engaged in analysis using the previous version of these data, you may still access the previous version through the end of 2009 ”
and there is a link to http://cdiac.ornl.gov/ftp/ndp070/ there.
Maybe someone with a decent server and bandwidth should download all that data (looks to be a couple of hundred MBytes) before it goes away on 31 December 2009?
Reply: I got it. ~ ctm
I have looked at the raw data straight from NOAA (GHCN) for a few dozen stations. I am comparing the data from September 2007 to the data from today.
So far there are *zero* changes to the raw GHCN data.
If this holds up for all the stations, then I think the differences that Anthony found either:
a. Occurred between June and September 2007
b. Occurred somewhere in the processing between GHCN and GISTEMP (is GISS now using USHCN v2?)
Everyone please calm down on the conspiracy theories and accusations of fraud.
After doing some semi-manual comparisons, I am now letting my computer do automated comparisons of all of the stations in the USA. I will report more when it’s done.
I’m not really swift at this stuff but it seems to me that with all these adjustments the “Warmers” have set themselves a trap that is going to burn them in the future. (Or maybe it already is and that is why temperatures have leveled off for the past 10 years). They can’t just keep adjusting temperatures ever upwards. Pretty soon, it is going to be obvious that the temperatures bear no relation to reality. Conversely, they can’t keep doing the Ministry of Truth routine of adjusting history downwards. There are too many copies of the current data sitting in skeptic archives. Therefore, they are trapped. The future has to level off and that is going to mean their AGW Theory which demands future warming is wrong and it will be proved so by thier own lies.
I have a copy downloaded with all fields as of June 28, 2009 in case we want to cross check data revisions. It appears that their last update was July 09, so there will be some “change” between my version and what is currently available.
How much of a change *might* be interesting. If anyone from WUWT shop wants a copy of my downloads of data email me. I downloaded about 80 gigs worth of different climate data (yes gigs) in June.
I can open up an FTP connection for downloading if you decide you want access to it. Email me for contact info.
Jack
“Nov. 14, 2009: USHCN_V2 is now used rather than the older version ”
Just in time for Copenhagen. /sarc off
NOTE TO RON DE HAAN
Your email address is invalid, I treid to email you and it was rejected. In keeping with policy here, your recent comments have been removed and you will not be able to post comments until you provide a valid email address.
FOR OTHER POSTERS:
I agree with John V. I’m not interested in conspiracy theory comments. My interest is in finding out the hows and whys.
SO PLEASE COOL IT!
I’ve already lost most of my day dealing with this thing when I wasn’t prepared to because a commenter posted up an erroneous blink comparator.
Moderators: snip/delete OT and way out comments at will.
Don’t quite know where to put this; how about here; but move if you like Chas.
SCIENCE for Nov 27 2009; latest issue page1256-59.
Michael E. Mann et al.
“Global Signatures and Dynamical Origins of the Little Ice Age and Medieval Climate Anomaly.
Hey what the heck; I thought those were supposed to be a figment of the imagination; and didn’t really happen.
Well they used a “global climate proxy network” to reconstruct surface temperature patterns (over the past 1500 years.
Well they basically assert that some spots were as warm as the last decade during the MWP which they call the MCA. I thought an anomaly was something that was out of kilter.
Why is the Mediaeval warm period now called the medieval climate anomaly, and the little ice age is not an anomaly ?
Anyway, Mann’s paper is now published in SCIENCE.