I’ve been following this issue a few days and looking at a number of stations and had planned to make a detailed post about my findings, but WUWT commenter Steven Douglas posted in comments about this curious change in GISS data recently, and it got picked up by Kate at SDA, which necessitated me commenting on it now. This goes back to the beginning days of surfacestations.org in June 2007 and the second station I surveyed.
Remember Orland? That nicely sited station with a long record?
Note the graph I put in place in June 2007 on that image.
Now look at the graph in a blink comparator showing Orland GISS data plotted in June 2007 and today:
NOTE: on some browsers, the blink may not start automatically – if so, click on the image above to see it
The blink comparator was originally by Steven Douglas. However he made a mistake in the “after” image which I have now corrected.What you see above is a graphical fit via bitmap alignment and scaling of the images to fit. This is why the dots and lines appear slightly smaller in the “after” image. I don’t have the GISS Orland data handy at the moment from 2007, but I did have the GISS station plots from Orland from that time and from the present, downloaded from the GISS website today. If I locate the prior Orland data, I’ll redo the blink comparator.
I believe this blink comparator representation accurately reflects the change in the Orland data, even is the dots and lines aren’t exactly the same thickness.
Douglas writes in his notice to me:
It appears that RAW station plots are no longer available, although NASA GISS (Hansen et al) do not say it in this way. Here is the notice on their site:
Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).
I don’t know about the “renamed” option, but the RAW data appears to be NO LONGER AVAILABLE.
Here’s a detailed blink comparison of Orland. All their options now give you an “adjusted” plot of some kind. The “AFTER” in this graph show the “adjustments” to Orland.
Here is what the GISS data selector looks like now, yellow highlight mine, click to enlarge:
Above clip from: http://data.giss.nasa.gov/gistemp/station_data/
Here is the “raw” GISS data plot of Orland I saved back in 2007:
And here is another blink comparator of Orland raw -vs- homogenized data posted by surfacestations.org volunteer Mike McMillan on 12/29/2008:
And here is the “raw” GISS data for Orland today, please note the vertical scale is now different since the pre-1900 data has been removed, the GISS plotting software autoscales to the most appropriate range:
Source:
And it is not just Orland, I’m seeing this issue at other stations too.
For example Fairmont, CA another well sited station well isolated, and with a long record:
Here is Fairmont “raw” from 11/17/2007:
And here is Fairmont from GISS today:
Source:
This raises a number of questions. for example: Why is data truncated pre-1900? Why did the slope change? The change appears to have been fairly recent, within the last month. I tried to pinpoint it using the “wayback machine” but apparently because this page:
http://data.giss.nasa.gov/gistemp/station_data/
is forms based, the change in this phrase:
Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).
Appears to span the entire “wayback machine” archive, even prior to 2007. If anyone has a screen cap of this page prior to the change or can help pinpoint the date of the change, please let me know.
It is important to note that the issue may not be with GISS, but upstream at GHCN data managed by NCDC/NOAA. Further investigation is needed to found out where the main change has occurred. It appears this is a system wide change.
The timing could not be worse for public confidence in climate data.
I’ll have more on this as we learn more about this data change.
UPDATE1 from comments:
GISS also just started using USHCN_V2 last month. See under “What’s New”:
http://data.giss.nasa.gov/gistemp/graphs/
“Nov. 14, 2009: USHCN_V2 is now used rather than the older version 1. The only visible effect is a slight increase of the US trend after year 2000 due to the fact that NOAA extended the TOBS and other adjustment to those years.
Sep. 11, 2009: NOAA NCDC provided an updated file on Sept. 9 of the GHCN data used in our analysis. The new file has increased data quality checks in the tropics. Beginning Sept. 11 the GISS analysis uses the new NOAA data set. ”
Sponsored IT training links:
Worried about N10-004 exam? Our 640-802 dumps and 70-680 tutorials can provide you real success on time.
I’ve seen the Google Earth overhead of Orland station. Very nice site — once upon a time.
But I notice a rather steep recent warming.
And I also notice a fairly new cement path running quite close to the station — in fact, designed specifically for the station. I wonder what effect that has on the raw data, eh?
*****************
David (18:12:03) :
Isn’t everyone a little tired from all this leaping to conclusions? Mr. Watts has asked an interesting question. I see no factual basis for the conspiracy “answers” on this post other than a deep mistrust. That isn’t nearly enough.
*******************
The FACT that climate scientists dodge FOI requests and “lose” data is more than enough. Now NASA seems to be going in the same direction. If We The People distrust them enough, we can get our reps to deal with them. We can tell our reps not to make policy based on a fairly tale. In an ideal world, we could have them canned or their funding cut off. It will be interesting to watch.
Alan S. Blue (12:40:26) :
“The data is available!” But there are ongoing revisions without any apparent version control strategy. It’s like a Monty Python skit or Lucy with the football.
I find it reminds me of the old “Who’s on First?” skit… “I donno! THIRD BASE!!!”
I was going to post that I thought it likely an artifact of the USHCN USHCN.v2 transition, but someone beat me to it.
I had a thread going at the time (the GISS site was suddenly down without notice as to why):
http://chiefio.wordpress.com/2009/11/15/giss-watch-wonder-what-is-happening/
Probably only of hysterical interest now. 😉
On my “to do someday” list is a comparison of USHCN and USHCN.v2 (they are different in a few ways…). The v2 file is in 1/10 F, no longer 1/100 F. The older stations now have MUCH more ‘missing data’ flags It looks like more of our past has been selected for “deletion for QA reasons”. There is also a significant difference in the actual values. (I was doing a before / after benchmark to see what the effect was of the “putting back in” of 2007 to date in GIStemp when I found I could not directly compare them to see what happened at the 2007 transition…)
Oh, and of course, the V2 version has the data from May 2007 to date that is not in the USHCN old format file.
This is a giant “Dig Here” that I’d have done more work on by now, but that whole Climategate thing interrupted… not that I’m complaining, mind you 😉
FWIW, I suspect the reason the USHCN.v2 was tossed in in such a hurry might have something to do with my showing how to do it and starting to run benchmarks. I put an “Easter Egg” in the comments in the posted FORTRAN just for NASA 8-0
http://chiefio.wordpress.com/2009/11/06/ushcn-v2-gistemp-ghcn-what-will-it-take-to-fix-it/
For some completely unknown reason, they suddenly put USHCN.v2 in 😉
I suspect it was a “rush job” to make it go, but their web site has not caught up yet with the “new” USHCN.v2 file format.
And yeah, the constantly shifting sands of old “data” shifting all the time just drives me nuts too… The FIRST thing I’d do in a new data series is leave the past ALONE.
“Check the rerun of tonight’s O’Reilly Factor. Laura Ingraham, sitting in for Windbag, slices and dices Tyson Slocum with a very sharp blade.”
Not as much as she might have, had she had more complete information at her disposal. It’s frustrating to watch these people – their hearts are in the right place, but they should have assigned one person to the file to update them on a daily basis.
All concerned citizens should be writing their US Senators and Reps. I know my efforts to do this with my US Senator have been considered in taking constructive action (see below email). Also, it would be good if someone started a class action lawsuit that concerned citizens could help fund.
==================================
Dear Peter,
Thank you for contacting me regarding global climate change. It is good to hear from you.
I understand your concerns with the recently disclosed e-mails from the University of East Anglia’s Climatic Research Unit (CRU). The American and British scientists that comprise this unit are major contributors to the United Nation’s Intergovernmental Panel on Climate Change (IPCC). Their work provided the foundation of climate data used to create the UN IPCC climate change reports.
The released e-mails raise serious questions about the data used in the IPCC climate reports. This includes the most recent Fourth Assessment. These e-mails demonstrate a coordinated effort by trusted climate scientists to suppress dissenting views and manipulate data and methods to skew the IPCC reports to reach a unified view of climate change. In addition, the elimination by CRU of all the raw data on which these scientists based their models prevents other scientists from replicating their work and raises additional questions about the accuracy of the data used in the IPCC reports.
The actions by these scientists and others to suppress data that contradicts their conclusions is unacceptable. I have sent a letter to the Chairman of the Senate Subcommittee on Oversight in the Senate Environment and Public Works Committee, Senator Sheldon Whitehouse (D-RI), requesting an immediate investigation into this matter. I have also sent a letter to EPA Administrator Lisa Jackson calling for a thorough and transparent investigation into the actions of the scientists that were disclosed in their own emails. Given that the EPA has relied on this data to formulate a variety of policies, I have asked Administrator Jackson to withdraw all climate change regulations until an investigation and independent analysis of the data can be completed.
Thanks again for taking the time to share your views with me. I look forward to hearing from you in the future.
John Barrasso, M.D.
United States Senator
@jim: I suggest that it isn’t NASA, but NOAA. And NOAA has an even more direct connection with corruption than has NASA. Remember the “would you please delete re AR4” e-mail? Guess whom Michael Mann was asked to forward that to? Gene Wahl at NOAA/NCDC! What does that tell you, put together with everything we’ve seen in the last six days?
If you trawl to ftp://ftp.ncdc.noaa.gov/pub/datasets/climatedataold/temp/station/
You can grab the files dated from 1995 which relate to a GHCN ground station dataset.
Plenty of other ‘raw’ data for examination, if anyone with a fast connection would like to create a mirror for the whole of ftp://ftp.ncdc.noaa.gov/pub/ .. in particular ftp://ftp.ncdc.noaa.gov/pub/data/ and ftp://ftp.ncdc.noaa.gov/pub/datasets/ .. for archival and preservation purposes.
Anthony my guess is that you’re contemplating checking more of the archived plots from surface stations.org against current GISS plots. If you’re looking for volunteers count me in.
Anthony Maybe it’s just my suspicious nature but it might be a good idea if you have a backup of your surfacestations.org temperature plots just in case someone with a perverse nature decides to hack it and delete the plots.
Jim (12:58:09) : I guess the Wayback machine wouldn’t work to get the old data since it was stored on a server.
Or is there just a basic time lag for what is available? When the (USAFF) forum was discontinued by the owner I went wayback to collect the 20 months of my “Stay warm, World…” thread (96 pages) and found only the early months available. My assumption is that more recent months will become available over time.
Is my assumption valid?
It has been 3 weeks. Why haven’t they identified the hacker/leaker?
I can no longer trust ANY of these data sets…
Note to AGW quacks, don’t bother trying to convince me of heating or cooling until you can regain my trust and clearly demonstrate (prove beyond show of doubt) that the data is sound. Until then, I will do everything I can to ensure not one bloody dime of my money goes towards this BS (bad science).
Tim Heyes, good point on the traceability in pharma. I can’t imagine an excuse for this, especially given that even Halo or Super Mario Brothers lets you look at the change logs and bug fixes. Heck, everybody’s mouse drivers seem to have more openness and accountability, even rollback in case the new version is junk!
I’d also again note that site relocations should have no statistical impact on the data unless a century ago weather folks kept deciding to move stations to a cooler place to write down the readings in greater comfort. Any other reason for a move should see as many stations go up as down, and by similar amounts, which should all cancel out.
In short, the adjustment to a station’s data should be a matter only of local concern, for the purpose of accurately recording and reporting the area’s temperature for local use and reference because it impacts many decisions in a variety of endeavors.
Yet knowing that truly justified adjustments to station data should produce, world-wide, a statistically insignificant effect on climate records, the only reason to undertake such a global and sweeping revision is to produce an altered account of the climate’s history.
Put simply, the idea that ALL those old geezers put ALL their temperature stations in the wrong place, and all in the same flawed way, and nobody noticed it till now, is absurd. Especially since the discovery of any such fundamental flaw in the site selection behavior of weather geeks would itself merit mention everywhere from Nature to USA Today, ,since AGW alarmists would’ve trumpeted it to the heavens.
oops .. should read : “shadow of doubt”
Want to here a true story that happened to me over the past two days?
Yesterday I posted this on my WUWT blog.
“Michael (16:44:43) :
I scan the comments to the articles on a daily basis at The Huffington Post et al, to measure what I call the “Mass Brainwash Index”, that publication being one of the best places to get accurate results from the populous. Six months ago my index was at a reading of 9.5, 10 being the most brainwashed and 0 being the least. Today my Index has fallen to a reading of 7.5. Something dramatic is happening to the psyche of the American population.”
Today I post this concerning Huffington post;
“Michael (01:15:38) :
Top story on Huffington Posts Green Tab has this as the first comment about The Copenhagen Summit. Is somebody handing out brains over there?
“Mogamboguru I’m a Fan of Mogamboguru I’m a fan of this user 328 fans permalink
” An Incredibly Expensive F o l l y ”
“Why Failure in Copenhagen Would Be a Success”
CO2 Emissions Cuts Will Cost More than Climate Change Itself
Based on conventional estimates, this ambitious program would avert much of the damage of global warming, expected to be worth somewhere around €2 trillion a year by 2100. However, Tol concludes that a tax at this level could reduce world GDP by a staggering 12.9% in 2100 — the equivalent of €27 trillion a year.
.
It is, in fact, an optimistic cost estimate. It assumes that politicians everywhere in the world would, at all times, make the most effective, efficient choices possible to reduce carbon emissions, wasting no money whatsoever. Dump that far-fetched assumption, and the cost could easily be 10 or 100 times higher.
To put this in the starkest of terms: Drastic carbon cuts would hurt much more than climate change itself. Cutting carbon is extremely expensive, especially in the short-term, because the alternatives to fossil fuels are few and costly. Without feasible alternatives to carbon use, we will just hurt growth.
Secondly, we can also see that the approach is politically flawed, because of the simple fact that different countries have very different goals and all nations will find it hard to cut emissions at great cost domestically, to help the rest of the world a little in a hundred years.”
Me;
Yes Virgina there is a Santa Clause.””””””””””””””””””””
Later today I put up this topic on Daily Paul
http://www.dailypaul.com/comment/reply/118748#comment-form
For those at the WUWT blog, the quotes are in the one with the Kid and the one with John Coleman.
True Story.
Hmmm…. Very Curious…
First up, the USHCN file has a different type of stationID:
[chiefio@tubularbells input_files]$ inin ^42572591004
42572591004 ORLAND 39.75 -122.20 77 73R -9FLxxno-9x-9MED. GRAZING C2 15
[chiefio@tubularbells input_files]$ grep 42572591004 ushcn.tbl
46506 42572591004 0
So we’re looking for station 046506
Here is a sample of the “old” Orland data from UHSCN:
grep ^046506 hcn_doe_mean_data | grep …………3A
046506 1883 3A 46.55 B 48.25 B M 63.30 B 59.50 B 67.50 B 81.70 B S 87.50 B 85.30 B 81.44 B 64.04 B 57.54 B 46.95 B 65.80
046506 1884 3A 48.65 B 49.55 B 55.40 B 59.40 B 72.40 B 73.30 B 84.70 B 85.40 B 73.44 B 68.04 B S 59.74 B 52.15 B S 65.18
046506 1885 3A 51.75 B 56.85 B S 66.10 B 66.80 B 75.70 B 75.70 B 83.70 B 87.40 B 80.34 B 72.74 B 57.24 B 53.05 B 68.95
046506 1886 3A 49.05 B 56.55 B 56.90 B 61.30 B 70.00 B 83.00 B 86.90 B 85.90 B 79.94 B 63.24 B 55.04 B 52.95 B 66.73
046506 1887 3A 52.85 B 44.75 B 59.40 B 59.60 B 69.20 B 77.80 B 84.90 B 81.20 B 78.04 B 72.54 B 61.64 B 49.95 B 65.99
046506 1888 3A 43.95 B 54.25 B 53.90 B 69.00 B 70.70 B 75.30 B 85.10 B 85.50 B 84.94 B 69.76 B M 58.14 B 50.55 B 66.76
046506 1889 3A 48.35 B 55.05 B S 57.23 B M 66.49 B M 71.49 B M 86.10 B 86.70 B 84.50 B 80.94 B 64.84 B 59.64 B 47.35 B 67.39
046506 1890 3A 42.85 B 48.35 B 55.00 B 65.00 B 71.00 B 75.70 B 84.70 B S 82.60 B 74.24 B 68.24 B 61.74 B 46.85 B 64.69
046506 1891 3A 51.15 B S 47.25 B 55.10 B 62.20 B 71.10 B S 76.80 B 89.60 B 89.40 B 77.44 B 70.24 B 58.74 B 45.65 B 66.22
046506 1892 3A 47.45 B 51.45 B 55.60 B 59.10 B 69.90 B 76.80 B 84.10 B 83.10 B 73.74 B 65.94 B 58.44 B 48.25 B 64.49
046506 1893 3A 42.65 B 47.95 B 52.10 B 58.50 B 71.00 B 77.70 B 89.90 B S 89.20 B S 70.53 B M 64.09 B M 57.84 B 52.55 B S 64.50
046506 1894 3A 45.05 B 47.95 B 53.00 B 62.40 B 66.50 B 71.30 B 88.30 B S 83.90 B 76.54 B 64.84 B 59.44 B 44.35 B S 63.63
046506 1895 3A 43.15 B 50.65 B 58.20 B 60.80 B 69.00 B 80.80 B 84.60 B 84.10 B 71.64 B 68.84 B 53.54 B 42.65 B 64.00
046506 1896 3A 46.95 B 49.95 B 51.80 B 53.80 B 64.30 B 80.60 B 90.30 B 83.90 B 74.24 B 68.04 B 52.64 B 47.25 B 63.65
046506 1897 3A 46.25 B 46.25 B 48.70 B 65.80 B 76.90 B 79.90 B 87.40 B 83.30 B 77.64 B 63.94 B 54.14 B 48.45 B 64.89
046506 1898 3A 43.25 B 49.65 B 56.40 B S 69.70 B S 70.40 B 81.60 B 89.90 B 84.60 B 76.34 B 66.64 B 54.14 B 44.45 B 65.59
046506 1899 3A 48.95 B 51.65 B 53.10 B 62.80 B 65.60 B 79.40 B S 84.20 B 74.10 B 78.94 B 63.74 B 57.54 B 46.25 B 63.86
046506 1900 3A 49.65 B 51.75 B 59.60 B 64.30 B 69.40 B 79.10 B 83.50 B 78.70 B 72.84 B 61.54 B 57.34 B 46.15 B 64.49
046506 1901 3A 43.65 B 49.25 B 55.50 B 60.99 B M 68.60 B 83.80 B S 89.30 B 86.80 B S 72.74 B 69.24 B 58.14 B 49.55 B 65.63
046506 1902 3A 43.45 B 51.25 B 51.60 B 56.60 B 65.70 B 82.00 B 85.39 B M 79.51 B M 76.84 B 63.31 B M 52.88 B M 47.95 B 63.04
046506 1903 3A 44.65 B 44.65 B 49.22 B 55.85 B 65.63 B 71.58 B S 76.64 B 78.32 B 74.34 B 66.40 B 54.48 B 47.67 B 60.79
046506 1904 3A 47.75 B 48.91 B 50.52 B 59.35 B 70.33 B 79.18 B 80.04 B 81.22 B 76.44 B 63.80 B 54.18 B 43.77 B 62.96
046506 1905 3A 46.45 B 51.61 B 56.32 B 62.35 B 65.43 B 74.28 B 81.44 B 79.52 B 74.64 B 64.80 B 54.08 B 44.87 B 62.98
and a sample of the “new”: Orland data:
Not only do we lose 1883 and 1884 in their entirety, be it looks to me like the “new” version has cooled the past. The thirteenth column is the annual mean, IIRC. The 12 before it are the monthlies.
At the recent end we have, OLD:
NEW:
Where we see the added data from May 2007 to when I downloaded the file.
It also looks like there is a somewhat smaller, but present, warming of the recent data (though the last couple of years are a bit odd and off pattern.
But especially interesting is the 1930’s:
Old:
New
I think we have a great big “DIG HERE!!” on the delta between old and new USHCN file “formats”… it’s not just a format thing…
1934, for example, is 1/2 F colder “now” than it was before…
So much for “raw” data…
So, I ask again: Anyone know how to do a FOIA request for the changes made, code, reasons, emails,…
Sorry, i posted the wrong link in my story.
Good link
http://www.dailypaul.com/node/118748#comment-1280319
E.M.Smith (20:12:58) :
Umm, am I missing something basic? This is an important product, I assume
it’s used in a lot of ongoing research, how could they possibly release
this without a decent explanation of what changed and why. Wasn’t there
even a README file to explain some of that?
Roger Carr (20:48:23) :
Jim (12:58:09):
Wayback machine won’t grab the old data because of the way the website is set up. The data is not embedded in the page, you complete a form that retrieves the data from elsewhere, so there is nothing for wayback to save.
Pull a Nixon: Delete the Decline.
Hiding is for amateurs.
Oh, wait, they basically did….
“So, I ask again: Anyone know how to do a FOIA request for the changes made, code, reasons, emails,…”
Surely someone knows more than I, but this page
http://www.ncdc.noaa.gov/oa/about/ncdccontacts.html
linked to
http://www.rdc.noaa.gov/~foia/index.html
I can see the run-around even now.
Re: E.M.Smith (21:10:33) :
So, I ask again: Anyone know how to do a FOIA request for the changes made, code, reasons, emails,…
http://www.fcc.gov/foia/
David (12:55:39) :
Ok, new version, same data. Why is the data incompatible with the version instead of the other way around? Is there a legit reason pre-1900 data showing more warmth has to be truncated from the record because of the version switch? I don’t get how that is an explanation, I guess.
Answer: It isn’t the same data. See above…
David (12:58:50) : And while I am on this, is there somewhere that one can acquire the station data that includes Tmin and Tmax? All I have found so far is the monthly means. I would like to look at data on a daily basis, but I can’t find it. Does it exist?
Try here:
http://lwf.ncdc.noaa.gov/oa/climate/surfaceinventories.html
It’s the closest I’ve found so far (other than images of printed pages..)
David (18:12:03) :
Isn’t everyone a little tired from all this leaping to conclusions? Mr. Watts has asked an interesting question. I see no factual basis for the conspiracy “answers” on this post other than a deep mistrust. That isn’t nearly enough.
OK, how about the mail where they talk about not releasing data, talk about how to get on IPCC committees and who from NCDC and CRU ought to right which parts of the IPCC report? Oh, and commiserate about the need to suppress FOIA requests. That good enough? It’s only two folks talking, but they reference others. Is 2 enough for a conspiracy?
http://chiefio.wordpress.com/2009/12/11/crut-email-1248902393-txt-ncdc-chums/
they also mention having “raw data” on a server in 2003, but it isn’t completely clear to me if this is the “raw raw data” or merely the “cooked raw data” or the “warmed over cooked raw data” 😉 This being the data from their prior data leak episode.
Anthony,
Am I the only one to spot this?
Is this is meant to be a joke or a climatological station …
…I mean where is the grass covered enclosure in which the Stevenson Screen is meant to be mounted, according to internationally recognized meteorological standards…it’s all outside the very area in which it is supposed to be growing …. Why? … that’s back to front!
…some may argue that we are comparing ‘apples with apples’, since it has always been exposed in this manner – over a shingle bed, coarse gravel/ stones, whatever, with inherent difference in albedo values, (to mention one thing)… from a grass covered surface, which is the ideal and recognized surface, internationally….
….has anyone checked to see how long the surface has been like this….has it been modified over it’s lifetime…( like from grass to shingle/ loose metal)… to say that the station has always been there, and that’s the end of the matter, really is not good enough…
That station is not acceptable in my books, for what its worth!
E.M. Smith,
Regarding 1934, my father will be delighted to hear that the dustbowl wasn’t as bad as he’d thought. Perhaps with further revisions he could avoid living through it entirely! 🙂
Anyway, is there a link between the revisions and the reformatting of data? I can understand why they’d update the format, given the amount of money pouring in to the field from tax dollars that allow a host of long-delayed programming projects to move forward, but I don’t see why that would involve temperature adjustments back to the Great Depression.
On a side note, or perhaps a point in itself, the story of all these unjustified adjustments will come out, whether wholescale or in dribs and drabs, and each will be another example of how the data was manipulated. We’ve all spent years watching every new photo of a glacier, every newly noticed bird behavior, and every polar bear suicide spun as anecdotal proof of man-made global warming.
Now the shoe is on the other foot. Every crazy adjustment made to the historical temperature data will be analyzed, debated, and probably hung like an albatross around somebody’s neck. The press likes a narrative, and though the climate is hard to understand, if not fundamentally unfathomable without gross oversimplification, fraud is simple to grasp. There’s only so many times AGW scientists can go on TV and recite “peer-reviewed literature” before the public notices their uncanny resemblance to tobacco executives. I will bet that by the time a few AGW proponents go on a news show and get unexpectedly grilled like the estranged ex-boyfriend in a triple homicide, this whole house of cards will collapse and we’ll get back to unbiased science.