I’ve been following this issue a few days and looking at a number of stations and had planned to make a detailed post about my findings, but WUWT commenter Steven Douglas posted in comments about this curious change in GISS data recently, and it got picked up by Kate at SDA, which necessitated me commenting on it now. This goes back to the beginning days of surfacestations.org in June 2007 and the second station I surveyed.
Remember Orland? That nicely sited station with a long record?
Note the graph I put in place in June 2007 on that image.
Now look at the graph in a blink comparator showing Orland GISS data plotted in June 2007 and today:
NOTE: on some browsers, the blink may not start automatically – if so, click on the image above to see it
The blink comparator was originally by Steven Douglas. However he made a mistake in the “after” image which I have now corrected.What you see above is a graphical fit via bitmap alignment and scaling of the images to fit. This is why the dots and lines appear slightly smaller in the “after” image. I don’t have the GISS Orland data handy at the moment from 2007, but I did have the GISS station plots from Orland from that time and from the present, downloaded from the GISS website today. If I locate the prior Orland data, I’ll redo the blink comparator.
I believe this blink comparator representation accurately reflects the change in the Orland data, even is the dots and lines aren’t exactly the same thickness.
Douglas writes in his notice to me:
It appears that RAW station plots are no longer available, although NASA GISS (Hansen et al) do not say it in this way. Here is the notice on their site:
Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).
I don’t know about the “renamed” option, but the RAW data appears to be NO LONGER AVAILABLE.
Here’s a detailed blink comparison of Orland. All their options now give you an “adjusted” plot of some kind. The “AFTER” in this graph show the “adjustments” to Orland.
Here is what the GISS data selector looks like now, yellow highlight mine, click to enlarge:
Above clip from: http://data.giss.nasa.gov/gistemp/station_data/
Here is the “raw” GISS data plot of Orland I saved back in 2007:
And here is another blink comparator of Orland raw -vs- homogenized data posted by surfacestations.org volunteer Mike McMillan on 12/29/2008:
And here is the “raw” GISS data for Orland today, please note the vertical scale is now different since the pre-1900 data has been removed, the GISS plotting software autoscales to the most appropriate range:
Source:
And it is not just Orland, I’m seeing this issue at other stations too.
For example Fairmont, CA another well sited station well isolated, and with a long record:
Here is Fairmont “raw” from 11/17/2007:
And here is Fairmont from GISS today:
Source:
This raises a number of questions. for example: Why is data truncated pre-1900? Why did the slope change? The change appears to have been fairly recent, within the last month. I tried to pinpoint it using the “wayback machine” but apparently because this page:
http://data.giss.nasa.gov/gistemp/station_data/
is forms based, the change in this phrase:
Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).
Appears to span the entire “wayback machine” archive, even prior to 2007. If anyone has a screen cap of this page prior to the change or can help pinpoint the date of the change, please let me know.
It is important to note that the issue may not be with GISS, but upstream at GHCN data managed by NCDC/NOAA. Further investigation is needed to found out where the main change has occurred. It appears this is a system wide change.
The timing could not be worse for public confidence in climate data.
I’ll have more on this as we learn more about this data change.
UPDATE1 from comments:
GISS also just started using USHCN_V2 last month. See under “What’s New”:
http://data.giss.nasa.gov/gistemp/graphs/
“Nov. 14, 2009: USHCN_V2 is now used rather than the older version 1. The only visible effect is a slight increase of the US trend after year 2000 due to the fact that NOAA extended the TOBS and other adjustment to those years.
Sep. 11, 2009: NOAA NCDC provided an updated file on Sept. 9 of the GHCN data used in our analysis. The new file has increased data quality checks in the tropics. Beginning Sept. 11 the GISS analysis uses the new NOAA data set. ”
Sponsored IT training links:
Worried about N10-004 exam? Our 640-802 dumps and 70-680 tutorials can provide you real success on time.
I added this over at DP.
“I don’t know? Maybe I can create a bridge between the WUWT blog and the Daily Paul Blog, a Political blog and a Science blog, to bookmark these hreads, and keep the conversation going between these to blogs. The concersation that would come out would be most fascinating.”
I think I just invented “Cross Blog Debate”. This is my idea. Please give me credit for it. Thanks.
wtf (21:55:54) : wrote
“Has anyone else come to the conclusion that from this point out, virtually all past climate data are possibly suspect?”
Oh yes! And now you will see how deep the rabbit hole goes.
Randy (14:12:09) : How many climate scientists rely on this ‘adjusted’ data believing it to be a solid foundation upon which they then do their thing?
As Anthony said, almost everyone. I’ve spent a year or so digging through GIStemp looking for the “magic sauce” and always accepted the data a valid. It was only when I started to “characterize the data” for doing a GIStemp benchmark series that I found “odd trends” in the “raw” data. That lead to what I thought at the time was a boondoggle, but turned out to be a gold mine. That GHCN was fudged, big time.
Since GHCN looks to be the input for the 4 major temperature series (GIStemp, CRUt, NCDC, and the Japanese series) and then the modelers and other investigators take their output… this all comes down to one thing:
Who does what to the really raw data at NCDC as it is turned into GHCN? And why?
It would involve anywhere from 1 to a dozen people. (One manager or key researcher setting a standard, up to a dozen managing parts of the data set process – personal max estimate).
THAT is the Grand Prize in this treasure hunt…
My natural scepticism is firming up daily and amongst the general public I am not alone. However probably the last remaining barrier to total disbelief is that I can’t believe that so many serious scientists are ‘on the take’.
They are not. Most are honest, but duped. Some are willingly duped. Many have simply fallen in love with their theories and are not seeing the warts. And frankly, of the “thousands of scientists”, most of them seem to produce reports of the form “IFF AGW is real, then this bad thing will happen in my field.” I’d put the number of ‘intentional rats’ at about a half dozen max, but in high places. They will likely have a couple of dozen of willing accomplices, but those folks will be a mix of the ‘duped’ and the ‘suspected but the boss said to’ and the ‘Hey, It was a paycheck’ folks.
If however the base science they are relying upon is seriously flawed they I would expect to see an increasing number standing up to be counted.
When the rocks, arrows, and bullets are flying, folks do “duck and cover” not “stand up and be shot”.
Probably a lame question.
Nope. A very fine question. One asked in every fraud investigation. Who is the Alpha, who are the grunts, who are the dupes, who are the marks.
Get your Program, can’t tell the action without Your Program!…
David,
Is this is meant to be a joke or a climatological station …
It’s a climatological station.
A joke involves the 200W aquarium heater they got from Petco to keep the thermometer from freezing, allowing the station to maintain accurate readings in the coldest of cold spells.
😀
Frankly, the weather stations are so wacky that years ago I figured we’d just as well use OnStar satellites to collect surface temperatures from the onboard computers of speeding luxury cars. The system already uses GPS to track car locations, so the only hangup is convincing Lincoln and Lexus drivers to speed through fields in the middle of Iowa at 3:00 AM, and convincing NOAA to quit deleting datasets with the claim that spurious readings came from impacts with sleeping cows.
Thanks. I’ll be here all night (or until Anthony tells me to get serious!). Try the veal. It’s delicious!
Now if we could just get comments to appear on both blogs talking abound the same subject at the same time linking the two in the blobosphere?
Wouldn’t that bee a Hoot.
David (21:21:47) :
Roger Carr (20:48:23) :
Jim (12:58:09):
Wayback machine won’t grab the old data because of the way the website is set up. The data is not embedded in the page, you complete a form that retrieves the data from elsewhere, so there is nothing for wayback to save.
It looks to me like an artifact of the USHCN vs USHCN.v2 change. USHCN data set is date stamped with 2007. Any changes before that are not available, but the USHCN vs USHCN.v2 are available for download from NOAA / NCDC (at least for now… but I have saved copies 😉 See here:
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn
where the old version is still there with 2007 date stamp. And here:
http://www1.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/
for the new one. Oh, and for completion, see here:
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2
if you want the GHCN copy.
So many data copies, so little reality…
Randy (14:12:09) : How many climate scientists rely on this ‘adjusted’ data believing it to be a solid foundation upon which they then do their thing?
As Anthony said, almost everyone. I’ve spent a year or so digging through GIStemp looking for the “magic sauce” and always accepted the data a valid. It was only when I started to “characterize the data” for doing a GIStemp benchmark series that I found “odd trends” in the “raw” data. That lead to what I thought at the time was a boondoggle, but turned out to be a gold mine. That GHCN was fudged, big time.
Since GHCN looks to be the input for the 4 major temperature series (GIStemp, CRUt, NCDC, and the Japanese series) and then the modelers and other investigators take their output… this all comes down to one thing:
Who does what to the really raw data at NCDC as it is turned into GHCN? And why?
It would involve anywhere from 1 to a dozen people. (One manager or key researcher setting a standard, up to a dozen managing parts of the data set process – personal max estimate).
THAT is the Grand Prize in this treasure hunt…
My natural scepticism is firming up daily and amongst the general public I am not alone. However probably the last remaining barrier to total disbelief is that I can’t believe that so many serious scientists are ‘on the take’.
They are not. Most are honest, but duped. Some are willingly duped. Many have simply fallen in love with their theories and are not seeing the warts. And frankly, of the “thousands of scientists”, most of them seem to produce reports of the form “IFF AGW is real, then this bad thing will happen in my field.” I’d put the number of ‘intentional rats’ at about a half dozen max, but in high places. They will likely have a couple of dozen of willing accomplices, but those folks will be a mix of the ‘duped’ and the ‘suspected but the boss said to’ and the ‘Hey, It was a paycheck’ folks.
If however the base science they are relying upon is seriously flawed they I would expect to see an increasing number standing up to be counted.
When the rocks, arrows, and bullets are flying, folks do “duck and cover” not “stand up and be shot”.
Probably a lame question.
Nope. A very fine question. One asked in every fraud investigation. Who is the Alpha, who are the grunts, who are the dupes, who are the marks.
Get your Program, can’t tell the action without Your Program!…
BTW, TWC is calling for 2 to 4 FEET of snow in the Sierra Nevada.
E.M.Smith (21:42:25) :
I am only the first two, and have changed my name to eliminate anyone confusing me (I hope).
I meant they have the same “raw” data. But I am not even sure that is the case. Does the data have its own versions too? That seems very odd.
“Want to Join the Debate” is another invention of mine. If you don’t mind me laying claim to it right here right now.
It is a website that brings all the Blogosphere together.
It connects blogs talking about similar subjects together. Tiers of competence are created, voted on by their peers. May the best blog debate WIN!
Great stuff! But as a scientifically educated simpleton, I’m wondering why they bother adjusting the raw data at all, if all we’re looking for is ‘change over time.’
Adjustments would presumably be in order if we were trying to establish the actual temperatures, but as we’re not, it doesn’t matter if instruments are out of calibration, or anything else, as long as the site hasn’t been moved or the instruments replaced/recalibrated, and even those changes would show up as “step changes,” easily corrected for.
Using only raw data from the earliest records (which can be confirmed, apparently, by newspaper accounts), simply compare them to today’s raw data and a trend (if any) should jump off the page. No doubt different stations will show different trends in both directions, so average them.
The only obvious problem would be UHI effects, so rather than attempt to correct for them, simply exclude stations that are affected. There should still be plenty remaining for the survey to be sufficiently accurate for the argument, or “government work” for that matter. UHI impact change could be corrected for each station only if a record of the times for the changes exists and that seems highly unlikely in most cases. So, blow ’em off.
Similarly, it shouldn’t matter when a station became operational, 1880 or 1950, the trend will show, one way or the other. It might even better if the set of stations showed different start dates as that too might show a trend if there is one. Forget everything that happened in between- that’s weather, not climate.
So, somebody with patience, show me why this wouldn’t work, instead of spending gazillions of dollars arguing over adjustments.
I know my idea hit our moderator with a 2×4. It’s all documented on the DP server.
We can track the comment to the millisecond to see who got there first.
“Want to Join the Debate” is another invention of mine, a web site managing all the debates all over the world. If you don’t mind me laying claim to it right here right now.
George T, odd that your post would be going up as I was writing mine!
The data from the USHCN.v2 I posted above is from the USHCN.v2 Adjusted copy.
I’m looking for where I stuck the “unadjusted” and I’ll post correct data for comparison “raw to raw” once I find it…
Sorry for the error. I’ve just got too many data copies floating around right now…
David (21:21:47) : to Roger Carr (20:48:23) and Jim (12:58:09): Wayback machine won’t grab the old data because of the way the website is set up. …
Thanks, David.
yes, and a group is forming to rebuild and make available an open source database of the “Raw Raw” results.
OK, this is getting annoying. The “raw” data, from which the adjusted is made, does not exist for the earlier period. OK, here we can see at least that the file name, 9641C_200908_raw.avg contains the word “raw” and I’ve got the right data set. But the first year of data is 1903.
What happened to the 1800’s?
So how does this compare with the OLD data from the same period?
Old:
So the first couple of years have more -9999 missing data flags. The annual averages are a bit flakey too, but once it stabilizes, the two annual series again show the old version has been made cooler in the new version.
Year NEW OLD and Delta
—- — — ———
1910 629 63.24 -0.34
1911 613 61.71 -0.41
1912 611 61.52 -0.42
1913 630 63.29 -0.29
1914 625 62.85 -0.35
1915 629 63.20 -0.3
Looking again at the 1930’s
Year NEW OLD and Delta
—- — — ———
1929 628 62.29 0.51
1930 626 62.14 0.46
1931 644 64.01 0.39
1932 633 62.84 0.46
1933 627 62.66 0.04
1934 646 64.61 -0.01
1935 620 62.01 -0.01
1936 644 64.43 -0.03
1937 620 62.04 -0.04
1938 628 62.81 -0.01
1939 643 64.29 0.01
1940 635 63.47 0.03
We have added warming in the first years, but a flat middle.
And the present:
Year NEW OLD and Delta
—- — — ———
2000 622 62.02 0.18
2001 629 62.86 0.04
2002 626 62.51 0.09
2003 631 63.09 0.01
2004 628 62.81 -0.01
2005 627 62.72 -0.02
2006 626 62.54 0.06
2007 628 n/a
2008 631 n/a
There is less adjustment in the unadjusted data in recent years.
Please forgive my earlier posting of the adjusted data labeled as unadjusted.
In my defense, I can only offer that trying to keep strait what adjusted data was adjusted a lot and which unadjusted data was adjusted almost as much vs the raw that that was also adjusted… well, it’s easy to get lost.
At any rate, at least now you can compare “Old ‘unadjusted’ we hope” with “New ‘adjusted a lot'” and with “New ‘unadjusted’ just changed some”…
And we still have the past cooled off relative to the present, though the 1930s have gone nutty in the early years.
And we’re supposed to get excited about things in the 1/10 or 1/100 C place when the “raw” data bounces around in 1/10 to 1/2 C jumps?
Sometime after I’ve gotten a nights sleep, I’ll do a more formal posting on this. Doing it “on the fly and live” is not my favorite way to work…
I think I am beginning to understand how “global warming” has been man “made”. Having once published a peer-reviewed scientific paper I am sickened. This is not science. Scientists DO NOT alter their data and say they just changed its format.
I also understand why they are getting desperate enough to try that.
IF they can ram through the carbon derivatives bubble before the whole world has noticed it has stopped warming, then they can lie and pretend that their scam has saved the world. If not, they are so busted.
After reading this post earlier today, I let it percolate a bit before checking out the info for Orland, CA, at USHCN, NCDC, & GISS.
First, though, I needed to do some adjusting of my own. Got out the pizza data (it’s not delivery) and adjusted it with an extra dose of shredded chesses, additional pepperoni, and a fiery heavy sprinkling of crushed red pepper. Consumption of the data was than further tweaked with a grape product (Sangria).
Now I was ready to peruse the data. Wow! They all have different start dates, the earliest 1895, the latest 1931. And USHCN’s data I found was in °F while the others were in °C. I also note that the graph I got from USHCN had nothing looking like the “missing” data in the earlier GISS plot for Orland (pre-1900); it’s slope is mostly flat.
Trying to better understand this mess, I looked up earlier postings on WUWT for Orland, including Steve’s discussion at:
http://climateaudit.org/2009/06/29/orland-ca-and-the-new-adjustments/
After looking through the mind-numbing jumble of data just for this one supposedly stable station is enough to make one wonder, “Does anyone know what the real temperature is anywhere at anytime? And in a few years, how many times will today’s temperature data have been ‘adjusted’ and by how many agencies?”
While taking a stroll through WUWT’s memory lane (June 2007), I found a couple posts that seem especially relevant now. With daily hits for WUWT now averaging over 200,000, just look at this one about volume experienced back on June 19, 2007:
http://wattsupwiththat.com/2007/06/20/june-19th-a-busy-day-at-watts-up/
Yes, those were the good old days when many posts elicited comments numbering in the single digits!
And finally, after noting all the shenanigans by all these adjustments to data from governmental agencies worldwide, here’s a light fun piece from the archives that we all ought to take a moment to enjoy:
http://wattsupwiththat.com/2007/06/12/new-element-discovered-governmentium/
Now, before I check out for the night, I’m doing one final bit of adjusting, liberally tweaking my coke data with some rum fudging. That ought to keep me out of the loop until noon.
Daniel Ferry (15:03:12) :
Daniel, the technique was not the cause of the error. It was your conversion of Tmin from F to C….. 64.8F is 18.2C, not 23.2C.
This manipulation of truth reminds me of the Secret Gospel of Mark, which reads:
For even if they say something true, still the lover of the truth should not agree with them. For not all true things are truth. One must not value what human opinion considers truth more than the true truth, which is recognized through faith.
Or, in modern terms:
For even if the Sceptics say something true, still the Warmists should not agree with them. For not all true things are truth. One must not value what human opinion considers truth more than the true truth, which is recognized through the new religion.
Spin was not invented in the 21st century.
http://www.historian.net/secmark.htm
Anthony asked
“If anyone has a screen cap of this page prior to the change or can help pinpoint the date of the change, please let me know.”
I think I found the answer.
By using the Wayback Machine and starting at http://www.giss.nasa.gov I was able to navigate to various versions of the form. If the “Last Updated” date on the form is reliable, then it looks like the change you’re talking about was made between 11/24/2000 and 03/20/2001.
Here are links to three versions of the page as archived by the Wayback Machine.
07/18/1999
http://web.archive.org/web/19991022020416/www.giss.nasa.gov/data/update/gistemp/station_data/
11/24/2000
http://web.archive.org/web/20010305024332/www.giss.nasa.gov/data/update/gistemp/station_data/
03/20/2001
http://web.archive.org/web/20010619230553/www.giss.nasa.gov/data/update/gistemp/station_data/
Friends:
wtf (21:55:54) asks:
Has anyone else come to the conclusion that from this point out, virtually all past climate data are possibly suspect?
Well, as my above post (Richard S Courtney (14:07:21) ) shows, 6 years ago I and the other 18 signatories to my paper tried to publish that all past climate data are certaily suspect. But our paper was blocked from publication and my complaint at the blocking is part of the hacked (?) Climate gate emails.
Kevin Kilty (14:57:24) commented on statements from my above post where I said of the Jones et al., GISS and GHCN data sets of mean global tmperature (MGT) time series:
“These teams each provide 95% confidence limits for their results. However, the results of the teams differ by more than double those limits in several years, and the data sets provided by the teams have different trends….”
His comment on those statements said;
“Now why wouldn’t anyone notice allegedly independent estimates of the same quantity that differ by two times their respective 95% confidence intervals? 95% means something specific and to differ by two times such an interval is highly improbable. If Richard is right about this, and I have interpreted what he says correctly, why didn’t more alarm bells go off? This is exactly the type of data consistency issue that eventually deflated the “Palmdale Bulge”.”
At least 19 of us did notice and we ‘heard alarm bells’ but our paper was blocked from publication.
1. I can prove that we submitted the paper for publication.
2. I can prove that Nature rejected it for a silly reason; viz.
“We publish original data and do not publish comparisons of data sets”
3. I can prove that whenever we submitted the paper to a journal one or more of the the Jones et al., GISS and GHCN data sets changed so either
4. The paper was rejected because
(a) it assessed incorrect data
or
(b) we had to withdraw the paper to correct the data it assessed.
But I cannot prove who or what caused this.
pat (17:06:38) makes a comment that goes to the heart of the problem when he says:
“This wholesale substitution of opinion for real data is simply scary.”
Yes, as I said in my above posting;
“It should also be noted that there is no possible calibration for the estimates of MGT. The data sets keep changing for unknown (and unpublished) reasons although there is no obvious reason to change a datum for MGT that is for decades in the past. It seems that the compilers of the data sets adjust their data in attempts to agree with each other.”
In the absence of possibility of calibration what can the data be compaed to except “opinion”?
I could have added that the recent reduced trends in the the Jones et al., GISS and GHCN data sets imply that they are now adjusted in attempt to also agree with the satellite (RSS and UAH) data sets.
E.M.Smith (21:10:33) seems to have understood the importance of my point that said;
“although there is no obvious reason to change a datum for MGT that is for decades in the past”
because he writes:
“Here is a sample of the “old” Orland data from UHSCN:
Not only do we lose 1883 and 1884 in their entirety, be it looks to me like the “new” version has cooled the past.
1934, for example, is 1/2 F colder “now” than it was before…
So, I ask again: Anyone know how to do a FOIA request for the changes made, code, reasons, emails,…”
Adjustments to individual station records are only one of the ways the data have been changed over the years.
So, to put it kindly, it has been known for at least 6 years that the data sets of mean global temperature (MGT) time series are uncalibrated guesswork that have been repeatedly altered for a variety of unknown and unpublished reasons but publication of this knowledge has been prevented until now.
Richard