CEI files suit on GISS regarding FOIA delays

From The American Spectator: CEI Suing NASA Over Climate Stonewall

By on 5.27.10 @ 10:57AM

This morning in U.S. District Court for the District of Columbia, the Competitive Enterprise Institute is filing suit against NASA, calling the erstwhile space agency to account for its nearly three-year stonewall of access to internal documents exposing an abuse of taxpayer funds to advance the global warming agenda.

Gavin Schmidt
Goddard Institute for Space Studies climate scientist Gavin Schmidt. (Image credit: GISS)

Along the way to this point, we have begun revealing how NASA is running a third-party advocacy website out of NASA facilities, at taxpayer expense, to assail “skeptics” and promote the highly suspect basis for a specific policy agenda.

This campaign also helped to elevate the particular fiefdom in question (James Hansen’s Goddard Institute for Space Studies, or GISS) in terms of budget and stature. It has also elevated the scientists involved, professionally, at the expense of the taxpayer they are working to stick with the biggest economic intervention in our history (one I detail in my new book “Power Grab“).

In this process, if only thanks to pressure on NASA after a December 2009 news story about their games, we have already obtained important emails among 2,000 or so pages released. These include an admission to USA Today’s weather editor that NASA GISS is just a modeling office, using the temperature record of …CRU, the ClimateGate outfit. That means their “independent temperature record” is actually a recapitulation of one that …doesn’t exist, but was withdrawn as a result of ClimateGate when the custodians admitted they actually lost all original data.

more at The American Spectator: CEI Suing NASA Over Climate Stonewall

0 0 votes
Article Rating
92 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Jimmy Mac
May 27, 2010 9:51 am

Probably way off topic, but here’s my own take on the field:

May 27, 2010 9:53 am

Chris Horner seems somewhat confused about the nature of GISTemp. It uses GHCN, USHCN, and Antarctic stations for land temps and Reynolds/HadISSTv1 for ocean temps.
HadCRUT, on the other hand, uses its own set of stations (including most stations in GHCN) for land temps, and uses HadSSTv2 for ocean temps.
The insinuation that:

is truly bizarre. Its not to hard to take the various databases used and replicate GISTemp…
http://i81.photobucket.com/albums/j237/hausfath/Picture416.png

James Sexton
May 27, 2010 9:54 am

Rock on CEI!!!!

David, UK
May 27, 2010 9:54 am

These crooks need taking down. And the government whilst you’re at it.

May 27, 2010 9:55 am

Moderator: looks like I broke the html in my prior comment. It should have been:
Chris Horner seems somewhat confused about the nature of GISTemp. It uses GHCN, USHCN, and Antarctic stations for land temps and Reynolds/HadISSTv1 for ocean temps.
HadCRUT, on the other hand, uses its own set of stations (including most stations in GHCN) for land temps, and uses HadSSTv2 for ocean temps.
The insinuation that:
“”their “independent temperature record” is actually a recapitulation of one that …doesn’t exist, but was withdrawn as a result of ClimateGate when the custodians admitted they actually lost all original data.”
is truly bizarre. Its not to hard to take the various databases used and replicate GISTemp: http://i81.photobucket.com/albums/j237/hausfath/Picture416.png

May 27, 2010 9:59 am

I hope there is not a short statute of limitations in the US, as there apparently is in England. It would be a shame to see these folks get off because of their actions.

Henry chance
May 27, 2010 10:09 am

Gavin Schmidt may have some time this weekend. They are very busy blogging. I am sure they have been deleting records and files. Lisa Jackson at the EPA said to not keep correspondence or records.

May 27, 2010 10:19 am

Maybe Zeke that is just the point. Think about it.

Allencic
May 27, 2010 10:25 am

Have some of the principals on the side of AGW been separated at birth? Does AGW climatology have some special appeal for dorky looking bald guys like Hansen, Mann, Romm, and Gavin Schmidt? They all look as if they’re related

Milwaukee Bob
May 27, 2010 10:38 am

Zeke at 9:55 am,
Zeke, woe be it for me to put words in anyone else’s mouth, but I think a more correct way to phrase what he is saying would be:
…..these include an admission that NASA GISS is just a modeling office, and contrary to their assertion of having an “independent temperature record”, we now know they are using the temperature record of CRU, whoes custodians admitted they actually lost all original data, which means the NASA GISS “independent temperature record” actually doesn’t exist.

Honest ABE
May 27, 2010 11:05 am

Gavin Schmidt is such an incredible weasel. That picture does him no justice – you really need to watch video of him to fully observe his snake oil salesmanship.

stan stendera
May 27, 2010 11:09 am

Maybe the Schmidtbird will be forced to sing!!

DesertYote
May 27, 2010 11:15 am

NASA has a lot of explaining to do.
http://climate.nasa.gov/kids/bigQuestions/climateChanging/
Poor Polar Bear.
I don’t know why this stuff is not getting any more attention, but NASA deliberately telling our children lies is a big deal.

Stephen Pruett
May 27, 2010 11:21 am

Zeke,
I think the “insinuation” is based on the fact that it has been popular among Gavin and others to argue that there are many (or at least 4) independent temperature records that all provide similar data. However, recent climategate revelations and related questions in subsequent interviews have revealed that all the major databases include at least some of the same data, and that the claims of independence have, at the very least, been greatly exaggerated.
What is truly bizarre (to borrow your phrase) is that the climate science community seems not to realize that this is a problem, particularly when it seems that the further we dig into the details about these databases, the more reasons we find to doubt their reliability. I have asked on RealClimate and in other places whether the raw data, algorithms by which it has been adjusted, annotated computer programs used to apply the algorithms, and annotations indicating the detailed reasons and justifications for each adjustment are publicly available. No one has indicated that this critical information is available for any of the major databases, and most comments I have seen suggest that it is not only not publicly available, but that it probably cannot be retrieved in an intelligible form at all. Certainly, “Harry’s” efforts to sort out such information from the CRU data, as revealed in a Climategate document, does not inspire confidence in the climate record.
Imagine the FDA considering approval of a drug based on a spreadsheet that did not include the raw data, but data that was normalized and adjusted. What would the FDA response be when told, we cannot determine exactly how the data were adjusted or why, but we are sure the people who did it were trustworthy and qualified? I think we all know what the response would be and should be.
If you can show me that all the information listed above is readily available for any of the major data sets, I will be happy to eat public crow. If not, the whole climate science community needs to wake up and start offering major mea culpas. Until this occurs, my opinion of the standards of scientific rigor in this field and the reliability of its conclusions will remain very, very low.

carrot eater
May 27, 2010 11:23 am

The last paragraph is just weird, and probably indicates that the author is not familiar with the land surface records at all. GISTemp has nothing to do with CRU or its record-keeping. GISS gets most of its land surface data from NOAA, and that’s all available to everybody. And if you really don’t like the numbers NOAA has, you can trace that data back to a step before them, if you’re willing to go to the library and go through the old record archive books, or if you want to sit there looking up CLIMAT reports online for the more recent data.
As Zeke says, you can go to NOAA, get the raw data for yourself, and recreate GISTEMP for yourself.

Mac the Knife
May 27, 2010 11:34 am

God Bless the CEI!
Let the Rule Of Law prevail, across our troubled land and around the globe!
1. The Laws of the United States
2. The Laws of Physics
3. The Laws of Chemistry
4. The Laws of Thermodynamics
5. The Fundamental Laws Of Honest Humanity
– Do not lie.
– Do not cheat.
– Do not steal.
I am revolted beyond sufficiency of words by the profound intellectual dishonesty exhibited by the Global Warming cadres, from lowest dissembling acolyte to high priests of hot air, et.al…..

Enneagram
May 27, 2010 11:39 am

All this is about GLOBAL GOVERNANCE these GISS guys are after, so this is a must read:
http://www.lavoisier.com.au/articles/climate-policy/science-and-policy/ray-evans-quadrant-Copenhagen-march-2010.pdf
The UNFCCC’s COP 15, or Copenhagen as it was generally called, was supposed to
create a post-Kyoto world of decarbonisation (the Kyoto Protocol expires in 2012).
This new world order would be created by means of a binding treaty in which the
nation states which now comprise the world’s polity were to surrender their
sovereignty in all matters involving the use of carbon-based energy. Since civilisation
in the West, since the mid-C19, has been based on the increasing use of energy for
our domestic, industrial and commercial life, and particularly for the transport of
goods and people within states and between them, and since the overwhelming
proportion of this energy comes from burning fossil fuels, notably coal, this new
world order, which we can describe as the Green Empire or Imperium Viridis, would
supplant the nation state as the basis of the world’s polity.

Ed Darrell
May 27, 2010 11:44 am

Who is funding the CEI project for FOIA requests to NASA?
REPLY: I don’t know what the total makeup is, there are a lot of independent donations, I do know that. But I also know it won’t matter what the answer is, as you’ll simply write another hate filled post and blame “deniers” and “big oil”. Your MO precedes you. Blogging on school time and their network today? Tsk.
You can read their about page here http://cei.org/about
-A

May 27, 2010 11:49 am

GISS revised temps.

May 27, 2010 11:56 am

Stephen Pruett,
If Climategate provided a revelation that all land temp reconstructions use GHCN data, well, folks sure must not have been paying much attention for the prior few decades :-p
GISTemp doesn’t exactly try to hide the fact on their website…
“The current analysis uses surface air temperatures measurements from the following data sets: the unadjusted data of the Global Historical Climatology Network (Peterson and Vose, 1997 and 1998), United States Historical Climatology Network (USHCN) data, and SCAR (Scientific Committee on Antarctic Research) data from Antarctic stations. The basic analysis method is described by Hansen et al. (1999), with several modifications described by Hansen et al. (2001) also included.”

P Walker
May 27, 2010 12:06 pm

Ed Darrel – Why don’t you ask CEI ?
REPLY: That’s why I gave him the “About” link. His MO though is to make some comment here, then write about the answer, whatever it is. Ed is a lot like Romm, he usually makes a weekly rant against “deniers”. It’s kind of like Friday afternoons in Tehran, always predictable. -A

bubbagyro
May 27, 2010 12:08 pm

carrot eater says:
May 27, 2010 at 11:23 am
Haha! Yeah, right. Nice try. You find “corrected data” only from any of these charlatans, and numbers that incredibly coincide with each other across agencies around the world. As the old saying goes, “one lies and the next swears to it”. BUT, no algorithms for the corrections, no raw sources, etc. Jones and Mann have said, “the dog ate the data”. That isn’t going to work anymore.
Hence the reason for the FOIs, and the lawsuits, many more I hope, going forward. There is merit in this lawsuit on many grounds, and the judges involved agreed to the merit or they would not allow the case.
More than a civil case, IMHO, these are criminal fraud cases, beyond any shadow of a doubt. Virginia courts, as the first of many states that will eventually jump on the bandwagon (much like the tobacco lawsuits), will soon ferret these weasels out, under criminal and civil penalties.

Anton
May 27, 2010 12:20 pm

Zeke said …
“Chris Horner seems somewhat confused about the nature of GISTemp. It uses GHCN, USHCN, and Antarctic stations for land temps and Reynolds/HadISSTv1 for ocean temps.
“HadCRUT, on the other hand, uses its own set of stations (including most stations in GHCN) for land temps, and uses HadSSTv2 for ocean temps.”
Chris Horner is not giving an opinion, he is referencing a NASA e-mail in which the writer tells a “USA Today’s weather editor that NASA GISS is just a modeling office, using the temperature record of …CRU, the ClimateGate outfit.” This e-mail will, doubtless, be featured in the lawsuit.
When the “experts” at NASA make astonishing blunders, you focus on the person exposing them. Hmmm. Were have we seen this before?
Suck it up. This is just the beginning, so you might as well get used to it. Before this has played out, NASA, NASA GISS, NOAA, CRU, and the rest will be permanently, laughably infamous.

James Sexton
May 27, 2010 12:40 pm

@ carrot eater
You’re almost correct, but NOAA doesn’t get most of it’s temps from CLIMAT, rather, they get it from GHCN. See here, http://www.ncdc.noaa.gov/oa/climate/research/ghcn/ghcngrid.html#development
As I understand it, HadCrut uses GHCN also. While GHCN allows a FTP download, they’ve already mucked with it before one gets to see it. I quote from the page with the FTP download instructions……….”GHCN homogeneity adjusted data was the primary source for developing the gridded fields. In grid boxes without homogeneity adjusted data, GHCN raw data was used to provide additional coverage when possible. “……………………………nice, they used raw data when forced to.
Whether GHCN uses CLIMAT or not, I really couldn’t say, because, well, GHCN didn’t say. I believe the point in the suit is: we were told, once, that various independent data sets all confirmed similar findings. (Dramatic global temp increase) As one digs away, we’re finding that there weren’t so many “independent” data sets, rather, the same temps mucked with in a different manner to get the same results.

Tim Clark
May 27, 2010 12:42 pm

carrot eater says:
May 27, 2010 at 11:23 am
The last paragraph is just weird, and probably indicates that the author is not familiar with the land surface records at all. GISTemp has nothing to do with CRU or its record-keeping. GISS gets most of its land surface data from NOAA, and that’s all available to everybody. And if you really don’t like the numbers NOAA has, you can trace that data back to a step before them, if you’re willing to go to the library and go through the old record archive books, or if you want to sit there looking up CLIMAT reports online for the more recent data.
As Zeke says, you can go to NOAA, get the raw data for yourself, and recreate GISTEMP for yourself.

Which has been done:
http://chiefio.wordpress.com/
and the results are worse than we thought.

Tim Clark
May 27, 2010 12:45 pm

Zeke says: May 27, 2010 at 9:55 am;May 27, 2010 at 11:56 am
Are you missing the point. They both use the GHCN stations, 90% of the data. Then fudge independently. GHCN is not raw data.

P Walker
May 27, 2010 12:45 pm

Anthony – I’d already posted the comment before I noticed that you had provided a link . Sorry . BTW , if it were not for Chris Horner , I wouldn’t have heard of WUWT .

May 27, 2010 12:49 pm

James Sexton,
GHCN raw data is here: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z
It is the same “unadjusted data of the Global Historical Climatology Network” that GISTemp uses as a base for land temps.
CLIMAT is the format in which stations report data to GHCN.
Tim Clark,
I’ve been able to get pretty damn close to replicating both GISTemp land-only and GISTemp land/ocean using the raw land data and the HadISST1/Reynolds ocean data (the ocean data uses interpolation and isn’t “raw” per se): http://i81.photobucket.com/albums/j237/hausfath/Picture416.png
I’m still checking a few more things, but I’ll have a post up at Lucia’s place in a few days describing the steps necessary for the replication and providing the STATA code that does it.
REPLY: Zeke, I question whether that data at the FTP link provided is really “raw”. Since it has V2 attached to it, it implies that it has been processed. Do you have any proof that it is truly raw, and not processed by any adjustment algorithms at NCDC? – Anthony

crosspatch
May 27, 2010 12:59 pm

They might just be playing the averages. La Niña conditions tend to result in active Atlantic hurricane seasons and if this is a fairly strong La Niña, we might have a fairly strong hurricane season.
Seems like a safe bet on NOAA’s part.

Editor
May 27, 2010 1:03 pm

Well it’s about time… I don’t understand how anyone, with any view point, would trust the climate data coming from an organization that is led by the individual who wrote this article:
http://www.guardian.co.uk/commentisfree/2009/feb/15/james-hansen-power-plants-coal
“Coal-fired power stations are death factories. Close them”
“The reason is this – coal is the single greatest threat to civilisation and all life on our planet.”
“The trains carrying coal to power plants are death trains. Coal-fired power plants are factories of death.”
“When I testified against the proposed Kingsnorth power plant, I estimated that in its lifetime it would be responsible for the extermination of about 400 species – its proportionate contribution to the number that would be committed to extinction if carbon dioxide rose another 100 ppm.”
James Hansen is obviously extremely biased, why would anyone expect that the climate reporting from his organization would be objective?
Possibly the only thing that I agree with James Hansen on is this:
“Remember that history, and your children, will judge you.”

TomB
May 27, 2010 1:12 pm

If it is the government’s contention that these scientists were “moonlighting” and thus the records are not the government’s property, then the only reasonable response would be to sue them – personally – as individuals and present a similar document discovery request.

May 27, 2010 1:36 pm

Anthony,
GHCN v2 refers to the collection set, rather than the adjustment. Its slightly confusing since USHCN v2 refers to the adjustment (compared to v1).
Specifically, ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/ contains a number of files:
1) ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z which are raw means from CLIMAT reports (and retrospective collection for earlier records pre-CLIMAT) with basic QC procedures
2) ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.failed.qc.Z which is the data that failed QC
3) ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean_adj.Z which is the adjusted data
GISTemp uses the raw data for STEP0, but they do some adjustments in later steps.
As for how we know v2.mean is the raw data, well, we’ve done a few spot-checks comparing it to filed CLIMAT reports (see the discussion with Gene in the latter part of this comment thread, for example: http://rankexploits.com/musings/2010/comparing-global-landocean-reconstructions/ ). Also, everyone in the literature refers to v2.mean as “unadjusted” or raw. To be perfectly honest, some of the older data might have undergone adjustments by national MET offices prior to being collected by GHCN, though this would mostly affect pre-1960s data when things weren’t as standardized. There is a good discussion with Torn8o and AMac in the discussion thread over here: http://rankexploits.com/musings/2010/another-land-temp-reconstruction-joins-the-fray/
REPLY:Thanks. I’d like to repeat it. It has been my understanding that the RAW file still undergoes some adjustment and infilling, such as TOBS and FILNET, and thus isn’t really raw. Just so I can convince myself, where did you do the CLIMAT spot checks? -A

Enneagram
May 27, 2010 1:39 pm

Don´t ask, don´t tell GISS, they won´t come out from the closet!

James Sexton
May 27, 2010 1:48 pm

@ Zeke
Sigh, I’m used to being ignored, so I won’t take it as a slight. However, I’m surprised you read only part of my post. I’ll repost. From the page that has the link you provided, ”GHCN homogeneity adjusted data was the primary source for developing the gridded fields. In grid boxes without homogeneity adjusted data, GHCN raw data was used to provide additional coverage when possible.“ This implies, that when you download from the FTP site, the data your getting is homogenized “adjusted data”. As Anthony pointed out, the “V2” is a dead giveaway. Unadjusted or raw data doesn’t have different versions.

May 27, 2010 1:56 pm

Anthony,
There are no specific TOBS adjustments or FILNET for GHCN, those are USHCN only. As far as checking how raw GHCN is, find a specific weather record that is adjusted in the non-raw version (say, Central Park in the early part of the century) and check its value in the unadjusted version.
Similarly, you could look at the raw GSN data at http://gosic.org/gcos/GSN-data-access.htm and compare it to v2.mean.
You could also email NCDC and ask them :-p
REPLY: I’ll have a look. I’ve spent a lot of time on USHCN, and its adjustments. Not nearly enough on GHCN’s nuances. Chiefio has been doing all that. -A

carrot eater
May 27, 2010 2:01 pm

You can view CLIMATs here:
http://www.ogimet.com/gclimat.phtml.en
The web page is not really set up for downloading a bunch of data, but you can do spot checks.
The Japanese (JMA) also collect data, and they say they (in post 1990 period) go from CLIMATs directly, thus eliminating the GHCN/NOAA middleman.
http://ds.data.jma.go.jp/tcc/tcc/products/climate/index.html
This seems to be true, as the JMA includes some stations that the GHCN v2.0 does not.
Anthony, for the GHCN, raw means raw. Or at least, it is the monthly averages sent in by the individual countries. There is no such thing as TOB or FILNET for the non-US stations in GHCN, anyway.
TOB and FILNET do however exist for the US stations in the USHCN. But you can easily download the USHCN raw file before those steps. And the adjusted file has flags for values that come from FILNET, so you can take them out if you want.
What GISS reads in for the US has TOB, homogenisation (Menne 2009), and FILNET. GISS then looks for the FILNET flags, and removes most (but not quite all) of the FILNETted values.
What GISS reads in for the non-US is raw, same as what the individual countries send in.

May 27, 2010 2:02 pm

James Sexton,
The passage you are quoting refers to the Global Historical Climatology Network (GHCN) Global Gridded Products, which is indeed not a raw/unadjusted dataset. However, v2.mean is not gridded, nor adjusted. Peterson and Vose (1997) provides a good overview of GHCN v2 (no paywall): http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/images/ghcn_temp_overview.pdf
The NCDC GHCNv2 site is here: http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php

carrot eater
May 27, 2010 2:07 pm

James Sexton
“As Anthony pointed out, the “V2″ is a dead giveaway. Unadjusted or raw data doesn’t have different versions.”
Then you’ll be really confused when GHCN v3.0 comes out later this year (or maybe next year). The raw data in v3 will be the same as the raw data in v2.mean. But there will probably be more of it, as they collect stuff that wasn’t electronically reported as CLIMATs after 1990, and also the adjusted data in v3 will be adjusted using a new algorithm than that in v2.mean.adj.
REPLY: That will be a good test, to see if the data changes. A number of changes between USHCN1 and 2 were noted. Of course NOAA never predicates anything on the raw data, only the adjusted data. Press releases touting hottest/coldest on record globally come from GHCN adjusted. As we’ve seen from Karl’s powerpoint, GHCN V3 adjusted is already warmer and GHCNV2 – Anthony

James Sexton
May 27, 2010 2:25 pm

Zeke
OIC, thanks!

May 27, 2010 2:25 pm

Actually, there is a good testable question there: for stations in USHCN v1 that are also in USHCN v2, was there any change in the data in the raw.avg files?
Bear in mind that USHCN (not talking about GHCN now, which is a completely different beast) provides three levels of data:
1) raw
2) TOBs
3) F52 (TOBs + Menne inhomogeneity adjustments)
Sounds like a new project to work on!
REPLY: Another question to test is why all these adjustments have a net positive bias. Also Zeke, why have you stopped putting Hausfather on your posts here? AFAIK you are the only climate regular that started out posting full name, and then went into the closet. -A

carrot eater
May 27, 2010 2:38 pm

Anthony,
“A number of changes between USHCN1 and 2 were noted.”
There were changes in the *adjusted* data between USHCN v1 and USHCN v2. After all, the adjustment methods were changed considerably. But the underlying raw data would not have changed. In USHCN v2, the raw is quite easy to access – it’s in the file named raw.
“Of course NOAA never predicates anything on the raw data, only the adjusted data.”
They think the adjusted data is closer to the truth; otherwise they wouldn’t bother. But the raw data is still there for you to work with, for both US and GHCN, and people like Zeke have done a lot of study, working exclusively with the raw data. As it happens, for GHCN v2 as a whole, the raw and adjusted are about the same, when it comes to the global average. So globally, adjustments don’t really even matter. But if you take certain regional subsets like the US, then you see some difference between the raw and adjusted.
And there are some papers where they show both the raw and adjusted data.
REPLY: And therein lies the rub, we don’t believe the adjusted data (USHCN1 or 2) is close to the truth. -A

May 27, 2010 2:38 pm

Stephen Pruett: However, recent climategate revelations and related questions in subsequent interviews have revealed that all the major databases include at least some of the same data, and that the claims of independence have, at the very least, been greatly exaggerated.
Climate-gate may have brought this to your intention but it wasn’t ever ‘secret.’ All major global temperature processors use GHCN because the CLIMAT data which GHCN uses to construct its modern temperature record and the review process used in selecting historical data, makes it the highest quality data set available.
However, precisely because of this ‘one source’ issue, I’ve begun translating the SYNOP data set, GSOD, into a format for use by the handful of ‘official’ and numerous ‘independent’ global temp programs.
http://rhinohide.wordpress.com/2010/05/23/gsod-global-surface-summary-of-the-day/

May 27, 2010 2:54 pm

Anthony,
I dropped the last name because its long and unwieldy when I post a lot on a blog (e.g. over at Lucia’s). I’ll make sure to add it in for my future posts here, since I know you dislike folks with strong opinions not going by their full names :-p

carrot eater
May 27, 2010 3:03 pm

“Another question to test is why all these adjustments have a net positive bias. ”
Globally, they do not have much of any bias. Only in some locations like the US, they do.
If a whole bunch of stations change their TOB in a similar way, then you’ll get a bias. That can be quantified, sitting at home. A good bit trickier is the systematic shift from LiG to MMTS units. A whole bunch of stations switched over a certain period, but it’s hard to predict ahead of time, what sort of change that will bring to any given site.
“And therein lies the rub, we don’t believe the adjusted data (USHCN1 or 2) is close to the truth. -A”
Believe what you like, or you can do the math and get into these things. The TOB adjustment is well founded; you can sit there with hourly data and come up with your own TOB adjustment schemes if you like. The rest of the homogenisation in US v2 gets popped out of the pairwise method of Menne 2009. I haven’t had the chance to really study that yet, so I can’t comment on how good or bad it is.
But in any case, the raw data are there for you to play with, and Zeke and others have done a great deal using the raw data.

AlansPower
May 27, 2010 4:03 pm

People this is urgent : http://climate.nasa.gov/kids/bigQuestions/climateChanging/
We know they set a “fail-safe” for the future of their plain (we have a man into Al gore’s group now-thx to former BofA dummy).
So they decieded to CHANGE kid’s mind. They realize nobody would pay attention now.
Then 10 years from now they would try lying again and everybody would agree with them bcuz they’ve grown believing in that.
I’m inviting everyone/anyone to fight for our kid’s future.
Let’s build another website exactly like the Nasa developed to lie to our kids, and make it a VIRAL through out the internet and show FOR KIDS, with kids language, that NASA’s website is a LIE.
How about that? Who’s with me?
alanspower@gmail.com
THIS IS URGENT PLS!

Paul Jackson
May 27, 2010 4:33 pm

from v2.temperature.readme

This is a very brief description of GHCN version 2 temperature data and
metadata (inventory) files, providing details, such as formats, not
available in http://www.ncdc.noaa.gov/ghcn/ghcn.html. …
The three raw data files are:
v2.mean
v2.max
v2.min
The versions of these data sets that have data which we adjusted
to account for various non-climatic inhomogeneities are:
v2.mean.adj
v2.max.adj
v2.min.adj

TOBS occures in the daily files from daily_readme.txt

README FILE FOR DAILY GLOBAL HISTORICAL CLIMATOLOGY NETWORK (GHCN-DAILY) …
These variables have the following definitions:
ID is the station identification code. Please see “ghcnd-stations.txt”
for a complete list of stations and their metadata.
YEAR is the year of the record.
MONTH is the month of the record.
ELEMENT is the element type. There are five core elements as well as a number
of addition elements.
The five core elements are:
PRCP = Precipitation (tenths of mm)
SNOW = Snowfall (mm)
SNWD = Snow depth (mm)
TMAX = Maximum temperature (tenths of degrees C)
TMIN = Minimum temperature (tenths of degrees C)
The other elements are: …
TOBS = temperature at the time of observation (tenths of degrees C) …

I’ve never seen a TOBS in a daily file, but I haven’t looked real hard, just getting a directory listing of the .dly files bring my computer to it’s knees. I’ve been trying to load all of it into a postgresql database, but it’s a slow-go, the data doesn’t seem to rigorously follow the published formats between different file types. Another thing we need to watch out for is I’m not sure what the definition of a mean temperature actually is or if it’s consistent between different sources. The most common seems to be halfway between Tmin and Tmax which is temporally erratic and I’d think statistically indefensible

Gail Combs
May 27, 2010 4:36 pm

Enneagram says:
May 27, 2010 at 11:39 am
All this is about GLOBAL GOVERNANCE these GISS guys are after….
_________________________________________________________________________
Correct
Millinium Project:
“ The Global Monitoring for Environment and Security (GMES) was launched in May 1998 and adopted in June and November 2001 by ESA and the EU Councils, respectively. It is an initiative to promote sustainable development and global governance through the supporting of environmental and security policies…”
The following is from the Clinton era: (The 25×25 Initiative is sponsored by the Energy Future Coalition, a project of the UN Foundation)
“House Concurrent Resolution 25
“The official title of the resolution [H. Con. Res. 25] as introduced is: “Expressing the sense of Congress that it is the goal of the United States that, not later than January 1, 2025, the agricultural, forestry, and working land of the United States should provide from renewable resources not less than 25 percent of the total energy consumed in the United States and continue to produce safe, abundant, and affordable food, feed, and fiber.”
WHY 25X25 IS GOOD FOR YOU”
“American’s farms, ranches and forests – our working lands – are well positioned to make significant contributions to the development and implementation of new energy solutions. Long known and respected for their contributions to providing the nation’s food and fiber, an emerging opportunity exists for crop, livestock and grass and horticultural producers, as well as forest land owners, to become major producers of another essential commodity – energy.”
And yes the “working land” this is talking about is private property. This is why the USDA has tried to shove Premises ID down the throats of US farmers for the last several years.
Originally written to prevent government from trespassing on the people’s right to contract, the Constitution states in Article I, Section 10, Clause 1, that
“No state shall … pass any … law impairing the obligation of contracts, …” It is this constitutional provision that allows the Federal government to implement Federal programs by using so-called “Cooperative Agreements” (basically, a certain type of contract) in lieu of legislation. Commencing in the late-1950s, the Federal government began to contract with other jurisdictions to implement Federal programs where Congress does not have legislative authority.
In 1976, the U.S. government signed a UN document that declared:
Land … cannot be treated as an ordinary asset controlled by individuals and subject to the pressures and inefficiencies of the market. Private land ownership is also a principal instrument of accumulation and concentration of wealth and therefore contributes to social injustice;
D-1. Government must control the use of land to achieve equitable distribution of resources;
D-2. Control land use through zoning and land-use planning;
D-3. Excessive profits from land use must be recaptured by government;
D-4. Public ownership of land should be used to exercise urban and rural land reform;
D-5. Owner rights should be separated from development rights, which should be held by a public authority.
This document was signed on behalf of the U.S. by Carla A. Hills, then secretary of housing and urban development, and William K. Reilly, then head of the Conservation Fund, who later became the administrator of the EPA.
Land-use controls found their way into the 1987 report of the World Commission on Environment and Development, “Our Common Future,” which first defined the term “sustainable development.” The meaning of sustainable development here defined was codified in another U.N. document called “Agenda 21,” which was signed by President George H.W. Bush in 1992. This document recommended that every nation create a national sustainable development initiative.
“On June 8, 2007, Under-Secretary of Agriculture Bruce Knight, speaking at the World Pork Expo in Des Moines, Iowa, said, “We have to live by the same international rules we’re expecting other people to do.” He is referring to the International Criminal Court.
” The ICC is in part modeled on the Vienna Diplomatic Relations Conventions text where [premises] is defined globally and with a global use intended with no recognition afforded to the rights of private individuals, national laws or protections, or the rights or recognition to private property ownership.” http://nonais.org/2009/01/16/bulletin-board-200901/
This is why farmers who have educated themselves are very angry.
And finally President Clinton took the UN NGOs a step further. By Presidential Executive Order the USA was divided into ten regions. These regions are governed by an unholy mix of unelected government bureaucrats and NGOs. The regions were set up by President Nixon but implementing “regional governance began in earnest with the Clinton-Gore administration. “On the heels of the President’s Council on Sustainable Development , came the President’s Community Empowerment Board, chaired by Vice President Al Gore,” http://www.rense.com/general63/ree.htm
These quasi-governmental regional authorities are slowly transforming the US from representative government to government by United Nations sponsored and directed NGOs and appointed bureaucrats.

kadaka (KD Knoebel)
May 27, 2010 5:17 pm

Is carrot eater still saying disgustingly insulting vile things about Anthony, the Chefio, and others over at Tamino’s site, as I sadly personally witnessed a few months back? Now Gavin gets mentioned, and he/she/it promptly pops up for polite discussion.
carrot eater’s message is always the same, GISS is infallible and fully accurate, Hansen is God, and Tamino is a perfect genius. All attempts at “discussion” end with ‘You lack the wisdom and education to understand the great brilliance of Hansen and his work!’
Which is two major reasons right there why I don’t give a flying SNIP about whatever he/she/it has come here to say.

Dave McK
May 27, 2010 5:38 pm

@ Paul Jackson
What to do is plot only one datum from each day taken at the same time of day relative to solar noon.
There is no homogenization, averaging or whatnot and it shows a proper sample, apples to apples.

Van Grungy
May 27, 2010 5:54 pm

Gail,
I do believe that the American Constitution and the Bill of Rights are the most hated documents ever created.
God bless Americans.

E.M.Smith
Editor
May 27, 2010 6:04 pm

Zeke says: HadCRUT, on the other hand, uses its own set of stations (including most stations in GHCN) for land temps, and uses HadSSTv2 for ocean temps.
The CRU crew have said their data is 90%+ identical with GHCN. GHCN is the bulk of all surface data in GIStemp (the USHCN is only used for the USA land area, and even that was left out from 2007 to November of 2009, then put back in after being recooked into a different version…). The sea temps are irrelevant as all they do is slather on a coating of the Hadley/CRU work (that we’ve already seen is broken).
So at the end of the day to say that GIStemp uses the same input at CRU is pretty much correct. The minor additions of some Antarctic data and the remixing of the US stations in GHCN with the US stations in USHCN don’t amount to a hill of beans. That’s why they could point at how much the ‘agree’ with each other…. Same “stuff” in, very similar “stuff” out.
Basically, GIStemp and HadCRUT differ in that GIStemp re-mixes the USA stations and HadCRUT re-mixes the UK stations (and maybe a few others) and both are substantially based on the GHCN that is based on ???… a hand picked set of biased records taylor made to show warming via locational bias and station changes over time. Baseline cherry picked to the cold side of the PDO cycle (so at most you can get back to ‘normal’ after 60 years, and no colder) and with colder stations in the baseline cold period (when their cold anomalies can be locked in) then replaced with warmer stations later (so no such cold anomaly can ever be found again… And no, GIStemp does NOT do anomaly of a thermometer compared to it’s own history. It computes anomalies based on one bucket of thermometers in the past compared to a different bucket in the present. Not really an “anomaly” in the proper sense at all. More like saying my car is getting newer over time because I owned an old one in high school and have a new one now…)
Basically, NOAA / NCDC, HadCRUT, and GIStemp all depend on the same biased and re-cooked data that is in GHCN. There is NO independence between them.

E.M.Smith
Editor
May 27, 2010 6:12 pm

carrot eater says: As Zeke says, you can go to NOAA, get the raw data for yourself, and recreate GISTEMP for yourself.
That is flat out WRONG. (at best… I’m being polite.)
At NOAA you can get highly PROCESSED data food product, NOT raw data.
Their QA process changes the “data”. Oh, and notice that the “monthly means” used by GIStemp are a computed product, so by definition are not “raw”.
Then what GIStemp does to it is truly bizarre. I suppose one could recreate that, but it’s a piece of work.
BTW, NOAA / NCDC recognize this when they name the GHCN data “Unadjusted” rather than “raw”. It includes only some of their total cookage, not all of it, so the “Adjusted” version is even more out of touch with reality. But it is very important to remember that “UN-adjusted” is still adjusted, just not fully adjusted…

E.M.Smith
Editor
May 27, 2010 6:29 pm

Zeke says: As far as checking how raw GHCN is, find a specific weather record that is adjusted in the non-raw version (say, Central Park in the early part of the century) and check its value in the unadjusted version.
That won’t work as a cross check. Most of the cooking comes in about 1990 or so in the “duplicate number three” batch of processing changes. So you need to compare “duplicate number three” with “really raw” to find the changes. (Note that for most stations, it’s the “3” mod flag, but for some with more changes, it can have a higher number.) I suspect it is an artificial “replace extreme values with ASOS / AWS compliant values” as is used on USHCN but have not had the time to chase it down to final details.
(In the USHCN QA process, any record that isn’t liked is tossed and replaced with an AVERAGE based on a collection of local ASOS / AWS stations (read airports). That is guaranteed to suppress low excursions. The GHCN data shows an artifact consistent with that type of processing.)
So, want to really help move truth along? Publish a link to a detailed description of the change in processing that began in the 1988-1993 range as the “duplicate number three” records began replacing the “duplicate number 0, 1, and 2” flags on new data.
I’ll take the resounding silence as confirmation that THAT is the key …

Doug in Seattle
May 27, 2010 7:12 pm

Chris:
Thanks for all your hard work and thanks to CEI for its steadfastness in supporting you. One suggestion however – lose the smirk when giving interviews. It does not make you more believable – just snarky.

May 27, 2010 7:16 pm

Doug in Seattle,
To put it in perspective, take another look at Gavin in the article.☺

Stephen Pruett
May 27, 2010 8:21 pm

Zeke and Ron,
Thanks for the response and information. Ron is exactly right that I first became aware of any of this after climategate, so I have much to learn.
However, I have seen a number of statements from various sources (both science and media) indicating that there are multiple independent climate records, so we don’t need to worry if CRU’s data are suspect. Maybe that is the case, and I am glad you have patiently explained the fine points of these data sets, but I am not reassured when a well informed person such as yourself still must go through gymnastics and spot checks to know whether the data is raw or not.
Furthermore, when it is not raw, are the actual programs used to adjust the data and the reasons some data were adjusted but not not other data, available anywhere? I know the basic methods used to backfill grids, etc. have been published, but programs that do these things can have mistakes, and if the programs are not available, it is impossible to determine if there were mistakes. Also, without an explanation/justification for each data adjustment, we are really just taking someone’s word that it was done properly. It’s not that I think climate scientists are lying scoundrels, but this is just a poor way to operate. As I mentioned before, you couldn’t get a drug approved based on data that you state were adjusted but for which you can’t provide documentation and justification of the adjustment.
In most biological studies with large data sets (e.g., genomics and proteomics), any custom programs used to analyze them are provided as part of publications (usually as online supplements) and raw data are deposited in databases, so everything done can be reviewed, analyzed, and replicated. I had assumed climate data were handled similarly, so I accepted the alarming predictions of warming. I don’t anymore, and it’s not just because of the data issues, but that’s a topic for another post.
Anyway, I hope I have made my perspective clearer, and if I am wrong and there are detailed explanations of the adjustments of the major data sets readily available, my concerns about the reliability of the data would be eased considerably. However, if this information is not readily available, I would be interested to know whether you see this as a problem.

May 27, 2010 8:41 pm

The difference between the datasets of the Hadley Centre and the Global Historical Climatology Network is the fact that Hadley maintained their climatology long before global warming was a hype, whereas GHCN was in their own wordsdesigned to be used to monitor and detect climate change. GISS remodels and gets even more warming, of course. When are they going to reconcile these surface models with the satellite measurements which show indicate less warming over the continents?

sdcougar
May 27, 2010 9:09 pm

” DesertYote says:
May 27, 2010 at 11:15 am
NASA has a lot of explaining to do.
http://climate.nasa.gov/kids/bigQuestions/climateChanging/
Poor Polar Bear.
I don’t know why this stuff is not getting any more attention, but NASA deliberately telling our children lies is a big deal.”
–I posted comments about the nasa/kids propaganda last winter on several sites, but there seemed to be no interest. I guess ’cause it’s just for kids [but there is also an emphasis for the educators who are ‘educating our kids]
I think its gone now, but last winter one of the questions for kids was “Who is helping? People who live in little houses.”
I sent them feedback asking, “You mean people like Al Gore and James Hansen???”

May 27, 2010 9:12 pm

Gail Combs, May 27, 2010 at 4:36 pm :

And finally President Clinton took the UN NGOs a step further. By Presidential Executive Order the USA was divided into ten regions. … http://www.rense.com/general63/ree.htm

A couple observations and maybe a question or two:
a) Sourcing things from Jeff Renses’s (a former Alex Jones associate) website (a veritable ‘conspiracy central’ site less than a decade ago) these days?
b) Can you make any references besides “House Concurrent Resolution 25” – to any US Codes or Code of Federal Regulation (CFRs)?
c) Where is the ‘regional’ office for the UN that oversees Texas? I’d like to pay them a visit …
These should be easy questions to answer if your posting isn’t the product of populist paranoic vamping.
(I see you even got a mention/reference to the anti-National Animal ID tagging website in there too; way to go girl!!)
.

May 27, 2010 9:30 pm

I’ll take the resounding silence as confirmation that THAT is the key …
That is your MO, isn’t it? Whenever you bump into something that you don’t understand and don’t desire to learn, you use THAT something as the key to … to what? Afraid to say it? Prefer to rely on vague claims and insinuations?
Anyway, I did the leg work you so clearly chose not to do. Raise issues, claim misconduct, and walk away. Its so terribly predictable.
————————–
Publish a link to a detailed description of the change in processing that began in the 1988-1993 range as the “duplicate number three” records began replacing the “duplicate number 0, 1, and 2? flags on new data
As you know, when multiple data sources are available for a single station in GHCN, they are distinguished by separate record numbers. So the appearance of a new record number indicates a new data source.
There are roughly 1000 stations with a ‘record 3’. Lets see what the first year for ‘record 3’ is for various stations:
1951; 70 stations with a new record 3
1961: 190 stations with a new record 3
1971: 115 stations with a new record 3
1981: nope! pattern is broken. no new record 3
1987: 320 stations with a new record 3
1991: 17 stations with a new record 3
2000: 126 stations with a new record 3
full list here
So how is it that ‘new data sources’ are lining up at the beginning of decades (for 1951,1961,1971, 1991?, 2000). What happened to the 1981 update?

NCDC undertakes activities to rescue and digitize historical climate records, in order to extend available time series for surface air temperature and precipitation as a part of its Global Historical Climatology Network (GHCN) dataset. This has involved digitizing station history files, old books, maps and paper records. Currently, some historical GSN data for 407 out of the 981 GSN stations worldwide have been made available in response to requests from WMO. Historical data for another 364 GSN stations are available to NCDC from other archival sources.
Figure displays the total number of GSN stations in comparison with the total amount of GSN stations for which data are actually available from the GHCN dataset, for mean surface air temperature. Curves for precipitation are similar in shape. The total number of stations belonging to the GSN increased steadily until 1960, when the growth in number reached a plateau near today’s 981 stations. Monthly and daily data holdings increased similarly, with monthly data generally outnumbering daily data until the past decade. The dip in monthly data around 1990 is due to delays in updating the GHCN, which depend on retroactive data compilations, such as the World Weather Records, which are processed in decadal steps (the last one 1991-2000). The decrease in monthly data around 1970 is mainly due to the suspension of extensive efforts to digitize historical data at that time. The recovery of monthly data in the mid-1990s can be attributed to facilitated data exchange, following the WMO initiative of CLIMAT (WMO, 1995)1 data transmission over the GTS.

http://www.wmo.ch/pages/prog/www/CBS/Meetings/MG_5/Doc-3-1(3).doc (DOC)
Your continued alarmism is noted.

Van Grungy
May 27, 2010 9:37 pm
May 27, 2010 9:57 pm

Anyway, I hope I have made my perspective clearer, and if I am wrong and there are detailed explanations of the adjustments of the major data sets readily available, my concerns about the reliability of the data would be eased considerably. However, if this information is not readily available, I would be interested to know whether you see this as a problem.
There are few different methods of ‘replication’ in science.
Method 1: exact data and exact methods (equations, algorithms, code)
This is an audit. Nothing new is generated.
Method 2: exact data and independent methods
Method 3: independent data and exact methods
Method 2 is what a bunch of different blog guys are doing now by feeding GHCN unadj into their own global gridders and anomalizers. I’m working on Method 3.
Method 4: independent data and independent methods
I’m no historian of science, but this the way that I was taught that the physicals sciences have usually operated. And if a previous result couldn’t bolster an earlier finding – well that causes a ‘conflict’ which is resolved by further research, new approaches.
So to answer your question, I’m not particularly concerned by not achieving ‘Confirmation Method 1’ for historical weather data. While it may be technically possible to gather all the physical records, the paper records kept in libraries and shoe-boxes (some of which may have been destroyed over the years), the chance of gathering the exact same set of paper records and extracting the exact same set of data is exceedingly small.
So we can confirm GISTEMP and CRUTEM by feeding their data sources into independently constructed programs. And we use Method 3 (go to other similar data sources and run them through the public code for GISTEMP or CRUTEM). And we can use Method 4 and look at independent temperature records such as satellite records and paleo-proxies. And then we weight the evidence to see if GHCN+GISTEMP/CRUTEM should be considered confirmed or rejected.
Or we can rely on CEI to spin a politically-driven narrative.
I know which methods I prefer.
And GHCN is documented better than the WUWT experts let on:
Peterson and Vose, 1997 An Overview of the Global Historical Climatology Network Temperature Database
Peterson, T.C., R. Vose, R. Schmoyer, and V. Razuvaev, 1998: Global Historical Climatology Network (GHCN) quality control of monthly temperature data. International Journal of Climatology, 18 (11), 1169-1179. (PDF Version)

SS
May 27, 2010 10:53 pm

No self respecting man wears a scarf.

May 27, 2010 11:11 pm

I hope they win the lawsuit.
Frankly I would just be happy if someone would do something to tape Hansen’s mouth to truly enforce his gag order that he’s been disobeying for the last 8 years.
The man constantly proves the old Chinese proverb…
Tis better to keep your mouth closed, and let people think you are a fool…. than to open your mouth and remove all doubt.

Pete Hayes
May 27, 2010 11:29 pm

I see our friend Ed took his finance question/moan over to the American Spectator.
Ed, who is financing RealClimate.org? Who is financing Gavins time on the site? Why did they drop the time/date stamps?
Its a pointless, shrill attempt to try to prove the sceptical side is financed by “Big Money”. Sheesh! They say we sceptics follow conspiracy theory!
Make sure mine is in a brown envelope please Anthony 😉

May 27, 2010 11:30 pm

I love it when People like Zeke and Carrot Eater swear by Jonesies temp data.
The man has as much as said he has crossed the line of his duty to faithfully keep accurate records and has changed from a friendly fellow scientist to someone who says now, Hell no I won’t give you the information you want, you’ll just used it against me… Damn the torpedos and the Freedom of information law (in Britain and the US mind you) and while I’m at it Damn You, and good day Sir. (emphasis and choice of words mine but it pretty much paints the picture correctly)
If you people really want to believe in the word of someone like Phile Jones and you really believe that there was a fair hearing in Britain that found him innocent of all mis-deeds… I have some guaranteed winning lottery tickets and ocean front property in Arizona with your name on it.
Phil Jones is exhonerated in Britain and Michael Mann is exhonerated by Penn State, there hasn’t been this much white paint going on fences since Ralph Maccio starred in the Karate Kid. Aiiii Painta Fence Daniel-San

John Murphy
May 27, 2010 11:33 pm

Zeke
I always love those graphs. 0.6C in 130 years. No wonder I feel like a boiled egg!

May 27, 2010 11:39 pm

By the way words of wisdom to people like Zeke and Carrot Eater… show a little respect to Mr. Watts and Mr. Sexton, these guys have more knowledge, wisdom, and class than most people in the AGW arena.
Don’t believe their side if you chose to blindly follow Phil Jones but unlike Jones, Mann et al. their science is not blind nor is it lacking a great deal of rock solid evidence behind it. That is your right. You two guys aren’t obnoxious like some but you seem to give credence and dead sea scroll truth to the words of James Hansen.
I’m conservative but in my opinion one of the few things Bush Jr. did before he handed over the keys to Obama was slap a gag order on one Mr. Hansen. The GISS has constantly mishandled data, published un-verified data and Mr. Hansen continues to violate a Presidential Gag Order which has not been lifted I might remind you.

May 27, 2010 11:42 pm

regarding Alan Power’s comments.
Yes last year Mr. Gore spoke before dozens of high schools.
He told them on numerous occasions throughout his cultic speach… “Your parents aren’t as wise as you, they don’t know about this stuff.” Go home and tell your parents they are dumb when it comes to global warming and demand they get with it.
I wonder how he would like it if I went to his son that he mentions in his Global warming rhetoric, and told him. He Gore Jr. Your father is an idiot and doesn’t know what he’s talking about, tell him to put his head back on his shoulders and get his facts straight?

May 27, 2010 11:43 pm

Not that I would ever speak ill of the former future president.
All hail Al Gore, the inventor of the internet.

carrot eater
May 27, 2010 11:43 pm

Stephen Pruett
“Furthermore, when it is not raw, are the actual programs used to adjust the data and the reasons some data were adjusted but not not other data, available anywhere?”
Releasing programs that recreate the entire thing is only possible when the process is 100% automated. This is the case for GISS, and their code is available on their website (http://data.giss.nasa.gov/gistemp/sources/), and it’s been picked over and re-written for clarity here: http://clearclimatecode.org/about/
For the USHCN v2.0 (which is the current version), the code is also available, and it should be available for GHCN v3.0 when that comes out.
“and if the programs are not available, it is impossible to determine if there were mistakes.”
This isn’t at all true. You can figure out that somebody made a mistake without looking at their code. You do it by independently doing your own calculations, and seeing that your results don’t match. You can try to implement their methods (which are described in detail in a bunch of papers) or come up with your own methods. Both have been done in the past months by several people.

carrot eater
May 28, 2010 12:08 am

EM Smith
“At NOAA you can get highly PROCESSED data food product, NOT raw data.
Their QA process changes the “data”. Oh, and notice that the “monthly means” used by GIStemp are a computed product, so by definition are not “raw”.”
The GHCN QC process only changes the data in that outlier months are tossed out and moved into the QC file, where you can see exactly what’s been tossed out. They are not replaced with anything but the missing data flag.
But yes, monthly means are the result of somebody taking an average, so they’re only raw inasmuch as an average is raw.
“Then what GIStemp does to it is truly bizarre. I suppose one could recreate that, but it’s a piece of work.”
It’s been recreated, both by following the exact same method, as well as making various changes to the method. As for whether it is “truly bizarre”: I think you still might not understand the reference station method, as you’ve gotten it quite very wrong when you’ve worked out toy problems.

May 28, 2010 12:50 am

Pete,
pffffft. According to James Hoggan and others at desmogblog.com everyone who speaks out against the warmists are millionares sucking up funding from Exxon or the tobacco companies.
I wish that were the case…. I’m a poor 50 year old student who’s so broke I’m living in my mom’s house with my mom and my wife and kid and 7 cats. (not because i’m a loser but because i spent over 50k to move from Washington state for a job in Alaska then got laid off 6 months later because of our lovely economy, then had to spend another 7 k to move back to Washington and been unemployed going on 2 years now and I’m going to school). While Phil Jones and Michael Mann still get money from the US Government and the IPCC and other agencies. Let’s get serious, the US gov’t alone has spent over 1.8 trillion with a “t” dollars in the last 5 years on global warming studies. Where is the real conflict of interest here folks?
Anyone on the warmist side is welcome to check my checking account statement.
Granted I’m a nobody in the fight against Darth Gore and the global warming dark side but… skeptic I am none the less.

Matt
May 28, 2010 3:07 am

To answer the earlier question about CEI funding – here’s a document listing their larger sponsors: http://legacy.library.ucsf.edu/cgi/getdoc?tid=jot72e00&fmt=pdf&ref=results
Here’s a short list:
Amoco Foundation, Inc.
Coca-Cola Company.
CSX Corporation,
FMC Foundation
Ford Motor Company Fund
Philip Morris Companies, Inc.
Pfizer Inc.
Precision Valve Corporation
Texaco, Inc.
Texaco Foundation
Is this a case of follow the money?

Capn Jack
May 28, 2010 3:09 am

Poor Old Gav I met him in a knife fite once. Way back.
Hands in the till on the Company time, come November he is toast.
Naughty Naughty.
He is one nasty piece of work in my honest opinion. Can’t fite he will roll.

Ryan
May 28, 2010 3:34 am

For the last time data from 1885 given in ASCII format is CLEARLY NOT RAW DATA! They didn’t have computers in 1885 so the raw data can only be in hand-written tables of temperature readings. Scanned versions of these could be considered raw data, but not the ASCII tables of the same. You cannot do any kind of sensible spot-check of the ASCII data provided unless you at least have some access to the hand-written temperature readings.
If you look at the ASCII data alleged to be raw you will, however, learn all kinds of things about that “raw” data which you would find disconcerting, such as:-
1] Much of the data obtained from outside the developed world is so patchy with so much missing data that no sensible attempt could be used to extract a trend from it.
2] The missing data is simply annoted “-9999”. A useful indication of why it is missing is not given.
3] For stations in the developed world, much if the missing data is actually from the last decade (e.g. that old favourite Darwin Australia)
4] The data is quoted allegedly accurate to 1/10th Celsius. This is despite the fact that max/min thermometers are only marked in gradations of 0.5Celsius normally and even the best modern mercury thermometers meeting DIN58654 only have graduations at 0.2Celsius intervals. It seems highly unlikely that data collected 100years ago in the more remote parts of the world can be quoted to anything like 0.1Celsius, given that it is not necessary for weather monitoring (you would need to be able to read a column of mercury accurate to within 0.1mm – that’s quite a talent). More than likely many of these readings are conversions from Fahrenheit to Celsius with the decimal point occuring as a result of long-division. Without the real RAW hand-written data it is difficult to check this.
5] The annual averages take these figures and then average them out over the whole year, which smooths out all the variation in temperatures in a quite unreasonable way. It would be preferable to take the temperatures from, say, the 1st July in the northern hemisphere and 1st Jan in the southern hemisphere (i.e. summer temps when CAGW is allegedly at its worst) to present the annual data with the underlying natural variability still intact.
Ryan.

MostlyHarmless
May 28, 2010 4:07 am

The full text of the CEI complaint can be found at:
http://legaltimes.typepad.com/files/nasa-complaint.pdf
It’s well worth a read.

Capn Jack
May 28, 2010 5:28 am

The requests are numerous, the statute mostly Presidential FOI order, which has no weight in state Jurisdictions.
NYC has the worst record for FOI in the US. That is why legally Nasa’s Real Climate blog site is based there, NASA owns the Real Climate site in that it’s employees operated it on Government time, 24 hours daily.
NYC has the worst law for RICO and also has the worst Law for Fraud against Government

Capn Jack
May 28, 2010 5:36 am

Oops Gav of NASA
May I recommend the truth,

You hit a mate of mine hard, you put me mate to the [snip] wall, lost everything.
Gav, toast.

Steve Keohane
May 28, 2010 6:53 am

E.M.Smith says: May 27, 2010 at 6:29 pm
So, want to really help move truth along? Publish a link to a detailed description of the change in processing that began in the 1988-1993 range as the “duplicate number three” records began replacing the “duplicate number 0, 1, and 2″ flags on new data.

That time range is interesting, as it seems to coincide with the introduction of the MMTS sensors, and the UHI they caused. What are your thoughts on the #3 records, have you covered this on your site? I should have time to swing by this weekend.

May 28, 2010 7:33 am

That time range is interesting, as it seems to coincide with the introduction of the MMTS sensors, and the UHI they caused.
MMTS caused UHI?

From the very first year when the MMTS was
installed at the Fort Collins weather station back in 1984, MMTS has consistently measured lower daily maximum temperatures with the largest
differences occurring in winter.
Daily minimum temperatures showed very small differences but with a consistent seasonal cycle. The patterns that were first observed in 1984 and 1985 continue to be repeated each year.

http://ams.confex.com/ams/pdfpapers/91613.pdf
REPLY: True, UHI isn’t caused by MMTS, he made a boo boo in phrasing. OTOH, since you brought it up I’ve got a paper in the works that shows the opposite, comparing CRS LIG thermomters with MMTS. -A

May 28, 2010 7:41 am

EM Smith,
While the number of duplicate records does increase over time (and, interestingly enough, not all duplicates have the exact same data; some have gaps and others do now), your focus on duplicate 3 is somewhat misplaced.
Percent of GHCN records with dup3 by year: http://i81.photobucket.com/albums/j237/hausfath/Picture421.png
Temperature reconstruction with and without dup3 records: http://i81.photobucket.com/albums/j237/hausfath/Picture422.png
Ironically, excluding dup3 slightly increases the trend in recent years vis-a-vis a GHCN reconstruction using all records.

Stephen Pruett
May 28, 2010 9:52 am

Thanks for responses, and the situation with the climate data is perhaps not as bad as I thought, but it’s also not consistent with standards in other fields. We could potentially be investing huge sums that will measurably hurt all of us based on pretty haphazard data sets. The differences among the major ones are also a little disconcerting, though I know some are very similar over most of the record. I suppose what I am looking for, what would restore some of my confidence in climate science would be for the field as a whole to say, you know these data sets are not as good as they should be, and many papers cited in IPCC reports used adjustments that cannot be reproduced because the data or the code or both are not available. We are going to collectively go back and in an open manner confirm that all raw data are correct (by comparing to original documents) and discard those that can’t be confirmed, decide on the best procedures to adjust the data, document the adjustments carefully, make it publicly available all in one place, and show it in the next IPCC report. I have not heard of an effort like this; is there one, and do you think the community has any interest in this? I know this would be costly and would take some time, but I think as long as there is reason to doubt the validity of the data sets, it will not be possible to get the public behind any legislative measures. Defending the data sets when there are obviously some very real problems (even if you regard them as minor) only makes the climate science community less credible. The other credibility issue is the pervasive effort to “tell a tidy story”, but that’s a topic for another time. However, on that as well an admission of error would be a really nice place to start. Otherwise, credibility will not return and people like me who were on board with alarmism, will not be persuaded back into that camp. Many of these people could be persuaded; they don’t have a lot invested in this and are ready to be convinced one way or the other. When they see a defensive group of people who don’t want to admit there are any problems, most who are on the fence get suspicious. The bottom line is that the response of the climate science community to climategate, as much as the climategate revelations, has and continues to decrease the credibility of the field.

May 28, 2010 11:52 am

Stephen Pruett….
Huge sums have already been invested in this fight, for that is what it really is you know. 1.8 Trillion has already been spent in less than 10 years by the US Government on this. The UN has spent that much or more, it has all gone to the same poeple. Why do you think they are fighting so hard to keep this rediculous charade going?

May 28, 2010 2:03 pm

However, on that as well an admission of error would be a really nice place to start.
How about an admission of uncertainty. The further back in time you go in the physical record, the greater the uncertainties that accrue. Not only due to different instrumentation and different data gather methods, but also to decreasing spatial coverage (fewer stations, bigger gaps). And if you look for them, its not hard to find the global anomaly graphs with the errors that the authors willingly recognize.
http://hadobs.metoffice.com/crutem3/diagnostics/global/nh+sh/
Otherwise, credibility will not return and people like me who were on board with alarmism, will not be persuaded back into that camp.
I would hope that you don’t feel you have to be ‘camped’ with a pro-alarmism or anti-alarmism tribe. I am quite comfortable learning and experimenting with the data and methods of AGW without being an alarmist (although I am pretty aggressive against those who use their own ignorance about a subject to suggest malfeance on the part of others – we are all ignorant about some things. Watts, Smith, myself included. Ignorance should be used as a door to learning – not a club to beat others).

May 28, 2010 2:08 pm

And error for GISTEMP
Note the green bars
http://data.giss.nasa.gov/gistemp/graphs/

Chuck Wiese
May 28, 2010 2:48 pm

Gavin Schmidt’s website “Real Climate” is a roos, fraud and propaganda machine in my opinion. As a meteorologist, I have tried to engage him, Ray Ladbury and others to debate critical points they make as well as refute many of them only to have my posts surgically edited or deleted completely when you can prove they made incorrect statements or pin them down to demonstrate they are wrong. They also have a nasty habit of taking any part of a post you make through editing, and if they can find a way to take part of a statement you make that is incomplete without the rest of the paragraph that they then can turn around on you they will do it relentlessly followed by some sort of a snarky insult that is usually directed by Ray Ladbury whom I fondly refer to as “Mr. Honeybucket” because he is so full of himself which is 90% crap.
In my history of blogging of many years, I have never seen or experienced such a phony, false and self serving website full of bad information as this. I would not trust a thing that comes out of Gavin’s or Ray Ladbury’s keyboard, especially there, and I wouldn’t be surprised if our tax dollars are funding it. It is nothing more than a full time propaganda machine visited and posted on frequently by all the special interests protecting climate research and promoting cap and trade.

ECE Georgia
May 29, 2010 7:19 am

For those of you making strident political statments, I come here for the ‘science’ part of the larger political and socio-economic discussion. I have been reading this wonderful open forum and discussion daily for many months getting here due to Chris’ posts on BigGovernment.
Anthony, forgive me:
If anyone here wants to effect the political change in the USA, work to elect a non-politician to Congress this Nov. Reset the ‘lobbiest’ form of government. A really simple goal, is to bring 3 others to the polls who have ‘common sense’. Moving a mountain can be accomplished one grain of sand at a time.
On a positive note:
From a scientific, climate standpoint, we have MANY years to proove that CO2 does not effect climate change in any appreciable way, since nothing being proposed will reduce CO2 due to fossil fuels, until they run out!

kadaka (KD Knoebel)
May 29, 2010 10:58 am

From: ECE Georgia on May 29, 2010 at 7:19 am

On a positive note:
From a scientific, climate standpoint, we have MANY years to proove that CO2 does not effect climate change in any appreciable way, since nothing being proposed will reduce CO2 due to fossil fuels, until they run out!

Not exactly true, as rising energy costs due to otherwise-ineffective regulations, including the indirect costs as in rising prices on goods and services from providers with increased energy costs, are already leading to reductions in consumption thus reductions in human-based CO2 emissions, quite often among the most vulnerable like those who are poor, elderly, and/or ill.
As witnessed during the recent winter in the UK and across Europe, such high energy costs lead to reduced use of heating among the elderly, which then lead to permanent reductions in CO2 emissions by individuals.

June 1, 2010 10:43 am

spent most of the morning digging in to this workshop –
1st – even though they don’t agree – carrot eater and zeke should be commended for coming by and be willing to work on (dont say argue) the various points…..
As a non stats non science (although familar with both) guy the question of the issue with the CEI lawsuit seems to be that (and these are quite different, though related, issues:)
1) The GISS original data base and adjustments, have not been published….
2) The location of the temp sensors has changed and/or deleted….
3) Is there a bias towards warming in the record….
What Zeke are carrot eater are arguing, (if you go to their website http://rankexploits.com/musings/author/zeke/)
will give you a interesting statistical reconstruction of the GHCN database information
which seems to point to saying that the GHCN database was not unfairly or wrongly manipulated (i.e. the statistical regression) to show a “warming bias”….
Is this correct ?
Howoever, some pesky questions remain, not convincingly argued….
The question as to whether the data was “pre-manipulated” (for want of a better word…) whether – the Temperature averaging and Temperature “picking” was done in such a way to show warming bias – where the numbers the original readings, or were they adjusted… the answer seems to be “maybe”… Temperatures were averaged, or backfilled, or gridded and then presented as “original.” It seems that there is some question whether the area that each station represents is an accurate average of an average…..and even if this is valid….)
There is some question whether hi temp/lo temp would produce a bias towards warming – it is not the same as “mean” – but is what is used to show warming… ?
(I think this is valid – but would appreciate any clarifying statements)
There is a question of the number of Temperature stations taken, which changed sharply in 1990s (downward) would produce the same readings, or rather would show a “warming bias”
Zeke and Carrot eater have proposed that a statistical historical check has been done, and found the numbers coincide – in other words – the fewer new stations are statistically the same as the many “old” stations (although I don’t think the confidence levels could be the same….) Could this amplify warming ?
Is this correct ?
Finally, does the Temperature record show and urban heat bias toward warming that is somehow amplified by the above questions ? This is one of the oldest argument against the AGW hypothesis… I realize that Hansen (1997 ?) tried to answer this – but also that these answers cannot be taken as gospel…… Is there any more information on this that I am missing ? (This relates to the placement of the temp sensors)
Have I missed something ? Have I correctly summarized the points of contention ?
regards

June 1, 2010 4:14 pm

mackinacnick says:
June 1, 2010 at 10:43 am
If you have any questions regarding temp anomalies and missing data or poorly placed weather stations I would encourage you to go to ■Surface-stations Gallery
■Surface-stations Main on Mr. Watts links above and you will see the history of bad sensor info and placement and this is what they are using for data.
Take particular attention to the Tucson, AZ picture. All the pictures are very telling but I can attest to Tucson. I’ve been in Tucson and lived in the area. I know that it has grown exponentially and the years of growth have a direct correlation with the temperature readings in the articles expressing temps in Tucson going higher than the rest of the nation.
Also if you want I can give you links to a study that was done in Barrow Alaska for the results of UHIE in an area.
Even a small community of 200 in a very small village near the North Pole shows that UHIE plays a large role in temperature readings and rather than account for them the other side simply scoffs and pats us on the head and tells us we’re too stupid to really understand. Jone’s peer reviewed article on UHIE effects is a good example of how corrupt the review process is and why peer review material isn’t what the warmist side tries to build it up to being.
You can go to my blog http://tinyurl.com/29mywmo and contact me if you want me to give you the info on that study in Barrow, AK.

Ed Darrell
June 2, 2010 11:55 am

mackinacnick said:

Take particular attention to the Tucson, AZ picture. All the pictures are very telling but I can attest to Tucson. I’ve been in Tucson and lived in the area. I know that it has grown exponentially and the years of growth have a direct correlation with the temperature readings in the articles expressing temps in Tucson going higher than the rest of the nation.

Just how far wrong are the readings from Tucson over the course of a year, and how wrong are they for a decade? How much does Tucson’s reading skew a claim that “temperatures have risen in the U.S.” over any period of time?
How do Tucson’s skewed temperature readings compare with the Department of Agriculture’s plant hardiness zones map? Is the map wrong, for Tucson?
What do you compare Tucson’s readings to, to know the Tucson readings are wrong?
http://timpanogos.wordpress.com/2008/12/16/waiting-for-the-new-president-doctoring-data-on-global-warming/