Much Ado About Very Little
Guest post by Zeke Hausfather and Steve Mosher
E.M. Smith has claimed (see full post here: Summary Report on v1 vs v3 GHCN ) to find numerous differences between GHCN version 1 and version 3, differences that, in his words, constitute “a degree of shift of the input data of roughly the same order of scale as the reputed Global Warming”. His analysis is flawed, however, as the raw data in GHCN v1 and v3 are nearly identical, and trends in the globally gridded raw data for both are effectively the same as those found in the published NCDC and GISTemp land records.
Figure 1: Comparison of station-months of data over time between GHCN v1 and GHCN v3.
First, a little background on the Global Historical Climatology Network (GHCN). GHCN was created in the late 1980s after a large effort by the World Meteorological Organization (WMO) to collect all available temperature data from member countries. Many of these were in the form of logbooks or other non-digital records (this being the 1980s), and many man-hours were required to process them into a digital form.
Meanwhile, the WMO set up a process to automate the submission of data going forward, setting up a network of around 1,200 geographically distributed stations that would provide monthly updates via CLIMAT reports. Periodically NCDC undertakes efforts to collect more historical monthly data not submitted via CLIMAT reports, and more recently has set up a daily product with automated updates from tens of thousands of stations (GHCN-Daily). This structure of GHCN as a periodically updated retroactive compilation with a subset of automatically reporting stations has in the past led to some confusion over “station die-offs”.
GHCN has gone through three major iterations. V1 was released in 1992 and included around 6,000 stations with only mean temperatures available and no adjustments or homogenization. Version 2 was released in 1997 and added in a number of new stations, minimum and maximum temperatures, and manually homogenized data. V3 was released last year and added many new stations (both in the distant past and post-1992, where Version 2 showed a sharp drop-off in available records), and switched the homogenization process to the Menne and Williams Pairwise Homogenization Algorithm (PHA) previously used in USHCN. Figure 1, above, shows the number of stations records available for each month in GHCN v1 and v3.
We can perform a number of tests to see if GHCN v1 and 3 differ. The simplest one is to compare the observations in both data files for the same stations. This is somewhat complicated by the fact that station identity numbers have changed since v1 and v3, and we have been unable to locate translation between the two. We can, however, match stations between the two sets using their latitude and longitude coordinates. This gives us 1,267,763 station-months of data whose stations match between the two sets with a precision of two decimal places.
When we calculate the difference between the two sets and plot the distribution, we get Figure 2, below:
Figure 2: Difference between GHCN v1 and GHCN v3 records matched by station lat/lon.
The vast majority of observations are identical between GHCN v1 and v3. If we exclude identical observations and just look at the distribution of non-zero differences, we get Figure 3:
Figure 3: Difference between GHCN v1 and GHCN v3 records matched by station lat/lon, excluding cases of zero difference.
This shows that while the raw data in GHCN v1 and v3 is not identical (at least via this method of station matching), there is little bias in the mean. Differences between the two might be explained by the resolution of duplicate measurements in the same location (called imods in GHCN version 2), by updates to the data from various national MET offices, or by refinements in station lat/lon over time.
Another way to test if GHCN v1 and GHCN v3 differ is to convert the data of each into anomalies (with baseline years of 1960-1989 chosen to maximize overlap in the common anomaly period), assign each to a 5 by 5 lat/lon grid cell, average anomalies in each grid cell, and create a land-area weighted global temperature estimate. This is similar to the method that NCDC uses in their reconstruction.
Figure 4: Comparison of GHCN v1 and GHCN v3 spatially gridded anomalies. Note that GHCN v1 ends in 1990 because that is the last year of available data.
When we do this for both GHCN v1 and GHCN v3 raw data, we get the figure above. While we would expect some differences simply because GHCN v3 includes a number of stations not included in GHCN v1, the similarities are pretty remarkable. Over the century scale the trends in the two are nearly identical. This differs significantly from the picture painted by E.M. Smith; indeed, instead of the shift in input data being equivalent to 50% of the trend, as he suggests, we see that differences amount to a mere 1.5% difference in trend.
Now, astute skeptics might agree with me that the raw data files are, if not identical, overwhelmingly similar but point out that there is one difference I did not address: GHCN v1 had only raw data with no adjustments, while GHCN v3 has both adjusted and raw versions. Perhaps the warming the E.M. Smith attributed to changes in input data might in fact be due to changes in adjustment method?
This is not the case, as GHCN v3 adjustments have little impact on the global-scale trend vis-à-vis the raw data. We can see this in Figure 5 below, where both GHCN v1 and GHCN v3 are compared to published NCDC and GISTemp land records:
Figure 5: Comparison of GHCN v1 and GHCN v3 spatially gridded anomalies with NCDC and GISTemp published land reconstructions.
If we look at the trends over the 1880-1990 period, we find that both GHCN v1 and GHCN v3 are quite similar, and lie between the trends shown in GISTemp and NCDC records.
1880-1990 trends
GHCN v1 raw: 0.04845 C (0.03661 to 0.06024)
GHCN v3 raw: 0.04919 C (0.03737 to 0.06100)
NCDC adjusted: 0.05394 C (0.04418 to 0.06370)
GISTemp adjusted: 0.04676 C (0.03620 to 0.05731)
This analysis should make it abundantly clear that the change in raw input data (if any) between GHCN version 1 and GHCN version 3 had little to no effect on global temperature trends. The exact cause of Smith’s mistaken conclusion is unknown; however, a review of his code does indicate a few areas that seem problematic. They are:
1. An apparent reliance on station Ids to match stations. Station Ids can differ between versions of GHCN.
2. Use of First Differences. Smith uses first differences, however he has made idiosyncratic changes to the method, especially in cases where there are temporal lacuna in the data. The method which used to be used by NCDC has known issues and biases – detailed by Jeff Id. Smith’s implementation and his method of handling gaps in the data is unproven and may be the cause.
3. It’s unclear from the code which version of GHCN V3 that Smith used.
STATA code and data used in creating the figures in this post can be found here: https://www.dropbox.com/sh/b9rz83cu7ds9lq8/IKUGoHk5qc
Playing around with it is strongly encouraged for those interested.
“The funniest interventions could be yours. We spoke with Steven Mosher of the ability to disable the implicit homogenization in BEST. This implicit homogenization is the result of the segments adjustments. If you disable this setting, there is simply no results.”
Huh? once again phi is talking about something he knows nothing about.
You can turn the scalpel on or off. readers HERE should know that because they will remember that when Muller first ran some tests on station quality ( anthony’s issue ) those tests
were run with the scalpel off.
At some point people will address the following issues.
1. sunshine tried to pass estimated data from Env canada off on you without mentioning it.
2. EM tried to pass his version of First differences off on you as being peer reviewed
and None of you, not a single one of you, even had the memory to realize that
we had discussed that method on climate audit and Jeff Ids Air vent.
3. Phi, anonymously, continues to hi jack threads with requests for citations that
a) are right on google
b) are in the bibliographies of papers he claims to have read
c) have been given to him before.
4. we get the same answer as Giss and Cru using
a) no GHCNv3 data
b) using methods developed by skeptics
All that said there are some valid criticisms of GISS and CRUtemp but they are
scientifically uninteresting. They are fascinating nit picks, but in the end using different data and better methods ( skeptic approved methods– gosh it was everything we asked for ) we get the same answer. Yup, its warmer now than in the LIA. we know that with a little more certainty
than we did before. more temporal coverage and more spatial coverage. transparent code.
open data. Everything we asked for. What nobody expected was that the answer would change so little.
Mosher: “he posts data that is estimated and does not disclose that fact”
Again, as I said above, the monthly data I got from EC did not say that.
However, my post also said that wasn’t the most important issue. It was data appearing BEST that had no basis in reality.
But I think I had a legitimate question as to where a -13.7C data point for Malahat came from in the BEST data. Even Nick Stokes admitted it appeared out of nowhere. You continue to try and divert peoples attention from mysterious data.
I got 4355 stations stations with a cooling trend from 2000 to 2011, most in the USA.
Further back the number drops but it is still significant.
http://sunshinehours.wordpress.com/2012/03/18/cooling-weather-stations-by-decade-from-1880-to-2000/
How many do you get by decade? Map them.
Anyway, frankly I think it is disgusting BEST only shows data from 1950 and totally ignores the warmer 20s and 30s. What a joke.
Lucy
“Like showing the HS is an unavoidable statistical result of the Team’s method of selection, I suspect CA would show that the warming is an unavoidable statistical result of dropping stations etc. But I am open to disproof.”
we have offered that proof in many forms, even on this blog.
1. We did a global average using around 100 of the longest stations: same answer
2. We ran the average with random selections of data: same answer
3. We ran a test that only included stations that survived the drop out: same answer
4. We ran with completely different datasets: same answer
we ran different methods. we ran sensitivity tests. everything.
Now, here is what we did find. I will call it Carricks find since he did most the work and made the best suggestions.
When you have missing months stations tend to COOL. thats right. this came out of me looking at the cooling station phenomena. The more complete the record, the warmer the station will be. Carrick had some theories about this that I’m going to get to test someday when the day gets extended to 36 hours.
One other thing that remains is the latitude bias that Carrick has identified. I think there may be a coastal station bias as well, that is too many stations on coasts will depress the record.
Mosher: “I don’t know anyone who argues that it is cooler now than in the LIA. but go ahead, fire away with your best data.”
1) Washington State. The last 5 years average temperature are cooler than ~1898 to 1908.
Or by “now” do you mean 1998. It isn’t 1998 anymore.
You can check here: http://www.ncdc.noaa.gov/oa/climate/research/cag3/wa.html
Arizona?
http://sunshinehours.wordpress.com/2012/06/24/arizonanoaa-5-year-averages-plotted-using-all-monthly-anomalies/
2) While the US temperature record starts at 1895, assuming that it was cooler before 1895 is not supportable. Looking at the Arizona graph or Washington, what would they look like if the data went back to 1800?
That wasn’t the point at all. I am a laymen. I come to sites such as this to find information and to digest it in as much capacity as I can do given that fact. What this ends up coming down to is usually one of two things –
1. Simple to follow ideas and
2. Trust
Because at the end of the day, to take what is being said and have faith in the person who said it really does underpin the average person’s knowledge of science, particularly in matters that are very complex.
Now one of my first forays into researching this science was Real Climate. In spending some time there I found so many arrogant posts – not just by the commenters, but by the moderators as well in the comments – to any question differing of opinion, and experienced this first hand. That initial experience had me so turned off to the people there that now I do not trust any of them. They could argue the sky was Blue, but I wouldn’t believe it.
It is true that you do not need to be liked to do good science, but when it is layment such as me and the general public and especially many of the politicians setting the policies on that science, it is important that we trust those giving their opinions.
The point, though it was long to get there, was that the last time I had seen the analysis used in the first section of this ost is was done intentionally to mislead people reading it into believing what was being said by an intentional twisting of data to suit their purpose. To be honest, I read the first paragraph, saw the first graphs and immediate had the gut reaction to ignore the rest of the post as not worth my time because of that. As I said, this is unfortunate as the rest may well be a valid post, and it you may not have had that intention in mind when you posted it.
All I suggested was caution when posting that kind of information because it doesn’t help anyone, particularly if that example is used as the lead in to the rest of the argument. You stated that it is meaningless without comparing trends. The thing is, if you read the initial post from a third party point of view, you will note you made clear distinction that these were two different methods to both show there is not a problem:
Fllowed by:
That says to me that you hold the first method to be substantial in showing no bias indipendantly of the second method – which is an issue as that is not exactly true. Now me as a layment then has to ask the questio, should I trust that the second section of the article?
Please excuse the typing errors, my keyboard is old and if I do not whack the key it can just miss letters. Also I have no idea why I keep adding a t onto “laymen” o_O.
Amino, I admit I did misconstrue your comments, and I apologize to all present in this conversation for my role in escalating and personalizing this. Let’s leave it at that, ok?
You said:
No cherry picking is something else entirely. I selected data based on trying to reliably estimate a trend. Briefly, cherry picking would be selecting a data to produce desired outcome. I had no idea when I did this analysis what I would find for sure. I repeated the Mueller “marble plot” analysis obviously because I questioned the result shown in his plot, and I designed the analysis based on best testing this… that includes common base line (data present over selected interval), not so much data “in sample” missing that the trend could not be reliably estimated, etc.
Um, you were talking about Mueller, who is responsible for BEST, which is land only. Why do you think we should consider ocean data for a land-only reconstruction? And why are you bringing in GISTEMP to a discussion about Mueller’s data (really GHCN in the figure in question)?
Willis
“Asking someone for a citation to their claim is not a “clueless question”, Carrick. It is everyday scientific practice.”
1. This thread is not about TOBS. I have had this discussion about TOBS over and over again
with the anonymous phi ( bad andrew always shows up ) Now, you often manage
the discussion on your posts by telling people what is on topic and off topic. You’
dont see me coming on to your threads and saying “hey willis, that seems on topic to me!”
No, I dont. because I respect your decision to engage those who believe are engaging
in good faith. You put good hard time into your posts and commenting on your posts.
trolls waste your time and they waste my time. Every second I spend on a troll is time
away from the real issues: First differences as a method; and the changes between GHCN v1
and GHCN v3.
2. Phi has been told before when he tried to hi jack other threads where he could look
to find the answer to his question. Its not that hard. Anybody who works in this field
and actually reads papers and reports knows that the US is not the only country
that has an issue with time of observation. Its received the most publicity to be sure
but one test i use to tell if another party is really interested in the topic is
have they read the core literature. Or are they popping up to ask their pet off topic
question to derail the conversation. Anybody who googles time of observation bias
will find the reference to japan. Anybody who looked through CRUTEM4 sources
( a sure test of interest in the subject ) would find the case of canada. Anybody
who read WUWT and is interested in the topic would have read the source
documents about the australian case. Derailers, thread jackers, are easy to
spot. They are typically anonymous. They talk about one topic and not in much
depth.
He has also been told that the long series have been adjusted for TOBS. I assume
that somebody truly interested in this issue would have read about the
most important series that we have: CET. you have read the CET papers, you didn’t forget the sections about adjusting for TOB changes. I expected you, Willis, to remember that Armagh has also been adjusted for Time of observation as I recall you wrote about
that station here on WUWT and assume you read the underlying literature. Anyone who has looked at that data and paper would have read about adjustments for TOBS.
basically, I have no time for people who refuse to read, especially when they have been given the information before, especially when their request is off topic. For those who cant remember, I have a bit more patience.
sunshine. If you want to change the topic of this thread for yet another time and bring up your crazy denial of the LIA, Let me suggest that you take it to your blog.
Pamela Gray:
The point of performing a 70-year trend estimate is to remove short-period fluctuations to expose whether long-term warming is present or not. If you did an annual estimate, all you’d be seeing is climate noise.
I did an analysis of the effect of climate noise on temperature trend as a function of the integration period (e.g., interval of measurement of the trend), and as you can see for very short periods, the trend is telling you nothing at all interesting about warming or cooling of climate. 30 years is the minimum period I would recommend for reliably estimating temperature trends.
By the way I also did 1970-2010 similar result.
We could do what you are suggesting in your movie (in fact something like this has been done), but I’m not sure what you’d learn from it.
Carrick
It’s land data. You advocate it. What’s up with this putting it off on the man that made the data you prefer?
Carrick says:
June 24, 2012 at 8:26 pm
“Amino, I admit I did misconstrue your comments, and I apologize to all present in this conversation for my role in escalating and personalizing this.”
Thanks for the apology. I accept it. And I appreciate it.
One thing I’d like to clear up is it did look like you were departing from the thread and not just temporarily to do some tiling. But it is easy to write a comment that appears to mean something other than it meant. I’m pretty sure we’ve all done it.
sunshine (double checked attribution this time >.< ):
Generally the period given is circa 1350 to 1850. See my comments about about needing to be careful when comparing temperatures that you use a long-enough averaging period to remove short-period climate noise. I didn’t show this, but the period you need to average over becomes larger as you make the geographical area smaller. 10 years would be too short for full global temperature, it’s virtually worthless for very small (on a global scale) geographical areas like Washington State.
Mosher: “he posts data that is estimated and does not disclose that fact”
Again, as I said above, the monthly data I got from EC did not say that.
############
Then before you come onto this web site to stalk me and derail the conversation
please read the web site where you get your data. Or you can use the software
that I wrote for people like you to read the data. That software was written for your benefit
because of questions you asked about environment canada. It allows you
to download all the data.and to analyze it.
But do not come onto a thread where something else is being discussed, throw up
data that you have not checked when I wrote a program that allows you to check
and make me waste my time correcting your mistakes.
Now in case the thread nanny Willis comes along and explains that I should help you
undo your mistake ( when I’ve already written a software package that allows you
to do it ) let me give you the links. click on the second one to get the csv.
or use my program to download and organize all of env canada. Its was only a few weeks work
http://www.climate.weatheroffice.gc.ca/climateData/monthlydata_e.html?timeframe=3&Prov=BC&StationID=65&mlyRange=1920-01-01|2005-04-01&Year=1920&Month=01&Day=01
http://www.climate.weatheroffice.gc.ca/climateData/bulkdata_e.html?timeframe=3&Prov=BC&StationID=65&mlyRange=1920-01-01|2005-04-01&Year=1920&Month=1&Day=01&format=csv
Amino:
I’m not sure where you are getting that I am “advocating” land data or what you mean by “putting it off on the man that made the data” I prefer. GHCN, the data set I used, was made by many thousands of researchers not one, so who is the “man that made the data” that I prefer???
Please make sense.
I was responding to a specific issue that related to “Mueller’s data”, which is GHCN, which is land-only data. I used an analysis method that selected stations that could reliably be used to estimate temperature for a given period to test that data. I’m not sure what’s left to discuss besides that other than to conclude 1/3 of the GHCN stations do not actually show cooling (this is the data set Mueller’s figure was showing, and again it did specifically does not include ocean data).
The original comment made above about the “1/3 of the data” is wrong. We should move on to discussing something else. If you want to discuss ocean data (e.g. GISTEMPs extrapolation into the Arctic Ocean during wintertime months from land-only stations), I’m game. But it’s a different subject…and let’s wrap up this other topic by agreeing about this much at least.
steven mosher says:
June 24, 2012 at 6:48 pm
And in this case he has succeeded in derailing the conversion … but only because you were unwilling to give a simple link. In addition, as I mentioned above, there are others, like myself, interested in the same question who didn’t follow your previous discussion.
So next time, how about you just give the freakin’ link and move on, and not turn it into some kind of battle on principle? How hard is it to give a single link?
w.
Mosher: “sunshine. If you want to change the topic of this thread for yet another time”
You asked and I quote: “I don’t know anyone who argues that it is cooler now than in the LIA. but go ahead, fire away with your best data.”
And for 3rd or 4th time you dodge the key point: Spurious data did appear in BEST.
And more importantly as I’ve shown with a few graphs: Starting BEST graphs in 1950 is dishonest when the distant past is just as warm as it is today (and I’m talking 5 year averages, not 1 data point).
It isn’t 1998 anymore. It has cooled. Why it has cooled is fascinating. But your blinkers are on. You have a project to protect.
California: http://sunshinehours.wordpress.com/2012/06/24/californianoaa-5-year-averages-plotted-using-all-monthly-anomalies/
“But I think I had a legitimate question as to where a -13.7C data point for Malahat came from in the BEST data. Even Nick Stokes admitted it appeared out of nowhere. You continue to try and divert peoples attention from mysterious data.”
1. are you using the right dataset. last time we went through this you made the mistake of using
the raw data. Remember that?
2. Download the program BerkeleyEarth.
3. Using that program download the quality controlled data.
4. Using that program read in the sources.txt file. This will tell you the data sources
for every line of data.
5. Then realize that values that look “out of range” are handled by the last two
steps of quality control in the programs. In short, if the data is garbage it
goes through one last quality check and then a kriging process. In short,
just because a data element is in the file does not mean that it gets used.
There is a regional consistency test and kriging weighting.
6. Download the package CHCN and look at all the env canada data.
To recap. You are thread jacking. I have discussed this before with you and pointed you to
the software tools I wrote so that you can do this without making mistakes.
And in this case he has succeeded in derailing the conversion … but only because you were unwilling to give a simple link. In addition, as I mentioned above, there are others, like myself, interested in the same question who didn’t follow your previous discussion.
##########################
willis, you handle derailers your way I handle them my way. phi has been told last time he derailed the conversation where to look. You yourself are well enough read in this field ( CET and Armagh) to know that the US is not the only record that does TOBS adjustments. Did you honestly forget that?
Carrick: ” that you use a long-enough averaging period to remove short-period climate noise.”
I used 1971-2000 for the 30 year period. But it isn’t the value, it is the value compared to to other values.
And who said only Washington State is the only region warmer at the turn of the 20th century than it is in the last 5 years. California and Oregon.
Other regions were a lot warmer in the 20s or 30s than the last 5 years.
NOAA: “The Little Ice Age (or LIA) refers to a period between 1350 and 1900”
http://www.ncdc.noaa.gov/paleo/ctl/resource1000.html
Mosher, I scraped the EC data here (as I’ve mentioned before):
http://climate.weatheroffice.gc.ca/prods_servs/cdn_climate_summary_report_e.html?intMonth=5&intYear=2012&prov=BC&txtFormat=text&btnSubmit=Submit
sunshine
once again with the derailing of the conversation. I’m guess that Willis is the only one here who gets to tell commenters that there comments are off topic
“I got 4355 stations stations with a cooling trend from 2000 to 2011, most in the USA.”
Without a doubt. You will find cooling stations throughout the record. But these are the important questions
1. How many of those records are complete? Carrick had a wonderful finding that as you
look at COMPLETE records, records that have no missing months, the fraction
of cooling stations goes down. Higher quality data appears to mean something.
This is a cool mystery.
2. How many of those trends are statistically significant? what kind of correction
for auto correlation did you use? Basically what I found in poking around with this
was that about 15% of long stations were cooling– 10% if you used complete records
and this number dimished more if you ask for statistical significance
Here is a CLUE. when a warmist says “Its warming” we all rightly pounce on them
and ask where are the error bars. It seems fair to ask that question when one
finds cooling stations.
Now dont get me wrong, I think cooling stations are very interesting. Especially
those in urban heat islands. in fact there are a couple papers written on the effect
you want the links? Willis has them. he is the link nanny
steven mosher says:
June 24, 2012 at 8:41 pm
Thanks for that explanation, Steven. If the issue is that phi’s question is off topic, that’s fine. As you point out, I say things are off topic on my threads. And so can you … but:
1. In this case, the person who introduced the TOBS topic was you, saying:
The fact that you were the person who introduced the topic into the thread means that it is far from obvious that you will later consider it off-topic …
2. You didn’t do what you correctly point out that I do, which is to tell a poster that their comment is off topic. Instead, you first brought up the topic of TOBS, and then when asked if you had a reference for your claims about TOBS you said:
“yes. do more reading and post your results.”
No matter how many times I read that, I don’t get “TOBS is off-topic for this thread” out of it. Nor do I get “I told you that before, phi, and you ignored it” out of it, nor do I get “phi, you are a troll, I’m going to ignore you.” We’re not mind readers out here.
My point is simple. Once again, your cryptic posting style has done nothing but cause confusion and dissension. If the issue is that phi’s question is off-topic, say so. If the issue is that phi has asked that question before and ignored the answer, say so. If the issue is that you think phi is a troll, say so.
Because saying what you said just makes it look like you have something to hide, even though you don’t, and even though you have valid reasons for your actions. And that is something that you don’t need.
You are brilliant man, Steven, I’ve learned lots from you, and I’m a huge fan of your work and your insights… but your posting style, not so much.
w.
Bruce, err sunshine, if want to continue your LIA denial, please go to this thread.
There you have a link, so the nanny will be happy
http://wattsupwiththat.com/2012/06/24/hh-lamb-climate-present-past-future-vol-2-in-review-part-i/#more-66176
Carrick
Again, and really, for the last time, you are biased. You prefer Meuller and Hansen. My goodness man, look around you.