NOTE: Please note that part 2 is now online, please see it here.
I recently plotted all four global temperature metrics (GISS, HadCRUT, UAH, RSS) to illustrate the magnitude of the global temperature drop we’ve seen in the last 12 months. At the end of that post, I mentioned that I’d like to get all 4 metrics plotted side-by-side for comparison, rather than individually.
Of course I have more ideas than time these days to collate such things, but sympathetic reader Earle Williams voluntarily came to my rescue by collating them and providing a nice data set for me in an Excel spreadsheet last night.
The biggest problem of course is what to do with 4 different data sets that have different time spans. The simplest answer, at least for a side by side comparison is to set their time scales to be the same. Satellite Microwave Sounder Unit (MSU) data from the University of Alabama, Huntsville (UAH), and Remote Sensing Systems (RSS) of Santa Rosa, CA only go back to 1979. So the January 1979-January 2008 period is what we’ll concentrate on for this exercise as it very nearly makes up a 30 year climate period. Yes, I know some may call this an arbitrary starting point, but it the only possible point that allows comparison between land-ocean data-sets and the satellite data-sets.
Here is the first graph, the raw anomaly data as it was published this month by all the above listed sources:
Here is the source data file for this plot and subsequent plots.
I also plotted a magnified view to show the detail of the lat 12 months with notations added to illustrate the depth of the anomaly over the past 12 months.
March 2005 to January 2008, magnified view – click for larger image
I was particularly impressed with the agreement of the 4 metrics during the 1998 El Niño year as well as our current 2008 La Niña year.
I also ran a smoothed plot to eliminate some of the noise and to make the trends a bit more visible. For this I used a 1 year (12 month) average.
Again there is good agreement in 1998 and in 2008. Suggesting that all 4 metrics picked up the ENSO event quite well.
The difference between these metrics is of course the source data, but more importantly, two are measured by satellite (UAH, RSS) and two are land-ocean surface temperature measurements (GISS, HadCRUT). I have been critical of the surface temperature measurements due to the number of non-compliant weather stations I’ve discovered in the United States Historical Climatology Network (USHCN) from my www.surfacestations.org project.
One of the first comments from my last post on the 4 global temperature metrics came from Jeff in Seattle who said:
Seems like GISS is the odd man out and should be discarded as an “adjustment”.
Looking at the difference in the 4 times series graphs, differences were apparent, but I didn’t have time to study it more right then. This post today is my follow up to that examination.
Over on Climate Audit, there’s been quite a bit of discussion about the global representivity of the GISS data-set due to all of the adjustments that seem to have been applied to the data at locations that don’t seem to need any adjustments to compensate for things like urban heat islands. Places like Cedarville, CA and Tingo Maria, Peru both illustrate some of the oddities with the adjustment methodology used by NASA GISS. One of the issues being discussed is the application of city nightlights (used as a measure of urbanization near the station) as a proxy for UHI adjustments to be applied to cities in the USA. Some investigation has suggested that the method may not work as well as one might expect. There’s also been the issue of whether of not stations classified as rural are truly rural.
So with all of this discussion, and with this newly collated data-set handed to me today, it gave me an idea. I had never seen a histogram comparison done on all four data-sets simultaneously.
Doing so would show how well the cool and warm anomalies are distributed within the data. If there is a good balance to the distribution, one would expect that the measurement system is doing a good job of capturing the natural variance. If the distribution of the histogram is skewed significantly in either the negative or positive, it would provide clues into what bias issues might remain in the data.
Of course since we have a rising temperature trend since 1979, I would expect all 4 metrics to be more distributed on the positive side of the histogram as a given. But the real test is how well they match. All four metrics correlate well in the time series graphs above, so I would expect some correlation to be present in the histogram as well. The histograms you see below were created from the raw data from 1979-2008. No smoothing or adjustments of any kind were made to the data. The “unadjusted” data in this source data file were used: 4metrics_temp_anomalies.txt
First we have the satellite data-set from UAH:
University of Alabama, Huntsville (UAH) Microwave Sounder Data 1979-2008 – click for larger image
The UAH data above looks well distributed between cool and warm anomaly. A slight warm bias, but to be expected with the positive trend since 1979.
Next we have the satellite data-set from RSS:
Remote Sensing Systems (RSS) Microwave Sounder Data 1979-2008 – click for larger image
At first I was surprised at the agreement between UAH and RSS in the percentages of warm and cool, but then I realized that these data-sets both came from the same instrument on the spacecraft and the only difference is methodology in preparation by the two groups UAH and RSS. So it makes sense that there would be some agreement in the histograms.
Here we have the land-ocean surface data-set from HadCRUT:
Hadley Climate Research Unit Temperature data 1979-2008 – click for larger image
Here, we see a much more lopsided distribution in the histogram. Part of this has to do with the positive trend, but other things like UHI, microsite issues with weather station placement, and adjustments to the temperature records all figure in.
Finally we have the GISS land-ocean surface data-set:
NASA Goddard Institute for Space Studies data 1979-2008 – click for larger image
I was surprised to learn that only 5% of the GISS data-set was on the cool side of zero, while a whopping 95% was on the warm side. Even with a rising temperature trend, this seems excessive.
When the distribution of data is so lopsided, it suggests that there may be problems with it, especially since there appears to be a 50% greater distribution on the cooler side in the HadCRUT data-set.
Interestingly, like with the satellite data sets that use the same sensor on the spacecraft, both GISS and HadCRUT use many of the same temperature stations around the world. There is quite a bit of data source overlap between the two. But, to see such a difference suggests to me that in this case (unlike the satellite data) differences in preparation lead to significant differences in the final data-set.
It also suggests to me that satellite temperature data is a more representative global temperature metric than manually measured land-ocean temperature data-sets because there is a more unified and homogeneous measurement system, less potential bias, no urban heat island issues, no need of maintaining individual temperature stations, fewer final adjustments, and a much faster acquisition of the data.
One of the things that has been pointed out to me by Joe D’Aleo of ICECAP is that GISS uses a different base period than the other data-sets, The next task is to plot these with data adjusted to the same base period. That should come in a day or two.
UPDATE1: I’ve decided to make this a 3 part series, as additional interest has been generated by commenters in looking at the data in more ways. Stay tuned for parts 2 and 3 and we’ll examine this is more detail.
UPDATE2: I had mentioned that I’d be looking at this in more detail in parts 2, and 3. However it appears many have missed seeing that portion of the original post and are saying that I’ve done an incomplete job of presenting all the information. I would agree for part1, but that is what parts 2 and 3 were to be about.
Since I’m currently unable to spend more time to put parts 2 and 3 together due to travel and other obligations, I’m putting the post back on the shelf (archived) to revisit again later when I can do more work on it, including show plots for adjusted base periods.
The post will be restored then along with the next part so that people have the benefit of seeing plots and histograms done on both ways. In part 3 I’ll summarize 1 and 2.
In the meantime, poster Basil has done some work on this of interest which you can see here.
UPDATE3: Part 2 is now online, please see it here.
I now see where Anthony linked to his original dataset.
When others get around to normalizing all four series on the series’ means, I think they will be intrigued by the results. Superficially, the satellite series appear more volatile, or extreme, than the land-sea series. Would there be a physical basis in climate science for this, or is it an artifact of the methodologies employed?
I’ll have more, later. If not before this thread gets old, then maybe we could take it up in a dedicated discussion on the CA forum.
REPLY: Our master of heat sinks, Evan should be able to tell you.
Evan Jones,
The 4-way link, ie. the text file with all four metrics, is captured directly from the spreadsheet I sent Anthony. The ‘Unadjusted’ label was mine, meaning those are the data exactly as distributed by the respective agencies. Whatever GISS or the others did to get their data into that form you will need to follow the original links to find out.
My notion of adjusted was what everyone else is describing with respect to subtracting out each series’ mean so that they are all centered around zero for the 1979 – 2008 time period. That is the adjustment I had in mind.
By the way, I tried not to use the word normalized, because in my mind normalizing implies scaling a series to fit within a defined range. The only thing I would consider doing to these data are to slide them up and down the Y axis, since as ‘temp anomaly’ profiles they reference an arbitrarily defined temp, in this case defined by the mean temperature over arbitrary time periods. The differences in those time periods is why the base temperatures differ.
I observed the same thing in the data as Basil, that the sat data appears to have a wider range. Given that these are completely differing methodologies not even measuring the same physical phenomena, I am more surprised at the degree of similarity.
Obsessive Ponderer says ” 1 The satellite data shows no significant warming or cooling for 19 years – explain that with the CO2 hypothesis.”
I’d venture to say that the sat data shows no SIGNIFICANT warming at all, ever, at least not in the realm where one can attribute what there is strictly to CO2. If you were to plot the change in population and the change in land use (forest now being farms, growing cities, etc.) and assume that said changes also have an influence, I don’t see how there’s enough signal to claim that the CO2 is even detectable. Problem is that I don’t know enough of the physics of this to know how the land use affects what the sats measure, but surely this is being measured as well.
I always thought that global warming meant more extreme weather, rather than specifically a warmer average temperature across large regions.
4 way link? I’m confused by this entire question.
Sorry. I meant the link in your above post to the 4 data series. It says the data is unadjusted.
Mr. Williams: Ah. Thanks for the clarification.
ANTHONY:
Blogger CCE at another site applied a twelve month “moving” average to the 4 global temperature metrics claiming that approach superior when examining anomalies. Assuming he used the same datasets, his results posted here:
http://cce.890m.com/temp-compare.jpg
show that GISS is in the mainstream of reporting temperature change and not at the high end.
When you write again, would you please comment on the efficacy, if any, of a “moving” average.
CCE also wrote that “[he keeps] hearing about how GISS is trending away from RSS. If anyone would bother to actually check, RSS and GISS have been converging over time.” Given your writings here and at surfacestations.org, and those at CA about the data difficulties, is there any basis for CCE’s claim?
EVAN CHECK YOUR EMAIL – Anthony
Brian Flynn,
The data are available above for you to look at and judge for yourself. If you don’t have software for displaying graphs and performing calculations on data, such as Excel or Microsoft Office, you can download a free software package called OpenOffice, which includes a spreadsheet application that will permit you to view the data yourself. I don’t mean to put you off in this regard, just mentioning that some simple but powerful software is available to anyone with a computer at http://www.openoffice.org .
If you plot the monthly data and calculate the difference between the GISS and RSS data you will see that they vary generally in the range of +/- 2 tenths of a degree Celsius over the 1979 – 2008 time period. One thing about smoothing the data is you may disregard the underlying variance in the data. There is no statistically significant trend in the monthly difference between GISS and RSS.
Please be aware that the GISS temperature calculations involve adjusting historical temperatures up or down to account for resumed biases and urban impacts in the observational data. Most recent temperatures are not adusted in calculating the GISS temperature anomaly. Any perceived recent convergence between the two measures could be an effect of the GISS method of adjusting the past temperatures.
Re Brian Flynn,
I quickly ran Anthony’s data through Excel 2007 Moving average using 12 months (putting 12 in the interval box). My initial reaction from looking at the link given was HUH?
Now I am far from an expert on this data manipulation stuff, but my chart (don’t know how to post it from Excel) looks absolutely nothing like the graph from the link(ie the exact opposite). I’m I doing something wrong, or are we looking a data manipulation par excellence?
I’ve normalized the data series, and fit a series of trend and step functions to the data. Rather than post a spaghetti graph, here’s a link to a plot of just the trend lines:
http://i31.tinypic.com/vzagj9.jpg
At least since 2001, it doesn’t look like RSS and GISS are converging. All four data sets show a downward trend since 2001, but the downward trend for GISS is much less than the downward trend for RSS.
Some quick observations:
From 1993 through 2001, all four series show a remarkably similar trend apart from the 1998 El Nino. Prior to 1993, the UAH series stands out from the other three. Since 2001, RSS and HadCRUT show almost identical downward trends, all the more remarkable for one being satellite, and the other a land-sea set.
When I get the time, I will try to collate the actual numbers into some kind of readable table.
To Obsessive Ponderer, all the data sets show some warming from 1979 to 1997, just as they all show cooling since 2001. But the warming was much less than is inferred from a simple trend line through the data for the entire time frame, as is common.
Brian: Anthony’s 12-month moving averages have been smoothed and CCE’s moving averages have not been smoothed. That’s the difference between the two charts. The idea of smoothing is to reduce the noise level.
I’m still struck by the headless El Nino (1998) of GISTEMP, including the one on CCE’s chart. This is a feature that is unique to GISTEMP, and it does not square up with HADCRUT, RSS or UAH.
In stock trading, the headless El Nino would be called a divergence. In our case, the equity GISTEMP is posting higher highs whereas the other three equities in the industry group are trending downward or sideways and are therefore failing to confirm GISTEMP’s uptrend. The obvious play for the trader here is to sell shares of GISTEMP short, meaning betting against the uptrend. (The assumption behind this trade is that part of a stock’s movements can be attributed to the overall movement of the peers in the industry group, especially if the moves are confirmed by hard data, such as news of declining sales, etc.)
Basil,
I beg to differ on the trend shown by UAH and RSS data from 1979 to approx Dec 1997.
I first noticed this on a bar graph of the satellite global temp. Line graphs seem to indicate an upward trend. If you look at a bar graph it does not.
If you put a simple trend line on a graph of satellite data from 1979 to Dec 1997 it will give you a very slight upward trend.
I asked W Briggs about this. Look at his website especially the wavelet analysis and his emphasis on going beyond simple trend lines. http://wmbriggs.com/blog/category/climatology/.
He doesn’t say it (being a causcious scientist) but I will – there is no discernable satellite temperature trend until the middle of the 1990s.
Then something happens – he indicates in 1996 or 1998.
If you do a histogram of the satellite data from the same dates (I did it from Jan 1979 to Dec 1997) the data is symmetrically centered on 0 with the the 0 mark’s cumulative %age = 54%. If I am not mistaken, this is about as perfect as you can get with real data. No warming, no cooling.
This is 19 years, surely a significant amount of time if CO2 was the most significant driver of global temperatures. It indicates to me that there is a more important influence or influences on climate than CO2. It concerns me that my government (Canadian) is spending 240 million dollars on sequestering CO2 (not 254 million as stated above-my bad) and all three US presidential candidates believe in CO2 catastrophic warming and one of them can’t wait to stop climate change.
Also, notice that the increase in global temperatures (if the sat data is more representative of the real situation) was just in time to give a impetus to the CO2 hypothesis (I believe the Rio conference was in 1992).
Simple trend lines will not convince me CO2 is the big culprit here.
There is something amiss in the GISS temperatures. We need to heed what the data is telling us.
Harold Vance
(The assumption behind this trade is that part of a stock’s movements can be attributed to the overall movement of the peers in the industry group, especially if the moves are confirmed by hard data, such as news of declining sales, etc.)
By the time the hard data becomes public (“MSM”) the damage is almost always done.
It is always good to check with the company’s suppliers before jumping on the “old bandwagon”. Several companies in an industry group could be “cooken the books”.
Obsessive Ponderer,
I looked at the Briggs web site. Do you know which version of the satellite data he used? I ask, because since last posting, I recalculated the trend lines using Cochrane-Orcutt to handle the serial correlation (what Briggs is probably alluding to when he talks about examining the residuals). I don’t have the results in front of me (they are at work, I am at home for the evening), but the UAH “trend” is nearly flat in the first period (given the way I broke the data up, that’s 1979-1992), and not significantly different than zero. I don’t recall the significance of the other data sets. I’ll look at it tomorrow.
I’m certainly not trying to infer anything about the influence of CO2 from simple trends. My interest is in discerning what the real trends are, regardless of the cause. And whatever they are, they are not uniform over very long periods of time, like is often inferred. While we might differ on how much, I don’t see a lot of increase from 1979 to 1992, a pretty obvious increase from 1992 through 2001, and then all the data sets show decreases since 2001. In fact, in the trends corrected for serial correlation, the trends since 2001 are even more sharply downward than in the image posted earlier today.
I’ll post more tomorrow, including statistical significance of the “trends” — whatever they are.
How does Jan 2008 show up on your graph of 12-month average temperature anomalies? Don’t you need 6 months of data AFTER Jan 2008 to calculate a 12-month average centered on January? Or is it a trailing average (average of the previous 12 months)?
REPLY: The data was supplied by the four agencies already in anomaly form, I did not calculate it.
GISS and RSS have been converging over time. The “divergence” of GISS from RSS over recent years is no different than any number of past “divergences.”
Other than the obvious satellite/instrument difference, the satellite coverage extends from 82.5 North to 70 South. All but the tip of Antarctica is excluded, as is the small area around the north pole (and the high altitude areas of the Himalayas and Andes). GISTEMP interpolates areas of poor coverage which gives it more polar coverage than the other analyses. GISTEMP and HadCRU show virtually identical warming over the past 30 years despite their differences. You will get periods where they are different and we expect differences.
This plot shows RSS subtracted from GISS, centered around 0.
http://cce.890m.com/gissvsrss.jpg
Over the entire Satellite era, RSS and GISS are converging. Over the most recent years, GISTEMP has trended upwards slightly in relation to RSS and the others, but this is no different than any number of times they have diverged (in either direction), and the magnitude of the divergence is not large compared to the differences with the early satellite data.
Slightly O.T. for the direction the thread has taken, but on the television network The C.W.’s morning news show “The Daily Buzz”, they reported this cooling trend. While their news coverage is a mile wide and an inch deep, the host said that all four temp records show a downward turn and have dropped so much in the last year that they have erased the rise of the past century.
This is where the war will be won or lost. It is the popular media, defined somewhat differently than the Main Stream Media, that influence the Dumb Masses and determine the duration of the hoax.
I find it interesting that Gavin hasn’t found the “spare time” in his day to post on this or any other topic in over a week.
Bob…L
As much as I appreciate and applaud Anthony’s work, he’s fueled a misleading view of the data when people make statements like “[temps] have dropped so much in the last year that they have erased the rise of the past century.” The drop from January 07 to January 08 was certainly of that order of magnitude, but January 07 was unusually warm. If you look at the trend lines I posted here:
http://i31.tinypic.com/vzagj9.jpg
Even with the downturn since 2001, we haven’t entirely erased the trend since 1979, let alone the past century. I’m with those who think we may be entering a cooling phase, and that the recent warming was more attributable to natural factors, than human, but I don’t think we should overstate what the data actually shows.
REPLY: Well the blame for the “misleading view” lies with Michael Asher, of Daily Tech. I complained within a few minutes of its posting about the use of the word “erase” and other issues, he was slow to react. It took three tries to get him to move, and by then hours had passed.
But, the same thing happens sometimes when NOAA or GISS puts out a press release. Unfortunately, our media is far from perfect, and they often don’t understand science well enough to translate it for the layman. I try to do that here, but even then, such terrible mistakes occur. I regret that it has happened but I can’t do much to put that genie back in the bottle.
Basil, Anthony obviously can’t control what people think, or prevent them from misinterpreting his presentations, happens constantly on both sides of the issue.
What he DOES show is that we just don’t know enough about global climate to make any major decisions about anything.
Please correct me if I’VE misinterpreted your data, Anthony.
Anthony,
I know you were not responsible. And you are right that the “warmists” do the same thing on the other side whenever they can get away with it.
FWIW, here’s the latest plot of “trends” I’ve fit through the four series:
http://i29.tinypic.com/302oxli.jpg
These are more “robust” than the previous trend lines, since I used Cochrane-Orcutt estimation to handle serial correlation. The resulting trends, in a tabular format (I hope this turns out readable):
1979:01-1992:12
---------------------------------------------
GISS 0.000783764** (0.094C/decade)
HadCRUT 0.000460122** (0.055C/decade)
RSS_MSU 0.000498964 (0.060C/decade)
UAH_MSU 1.71035E-05 (0.002C/decade)
1993:01-2001:12
---------------------------------------------
GISS 0.00174741** (0.210C/decade)
HadCRUT 0.00147990** (0.178C/decade)
RSS_MSU 0.00221135** (0.265C/decade)
UAH_MSU 0.00217023** (0.260C/decade)
2002:01-2008:1
---------------------------------------------
GISS -0.00091450 (-0.110C/decade)
HadCRUT -0.00270338** (-0.324C/decade)
RSS_MSU -0.00208111 (-0.250C/decade)
UAH_MSU -0.00130882 (-0.157C/decade)
The double asterisks indicate a 95% CI level of significance. No asterisks indicate that the numbers are not significantly different than zero. At some point, and in some venue, I’ll have more to say about what I’m doing here. But next up will be a composite trend weighted by inverse variance, and tests of each series against that composite.
REPLY: Thanks, I’ll put this up in part 2
Basil,
I appreciate what you say and agree. My intended point was my surprise at seeing these findings in that setting and to reiterate a point I have made repeatedly here. It is interesting to debate the fine minutia of the science of the AGW argument. The economy / freedom wrecking policies being discussed will be driven by politicians whose main interest is power and reelection. It is opinion of the “Dumb Masses”, who avoid most news programming, that will influence the politicians much more than any group of climate scientists.
The information presented by the host is typical of the treatment of most scientific information. Just rare to see the skeptic side shown.
cce,
Trying to detect fluctions in a trend that are an order of magnitude smaller than the standard deviation in your data is an exercise in futility.
The trend (LS linear fit) over the satellite era of the centered difference between GISS – RSS is minute, and of a negative slope at -0.0012 deg C per year with a mean of 0.
The trend over the last ten years is positive at 0.0145 deg C per year with a mean of -0.0216.
The trend over the last five years is positive at 0.0187 deg C per year with a mean of 0.0045. Above zero and climbing is not convergence.
However, the standard deviation in the GISS – RSS data is 0.1329 deg C.
The data do not support a conclusion that the two metrics are converging. There is simply no significant change over time. So if you wish to refute the notion that the two metrics are diverging, the data will back you up. But they won’t go so far as to support the conclusion that they are in fact converging.
Have a look at the main page, I just posted Basil’s graph and table.
Basil
I think Briggs just used the info available on the internet, but you could ask him. I sent him data in Excel format which I got from the UAH and RSS sites which I could sent to you if you like. Might save you some time. I’ll comment later as I got to do some work.