HADCRUT4 Adjustments – Discovering Missing Data or Reinterpreting Existing Data? (Now Includes September Data)

Guest Post by Werner Brozek and Just The Facts:

To begin, we would like to sincerely thank Tim Osborn of the University of East Anglia (UEA), Climatic Research Unit (CRU) for his responses (1, 2) to this recent WUWT article. Unfortunately, by the time the dialogue got going, many readers had already moved on.

Tim’s responses did leave few loose ends so we would like to carry on from where we left off. Hopefully Tim Osborn will respond again so we can continue the dialogue. In comments of the prior article both Tim Osborn and Steven Mosher argued that recent adjustments to HadCRUT4 were associated with missing data, i.e.:

Steven Mosher October 5, 2014 at 8:01 pm

Yes. the new data comes from areas that were undersampled.

We always worried that undersampling was biasing the record warm. remember?

Nobody had an issue when I argued that missing data could bias the record warm.

Now the data comes in.. Opps, the old record was biased cold.

So people wanted more data now they got it. read it and weep.

In particular the warming in the present comes from more samples at higher latitudes.

Tim Osborn (@TimOsbornClim) October 7, 2014 at 2:30 pm

“As to whether, or indeed why, our updates always strengthen the warming… overall they often have this effect and the reason is probably that the regions with most missing data have warmed more than the global average warming. This includes the Arctic region, but also many land regions. This is because, on average, land regions have warmed more than the ocean regions and so filling in gaps over the land with additional data tends to raise the global-mean warming.”

Looking at the most recent adjustments to HadCRUT4:

it appears they found more warming during the last 18 years in the Northern Hemisphere;

whereas in the Southern Hemisphere they found warming that had been missing since World War I:

When looking at the warming over last 18 years found in the Northern Hemisphere, it appears that a significant portion of it was found in Asia;

and Tim noted in his comment that;

“the big changes in the China station database that were obtained from the (Cao et al., 2013, DOI: 10.1002/jgrd.50615) and Xu et al. (2013, doi:10.1002/jgrd.50791) studies listed in the release notes, which is clearly not a high latitude effect.”

Looking at the recent warming obtained from Cao, Lijuan Cao, Ping Zhao, Zhongwei Yan, Phil Jones et al. appears to have found recent warming as a result of a reconstruction;

whereby existing station data;

was adjusted to account for “relocation of meteorological station”, “instrument change” and “change points without clear reason”:

Furthermore, this recent paper by co-author Yan ZhongWei states that:

Long-term meteorological observation series are fundamental for reflecting climate changes. However, almost all meteorological stations inevitably undergo relocation or changes in observation instruments, rules, and methods, which can result in systematic biases in the observation series for corresponding periods. Homogenization is a technique for adjusting these biases in order to assess the true trends in the time series. In recent years, homogenization has shifted its focus from the adjustments to climate mean status to the adjustments to information about climate extremes or extreme weather. Using case analyses of ideal and actual climate series, here we demonstrate the basic idea of homogenization, introduce new understanding obtained from recent studies of homogenization of climate series in China, and raise issues for further studies in this field, especially with regards to climate extremes, uncertainty of the statistical adjustments, and biased physical relationships among different climate variables due to adjustments in single variable series.

In terms of Xu et al. 2013, in this recent article in the Journal of Geophysical Research Xu et al. state that:

This study first homogenizes time series of daily maximum and minimum temperatures recorded at 825 stations in China over the period from 1951 to 2010, using both metadata and the penalized maximum t test with the first-order autocorrelation being accounted for to detect change points and using the quantile-matching algorithm to adjust the data time series to diminish discontinuities. Station relocation was found to be the main cause for discontinuities, followed by station automation.

As such the recent warming in the Asia region appears to be the result of adjusting existing station data to account for “relocation of meteorological station”, “instrument change”, “station automation” and “change points without clear reason” versus “filling in gaps” for “missing data”.

A second aspect of the recent HadCRUT4 adjustments that raises questions, is related to Tim Osborn’s explanation for why “our updates always strengthen the warming”, i.e.;

“the reason is probably that the regions with most missing data have warmed more than the global average warming. This includes the Arctic region, but also many land regions. This is because, on average, land regions have warmed more than the ocean regions and so filling in gaps over the land with additional data tends to raise the global-mean warming.”

Per Morice et al., 2012;

“HadCRUT4 remains the only one of the four prominent combined land and SST data sets that does not employ any form of spatial infilling and,as a result, grid-box anomalies can readily be traced back to observational records. The global near-surface temperature anomaly data set of the Goddard Institute for Space Studies (GISS) [Hansen etal., 2010], is again a blend of land and SST data sets.The land component is presented as a gridded data set in which grid-box values are a weighted average of temperature anomalies for stations lying within 1200 km of grid-box centres. The sea component is formed from a combination of the HadISST1 data set [Rayner et al., 2003] with the combined in situ and satellite SST data set of Reynolds et al. [2002].”

And per the CRU’s Frequently Asked Questions:

How are the land and marine data combined?

Both the component parts (land and marine) are separately averaged into the same 5° x 5° latitude/longitude grid boxes. The combined version (HadCRUT4 ) takes values from each component and weights the grid boxes according to the area, ensuring that the land component has a weight of at least 25% for any grid box containing some land data. The weighting method is described in detail in Morice et al. (2012). The previous combined versions (HadCRUT3 and HadCRUT3v) take values from each component and weight the grid boxes where both occur (coastlines and islands) according their errors of estimate (see Brohan et al., 2006 for details).

UEA CRU and the Met Office Hadley Center moved to this methodology with the introduction of HADCRUT4 in 2012, i.e.:

“The blending approach adopted here differs from that used in the Brohan et al. [2006] data set. Here, land and sea components are combined at a grid-box level by weighting each component by fractional areas of land and sea within each grid-box, rather than weighting in inverse proportion to error variance. This approach has been adopted to avoid over representation of sea temperatures in regions where SST measurements dominate the total number of measurements in a grid-box. The grid-box average temperature for grid-box i is formed from the grid-box average temperature

anomalies for the land component, for the SST component, and the fractional area of land in the grid-box”

“Coastal grid-boxes for which the land fraction is less than 25% of the total grid-box area are assigned a land fraction weighting of 25%. Here, we are making the assumption that land near-surface air temperature anomalies measured in grid-boxes that are predominantly sea covered are more representative of near-surface air temperature anomalies over the surrounding sea than sea-surface temperature anomalies. These fractions ensure that islands with long land records are not swamped by possibly sparse SST data in open ocean areas (where the island is only a small fraction of the total grid-box area).”

The more cynical might wonder if this change in methodology had anything to do with the fact that;

“Land-based temperatures averaged 0.04°C below sea temperatures for the period 1950 to 1987 but after 1997 averaged 0.41°C above sea temperatures.”

With oceans covering about 70% and land covering about 30% of the total area, it seems that there would be more missing data from oceans than for land, especially since few people live in the middle of the southern ocean. However, any new data that may be found after 1998 would likely be land areas and not remote ocean areas, thus these land areas could have an inordinate influence on global temperatures.

Suppose for example that a landlocked 5 X 5 grid had thermometer readings for 25% of its land area in 1998 and that the anomaly had increased by 0.8 C from a historical baseline. Assume that we knew nothing about the other 75% of the land area in 1998. Also, assume that the ocean average temperature increased by 0.4 C during the same period.

Suppose that in 2014 readings were found for the other 75% of this same landlocked 5 X 5 grid and it was found that this remaining area increased by only 0.7 C. Then even though the 1998 anomaly for this particular landlocked 5 X 5 grid appeared cooler in 2014 than was thought to be the case in 1998, the global anomaly would still go up since 0.7 C is greater than the 0.4 C that the oceans went up by. Is this correct based upon the HadCRUT4 gridding methodology?

A third aspect of the HadCRUT4.3 record and adjustments that raises questions can be found in the Release Notes, where it notes the “Australia – updates to the ‘ACORN’ climate series and corrections to remote island series”. However, as Jo Nova recently wrote;

“Ken Stewart points out that adjustments grossly exaggerate monthly and seasonal warming, and that anyone analyzing national data trends quickly gets into 2 degrees of quicksand. He asks: What was the national summer maximum in 1926? AWAP says 35.9C. Acorn says 33.5C. Which dataset is to be believed?”

As such, if there is a problem with source data such as ACORN in Australia, then this problem is reflected in HadCRUT4.3 record Furthermore, this problem would be compounded by the HADCRUT4 methodology of giving the “land component has a weight of at least 25% for any grid box containing some land data”, as the ACORN network is heavily weighted towards coastal overage, i.e.:

—

In the sections below, as in previous posts, we will present you with the latest facts. The information will be presented in three sections and an appendix. The first section will show for how long there has been no warming on several data sets. The second section will show for how long there has been no statistically significant warming on several data sets. The third section will show how 2014 to date compares with 2013 and the warmest years and months on record so far. The appendix will illustrate sections 1 and 2 in a different way. Graphs and a table will be used to illustrate the data.

Section 1

This analysis uses the latest month for which data is available on WoodForTrees.com (WFT). All of the data on WFT is also available at the specific sources as outlined below. We start with the present date and go to the furthest month in the past where the slope is a least slightly negative. So if the slope from September is 4 x 10^-4 but it is – 4 x 10^-4 from October, we give the time from October so no one can accuse us of being less than honest if we say the slope is flat from a certain month.

1. For GISS, the slope is flat since October 2004 or an even 10 years. (goes to September)

2. For Hadcrut4, the slope is flat since January 2005 or 9 years, 9 months. (goes to September). Note that WFT has not updated Hadcrut4 since July.

3. For Hadsst3, the slope is not flat for any period.

4. For UAH, the slope is flat since January 2005 or 9 years, 9 months. (goes to September using version 5.5)

5. For RSS, the slope is flat since September 1, 1996 or 18 years, 1 month (goes to September 30).

The following graph shows just the lines to illustrate the above. Think of it as a sideways bar graph where the lengths of the lines indicate the relative times where the slope is 0. In addition, the upward sloping brown line at the top indicates that CO2 has steadily increased over this period.

When two things are plotted as I have done, the left only shows a temperature anomaly.

The actual numbers are meaningless since all slopes are essentially zero. As well, I have offset them so they are evenly spaced. No numbers are given for CO2. Some have asked that the log of the concentration of CO2 be plotted. However WFT does not give this option. The upward sloping CO2 line only shows that while CO2 has been going up over the last 18 years, the temperatures have been flat for varying periods on various data sets.

Section 2

For this analysis, data was retrieved from Nick Stokes’ Trendviewer available on his website <a href=”http://moyhu.blogspot.com.au/p/temperature-trend-viewer.html”. This analysis indicates for how long there has not been statistically significant warming according to Nick’s criteria. Data go to their latest update for each set. In every case, note that the lower error bar is negative so a slope of 0 cannot be ruled out from the month indicated.

On several different data sets, there has been no statistically significant warming for between 14 and almost 22 years according to Nick’s criteria. Cl stands for the confidence limits at the 95% level.

Dr. Ross McKitrick has also commented on these parts and has slightly different numbers for the three data sets that he analyzed. I will also give his times.

The details for several sets are below.

For UAH: Since May 1996: CI from -0.024 to 2.272

(Dr. McKitrick says the warming is not significant for 16 years on UAH.)

For RSS: Since November 1992: CI from -0.001 to 1.816

(Dr. McKitrick says the warming is not significant for 26 years on RSS.)

For Hadcrut4.3: Since March 1997: CI from -0.016 to 1.157

(Dr. McKitrick says the warming is not significant for 19 years on Hadcrut4.2. Hadcrut4.3 could be very slightly different.)

For Hadsst3: Since October 1994: CI from -0.015 to 1.722

For GISS: Since February 2000: CI from -0.060 to 1.326

Note that all of the above times, regardless of the source, with the exception of GISS are larger than 15 years which NOAA deemed necessary to “create a discrepancy with the expected present-day warming rate”.

Section 3

This section shows data about 2014 and other information in the form of a table. The table shows the five data sources along the top and other places so they should be visible at all times. The sources are UAH, RSS, Hadcrut4, Hadsst3, and GISS.

Down the column, are the following:

1. 13ra: This is the final ranking for 2013 on each data set.

2. 13a: Here I give the average anomaly for 2013.

3. year: This indicates the warmest year on record so far for that particular data set. Note that two of the data sets have 2010 as the warmest year and three have 1998 as the warmest year.

4. ano: This is the average of the monthly anomalies of the warmest year just above.

5. mon: This is the month where that particular data set showed the highest anomaly. The months are identified by the first three letters of the month and the last two numbers of the year. Note that this does not yet include records set so far in 2014 such as Hadsst3 in June, etc.

6. ano: This is the anomaly of the month just above.

7. y/m: This is the longest period of time where the slope is not positive given in years/months. So 16/2 means that for 16 years and 2 months the slope is essentially 0.

8. sig: This the first month for which warming is not statistically significant according to Nick’s criteria. The first three letters of the month are followed by the last two numbers of the year.

9. sy/m: This is the years and months for row 8. Depending on when the update was last done, the months may be off by one month.

10. McK: These are Dr. Ross McKitrick’s number of years for three of the data sets.

11. Jan: This is the January 2014 anomaly for that particular data set.

12. Feb: This is the February 2014 anomaly for that particular data set, etc.

20. ave: This is the average anomaly of all months to date taken by adding all numbers and dividing by the number of months. However if the data set itself gives that average, I may use their number. Sometimes the number in the third decimal place differs slightly, presumably due to all months not having the same number of days.

21. rnk: This is the rank that each particular data set would have if the anomaly above were to remain that way for the rest of the year. It may not, but think of it as an update 45 minutes into a game. Due to different base periods, the rank is more meaningful than the average anomaly.

If you wish to verify all of the latest anomalies, go to the following:

For UAH, version 5.5 was used since that is what WFT used, see: http://vortex.nsstc.uah.edu/public/msu/t2lt/tltglhmam_5.5.txt

For RSS, see: ftp://ftp.ssmi.com/msu/monthly_time_series/rss_monthly_msu_amsu_channel_tlt_anomalies_land_and_ocean_v03_3.txt

For Hadcrut4, see: http://www.metoffice.gov.uk/hadobs/hadcrut4/data/current/time_series/HadCRUT.4.3.0.0.monthly_ns_avg.txt

For Hadsst3, see: http://www.cru.uea.ac.uk/cru/data/temperature/HadSST3-gl.dat

For GISS, see: http://data.giss.nasa.gov/gistemp/tabledata_v3/GLB.Ts+dSST.txt

To see all points since January 2014 in the form of a graph, see the WFT graph below. Note that HadCRUT4 is the old version that has been discontinued. WFT does not show HadCRUT4.3 yet.

As you can see, all lines have been offset so they all start at the same place in January 2014. This makes it easy to compare January 2014 with the latest anomaly.

Appendix

In this part, we are summarizing data for each set separately.

RSS

The slope is flat since September 1, 1996 or 18 years, 1 month. (goes to September 30)

For RSS: There is no statistically significant warming since November 1992: CI from -0.001 to 1.816.

The RSS average anomaly so far for 2014 is 0.252. This would rank it as 7th place if it stayed this way. 1998 was the warmest at 0.55. The highest ever monthly anomaly was in April of 1998 when it reached 0.857. The anomaly in 2013 was 0.218 and it is ranked 10th.

UAH

The slope is flat since January 2005 or 9 years, 9 months. (goes to September using version 5.5 according to WFT)

For UAH: There is no statistically significant warming since May 1996: CI from -0.024 to 2.272. (This is using version 5.6 according to Nick’s program.)

The UAH average anomaly so far for 2014 is 0.196. This would rank it as 8th place if it stayed this way. 1998 was the warmest at 0.419. The highest ever monthly anomaly was in April of 1998 when it reached 0.662. The anomaly in 2013 was 0.197 and it is ranked 7th.

Hadcrut4.3

The slope is flat since January 2005 or 9 years, 9 months. (goes to September)

For Hadcrut4: There is no statistically significant warming since March 1997: CI from -0.016 to 1.157.

The Hadcrut4 average anomaly so far for 2014 is 0.560. This would rank it as 1st place if it stayed this way. 2010 was the warmest at 0.555. The highest ever monthly anomaly was in January of 2007 when it reached 0.835. The anomaly in 2013 was 0.492 and it is ranked 9th.

HadSST3

For HadSST3, the slope is not flat for any period. For HadSST3: There is no statistically significant warming since October 1994: CI from -0.015 to 1.722.

The HadSST3 average anomaly so far for 2014 is 0.477. This would rank it as 1st place if it stayed this way. 1998 was the warmest at 0.416 prior to 2014. The highest ever monthly anomaly was in July of 1998 when it reached 0.526. This is also prior to 2014. The anomaly in 2013 was 0.376 and it is ranked 6th.

GISS

The slope is flat since October 2004 or an even 10 years. (goes to September)

For GISS: There is no statistically significant warming since February 2000: CI from -0.060 to 1.326.

The GISS average anomaly so far for 2014 is 0.65. This would rank it as third place if it stayed this way. 2010 was the warmest at 0.66, so the 2014 anomaly is only 0.01 colder at this point. The highest ever monthly anomaly was in January of 2007 when it reached 0.92. The anomaly in 2013 was 0.60 and it is ranked 6th.

Conclusion

At present, the average HadCRUT4.3 anomaly for the first 9 months of 2014 is 0.560, which is above the previous record of 0.555 set in 2010. With a September anomaly of 0.595, a new record for 2014 is a definitive possibility. However, can we trust any HADCRUT4 record or might it simply be an artifact of questionable adjustments to the temperature record?