Guest Essay by Renee Hannon
Introduction
This post is a coarse screening assessment of HadCRUT4 global temperature anomalies to determine the impact, if any, of data quality and data coverage. There has been much discussion on WUWT about the quality of the Hadley temperature anomaly dataset since McLean’s Audit of the HadCRUT4 Global Temperature publication which is paywalled. I purchased a copy to see what all the hub-bub was about, and it is well worth the $8 in my view. Anthony Watts’ review of McLean’s findings and executive summary can be found here.
A key chart for critical study is McLean’s Figure 4.11 in his report. McLean suggests that HadCRUT4 data prior to 1950 is unreliable due to inadequate global coverage and high month-to-month temperature variability. For this post, I subdivided McLean’s findings into three groups shown with added shading: Good data which covers the years post-1950. During this period global data coverage is excellent at greater than 75% and month-to-month temperature variation is low. Questionable data occurs from 1880 to 1950. During this period global data coverage ranged from 40% to 70% with higher monthly temperature variations. Poor data is pre-1880 when global coverage ranged from 14 to 25% with extreme monthly temperature variations.
An obvious question is how does data coverage and data quality impact the technical evaluation and interpretation of HadCRUT4 global temperature anomalies?
Detrending Methods
The monthly HadCRUT4 global temperature anomaly dataset, referred to as temperature hereafter, was detrended to compare temperature trends, deviations and the impact of noise. The focus was on interannual temperature anomalies. Therefore, it is important to remove underlying longer-term trends from the data.
A common method to detrend temperature data uses a slope of linear regression over a period of time. The linear trend from 1850 to 2017 is used to detrend the HadCRUT4 temperature data shown in Figure 1a. A simple linear regression through the entire dataset leaves a secondary trend in the data. As seen in Figure 1a, the underlying longer-term signal is not completely removed, and the remaining data is not flattened. Several linear regression slopes are required to completely remove secondary trends.
Figure 1: Comparison of commonly used detrending methods on temperature datasets. Gray boxes show statistics associated with each method. a) HadCrut4 annual monthly temperature anomalies detrended using a single linear regression slope. b) HadCrut4 annual monthly temperature anomalies detrended using a 21-year running average. Several temperature anomalies greater than 2 σ are noted on the graph.
Another method used here is a centered running average of 21 years to detrend the temperature dataset. Since averaging degrades the tail end of the data, the last 10 years used a simple linear regression to extend past the running average. The running average with linear tail is subtracted from the HadCRUT4 temperature anomaly data. This removes the influence of underlying longer-term trends and the remaining data appears flattened. As shown in Figure 1b, temperature data detrended using this method produces an average temperature close to zero indicating a flattened trend. It is now easier to compare temperature spikes, amplitude of key events, or changes in the baseline noise.
Figure 1 enables comparison of the two detrending methods. Temperature standard deviations and ranges between the two methods are different. The detrended data using the running average has a smaller standard deviation and narrower temperature range of 0.4 degrees C versus 0.7 degrees C for the linear detrended data. This narrower range is more indicative of short-term climate variability compared to the linear detrended data which is a combination of both short-term and underlying longer-term trends.
Temperature Spike Frequency
At irregular intervals on the detrended data, a temperature spike lasting approximately 1-2 years emerges through the background noise such as in 1878, 1976, 1998, and 2016 to name a few. These warm and cold temperature spikes are attributed to El Niño and La Niña conditions of the El Niño-Southern Oscillation (ENSO) which effect temperature, wind and precipitation (Trenberth, et. al.). Figure 2 highlights the temperature spikes which exceed two standard deviations from the zero baseline.
Figure 2: HadCRUT4 detrended by running average. Red and blue dots show temperature anomaly spikes greater than 2 σ from the zero baseline.
Post-1950 there are only two cold and two warm spikes that exceed 2σ. The 1998 and 2016 warm spikes are associated with well documented El Nino conditions (Trenberth). The period from 1950 to 1998 is unusually devoid of warm temperature spikes. During this interval, there were several large volcanic eruptions at Mount Pinatubo in 1991, El Chichon in 1982 and Mount Agung in 1963 with continued weaker eruptions. Volcanoes produce sulfate aerosols that slow warming by reflecting incoming solar radiation and can cause global surface temperatures to decrease. Studies suggest that volcanic eruptions impact global temperatures about 1 year after the eruption and for approximate 2-3 years depending on the size of the eruptions.
The 1970’s were years of record cold temperatures in parts of North America. These years coincide with the ice age scare and the Cold Wave of 1977 when, in the U.S., the Ohio River froze solid for the first time since 1918 and snow fell in Miami with record low temperatures of less than 28 degrees F. La Nina conditions around 1956 resulted in a European cold wave. The years 1976/77 are commonly recognized as an abrupt climate shift from cooler to warmer global temperatures (Giese, et. al). Post-1976 there is a lack of cold temperature spikes.
In contrast, pre-1950 there are double the number of warm and cold temperature spikes compared to post-1950, especially between 1880 and 1920. One hypothesis is some warm and cold temperature spikes are the result of the increased noise in the data due to sparse data coverage as described by McLean.
Other interpretations suggest the temperature spikes are related to increased frequency of El Nino and La Nina events. For example, the period from 1900 to 1940 has been described as being dominated by the Era of El Ninos based on the Nino3.4 index summarized by Donald Rapp. However, during this period there were also three temperature cold spikes exceeding 2 𝜎 and several warm temperature spikes immediately preceding the referenced Era of El Ninos. Additional study is warranted to determine if some of the warm and cold temperature spikes are caused by noisy data or are the result of true El Nino and La Nina episodes. It is difficult to distinguish background noise from anomalous spikes during this period.
Temperature Spike Amplitudes
The amplitude of warm and cold temperature spikes on detrended data were compared from 1860 to 2017 and are shown in Figure 3. The 1878 warm temperature spike is the most outstanding anomaly in the past 160 years. This warm spike is an order of magnitude greater than any other temperature excursion from a zero baseline and occurs in McLean’s sparse data coverage zone with the highest amount of noisy data. The 1878 temperature warm spike may indeed be related to an El Nino episode. Several authors have reported large-scale drought in northern China and famine in Southern India during this time. However, the 1878 temperature could be over-estimated because of poor data coverage and quality. This outlier appears to be largely driven by the Northern Hemisphere global temperature record.
Figure 3: Comparison of warm and cold global surface temperature spikes from 1860 to 2017. Data is the detrended data shown in figure 1b using a running average. Spikes were aligned close to maximum departure. Note temperature scale on warm spikes is zoomed out due to the 1878 anomaly. Warm average temperature maximum does not include 1878.
Except for 1878, the other warm temperature spikes appear very similar and all peak just slightly above 0.2 degrees C. This casts additional suspicion upon the accuracy of the 1878 warm spike. Further, the cold spikes do not show any unusual trends. The 1976 cold spike is the coldest over the past 160 years, but not significantly so. Also, note both warm and cold spikes show nearly equal temperature departures of +0.23 and -0.22 degrees C from a zero baseline, respectively. Again, 1878 is an exception.
Observations
HadCRUT4 global temperature anomalies were assessed to determine the impact of varying data coverage and noise as described by McLean. It is recognized that sparser data coverage and increased noise have influenced interannual HadCRUT4 temperature anomalies pre-1950.
Interannual temperature spikes, both warm and cold, increase in frequency during the Questionable Data timeframe pre-1950. While several of these spikes may be associated with El Nino and La Nina events, others may be the result of increased noise within the data. This period of questionable data, especially the cluster of warm and cold spikes from 1880 to 1920, warrants further study.
The 1878 warm temperature spike is the most significant interannual temperature anomaly over the past 160 years. The temperature amplitude is almost double in magnitude over all other temperature amplitude spikes. This recorded temperature outlier is most likely erroneously high due to sparse global coverage and increased data noise.
Warm and cold spike maximums do not demonstrate any obvious increasing or decreasing trends from 1880 to present day, except for 1878. The recent 1998 and 2016 warm spikes are within one standard deviation of past warm spikes.
The HadCRUT4 global data pre-1950 may contain useful information but should be used with caution with an understanding that increases in noise due to poor data sampling may create false or enhanced temperature anomalies.
A future post will evaluate the data quality impact on the underlying HadCRUT4 decadal and multi-decadal trends.
Acknowledgements: Special thanks to Andy May and Donald Ince for reviewing and editing this article.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Why moving average filters are bad:
https://www.researchgate.net/post/What_are_the_disadvantages_of_moving_average_filter_when_using_it_with_time_series_data
You can have up to 25% error on peak analysis that you are doing because the first bounce in the stop band is about 25% of the original signal AND its phase is inverted.
It’s better to use a segmented linear regression for detrending.
It is better to use a proper low-pass filter.
some suggested here:
https://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/
Then subtract the filtered series (low frequency components) for the original one so that only the high-frequency components are left, correct?
Still, I wouldn’t expect much difference between different valid methods.
On the other hand, the frequency response of running means is bad.
Yes, HP = 1 – LP . Standard filter basics.
The key word is “valid” and RM is not a “valid” filter. Others have pros and cons you can debate about and chose. If you think RM is not too good but probably OK in practice, try some data. Here is one of the comparisons from my article.
Just look at what happens around the largest troughs and peaks: RM often inverts a peak. That is why I am so scathing about this article with aims to look at peaks after such a distorting filter has been used.
Look at the SSN example there too. The peak of the last cycle gets clobbered and the 13mo RM ends up with a later peak. Despite my bringing this to attention of the maintainers and their having admitted it was a problem, This is still the preferred filter they use.
RM is so ingrained no one seems to be able to let it go. The fact they you may actually have to do some work and learn seems to be an insurmountable barrier for most people, even Phd scientists.
Thank you Peter. Same point I have just made.
Looking at fig1b : the height of the 1878 spike is about 10% bigger after filtering. How does that work ?
What is the basis of that conclusion? Where is the expectation that all spikes should be the same size and if they are not they must be erroneous? That’s without noting that the spike was artificially increased by the crappy data processing.
This article is severely flawed from the outset and is not a basis for criticizing HadCRUFT4.
It would make more sense to ask you have a dataset which mixes incompatible land and sea data.
“During this period global data coverage is excellent at greater than 75% …”
Is this science, or comedy?
Oh for pity’s sake , here we go again.
RUNNING MEANS ARE CRAP. Find a decent filter.
Please read and understand : Data corruption by running mean “smoothers”
https://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/
The idea of using a high-pass filter for “detrending” is fine and you can subtract the result of a low-pass filter to achieve that end.
Averaging does not “degrades the tail end ” it removes it , since you can’t find the average. If you switch to a different method you are just introducing mess into the data. If there’s not data , live with it.
The rest of the article is not worth reading, since the author simple does not have the first idea what he is doing to the data. Many of the “spikes” he is looking will either a result of or at least corrupted by the data distortions of the running he starts with.
Look at the height of the 1978 peak above the previous trough. In fig 1b it is about 10% higher than in fig1a.
What’s up with that ?
The author then does on the conclude it is anomalously high. Some of that was his fault.
I agree using the previous trough, then the 1878 peak is slightly higher in fig 1b than 1a (0.58 versus 0.55 deg C), not quite 10%. If you use the following small trough, then the 1878 peak is the about the same in fig 1a and fig 1b (0.40 versus 0.39 deg C, respectively).
However, if you use the following large trough, then the 1878 peak is larger in fig 1a than 1b (0.61 versus 0.55 delta deg C, respectively). So which trough do you use, the previous or the following? That’s why it’s important to flatten the data to remove underlying trends which is what fig 1b attempts to do.
From your first para , it is clear that the 21y was sticking bump under the 1878 peak This is because the filter turned the peak into a trough which gets inverted it when subtracting the two. This is exactly the kind of thing I was pointing to.
You will see how big it is if you plot the gaussian and RM together.
It may be not big enough to change the identification of this as the largest peak, but ‘phew I got away with it’ is not a good basis for any analysis. How big the effect is depends on the nature of the data in relation to the window length of the filter. It is totally unnecessary distortion which can be avoided by using a proper filter. If you’ve equipped yourself with some better options, I’m glad you picked up on my comments.
Greg
Greg,
I specifically used 21y because it did not stick a bump under any peaks and invert them. Now if you use an RM less than 21 years, like 16 years for example, then you start seeing bumps.
https://imgur.com/51SVjTr
If you do a tripple RM ( close to a gaussian ) you will find there is some deviation around that period. You can also the see the amount of short term variation that gets past even a very long period RM.
http://woodfortrees.org/plot/hadcrut4gl/mean:252/mean:188/mean:140/plot/hadcrut4gl/mean:252
You will also see that there are RM runs persistently higher and lower over extended periods. All of this is very dependent on the data so you can not know whether there is a data inversion without comparing to a clean filter.
You can usually get a better “smoothing” with comparably shorter 3RM eg.
http://woodfortrees.org/plot/hadcrut4gl/mean:144/mean:107/mean:80/plot/hadcrut4gl/mean:252
I actually don’t see what the point of this article was, since there is no way to say whether there was more climate variability at the end of 19th c. or not , certainly not just by looking at this data. That would very well be a true difference at that time.
BEST land data goes back further and shows much greater swings in the early part of the record.
Climate does change, we have to live with that.
Greg,
I actually did look at triple RM, it is smoother but you lose even more data on the tail ends as shown in your graphs.
Greg,
Of course there are more sophisticated filters than a running average such as gaussian or loess filtering. As I mention in the first sentence of the post, this was a coarse screening analyses.
Switching methods only affected the last 10 years of data out of 160 years and impacts one peak, 2016.
Neither gaussian nor RM will provide you a value for the last 10y on a 21y window. So I don’t quite get how that change can affect 2016.
Applying an ad hoc , home made pastiche of techniques is not good. ( typical climatology games ).
Maybe you used loess which applies different filtering in order to fudge a result for the end sections, so I won’t use it.
A gaussian would be fairly typical choice for this kind of job. If you have sussed out how to use some well behaved filters, that will be a bonus for future work. Good man.
Here’s the HadCRUT global dataset isolated with a loess filter. Looks very similar to the running average. 1878 still looks interesting.
https://imgur.com/yc1rCHb
Speaking of bad data, the weather station at the Cornell Cooperative Farm south of Canton NY used to be a mile away at a higher altitude in the middle of a field on a flat bit of land. Now it is at the farm itself, 15′ from a parking lot , 20′ from a house, on the north slope of a hill.
None of that shit is in the official record of the site.
Global warming is nonsense.
As for the 1878 spike there is little doubt that the 1877-78 El Nino was an exceptionally strong one, quite possibly the strongest for several centuries. It caused very large famines in Asia and and extreme weather events in South America.
http://repositorio.uchile.cl/bitstream/handle/2250/125772/Aceituno_Patricio.pdf?sequence=1&isAllowed=y
The interesting part of that answer is how confident are you about the readings in 1878. You may want to look at history of temperature measurement. The Stevenson screen doesn’t even take it’s standard form in publication until 1884 and then there is some implementation lags. Your great famines may be nothing more that a moderate weather event exacerbated by other population or local events. The trouble with historical events with little actual reliable data is it is easy to make a story and that cuts both ways to both sides of the argument.
No matter how unreliable temperature readings were in 1878 they were probably not much worse than in 1877 or 1879, and meteorologists back then were well aware of the need for screening thermometers even though they used somewhat different screens.
“Your great famines may be nothing more that a moderate weather event exacerbated by other population or local events.”
They killed about 3 % of the world population, so probably were rather more than local events.
tty,
I completely agree with you and Scott (who commented upthread). There is very strong evidence that the temperature spike in 1878 (which may or may not be as big as shown) was largely a consequence of a major El Niño. This would be entirely consisent with the more recent temperature spikes, which also correspond with El Niños (as well as spikes in CO2 growth rate). Exceptions appear to reflect”interference” effects of volcanic eruptions or other issues, which may well include data busts or coverage bias. I think that Renee’s analysis is a good start and I do not wish to undermine that in any way. At the same time (as I commented above) we need to be very careful trying to interpret data that reflect multiple sources and/or the whole globe, when regional effects may be a significant complicating factor. For example the spike in 1878 would seem to show up best in the tropics SST data, as would be expected if the primary cause is El Niño.
Science 101 if you cannot accurately measure it you can only ‘guess it ‘ and calling it anything but a guess , such as theory or ‘projection ‘ and using models does not change the reality that you need to take these approaches because you are able to measure. in the first place.
Just a couple of nit-picky things:
“which effect temperature, wind and precipitation” looks like it should have “affect” instead of “effect.” While some things do create temperature, wind, and precipitation, I think you meant that they change them.
“This warm spike is an order of magnitude greater than any other temperature excursion from a zero baseline” made me think the spike would be 2° C larger since the others seem to be 0.2° C above the baseline. However, it was only 0.4° C above the baseline. I may be misusing the term, order of magnitude, but I thought it meant that we had something ten times smaller or larger than the items to which we’re comparing them since we use a base-10 system. My head cold might be affecting my thoughts on this, though.