This method may or may not have merit – readers are invited to test the merit of it themselves, the method is provided – Anthony
Guest essay by Ron Clutz
People in different places are wondering: What are temperatures doing in my area? Are they trending up, down or sideways? Of course, from official quarters, the answer is: The globe is warming, so it is safe to assume that your area is warming also.
But what if you don’t want to assume and don’t want to take someone else’s word for it. You can answer the question yourself if you take on board one simplifying concept:
“If you want to understand temperature changes, you should analyze temperature changes, not the temperatures.”
Analyzing temperature change is in fact much simpler and avoids data manipulations like anomalies, averaging, gridding, adjusting and homogenizing . Temperature Trend Analysis starts with recognizing that each microclimate is distinct with its unique climate patterns. So you work on the raw, unadjusted data produced, validated and submitted by local metrorologists. This is accessed in the HADCRUT3 dataset made public in July 2011.
The dataset includes 5000+ stations around the world, and only someone adept with statistical software running on a robust computer could deal with all of it. But the Met Office provides it in folders that cluster stations according to their WMO codes.
Anyone with modest spreadsheet skills and a notebook computer can deal with a set of stations of interest. Of course, there are missing datapoints which cause much work for temperature analysts. Those are not a big deal for trend analysis.
The method involves creating for each station a spreadsheet that calculates a trend for each month for all of the years recorded. Then the monthly trends are averaged together for a lifetime trend for that station. To be comparable to others, the station trend is normalized to 100 years. A summary sheet collects all the trends from all the sheets to provide trend analysis for the geographical area of interest.
I have built an Excel workbook to do this analysis, and as a proof of concept, I have loaded in temperature data for Kansas . Kansas is an interesting choice for several reasons:
1) It’s exactly in the middle of the US with no (significant) changes in elevation;
2) It has a manageable number of HCN stations:
3) It has been the subject lately of discussion about temperature processing effects;
4) Kansas legislators are concerned and looking for the facts; and
5) As a lad, my first awareness of extreme weather was the tornado in OZ, after which Dorothy famously said: “We’re not in Kansas anymore, Toto.”
I am not the first one to think of this. Richard Wakefield did similar analyses in Ontario years ago, and Lubos Motl did trend analysis on the entire HADCRUT3 in July 2011. With this simplying concept and a template, it is possible for anyone with modest spreadsheet skills and a notebook computer to answer how area temperatures are trending. I don’t claim this analysis is better than those done with multimillion dollar computers, but it does serve as a “sanity check” against exaggerated claims and hype.
For the Kansas example, we see that BEST shows on its climate page that the State has warmed 1.98 +/-0.14°C since 1960. That looks like temperatures will be another 2°C higher in the next 50 years, and we should be alarmed.
Well, the results from temperature trend analysis tell a different story.
From the summary page of the workbook:
| Area | State of Kansas, USA | |
| History | 1843 to 2011 | |
| Stations | 26 | |
| Average Length | 115 | Years |
| Average Trend | 0.70 | °C/Century |
| Standard Deviation | 0.45 | °C/Century |
| Max Trend | 1.89 | °C/Century |
| Min Trend | -0.04 | °C/Century |
So in the last century the average Kansas station has warmed 0.70+/-0.45°C , with at least one site cooling over that time. The +/- 0.45 deviation shows that climate is different from site to site even when all are located on the same prairie.
And the variability over the seasons is also considerable:
| Month | °C/century | Std Dev |
| Jan | 0.59 | 1.30 |
| Feb | 1.53 | 0.73 |
| Mar | 1.59 | 2.07 |
| Apr | 0.76 | 0.79 |
| May | 0.73 | 0.76 |
| June | 0.66 | 0.66 |
| July | 0.92 | 0.63 |
| Aug | 0.58 | 0.65 |
| Sep | -0.01 | 0.72 |
| Oct | 0.43 | 0.94 |
| Nov | 0.82 | 0.66 |
| Dec | 0.39 | 0.50 |
Note that February and March are warming strongly, while September is sideways . That’s good news for farming, I think.
Temperature change depends on your location and time of the year. The rate of warming is not extreme and if the next 100 years is anything like the last 100, in Kansas there will likely be less than a degree C added.
Final point:
When you look behind the summary page at BEST, it reports that the Kansas warming trend since 1910 is 0.75°C +/-0.08, close to what my analysis showed. So the alarming number at the top was not the accumulated rise in termperatures, it was the Rate for a century projected from 1960. The actual observed century rate is far less disturbing. And the variability across the state is considerable and is much more evident in the trend analysis. I had wanted to use raw data from BEST in this study, because some stations showed longer records there, but for comparable years, the numbers didn’t match with HADCRUT3.
Not only does this approach maintain the integrity of the historical record, it also facilitates what policy makers desperately need: climate outlooks based on observations for specific jurisdictions. Since the analysis is bottom-up, microclimate trends can be compiled together for any desired scope: municipal, district, region, province, nation, continent.
If there is sufficient interest in using this method and tool, I can provide some procedural instructions along with the template.
The Kansas Excel workbook is provided as an example: HADCRUT3 Kansas.xls
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Nice post Ron. I’m a DIY data nerd and this post is right up my alley. I’ve been doing some basic pivot table analysis on various datasets since last week.
If these next questions seem a bit on the under side of the over-under scale of eruditeness, bear in mind, I’m also a numskull.
Are there any datasets with temperatures captured from the darkest hour of the day? Or the time of day when direct sunlight has the least effect on surface temperature? Is that what TMIN represents?
Would trending the temp at the darkest hour of day give us a good idea of “trapped” warmth?
Matt L.
Good questions. Glad to hear you you do pivot tables; with that skill you can import from BEST data tables. There when you select all of the rows, you also get all of the columns even though the raw data is only in the first 3. With a pivot table on those 3, you get the array with years in the left column and months in the top row. Paste into the template and you are good to go.
Usually TMin occurs in the early am before sunrise, and TMax in the mid to late afternoon. Of course, especially in winter, a storm with sharp temperature changes or a heat wave could throw off the coolest or hottest time of the day. How fast or slow the air temperature cools overnight is a function of many things, but principally, humidity. Dry deserts can get freezing cold after sundown, while moist tropical places are mildly warm all night.
In the HADCRUT3 land dataset, the base data in a monthly average of daily averages:(TMin+Tmax)/2. . Some datasets do include maxs and mins as well, and these are quite informative, as mentioned by Wayne Delbeke upthread.
Richard Wakefield is my mentor on this so have a look here:
http://cdnsurfacetemps.wordpress.com/
Brian H I have evidence that the BOM wiped approximately 2 Mj/M^2/day off the insolation record across Australia for 2013. In the case of Sydney, the record tampering goes back to 1 Jul 2011. The high levels of insolation were an inconvenient truth that does not fit the BOM’s political agenda or the CAGW mantra. I used to have a healthy respect for the BOM but these days I question everything that they produce.
I’ve been asked why do this analysis. That’s really asking why are people wondering about the temperature trends. The underlying issue is: Are present temperatures and weather unusual and something we should worry about? In other words, “Is it weather or a changing climate?”
The answer in the place where you live depends on knowing your climate, that is the long-term weather trends. This analysis enables you to find from relevant station histories both the long-term trend and the normal range. With this context, you can judge for yourself between weather and climate change.
The workbook deals only with temperature monthly station averages (HADCRUT3 raw data), but can be expanded to TMaxes, TMins and precipitation, if you have access to those data.
BTW, for those interested, here is:
Why adjusting termperature records is a bad idea, by JR Wakefield
Matt L
Most weather stations record temperatures at hourly or quarter hourly obsevations. Calculating the average using these numbers is more accurate than (max+min)/2, but it also takes more computing time.
“Ron C. says:
July 13, 2014 at 8:32 am
Wayne Delbeke says:
July 12, 2014 at 11:25 am
That’s interesting. My next project is Canadian long term stations. Can you provide a link to raw station data from EC? I have data from HADCRUT3, and comparing could be interesting, also maybe EC records are longer.
HADCRUT3 and 4 use a homogenized version of canadian data
ENV canada data can be downloaded but its a nightmare. I wrote a R package to do this its called CHCN.
a few years back we did a station by station comparison between Env Canada and BEST input data.
There might be a few stations we miss.
ENV canada take their data and submit to GHCN-D as well as other collections.
so you will find duplication between ENV canada and GHCN-D
Your biggest problem is trusting ENV canada. And you have to ask “why didnt they submit this station to official collections” and why did they submit this station instead.
The biggest problem is that people assume there is a historical truth to get at here. There isnt.
There is a estimation of what the past was. The biggest assumption is that there is a thing called
‘raw’ data. What there is, in fact, is data that has been declared “adjusted” and “other” data.
You have some evidence that adjusted data is adjusted. somebody called it adjusted. you have no evidence that “other’ data or data that is reported to be raw, is in fact raw.
In short, skeptics for the most part are not skeptical enough when it comes to the distinction between raw and adjusted.
If we define raw as ‘what the sensor showed’
then we never have raw data. we never have what the sensor showed. we have what the observer wrote down. or we have what the electronic system reported or stored. Neither of these is exactly what the sensor showed. one can assume zero transmission, transcription errors.. but of course we know this assumption is wrong. In fact what the sensor showed is lost. in practice however people refer to stuff the observer wrote down as a raw report. and people refer to the electronic transmissions of sensor data as raw. but a good skeptic knows that neither is these is the same thing as what the sensor sensed. Otherwise we would believe an observer when they write down 100C in the dead of winter or we would believe the automated reporting system when it records 15000C.
So there are two classes of data. Data which somebody declares has been adjusted. and data that is either silent on the issue of adjustment or declares itself as raw. but we have no evidence that raw is in fact raw, that raw is in fact what the sensor showed in the past.
we can call it raw, and many of us do call it raw, but in fact if we are skeptical we dont know that it is raw. we only know that somebody called it raw, or that no record of adjustment is extant.
how do you proceed?
To do this right you first start by developing a method. Or using a known proven method.
You then test that method by using synthetic data to see any systematic bias in the method.
Then you take all the data you can find. You take a subset of that data. you hold the rest out
Then you use the data selected to create an estimation of the past.
You then use the out of sample data to test this estimate.
What you dont do is start with a method that you think is “good” you FIRST have to show the method is good APART FROM the data you are looking at. This post doesnt do that. grade; F
Then you dont start by cherry picking data or picking data according to some criteria that you have never tested. You have the universe of data. You select a random subset. you create a estimation.
Then you test other subsets. You will find cases where the estimate fails because of data issues
and cases where it fails because of estimation ( statistical model ) issues.
Thanks, Steven, for your comment.
As you say, measurements in the past have uncertainty, including today’s readings, which despite our best efforts to be accurate, will soon join the history and its uncertainty. I do hope we can agree that there is an important difference between a fact estimated to have already occurred in the past and a fact estimated to occur in the future.
You confirm what HADCRU also says, that NWSs like Environment Canada (EC) do their own adjusting and homogenizing in the interest of highest quality data.
Now I believe that NWSs are sincere and competently trying to get the record right. Since the global warming cause, unfortunately some of the people working with these data do have an agenda. A skeptic has an additional uncertainty: Are they altering the record to suit their cause?
As you say, the “raw data” is already processed, and hopefully improved by removing errors. In the past, we could assume that the adjustments would be randomly distributed, but today bias could be creeping in. So we look for data as close to the instruments as possible, with as little processing as possible, verified by meteorologists whose mission is to report the weather as it happens.
I have found access to the EC history of monthly averages for Canadian stations (including TMaxs and TMins BTW). The record has gone through 2 homogenizations, and I will take their word that they have only improved the accuracy by that process. As you say, getting the data into my workbook is labor-intensive, cut-and-paste stuff, and I can myself introduce errors if I take my eye off the ball.
But I am following a focused method rather than a global one, and so the numbers of stations make the project doable.
I am pleased with the proof of concept, having verified that the spreadsheet calculations are working as intended. I also take comfort that my result was so close to that obtained by BEST with a far more sophisticated method and tools.
Maybe not a supercomputer, how about an old Olivetti Programma 101. NASA used one in calculating orbits for the Moon missions.
In response to some comments above:
The rationale for this method of analysis is simple and compelling.
Temperature is an intrinsic quality of the object being measured. We can take the temperature of a rock and a pail of water, and the average of the two tells us . . . Nothing. However, if we have a time series, the two temperature trends do inform us about changes in the two environments where the rock and water are located.
Weather stations measure the temperature of air in thermal contact with the underlying terrain. Each site has a different terrain, and for a host of landscape features documented by Pielke Sr., the temperature patterns will differ, even in nearby locations. However, if we have station histories (and we do), then trends from different stations can be compared to see similarities and differences..
In summary, temperatures from different stations should not be interchanged or averaged, since they come from different physical realities. The trends can be compiled to tell us about the direction, extent and scope of temperature changes
In reply to Ron C in particular :-
Here are the results from the 32 stations with the best CRUTem4 coverage before and after 1850 :-
1700-2013 Maximum +0.81 (St. Petersburg) Minimum +0.04 (Paris le Bourget and Vilnius) Average +0.41 C/century
1700-1850 Maximum +0.97 (Vilnius) Minimum -1.09 (Kobenhavn) Average -0.39 C/century
1850-2013 Maximum +1.45 (St. Petersburg) Minimum -0.01 (Vilnius) Average +0.94 C/century
Difference Maximum +2.44 (Kobenhavn) Minimum -0.98 (Vilnius) Average +1.33 C/century
Richard,
Interesting results. Let me see if I get the meaning.
We have a trend over 313 years of 0.41 C/century. The first 150 years cooled at -0.39 C/century. The last 163 years warmed at +0.94 C/century.
Don’t know the time frame for your fourth line; I do note that Kobenhavn was the minimum in 1700-1850, and the maximum in the last interval, whatever it is. Can you clarify?
Also, am I right to think that these are European sites?
Reply to Ron C :-
Yes, those (32) sites with the most coverage both before and after 1850 were all European sites. Philadelphia started in 1758 but only had 106 years with 12 months of data.
The last line is just the change from the pre-1850 trend to the post-1850 trend. So Vilnius was +0.97 C/century before 1850 and -0.01 C after 1850, for a difference of -0.98 C/century. Kobenhavn was -1.09 C/century before 1850 and +1.35 C/century after 1850, for a difference of +2.44 C/century.
Richard,
We would expect cooling before 1850, and warming from 1850 to present. Usually the graphs show 0.5 C/century from end of LIA (1850). Vilnius is really counter to that pattern, and Kobenhavn appears quite volatile being the outlier in both periods.
Is there a way to have a copy of your workbook? I would like to look deeper at this.
Reply to Ron C :-
How would I get it to you ? I produced workbooks for the individual stations, calculated the annual values, then copied and pasted the annual values into the ‘master’ workbook.
Richard
That sounds like a lot of work. Can you briefly describe your method? I.e, raw data and source, sequence of calculations, end results?
Also, I have another idea. If you provide al list of station names (with WMO ID numbers, if handy), I can get GHCN data for the stations, and amalyze the trends since 1850.
Reply to Ron C :-
1. I went to http://www.cru.uea.ac.uk/cru/data/temperature/crutem4/station-data.htm then scrolled down to the bottom and downloaded
crutem4_asof020611_stns_used.zip Station data
crutem4_asof020611_stns_used_hdr.txt Header lines from above
2. The header lines formed the master workbook.
3. Sorted the header lines in order of ‘First Year’
4. For each station with ‘First Year’ 1850 or before, searched for it in the station data.
5. Copied and pasted the station data into a new text file.
6. Imported the text file into a new workbook.
7. Calculated the temperatures by dividing by 10.
8. Calculated the average temperature for each year.
9. Cleared each year’s average for which there was a month with ‘-99.9’ indicating missing value.
10. Plotted graph of yearly average, 10 year average and linear trend line for each station.
11. Copied and pasted each station’s yearly values (169 rows) into master workbook.
12. Calculated (for each station) :-
(a) Range = Last Year – First Year
(b) % Coverage = % Years with values
(c) Number of years with values to 1850
13. Selected those 32 stations with the top values in (a) (b) (c) – this eliminated Frankfurt, Philadelphia and Moscow for sparse coverage.
14. Calculated (for each of 32 stations) :-
(a) slope over whole period
(b) slope before 1850
(c) slope after 1850
(d) difference between (b) and (c)
WMO Station No.
103840
62600
24581
260630
160590
67000
66450
24851
24640
106380
724080
160600
12710
160811
71500
110120
61860
115180
107760
160950
110350
113200
267300
66310
107270
123750
276120
128390
128430
109620
108650
Station Name Country
BERLIN GERMANY
DE BILT NETHERLANDS
UPPSALA SWEDEN
ST.-PETERSBURG RUSSIA
Torino ITALY
Geneve-Cointrin SWITZERLAND
Basel-Binningen SWITZERLAND
STOCKHOLM SWEDEN
STOCKHOLM/BROMMA SWEDEN
FRANKFURT A MAIN GERMANY
PHILADELPHIA, PA USA
Torino ITALY
Trondheim/Vaernes NORWAY
Milano-Brera ITALY
PARIS/LE BOURGET FRANCE
Kremsmunster AUSTRIA
Koebenhavn DENMARK
PRAHA/RUZYNE AIRPORT CZECH REPUBLI
Regensburg GERMANY
Padova ITALY
Wien-Hohe Warte AUSTRIA
Innsbruck-Universita AUSTRIA
VILNIUS LITHUANIA
Bern-Zollikofen SWITZERLAND
Karlsruhe GERMANY
WARSAW-OKECIE POLAND
MOSKVA RUSSIA
Budapest – Lorinc Ai HUNGARY
Budapest – Lorinc Ai HUNGARY
Hohenpeisenberg GERMANY
Munchen-Stadt GERMANY
Thanks for the info and link.
Impressive–full marks for dedication and persistence.
Assuming you are paid normal hourly rates, it seems you have serious money from Big Oil behind you. ;>)
The only difference in method I can see is my workbook also does trends for each month to get at seasonality. Were your trend lines done on the monthly TAvg values? Did you also do Std. Dev.?
This source appears preferable to Met Office, since at least in the Kansas subset, some records inexplicably stopped in 1990. You have of course selected long-service sites still active, so this is not an issue. Others have written, Jeff Id for example, on the dying out on stations, a lot of them in US and also Canada.
I am also considering the NOAA GHCN dataset here:
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/
It has the advantage of being up to date (June 2014), but it does have some unwanted columns to work around. The data is called qcu (quality controlled unadjusted) so that looks good. They also provide TMaxs and TMins in separate additional files, which could be interesting.
Looks like you’ve put the ball in my court.
There may be a delay in getting this done as my Big Oil check seems lost in the mail.
Reply to Ron C :-
No pay, I just love Excel 🙂 The trend lines were on the calculated annual averages. No, I didn’t calculate standard deviation. As I say, if you want any of my files, and can tell me a way to get it to you, please let me know.
Normally, this is not recommended, but since my name is already posted above, here is my email for you to send to me, maybe the master and a sample, say Vilnius.
rclutz@videotron.ca
Reply to Ron C :- Files now sent. Hope you like them 🙂
Richard, thanks for sending the files. I have learned a lot today.
I sent back your Vilnius spreadsheet with a Vilnius TTA spreadsheet added.
I see now that you have a temperature averaging method, including averaging all stations together for each year. As I have said in the WUWT post, I do not support averaging (or exchanging) temperatures among different stations. For reasons I have explained above, only the trends should be averaged.
In the amended Vilnius file, you can see the difference between the two methods.
TTA results for Vilnius:
C/Century 0.0266
C/Lifetime 0.0625
C/1849 0.0006
Yrs. Lifetime 235
Ave Annual C 6.35
Std. Dev C 2.23
As you can see, the results are quite different. According to the TTA, Vilnius temperatures are trending sideways since 1850, gained 1C in the 73 years before 1850. The overall trend is 0.03 C/century, the Average Annual temperature is 6.5C, with Std Dev. Of +/- 2.2C.
Please do check these calculations to see if I have made any errors.
The table didn’t paste completely above:
C/Century 0.0266
C/Lifetime 0.0625
C/1849 0.0006
Yrs. Lifetime 235
Ave Annual C 6.35
Std. Dev C 2.23
Still doesn`t like the table
C/before 1849 1.0811
C/since 1849 0.0006