I’m happy to present this essay created from both sides of the aisle, courtesy of the two gentlemen below. Be sure to see the conclusion. I present their essay below with only a few small edits for spelling, format, and readability. Plus an image, a snapshot of global temperatures. – Anthony
By Zeke Hausfather and Steven Mosher
There are a variety of questions that people have about the calculation of a global temperature index. Questions that range from the selection of data and the adjustments made to data, to the actual calculation of the average. For some there is even a question about whether the measure makes any sense or not. It’s not possible to address all these questions in one short piece, but some of them can be addressed and reasonably settled. In particular we are in a position to answer the question about potential biases in the selection of data and biases in how that data is averaged.
To move the discussion onto the important matters of adjustments to data or, for example, UHI issues in the source data it is important to move forward on some answerable questions. Namely, do the methods for averaging data, the methods of the GISS, CRU and NCDC bias the result? There are a variety of methods for averaging spatial data, do the methods selected and implemented by the big three bias the result?
There has been a trend of late among climate bloggers on both sides of the divide to develop their own global temperature reconstructions. These have ranged from simple land reconstructions using GHCN data
(either v2.mean unadjusted data or v2.mean_adj data) to full land/ocean reconstructions and experiments with alternative datasets (GSOD , WDSSC , ISH ).
Bloggers and researchers who have developed reconstructions so far this year include:
Steven Mosher
And, just recently, the Muir Russell report
What is interesting is that the results from all these reconstructions are quite similar, despite differences in methodologies and source data. All are also quite comparable to the “big three” published global land temperature indices: NCDC , GISTemp , and CRUTEM .
[Fig 1]
The task of calculating global land temperatures is actually relatively simple, and the differences between reconstructions can be distilled down to a small number of choices:
1. Choose a land temperature series.
Ones analyzed so far include GHCN (raw and adjusted), WMSSC , GISS Step 0, ISH , GSOD , and USHCN (raw, time-of-observation adjusted, and F52 fully adjusted). Most reconstructions to date have chosen to focus on raw datasets, and all give similar results.
[Fig 2]
It’s worth noting that most of these datasets have some overlap. GHCN and WMSSC both include many (but not all) of the same stations. GISS Step 0 includes all GHCN stations in addition to USHCN stations and a selection of stations from Antartica. ISH and GSOD have quite a bit of overlap, and include hourly/daily data from a number of GHCN stations (though they have many, many more station records than GHCN in the last 30 years).
2. Choosing a station combination method and a normalization method.
GHCN in particular contains a number of duplicate records (dups) and multiple station records (imods) associated with a single wmo_id. Records can be combined at a single location and/or grid cell and converted into anomalies through the Reference Station Method (RSM), the Common Anomalies Method (CAM), and First Differences Method (FDM), or the Least Squares Method (LSM) developed by Tamino and Roman M . Depending on the method chosen, you may be able to use more stations with short records, or end up discarding station records that do not have coverage in a chosen baseline period. Different reconstructions have mainly made use of CAM (Zeke, Mosher, NCDC) or LSM (Chad, Jeff Id/Roman M, Nick Stokes, Tamino). The choice between the two does not appear to have a significant effect on results, though more work could be done using the same model and varying only the combination method.
[Fig 3]
3. Choosing an anomaly period.
The choice of the anomaly period is particularly important for reconstructions using CAM, as it will determine the amount of usable records. The anomaly period can also result in odd behavior of anomalies if it is too short, but in general the choice makes little difference to the results. In the figure that follows Mosher shows the difference between picking an anomaly period like CRU does, 1961-1990, and picking an anomaly period that maximizes the number monthly reports in a 30 year period. The period that maximizes the number of monthly reports over a 30 year period turns out to be 1952-1983. 1953-82 (Mosher). No other 30 year period in GHCN has more station reports. This refinement, however, has no appreciable impact.
[Fig 4]
4. Gridding methods.
Most global reconstructions use 5×5 grid cells to ensure good spatial coverage of the globe. GISTemp uses a rather different method of equal-size grid cells. However, the choice between the two methods does not seem to make a large difference, as GISTemp’s land record can be reasonably well-replicated using 5×5 grid cells. Smaller resolution grid cells can improve regional anomalies, but will often result in spatial bias in the results, as there will be large missing areas during periods when or in locations when station coverage is limited. For the most part, the choice is not that important, unless you choose extremely large or small gridcells. In the figure that follows Mosher shows that selecting a smaller grid does not impact the global average or the trend over time. In his implementation there is no averaging or extrapolation over missing grid cells. All the stations within a grid cell are averaged and then the entire globe is averaged. Missing cells are not imputed with any values.
[Fig 5]
5. Using a land mask.
Some reconstructions (Chad, Mosh, Zeke, NCDC) use a land mask to weight each grid cell by its respective land area. The land mask determines how much of a given cell ( say 5×5) is actually land. A cell on a coast, thus, could have only a portion of land in it. The land mask corrects for this. The percent of land in a cell is constructed from a 1 km by 1 km dataset. The net effect of land masking is to increase the trend, especially in the last decade. This factor is the main reason why recent reconstructions by Jeff Id/Roman M and Nick Stokes are a bit lower than those by Chad, Mosh, and Zeke.
[Fig 6]
6. Zonal weighting.
Some reconstructions (GISTemp, CRUTEM) do not simply calculate the land anomaly as the size-weighted average of all grid cells covered. Rather, they calculate anomalies for different regions of the globe (each hemisphere for CRUTEM, 90°N to 23.6°N, 23.6°N to 23.6°S and 23.6°S to 90°S for GISTemp) and create a global land temp as the weighted average of each zone (weightings 0.3, 0.4 and 0.3, respectively for GISTemp, 0.68 × NH + 0.32 × SH for CRUTEM). In both cases, this zonal weighting results in a lower land temp record, as it gives a larger weight to the slower warming Southern Hemisphere.
[Fig 7]
These steps will get you a reasonably good global land record. For more technical details, look at any of the many http://noconsensus.wordpress.com/2010/03/25/thermal-hammer-part-deux/different http://residualanalysis.blogspot.com/2010/03/ghcn-processor-11.html models http://rankexploits.com/musings/2010/a-simple-model-for-spatially-weighted-temp-analysis/ that have been publicly http://drop.io/treesfortheforest released http://moyhu.blogspot.com/2010/04/v14-with-maps-conjugate-gradients.html
].
7. Adding in ocean temperatures.
The major decisions involved in turning a land reconstruction into a land/ocean reconstruction are choosing a SST series (HadSST2, HadISST/Reynolds, and ERSST have been explored http://rankexploits.com/musings/2010/replication/ so far), gridding and anomalizing the series chosen, and creating a combined land-ocean temp record as a weighted combination of the two. This is generally done by: global temp = 0.708 × ocean temp + 0.292 × land temp.
[Fig 8]
8. Interpolation.
Most reconstructions only cover 5×5 grid cells with one or more station for any given month. This means that any areas without station coverage for any given month are implicitly assumed to have the global mean temperature. This is arguably problematic, as high-latitude regions tend to have the poorest coverage and are generally warming faster than the global average.
GISTemp takes a somewhat different approach, assigning a temperature anomaly to all missing grid boxes located within 1200 km of one or more stations that do have defined temperature anomalies. They rationalize this based on the fact that “temperature anomaly patterns tend to be large scale, especially at middle and high latitudes.” Because GISTemp excludes SST readings from areas with sea ice cover, this leads to the extrapolation of land anomalies to ocean areas, particularly in the Arctic. The net effects of interpolation on the resulting GISTemp record is small but not insignificant, particularly in recent years. Indeed, the effect of interpolation is the main reason why GISTemp shows somewhat different trends from HadCRUT and NCDC over the past decade.
[Fig 9]
9. Conclusion
As noted above there are many questions about the calculation of a global temperature index. However, some of those questions can be fairly answered and have been fairly answered by a variety of experienced citizen researchers from all sides of the debate. The approaches used by GISS and CRU and NCDC do not bias the result in any way that would erase the warming we have seen since 1880. To be sure there are minor differences that depend upon the exact choices one makes, choices of ocean data sets, land data sets, rules for including stations, rules for gridding, area weighting approaches, but all of these differences are minor when compared to the warming we see.
That suggests a turn in the discussion to the matters which have not been as thoroughly investigated by independent citizen researchers on all sides:
A turn to the question of data adjustments and a turn to the question of metadata accuracy and finally a turn to the question about UHI. Now, however, the community on all sides of the debate has a set of tools to address these questions.
“If all the warming is raised min’s caused by UHI and station dropout, we have zero to worry about.
So, Zeke and Steve, is there a reliable record for min/max anywhere?”
1. station dropout is not an issue as far as Bias goes.
2. GHCN has daily min/max. others have as well.
George E. Smith says:
July 13, 2010 at 3:27 pm (Edit)
On #4 Gridding Methods; just what the heck grid are we talking about ?
I thought both Hadcrud, and GISStemp used data from some small number of thermometers spread around the world; so what the heck are these grid cells and what do 5 x 5 and 3 x 3 grids mean ?
************************************************************
There are roughly 7000-7200 hundred stations in GHCN.
they provide data from 1701 to the present.
They are distributed around the globe.
In my case I start by selecting stations using the following rule: A station must have
15 complete years in the 1953-1982 rime frame. I can vary this rule. and determine the sensitivity to that selection rule.
Then the stations are combine according the the latlon grid they are in. I can combine all stations in a 1 deg, 2 degree, 3 degree and 5 degree bin.
Each station is standardized by subtracting its mean during the 1953-82 period.
The stations are averaged per grid (lat lon)
Then all the grids are weighted by their area on the sphere. ( with/without land mask)
Then all grids are combined and an area weighted series.
By varying the size of the grid we can see the effect of gridding approaches.
Gisstemp uses an equal area grid. I use regular grids ( lat lon) and weight by area
DirkH says:
July 13, 2010 at 3:50 pm (Edit)
“And right after 82, a steep temp rise (and declining thermometer population).
Not accusing anyone of anything, just saying.”
That’s largely incorrect.
1. the answer is not changed by station drop off and
2. The stations do not drop that quickly after 1982.
3. If I change the period you will still get a rise.
4. If I pick 1000 stations for the whole period you will still see a rise.
here are the Stations that report BY MONTH ( months expressed as fractions)
1950 3342
1950.08333333333 3360
1950.16666666667 3368
1950.25 3393
1950.33333333333 3394
1950.41666666667 3382
1950.5 3401
1950.58333333333 3413
1950.66666666667 3407
1950.75 3405
1950.83333333333 3421
1950.91666666667 3425
1951 3974
1951.08333333333 3972
1951.16666666667 3974
1951.25 4001
1951.33333333333 4012
1951.41666666667 4016
1951.5 4031
1951.58333333333 4044
1951.66666666667 4062
1951.75 4065
1951.83333333333 4069
1951.91666666667 4061
1952 4119
1952.08333333333 4125
1952.16666666667 4135
1952.25 4137
1952.33333333333 4132
1952.41666666667 4156
1952.5 4172
1952.58333333333 4174
1952.66666666667 4184
1952.75 4197
1952.83333333333 4195
1952.91666666667 4190
1953 4238
1953.08333333333 4250
1953.16666666667 4259
1953.25 4266
1953.33333333333 4269
1953.41666666667 4288
1953.5 4286
1953.58333333333 4292
1953.66666666667 4281
1953.75 4283
1953.83333333333 4291
1953.91666666667 4299
1954 4343
1954.08333333333 4346
1954.16666666667 4351
1954.25 4347
1954.33333333333 4351
1954.41666666667 4366
1954.5 4361
1954.58333333333 4360
1954.66666666667 4378
1954.75 4375
1954.83333333333 4385
1954.91666666667 4374
1955 4362
1955.08333333333 4355
1955.16666666667 4370
1955.25 4363
1955.33333333333 4368
1955.41666666667 4370
1955.5 4371
1955.58333333333 4371
1955.66666666667 4364
1955.75 4378
1955.83333333333 4375
1955.91666666667 4382
1956 4419
1956.08333333333 4407
1956.16666666667 4420
1956.25 4424
1956.33333333333 4426
1956.41666666667 4420
1956.5 4430
1956.58333333333 4427
1956.66666666667 4435
1956.75 4429
1956.83333333333 4427
1956.91666666667 4432
1957 4435
1957.08333333333 4437
1957.16666666667 4450
1957.25 4456
1957.33333333333 4459
1957.41666666667 4456
1957.5 4474
1957.58333333333 4471
1957.66666666667 4453
1957.75 4455
1957.83333333333 4472
1957.91666666667 4465
1958 4486
1958.08333333333 4484
1958.16666666667 4488
1958.25 4510
1958.33333333333 4499
1958.41666666667 4499
1958.5 4499
1958.58333333333 4498
1958.66666666667 4503
1958.75 4507
1958.83333333333 4501
1958.91666666667 4504
1959 4525
1959.08333333333 4534
1959.16666666667 4536
1959.25 4538
1959.33333333333 4527
1959.41666666667 4537
1959.5 4540
1959.58333333333 4543
1959.66666666667 4541
1959.75 4541
1959.83333333333 4540
1959.91666666667 4547
1960 4576
1960.08333333333 4601
1960.16666666667 4606
1960.25 4616
1960.33333333333 4614
1960.41666666667 4599
1960.5 4606
1960.58333333333 4595
1960.66666666667 4614
1960.75 4619
1960.83333333333 4629
1960.91666666667 4621
1961 4776
1961.08333333333 4781
1961.16666666667 4794
1961.25 4807
1961.33333333333 4821
1961.41666666667 4823
1961.5 4811
1961.58333333333 4824
1961.66666666667 4820
1961.75 4817
1961.83333333333 4831
1961.91666666667 4817
1962 4877
1962.08333333333 4879
1962.16666666667 4892
1962.25 4875
1962.33333333333 4888
1962.41666666667 4883
1962.5 4891
1962.58333333333 4897
1962.66666666667 4921
1962.75 4918
1962.83333333333 4932
1962.91666666667 4927
1963 4994
1963.08333333333 5010
1963.16666666667 5004
1963.25 5020
1963.33333333333 5014
1963.41666666667 5027
1963.5 5025
1963.58333333333 5033
1963.66666666667 5037
1963.75 5028
1963.83333333333 5038
1963.91666666667 5030
1964 5066
1964.08333333333 5071
1964.16666666667 5087
1964.25 5072
1964.33333333333 5063
1964.41666666667 5070
1964.5 5077
1964.58333333333 5068
1964.66666666667 5068
1964.75 5077
1964.83333333333 5061
1964.91666666667 5049
1965 5143
1965.08333333333 5147
1965.16666666667 5143
1965.25 5143
1965.33333333333 5151
1965.41666666667 5143
1965.5 5147
1965.58333333333 5146
1965.66666666667 5143
1965.75 5151
1965.83333333333 5161
1965.91666666667 5156
1966 5176
1966.08333333333 5186
1966.16666666667 5196
1966.25 5195
1966.33333333333 5195
1966.41666666667 5184
1966.5 5192
1966.58333333333 5194
1966.66666666667 5186
1966.75 5194
1966.83333333333 5193
1966.91666666667 5186
1967 5192
1967.08333333333 5198
1967.16666666667 5196
1967.25 5193
1967.33333333333 5202
1967.41666666667 5193
1967.5 5201
1967.58333333333 5203
1967.66666666667 5199
1967.75 5199
1967.83333333333 5193
1967.91666666667 5178
1968 5202
1968.08333333333 5199
1968.16666666667 5202
1968.25 5206
1968.33333333333 5200
1968.41666666667 5204
1968.5 5205
1968.58333333333 5197
1968.66666666667 5209
1968.75 5208
1968.83333333333 5208
1968.91666666667 5193
1969 5211
1969.08333333333 5218
1969.16666666667 5221
1969.25 5223
1969.33333333333 5222
1969.41666666667 5219
1969.5 5217
1969.58333333333 5213
1969.66666666667 5215
1969.75 5186
1969.83333333333 5196
1969.91666666667 5188
1970 5176
1970.08333333333 5193
1970.16666666667 5179
1970.25 5194
1970.33333333333 5171
1970.41666666667 5181
1970.5 5164
1970.58333333333 5186
1970.66666666667 5188
1970.75 5186
1970.83333333333 5192
1970.91666666667 5185
1971 5054
1971.08333333333 5065
1971.16666666667 5045
1971.25 5050
1971.33333333333 5058
1971.41666666667 5049
1971.5 5033
1971.58333333333 5045
1971.66666666667 5064
1971.75 5050
1971.83333333333 5044
1971.91666666667 5023
1972 5039
1972.08333333333 5045
1972.16666666667 5047
1972.25 5037
1972.33333333333 5037
1972.41666666667 5043
1972.5 5033
1972.58333333333 5030
1972.66666666667 5035
1972.75 5019
1972.83333333333 5023
1972.91666666667 5019
1973 5020
1973.08333333333 5034
1973.16666666667 5024
1973.25 5007
1973.33333333333 5022
1973.41666666667 5006
1973.5 5008
1973.58333333333 5014
1973.66666666667 5005
1973.75 4998
1973.83333333333 4959
1973.91666666667 4986
1974 5019
1974.08333333333 5023
1974.16666666667 5016
1974.25 5004
1974.33333333333 4992
1974.41666666667 5004
1974.5 5001
1974.58333333333 4993
1974.66666666667 4985
1974.75 4991
1974.83333333333 4986
1974.91666666667 4967
1975 4974
1975.08333333333 4988
1975.16666666667 4960
1975.25 4982
1975.33333333333 4961
1975.41666666667 4966
1975.5 4951
1975.58333333333 4942
1975.66666666667 4941
1975.75 4945
1975.83333333333 4929
1975.91666666667 4924
1976 4873
1976.08333333333 4892
1976.16666666667 4872
1976.25 4866
1976.33333333333 4873
1976.41666666667 4879
1976.5 4867
1976.58333333333 4864
1976.66666666667 4867
1976.75 4860
1976.83333333333 4869
1976.91666666667 4849
1977 4850
1977.08333333333 4864
1977.16666666667 4863
1977.25 4843
1977.33333333333 4847
1977.41666666667 4845
1977.5 4823
1977.58333333333 4835
1977.66666666667 4831
1977.75 4829
1977.83333333333 4816
1977.91666666667 4825
1978 4838
1978.08333333333 4861
1978.16666666667 4853
1978.25 4844
1978.33333333333 4864
1978.41666666667 4845
1978.5 4829
1978.58333333333 4818
1978.66666666667 4810
1978.75 4794
1978.83333333333 4779
1978.91666666667 4777
1979 4779
1979.08333333333 4779
1979.16666666667 4776
1979.25 4770
1979.33333333333 4765
1979.41666666667 4772
1979.5 4741
1979.58333333333 4760
1979.66666666667 4753
1979.75 4746
1979.83333333333 4741
1979.91666666667 4713
1980 4731
1980.08333333333 4741
1980.16666666667 4742
1980.25 4730
1980.33333333333 4726
1980.41666666667 4741
1980.5 4745
1980.58333333333 4746
1980.66666666667 4749
1980.75 4746
1980.83333333333 4728
1980.91666666667 4700
1981 4448
1981.08333333333 4498
1981.16666666667 4519
1981.25 4501
1981.33333333333 4490
1981.41666666667 4513
1981.5 4483
1981.58333333333 4497
1981.66666666667 4518
1981.75 4457
1981.83333333333 4436
1981.91666666667 4431
1982 4286
1982.08333333333 4326
1982.16666666667 4292
1982.25 4289
1982.33333333333 4282
1982.41666666667 4297
1982.5 4266
1982.58333333333 4275
1982.66666666667 4267
1982.75 4265
1982.83333333333 4235
1982.91666666667 4234
1983 4220
1983.08333333333 4265
1983.16666666667 4273
1983.25 4265
1983.33333333333 4244
1983.41666666667 4266
1983.5 4234
1983.58333333333 4244
1983.66666666667 4238
1983.75 4224
1983.83333333333 4205
1983.91666666667 4191
1984 4168
1984.08333333333 4168
1984.16666666667 4154
1984.25 4155
1984.33333333333 4152
1984.41666666667 4164
1984.5 4164
1984.58333333333 4160
1984.66666666667 4116
1984.75 4131
1984.83333333333 4137
1984.91666666667 4098
1985 4098
1985.08333333333 4103
1985.16666666667 4113
1985.25 4157
1985.33333333333 4117
1985.41666666667 4122
1985.5 4107
1985.58333333333 4085
1985.66666666667 4099
1985.75 4095
1985.83333333333 4032
1985.91666666667 4003
1986 4032
1986.08333333333 4056
1986.16666666667 4045
1986.25 4061
1986.33333333333 4083
1986.41666666667 4031
1986.5 4059
1986.58333333333 4036
1986.66666666667 4021
1986.75 3988
1986.83333333333 3980
1986.91666666667 3953
1987 4018
1987.08333333333 4004
1987.16666666667 4015
1987.25 3996
1987.33333333333 3977
1987.41666666667 3992
1987.5 3995
1987.58333333333 3977
1987.66666666667 3999
1987.75 4017
1987.83333333333 3975
1987.91666666667 3975
1988 3992
1988.08333333333 3963
1988.16666666667 3992
1988.25 3972
1988.33333333333 3953
1988.41666666667 3936
1988.5 3941
1988.58333333333 3977
1988.66666666667 3945
1988.75 3949
1988.83333333333 3965
1988.91666666667 3959
1989 3899
1989.08333333333 3905
1989.16666666667 3887
1989.25 3922
1989.33333333333 3904
1989.41666666667 3844
1989.5 3894
1989.58333333333 3847
1989.66666666667 3864
1989.75 3848
1989.83333333333 3845
1989.91666666667 3830
1990 3646
1990.08333333333 3670
1990.16666666667 3675
1990.25 3671
1990.33333333333 3538
1990.41666666667 3495
1990.5 3367
1990.58333333333 3367
1990.66666666667 3325
1990.75 3424
1990.83333333333 3388
1990.91666666667 3367
1991 2664
1991.08333333333 2601
1991.16666666667 2606
1991.25 2649
1991.33333333333 2591
1991.41666666667 2570
1991.5 2541
1991.58333333333 2535
1991.66666666667 2683
1991.75 2573
1991.83333333333 2534
1991.91666666667 2512
1992 2564
1992.08333333333 2524
1992.16666666667 2475
1992.25 2587
1992.33333333333 2582
1992.41666666667 2576
1992.5 2538
1992.58333333333 2596
1992.66666666667 2538
1992.75 2565
1992.83333333333 2463
1992.91666666667 2413
1993 2441
1993.08333333333 2395
1993.16666666667 2391
1993.25 2409
1993.33333333333 2361
1993.41666666667 2372
1993.5 2418
1993.58333333333 2474
1993.66666666667 2439
1993.75 2470
1993.83333333333 2323
1993.91666666667 2395
1994 2426
1994.08333333333 2395
1994.16666666667 2402
1994.25 2389
1994.33333333333 2379
1994.41666666667 2325
1994.5 2421
1994.58333333333 2370
1994.66666666667 2381
1994.75 2366
1994.83333333333 2268
1994.91666666667 2208
1995 2285
1995.08333333333 2183
1995.16666666667 2322
1995.25 2308
1995.33333333333 2250
1995.41666666667 2230
1995.5 2212
1995.58333333333 2277
1995.66666666667 2284
1995.75 2212
1995.83333333333 2266
1995.91666666667 2238
1996 2362
1996.08333333333 2306
1996.16666666667 2262
1996.25 2353
1996.33333333333 2374
1996.41666666667 2366
1996.5 2337
1996.58333333333 2315
1996.66666666667 2331
1996.75 2309
1996.83333333333 2294
1996.91666666667 2352
1997 2316
1997.08333333333 2377
1997.16666666667 2367
1997.25 2354
1997.33333333333 2362
1997.41666666667 2281
1997.5 2294
1997.58333333333 2337
1997.66666666667 2341
1997.75 2360
1997.83333333333 2317
1997.91666666667 2296
1998 2326
1998.08333333333 2358
1998.16666666667 2335
1998.25 2350
1998.33333333333 2287
1998.41666666667 2353
1998.5 2325
1998.58333333333 2354
1998.66666666667 2345
1998.75 2301
1998.83333333333 2268
1998.91666666667 2277
1999 2300
1999.08333333333 2324
1999.16666666667 2364
1999.25 2305
1999.33333333333 2341
1999.41666666667 2334
1999.5 2343
1999.58333333333 2315
1999.66666666667 2315
1999.75 2305
1999.83333333333 2285
1999.91666666667 2278
2000 2282
2000.08333333333 2271
2000.16666666667 2282
2000.25 2269
2000.33333333333 2255
2000.41666666667 2319
2000.5 2225
2000.58333333333 2166
2000.66666666667 2280
2000.75 2288
2000.83333333333 2254
2000.91666666667 2238
2001 2240
2001.08333333333 2261
2001.16666666667 2273
2001.25 2221
2001.33333333333 2204
2001.41666666667 2279
2001.5 2223
2001.58333333333 2234
2001.66666666667 2271
2001.75 2282
2001.83333333333 2250
2001.91666666667 2242
2002 2224
2002.08333333333 2184
2002.16666666667 2270
2002.25 2289
2002.33333333333 2266
2002.41666666667 2272
2002.5 2276
2002.58333333333 2220
2002.66666666667 2281
2002.75 2301
2002.83333333333 2146
2002.91666666667 2141
2003 2291
2003.08333333333 2252
2003.16666666667 2233
2003.25 2289
2003.33333333333 2285
2003.41666666667 2271
2003.5 2237
2003.58333333333 2189
2003.66666666667 2261
2003.75 2289
2003.83333333333 2243
2003.91666666667 2238
2004 2177
2004.08333333333 2231
2004.16666666667 2241
2004.25 2248
2004.33333333333 2228
2004.41666666667 2212
2004.5 2242
2004.58333333333 2045
2004.66666666667 2080
2004.75 2064
2004.83333333333 2070
2004.91666666667 2075
2005 2074
2005.08333333333 2078
2005.16666666667 2083
2005.25 2095
2005.33333333333 2096
2005.41666666667 2071
2005.5 2047
2005.58333333333 2040
2005.66666666667 2088
2005.75 2037
2005.83333333333 2056
2005.91666666667 2140
2006 2091
2006.08333333333 2189
2006.16666666667 2161
2006.25 1165
2006.33333333333 1166
2006.41666666667 1154
2006.5 1152
2006.58333333333 1114
2006.66666666667 1181
2006.75 1161
2006.83333333333 1130
2006.91666666667 1141
2007 1164
2007.08333333333 1147
2007.16666666667 1155
2007.25 1162
2007.33333333333 1160
2007.41666666667 1195
2007.5 1176
2007.58333333333 1168
2007.66666666667 1166
2007.75 1197
2007.83333333333 1170
2007.91666666667 1159
2008 1000
2008.08333333333 994
2008.16666666667 993
2008.25 1011
2008.33333333333 1018
2008.41666666667 1013
2008.5 991
2008.58333333333 1013
2008.66666666667 1272
2008.75 1261
2008.83333333333 1293
2008.91666666667 1075
2009 1286
2009.08333333333 1289
2009.16666666667 1271
2009.25 1219
2009.33333333333 1256
2009.41666666667 1279
2009.5 1275
2009.58333333333 1271
2009.66666666667 1266
2009.75 1274
2009.83333333333 1279
2009.91666666667 1269
CE
“Had to chuckle at that one. For all the methodology choices that turn out not to matter much, that’s got to be one of the least consequential. No?”
I’m not so sure. For this excercise I used simple averaging. I will probably post on the other methods. I did not implement hansens RSM, but I probably will. The differences are small.. <.1C. It was a bitch. Also I think Roman is looking at that issue as well
“Would certain altitudes be over-represented in the cell’s average, and would it matter?”
no. it would not matter. each station is standardized by subtracting its mean.
“This is a well written overview of the problem by Steve Mosher!”
Asute readers (like my roommate charles the moderator) should be able to pick out the relatively minor contribution I made to the words. Zeke was kind enough to take my numbers and produce text and charts to describe what SEVERAL people have done.
without Roman’s help, jeffids help, stevemc, Ron, chad, zeke, and a bunch of people on the R help list this thing would not be done.
Mosher: “2. GHCN has daily min/max. others have as well.”
2. GHCN has daily min/max. others have as well.
GHCN v2 max/min for Canada ( for example) drops from 500-600 stations to 20-30 in the 1990s. Its a joke.
The raw data shows the max is cooling by the way.
For examples this is raw GHCN V2 max data for June/July/Aug ranked in 10 year periods for the USA. The 30s were the hottest (max) period.
Decade JJA
1930 – 1939 30.24
1929 – 1938 30.19
1931 – 1940 30.17
1928 – 1937 30.07
1932 – 1941 30.06
1933 – 1942 29.99
1934 – 1943 29.93
1927 – 1936 29.87
1935 – 1944 29.77
1925 – 1934 29.71
1926 – 1935 29.68
1936 – 1945 29.65
1924 – 1933 29.53
1917 – 1926 29.49
1893 – 1902 29.49
1952 – 1961 29.47
1916 – 1925 29.46
1892 – 1901 29.46
1913 – 1922 29.44
1951 – 1960 29.42
1937 – 1946 29.41
1923 – 1932 29.40
1994 – 2003 29.40
EC: That’s what’s bothered me. How can we have AGW that exempts the US
There are a couple of answers to that question.
The one I’ll go with is that, according to the IPCC AR4 WG1,
the “A” part of “GW” is only, in just the last several decades,
beginning to stand out from other natural forcings.
http://www.ipcc.ch/publications_and_data/ar4/wg1/en/spmsspm-understanding-and.html
In other words, the earlier highs were dominated by natural causes. The current highs are to some degree greater than zero a product of increasing CO2 (as determined by modeling analysis )
.DR says:
July 13, 2010 at 5:18 pm (Edit)
So once again the accuracy and precision of the data is still not addressed.
Sorry, I still fail to see the significance of reproducing the same results over and over without investigating the quality of measurement at each individual station and also the inclusion of land use change (which alters the climate over time i.e. boundary layer ) and UHI. It would seem those are the most important factors that need to be ironed out.
****************
Its very simple. You cant do that analysis competently without a tool. The tools were being questioned. So several people built their own tools. SHowing the tool doesnt bias the answer is the good first step for an analyst.
CE
Thanks for the kind remarks..
BillyBob:
“GHCN v2 max/min for Canada ( for example) drops from 500-600 stations to 20-30 in the 1990s. Its a joke.”
Generally speaking I try to avoid making statements like this without backing it up with solid analysis. Its very simple to test. take the average with only those 20-30 over the whole time period and compare. or take the period 1940-1990 and randomly select any 30, do this several times. Canada being at high latitude is more
homogenous ( highly correlated in space) than lower latitudes.
At the highest latitude (90degrees) its cold in every direction you choose to walk.
High spatial correlation means fewer stations are required to capture the signal.
““Consistent with,” perhaps. CO2 went up and temperatures went up. This does not prove causation. There have been periods in the past where temperatures went down and CO² remained high.”
Causation is never proved. the mechansim by whch GHG warm the planet is a physical theory. A physical theory that engineers use in the everyday construction of devices that we all enjoy. Its a physical theory which, Monkton, Lindzen, Christy, spencer, all agree with. we bicker over the MAGNITUDE of the effect, but no serious skeptic denies the basics of radiative transfer.
“But “confirm the theory of GHG warming?” How did you reach that conclusion?”
Confirm. note, I dont say verify.
I’m a confirmational holist.
http://en.wikipedia.org/wiki/Confirmation_holism
Steven Mosher says:
July 13, 2010 at 11:03 pm
“DirkH says:
July 13, 2010 at 3:50 pm (Edit)
“And right after 82, a steep temp rise (and declining thermometer population).
Not accusing anyone of anything, just saying.”
That’s largely incorrect.
1. the answer is not changed by station drop off and
2. The stations do not drop that quickly after 1982.
3. If I change the period you will still get a rise.
4. If I pick 1000 stations for the whole period you will still see a rise.”
Your points (1),(2),(3) and (4) might be correct, but they don’t interfere with my words:
“And right after 82, a steep temp rise (and declining thermometer population).”
Your “That’s largely incorrect.” talks about a possible conclusion that i intentionally did not write down.
This whole issue is so over complicated by failure to either transparently clean up poor temperature records and audit the sites for compliance and, then rate them properly for adjustment of things like UHI, along with cavalier one size fits all global extrapolation that is also on the face of it “not transparent” Its hardly surprising we are where we are.
I can also understand the frustration of Anthony when a site is proved to be so badly sited that it is withdrawn from the present system, but its rotten data is left to contaminate the historical record. (If that’s wrong feel free to correct A)
When looking back over historical hottest, coldest records its bad enough that some of these may be due to historical errors, deliberate skewing to maintain a locations claim (that happened in Australia) (Source BOM History The Weather Watchers) or poorly sited equipment that is poorly maintained.
But then applying guesswork and adjusting temperatures down in historical records can also skew modern interpretation, especially where trends are constantly used to illustrate extreme views. It takes very little to adjust a model bias, and when trust is lost within the scientific process, suspicion abounds.
Hopefully with co-operation on sites such as this eventually, some real consensus on data interpretation and weighting methods to be applied might be reached and confidence returned.
my 2 cents !
Ron Broberg says:
July 13, 2010 at 4:00 pm
“Dirk H: Here’s a guy who did a very simple analysis of raw data who comes to the conclusion that there is no discernible trend:
That guy freely admits that he did no geographic weighting. GHCN has a high percentage of US stations – and a low percentage of Southern Hemisphere stations. ”
Is the SH warming faster than the NH? It didn’t seem so to me in GISS’s famous global anomaly maps. Here’s one from Dec 2008:
http://global-warming.accuweather.com/2009/01/despite_recent_trends_giss_sti_1.html
So global warming seems to affect foremost landmasses with a lack of thermometers. Hmm, what could one do to find out more?
Add thermometers? I don’t know if that is a scientific answer, though, me not being a scientist…
the mechansim by whch GHG warm the planet is a physical theory. A physical theory that engineers use in the everyday construction of devices that we all enjoy. Its a physical theory which, Monkton, Lindzen, Christy, spencer, all agree with. we bicker over the MAGNITUDE of the effect, but no serious skeptic denies the basics of radiative transfer.
Over the magnitude once the physical is inserted into a system as complex as the climate. It is possible that there is no effect from CO² in the climate but no-one has yet completed a full mathematical model (SteveMc’s engineering study) which proves it one way or the other.
On another note, Mosh et al thanks for this example and I really do appreciate all this effort. I think a clear statement of purpose, etc at the beginning would have negated many of the comments received and I am utterly convinced that the introduction would have mentioned all that George Smith wrote.
I believe you know he is correct in what he says but that was not the purpose of your essay. In the final analysis it is only energy balance / lapse rate that matters and globa temp is for the public/media.
Lastly, I have become aware that GISS have been adjusting historic data along with the latest data and that these adjustments have tended to lower the historic data relative to the later. Am I correct, if so how has this been accounted for in your examples?
Amusing article that comes to the conclusion that all the simple arithmetic is done correctly. So it seems even climate scientists can add up.
Still, it puzzles me that the theory states that the additional CO2 should make most impact when the insolation is highest but they insist on averaging temperatures over a year. Seems to me that a more sensitive apporach would be to look only at temperature measurements on the Longest Day for both Northern and Southern hemispheres to see if there is any real evidence that CO2 is doing its evil work, rather than watering down the possibility of detection by adding in all the winter temperatures too. This approach would also mean you would retain the natural variation of temperatures due to “weather” which would allow mathemiticians to consider the statistical significance of any anomoly.
stephen richards
GISS adjustments do not enter into the above, at all. The source data are taken from a source upstream from GISS.
As for what GISS does, see the effect here.
http://clearclimatecode.org/gistemp-urban-adjustment/
DirkH
Can you clarify what methods and data your source used? It’s very difficult to tell.
By parsing language, I kind of think he used New Zealand data only, and may have used the First Difference Method for the calculation, but it’s not obvious to me.
George E. Smith says:
July 13, 2010 at 3:19 pm
Why not admit, that ALL of these records; are simply the result of statistical manipulations on the output of a certain set of thermometers; and that any connection between the result of that computation and the actual temperature of the earth is without any scientific foundation; it’s a fiction.
AGREED: It is a fiction.
George E. Smith says:
July 13, 2010 at 3:31 pm
is not correction simply making up false data?
AGREED: That is what scientists seem to do these days.
George E. Smith says:
July 13, 2010 at 5:31 pm
The point is that NOWHERE in this process, can the result be connected to the planet to “Calculate the Global Temperature” It simply calculates the variations of some quite arbitrary set of thermometers from themselves.
AGREED: End of story.
George E. Smith says:
July 13, 2010 at 5:45 pm
The GISStemp process, and the HADcrud process calculate GISStemp and HADcrud respectively; and nothing else. They have no connection to the mean global temperature of the planet; which in turn has no connection to the energy balance of the earth energy budget.
AGREED: Anomalies just seem to be about fear and obscuration.
Luis Dias says:
July 13, 2010 at 6:14 pm
George Smith, that’s quite an astonishing nihillist (and paranoid) vision of reality. I never thought I’d see that kind of thing. Even here.
Welcome to the real world.
rbateman says:
July 13, 2010 at 5:34 pm
If we plotted the yearly mean temp instead of the anomaly, set the bottom of the graph at ZERO, set the top at 2x the mean, then we’d see how this Global Temp scare is making a mountain out of a molehill.
AGREED: A very simple, very sensible and very correct way to look at their data
A very interesting post. I was particularly surprised at the lack of difference in the results from the various methods used thereby eliminating method as a problem area.
Would overlaying the standard deviation of at least the baseline data be helpful in interpreting the graphs?
Slightly O/T but I was looking at the 234 year long Central England Temperature record (link below) and noticed that the global average curve was almost entirely below the CET one. Now given that (just considering the northern hemisphere) there is a great deal more of the earth’s (warmer) surface south of UK than north of it I would have thought, intuitively, that the global curve would therefore have been above CET. Or am I missing something fundamental?
Thoughts anyone?
http://www.decc.gov.uk/assets/decc/statistics/climate_change/1_20100319151831_e_@ur momisugly@_surfacetemperaturesummary.pdf
Global temperature from 1979 plotted on a normal y-axis.
Ron Broberg says at 11:31 pm:
“The current highs are to some degree greater than zero a product of increasing CO2 (as determined by modeling analysis )”
That may or may not be correct, but it should be kept in mind that it is an assumption based on a computer model. It is not real world data, and quoting the IPCC still doesn’t make it anything more than a conjecture.
There has to be a large component of guessing in all these reconstructions. If the old thermometer method was adequate, why was there a change to thermocouples and thermistors? As the daily sampling rate went from 1 a day to hundreds a day, did not this cause different smoothing assumptions, removal of spikes, etc? Why should the pre- and post- mercury be able to be spliced; what was the duration and magnitude of the splice? Would one not expect spike removal in recent times to drive down maxima? How can you use old records when the metadata sheets are still being studied and adjustments made as we speak? How can you define a generic term “rural” and know it was always thus through instrumented time? A station in the middle of a vast paddock will respond to the heat of a lamp used to light it for reading before sunrise – is this everywhere quantified and corrected? Even the height of the grass growing around it can cause substantial change.
The error terms, when teated in a high quality manner such as for data that really matter for health or safety, are so large that it is impossible to draw solid conclusions. I’m with George E. Smith – July 13, 2010 at 5:45 pm
I think the comment of Luis Dias – July 13, 2010 at 6:14 pm is harsh and unjustified. The more temperature series I plot, the more I find stations with no change over the last 40 years, within reasonable interpretation. It takes only one such station, in theory, to disprove global warming; but when large numbers of them exist, then the concept of global warming has to explain a negative temperature driver at each one, which global warming theorists have failed to do.
Mosh says:
“underlying mechanism. Well, the results are consistent with and confirm the theory of GHG warming, espoused BEFORE this data was collected.”
Which of these two graphs suggests the better correlation Mosh?
http://tallbloke.files.wordpress.com/2010/07/soon-arctic-tsi.jpg
Take your time…