I’m happy to present this essay created from both sides of the aisle, courtesy of the two gentlemen below. Be sure to see the conclusion. I present their essay below with only a few small edits for spelling, format, and readability. Plus an image, a snapshot of global temperatures. – Anthony
By Zeke Hausfather and Steven Mosher
There are a variety of questions that people have about the calculation of a global temperature index. Questions that range from the selection of data and the adjustments made to data, to the actual calculation of the average. For some there is even a question about whether the measure makes any sense or not. It’s not possible to address all these questions in one short piece, but some of them can be addressed and reasonably settled. In particular we are in a position to answer the question about potential biases in the selection of data and biases in how that data is averaged.
To move the discussion onto the important matters of adjustments to data or, for example, UHI issues in the source data it is important to move forward on some answerable questions. Namely, do the methods for averaging data, the methods of the GISS, CRU and NCDC bias the result? There are a variety of methods for averaging spatial data, do the methods selected and implemented by the big three bias the result?
There has been a trend of late among climate bloggers on both sides of the divide to develop their own global temperature reconstructions. These have ranged from simple land reconstructions using GHCN data
(either v2.mean unadjusted data or v2.mean_adj data) to full land/ocean reconstructions and experiments with alternative datasets (GSOD , WDSSC , ISH ).
Bloggers and researchers who have developed reconstructions so far this year include:
Steven Mosher
And, just recently, the Muir Russell report
What is interesting is that the results from all these reconstructions are quite similar, despite differences in methodologies and source data. All are also quite comparable to the “big three” published global land temperature indices: NCDC , GISTemp , and CRUTEM .
[Fig 1]
The task of calculating global land temperatures is actually relatively simple, and the differences between reconstructions can be distilled down to a small number of choices:
1. Choose a land temperature series.
Ones analyzed so far include GHCN (raw and adjusted), WMSSC , GISS Step 0, ISH , GSOD , and USHCN (raw, time-of-observation adjusted, and F52 fully adjusted). Most reconstructions to date have chosen to focus on raw datasets, and all give similar results.
[Fig 2]
It’s worth noting that most of these datasets have some overlap. GHCN and WMSSC both include many (but not all) of the same stations. GISS Step 0 includes all GHCN stations in addition to USHCN stations and a selection of stations from Antartica. ISH and GSOD have quite a bit of overlap, and include hourly/daily data from a number of GHCN stations (though they have many, many more station records than GHCN in the last 30 years).
2. Choosing a station combination method and a normalization method.
GHCN in particular contains a number of duplicate records (dups) and multiple station records (imods) associated with a single wmo_id. Records can be combined at a single location and/or grid cell and converted into anomalies through the Reference Station Method (RSM), the Common Anomalies Method (CAM), and First Differences Method (FDM), or the Least Squares Method (LSM) developed by Tamino and Roman M . Depending on the method chosen, you may be able to use more stations with short records, or end up discarding station records that do not have coverage in a chosen baseline period. Different reconstructions have mainly made use of CAM (Zeke, Mosher, NCDC) or LSM (Chad, Jeff Id/Roman M, Nick Stokes, Tamino). The choice between the two does not appear to have a significant effect on results, though more work could be done using the same model and varying only the combination method.
[Fig 3]
3. Choosing an anomaly period.
The choice of the anomaly period is particularly important for reconstructions using CAM, as it will determine the amount of usable records. The anomaly period can also result in odd behavior of anomalies if it is too short, but in general the choice makes little difference to the results. In the figure that follows Mosher shows the difference between picking an anomaly period like CRU does, 1961-1990, and picking an anomaly period that maximizes the number monthly reports in a 30 year period. The period that maximizes the number of monthly reports over a 30 year period turns out to be 1952-1983. 1953-82 (Mosher). No other 30 year period in GHCN has more station reports. This refinement, however, has no appreciable impact.
[Fig 4]
4. Gridding methods.
Most global reconstructions use 5×5 grid cells to ensure good spatial coverage of the globe. GISTemp uses a rather different method of equal-size grid cells. However, the choice between the two methods does not seem to make a large difference, as GISTemp’s land record can be reasonably well-replicated using 5×5 grid cells. Smaller resolution grid cells can improve regional anomalies, but will often result in spatial bias in the results, as there will be large missing areas during periods when or in locations when station coverage is limited. For the most part, the choice is not that important, unless you choose extremely large or small gridcells. In the figure that follows Mosher shows that selecting a smaller grid does not impact the global average or the trend over time. In his implementation there is no averaging or extrapolation over missing grid cells. All the stations within a grid cell are averaged and then the entire globe is averaged. Missing cells are not imputed with any values.
[Fig 5]
5. Using a land mask.
Some reconstructions (Chad, Mosh, Zeke, NCDC) use a land mask to weight each grid cell by its respective land area. The land mask determines how much of a given cell ( say 5×5) is actually land. A cell on a coast, thus, could have only a portion of land in it. The land mask corrects for this. The percent of land in a cell is constructed from a 1 km by 1 km dataset. The net effect of land masking is to increase the trend, especially in the last decade. This factor is the main reason why recent reconstructions by Jeff Id/Roman M and Nick Stokes are a bit lower than those by Chad, Mosh, and Zeke.
[Fig 6]
6. Zonal weighting.
Some reconstructions (GISTemp, CRUTEM) do not simply calculate the land anomaly as the size-weighted average of all grid cells covered. Rather, they calculate anomalies for different regions of the globe (each hemisphere for CRUTEM, 90°N to 23.6°N, 23.6°N to 23.6°S and 23.6°S to 90°S for GISTemp) and create a global land temp as the weighted average of each zone (weightings 0.3, 0.4 and 0.3, respectively for GISTemp, 0.68 × NH + 0.32 × SH for CRUTEM). In both cases, this zonal weighting results in a lower land temp record, as it gives a larger weight to the slower warming Southern Hemisphere.
[Fig 7]
These steps will get you a reasonably good global land record. For more technical details, look at any of the many http://noconsensus.wordpress.com/2010/03/25/thermal-hammer-part-deux/different http://residualanalysis.blogspot.com/2010/03/ghcn-processor-11.html models http://rankexploits.com/musings/2010/a-simple-model-for-spatially-weighted-temp-analysis/ that have been publicly http://drop.io/treesfortheforest released http://moyhu.blogspot.com/2010/04/v14-with-maps-conjugate-gradients.html
].
7. Adding in ocean temperatures.
The major decisions involved in turning a land reconstruction into a land/ocean reconstruction are choosing a SST series (HadSST2, HadISST/Reynolds, and ERSST have been explored http://rankexploits.com/musings/2010/replication/ so far), gridding and anomalizing the series chosen, and creating a combined land-ocean temp record as a weighted combination of the two. This is generally done by: global temp = 0.708 × ocean temp + 0.292 × land temp.
[Fig 8]
8. Interpolation.
Most reconstructions only cover 5×5 grid cells with one or more station for any given month. This means that any areas without station coverage for any given month are implicitly assumed to have the global mean temperature. This is arguably problematic, as high-latitude regions tend to have the poorest coverage and are generally warming faster than the global average.
GISTemp takes a somewhat different approach, assigning a temperature anomaly to all missing grid boxes located within 1200 km of one or more stations that do have defined temperature anomalies. They rationalize this based on the fact that “temperature anomaly patterns tend to be large scale, especially at middle and high latitudes.” Because GISTemp excludes SST readings from areas with sea ice cover, this leads to the extrapolation of land anomalies to ocean areas, particularly in the Arctic. The net effects of interpolation on the resulting GISTemp record is small but not insignificant, particularly in recent years. Indeed, the effect of interpolation is the main reason why GISTemp shows somewhat different trends from HadCRUT and NCDC over the past decade.
[Fig 9]
9. Conclusion
As noted above there are many questions about the calculation of a global temperature index. However, some of those questions can be fairly answered and have been fairly answered by a variety of experienced citizen researchers from all sides of the debate. The approaches used by GISS and CRU and NCDC do not bias the result in any way that would erase the warming we have seen since 1880. To be sure there are minor differences that depend upon the exact choices one makes, choices of ocean data sets, land data sets, rules for including stations, rules for gridding, area weighting approaches, but all of these differences are minor when compared to the warming we see.
That suggests a turn in the discussion to the matters which have not been as thoroughly investigated by independent citizen researchers on all sides:
A turn to the question of data adjustments and a turn to the question of metadata accuracy and finally a turn to the question about UHI. Now, however, the community on all sides of the debate has a set of tools to address these questions.
So… the US hasn’t warming. the Russian dataset is compromised for a variety of reasons. The SH is warming more slowly than the NH.
Where is there a clear warming signal coming from?
Civilly,
EC
Zeke and Mosh,
Thanks for this post; it is an excellent summary.
The issue of UHI is for sure real, but it’s size does need better definition.
FWIW… A modest UHI effect combined with slower than expected (based on models) heat accumulation in the 0ceans (ARGO data) means that the long term (200+ yrs) climate sensitivity is likely on the low end of the IPCC range…. on the order of 1.5C – 2.0 C per doubling of CO2, 300 years out. The immediate sensitivity (20-30 yrs) looks more like 0.75 C – 1.0 C per doubling. Since the “age of carbon” will start declining within the next 40-50 years, due to supply limitations, the long term sensitivity is just never going to be seen. Ocean and biosphere absorption of CO2 will overtake emissions of CO2 within ~50 years, and atmospheric CO2 concentration will start declining.
Immediate and forced draconian reductions in CO2 emissions, at huge economic and human cost, can’t be justified based on any reasonable estimate of future warming.
“”” Steve Fitzpatrick says:
July 14, 2010 at 11:19 am
Zeke and Mosh,
Thanks for this post; it is an excellent summary.
The issue of UHI is for sure real, but it’s size does need better definition.
FWIW… A modest UHI effect combined with slower than expected (based on models) heat accumulation in the 0ceans (ARGO data) means that the long term (200+ yrs) climate sensitivity is likely on the low end of the IPCC range…. on the order of 1.5C – 2.0 C per doubling of CO2, 300 years out. The immediate sensitivity (20-30 yrs) looks more like 0.75 C – 1.0 C per doubling. Since the “age of carbon” will start declining within the next 40-50 years, due to supply limitations, the long term sensitivity is just never going to be seen. Ocean and biosphere absorption of CO2 will overtake emissions of CO2 within ~50 years, and atmospheric CO2 concentration will start declining.
Immediate and forced draconian reductions in CO2 emissions, at huge economic and human cost, can’t be justified based on any reasonable estimate of future warming. “””
Steve where is to be found, the definitive paper; presumably some peer reviewed recognised climate journal; that proves that the concept of “climate sensitivity” is valid; which is to say that T2-T1 = cs.log (CO22/CO21) ; of course base 2 logarithms.
Such a paper would presumably incorporate credible (peer recognised) data sets of Mean Global Surface Temperature; and also of Atmospheric CO2 abundance for that same period of time; showing that the graph is clearly Logarithmic; as distinct from Linear or of any other mathematical functional possibility; within the uncertainty limits of that recognised data; and /or such a paper, would provide some rational and peer reviewed and recognised Physical theoretical basis for believeing that those two data set should be connected by a logarithmic function; as distinct from any other mathematical relationship.
I’ve been trying for years to locate either the theoretical Physics basis for “Climate Sensitivity” (cs) or the empirical actually measured data (as distinct from proxy “data”) that establishes the concept of Climate Sensitivity; as a fixed Temperature increase in mean global surface Temperature per CO2 doubling in the atmosphere; so far with no such luck.
I have even read/heard anecdotally that in fact Professor Stephen H. Schneider of Stanford University, is the inventor; and father of “Climate Sensitivity”; and I haven’t been able to confirm either thato rlocate his defining paper that establishes what seems to be the Rosetta sStone of Climate Science.
(cs) is apparently the “Planck’s constant” or “Boltzmann’s Constant” of Climate Science; a fundamental constant of Physics; yet I can’t locate the origins of the theory.
You seem to be knowledgeable on the subject; so where are the seminal papers on “Climate Sensitivity” ?
George Smith 11:10AM
Spot on, George!
The prior ocean temp data is functionally worthless, for all of the reasons you accurately describe… and more. Measuring ship engine cooling water temperature ‘at the inlet source’ not only varies in sampling depth from ship to ship based on the inlet location, it varies continuously on each ship as the fuel and cargo load changes. We have no way of determining what depth (within roughly the range of minimum to maximum draft depth per ship) any given data point relates to, rendering the entire data set meaningless.
Add to that the lack of correlation of ocean water temp to near ocean atmospheric temps, and all attempts at using historical ocean temp measurements as ‘proxies’ for historical atmospheric temps becomes valueless as well.
GIGO and GIGO…..
But, perhaps if we adjusted them a bit…….. ? };>)
“underlying mechanism. Well, the results are consistent with and confirm the theory of GHG warming, espoused BEFORE this data was collected.”
Which of these two graphs suggests the better correlation Mosh?
http://tallbloke.files.wordpress.com/2010/07/soon-arctic-tsi.jpg
Take your time…”
Well since the slide you refer to
1. doesnt consider the entire globe.
2. Doesnt even transform C02 properly ( if the effect is a LOG effect you had best transform the predictor variable, like DUH)
I much prefer this:
http://comp.uark.edu/~jgeabana/gwfig1ab2.png
if you think that GHGs do not cause warming ( as in water vapor, c02, methane ) then
1. Talk to Christy, Monkton, Lindzen, Spenser, Willis,
2. never trust a another satillite sensor again.
3. Return your cell phone, it’s violating laws of physics.
Steve Fitzpatrick says:
July 14, 2010 at 11:19 am (Edit)
Zeke and Mosh,
Thanks for this post; it is an excellent summary.
The issue of UHI is for sure real, but it’s size does need better definition.”
*******************************************************************
The size of the UHI effect can be approached in several ways. I prefer to bound it from above, as in “not higher than” There are a couple ways to do that, but I’d suggest that looking at the UHA record of trop temps, specifically the trends there from 1979 onward, gives you an upper bound for the effect from 1979 to present.
prior to 1979, that’s a lot harder because the effect ( as Spencer notes) could be non linear at least initially. There are some folks working on estimates, But the bottom line is that the UHI effect is a Small contribution to the overall warming. The relative size of land to ocean makes it hard to the UHI bias to have any leverage in the final numbers.
“FWIW… A modest UHI effect combined with slower than expected (based on models) heat accumulation in the 0ceans (ARGO data) means that the long term (200+ yrs) climate sensitivity is likely on the low end of the IPCC range…. on the order of 1.5C – 2.0 C per doubling of CO2, 300 years out. The immediate sensitivity (20-30 yrs) looks more like 0.75 C – 1.0 C per doubling. Since the “age of carbon” will start declining within the next 40-50 years, due to supply limitations, the long term sensitivity is just never going to be seen. Ocean and biosphere absorption of CO2 will overtake emissions of CO2 within ~50 years, and atmospheric CO2 concentration will start declining.
Immediate and forced draconian reductions in CO2 emissions, at huge economic and human cost, can’t be justified based on any reasonable estimate of future warming.”
I’d associate myself with those comments, they are in the Lukewarmer camp.
A. Adding GHGs will warm the planet, not cool it.
B. The true sensitivity is at the low end of the IPCC range.
Yes, I’m a true believer, whatever that means.
George Smith:
“I’ve been trying for years to locate either the theoretical Physics basis for “Climate Sensitivity” (cs) or the empirical actually measured data (as distinct from proxy “data”) that establishes the concept of Climate Sensitivity; as a fixed Temperature increase in mean global surface Temperature per CO2 doubling in the atmosphere; so far with no such luck.”
You should well know that a sensitivity or gain cannot be derived. In the same way that the gains for a flight control system cannot be derived. Sensitivity can be bounded by simulation and modelling ( within big boundaries) and it can be inferred from observations if you are lucky.
If you have questions.. start with James, and try to look a bit harder.
http://julesandjames.blogspot.com/2006/09/can-we-believe-in-high-climate.html
http://julesandjames.blogspot.com/2006/03/climate-sensitivity-is-3c.html
Given that the general agreement of these independent reconstructions provide strong evidence that nobody has been fiddling the figures, it makes the UEA et al obstructions and prevarications all the more baffling and ultimately pyrhhic.
REPLY: Data fiddling take place in adjustments to the raw data, that is where the real issue lies, along with UHI/microsite effects -A
Steve where is to be found, the definitive paper; presumably some peer reviewed recognised climate journal; that proves that the concept of “climate sensitivity” is valid; which is to say that T2-T1 = cs.log (CO22/CO21) ; of course base 2 logarithms.
Why do people who refuse to read the primary texts, argue that they dont exist.
odd.
Start here
http://www.agu.org/pubs/crossref/1998/98GL01908.shtml
http://folk.uio.no/gunnarmy/paper/myhre_grl98.pdf
The basic inputs would be from HITRAN
HITRAN is where we start (its a database anybody who has worked with sensors
can tell you about, cant make good bombs or missiles actually WORK without it)
Then you would work with an RTE, Hmm I think they used
GENLN2 and some others
http://adsabs.harvard.edu/abs/1992ggll.rept…..E
or a better list
http://www.cramster.com/reference/wiki.aspx?wiki_name=List_of_atmospheric_radiative_transfer_codes
I’m most familiar with MODTRAN.
So ya, you start with the database of molecules. A database that those of us who built weapon systems to protect the free world rely on. Then you use RTE, radiative transfer equations. ya, the same physics we use to design weapons systems and sensor systems that protect the free world. So, basically same data, same theory, different purpose.
If you want to blame somebody for the advances in climate science ( seeing the role of C02) blame the Air Force. Its data and code generated initially for them. Go figure.
Thanks Zeke,
The code will be posted as aturn-key package with the basics covered.
I’ve pulled out a subset of all the code and passed that onto RonB as I would like to help him ( if possible) get something that is rewritten from the ground up.
I’ll make a version available in a week or so that folks can play with. It’s directed at beginers in R.
“if you think that GHGs do not cause warming ( as in water vapor, c02, methane ) then 2. never trust a another satillite sensor again.”
You have satellite data for 1930’s!!!!!!!!!!! Great. Can I see it?
As for your UHI comments … without min/max, you cannot even being to consider whether UHI is contaminating the GTR.
George E. Smith says:
July 13, 2010 at 5:31 pm
“The min/max daily temperature reading fails the Nyquist sampling criterion; and the spatial distribution of the set of thermometers also fails the Nyquist Test; and by orders of magnitude; so there’s no way that any recovered average can be correct because of the aliassing noise; and the result is meaningless anyway.
As I have said many times GISStemp calculates GISStemp; and nothing else; same goes for HADcrud.
And even if one did actually measure the true average Temperature of the globe; it is not related in any way to the energy flows; so it tells us nothing about the stability of earth’s energy balance.”
I’m with you all the way George. GMT is a useless proxy for the amount of energy held in our climate system and because climate is driven by deterministic chaos. Even if you could measure it temperature accurately enough to be able to isolate the tiny bit caused by carbon based GHG’s, linear trends wouldn’t tell us anything worthwhile. However, this is the marker which has been chosen by climate scientists to confirm or deny CAGW, so we have to live with it, even though it does have feet of clay.
Steven Mosher’s post shows that using the same basic ‘rawish’ data sets and applying various models all seem to give the same sort of time series result for GMT anomaly, which is very small. Steven has demonstrated that the discussion needs to move on to identify factors other than man-made GHG’s, so we can see the true magnitude of the effect.
I think this needs to be split between natural caused changes (e.g. solar, geothermal, ocean cycles,biosphere) and non-GHG effects of man (e.g. UHI, roads, farming, energy use, forestry, industrial pollution, energy use). All factors need to be considered f a true estimate of climate sensitivity is to be produced.
Steve Mosher,
Thanks for answering George Smith… I have not the time.
Cheers.
“”” anna v says:
July 13, 2010 at 8:56 pm
It is true that the climate community has managed to focus the world’s attention to the temperature anomalies, and by sleight of hand taken the attention from the temperatures themselves and how badly the models reproduce them.
I am with George Smith on this . “””
Anna; coming from somebody with your background; it is comforting to know that I am not alone, in seeing what is in the Emperor’s wardrobe.
Imagine a different playing field; say one that is actually more in tune with Professor Stephen H. Schneider’s actual academic qualifications; the field of biology.
Suppose we pick say 7000-8000 locations around the globe; here and there of a size not too unlike the space taken up by a typical “weather station” much like Anthony’s survey has shown us.
And we want to study data on animals for the World Wildlife Fund; that never used to be known as the WWF, until Vince McMahon put the WWF on the map.
So maybe we can assign say a Hectare to each of the 7000 odd sites. We go out periodically; and we count the animals present on the station; well we stipulate here that anything from the size of an ant, on up, is an animal. Doesn’t really matter what kind of animal it is. Does GISS take into account what make or model of barbecue grill it is at each official weather station; so an animal is as good as any other animal.
Now we only keep track of the changes in the station’s animal population; it matters not a jot; whether the elephants, all left on safari; or whether a locust plague just moved in; one animal is as good as another on our animal anomaly chart.
So now we have the means of monitoring animal anomalies all over the world; without regard to the fact that there are no Penguins in the Arctic Ocean.
For 150 years through thick and thin, we simply report on the statistical summation of the animal anomalies garnered from our seven thousand locations; well from time to time, we may add some stations, or take some away. Well there’s the Urban Animal Dearth (UAD) problem that arises when some sheep farm is converted into an 8 lane freeway; we have to make corrections to the data.
Well this is all very interesting and can employ countless numbers of otherwise unemployed and maybe unemployable biologists; but bottom line is.
Does this process really tell the World Wildlife fund anything meaningful about the state of the earth’s fauna; or flora if we did that instead.
You see there’s a basic assumption that the state of the earth’s animal populations as represented by those 7000 chosen locations originally is a GOOD representation of the original or baseline condition.
Now I understand the anomaly concept as it levels mountains and canyons, and reduces the planetary surface to a billiard ball; rather clever idea actually; but it isn’t a good facsimile of reality.
Now Mother Gaia really has a nicely equipped Laboratory; which is why she always gets the right answer. She literally has a thermometer on board each and every atom or molecule. Well maybe not formally; we like to statistically gather a number of molecules at any instant and take some form of average kinetic energy for those, as a representation of the Temperature; but we could add an asterisk, and simply say that we could do a time average of the energy of any single molecule for some short period and declare that to be the asterisked Temperature of that molecule. So in that sense Mother gaia’s thermometers are everywhere and she integrates the consequences of all of them to decide over time; just what weather and climate she is going to allow.
Well we don’t have that many thermometers; or animal accountants either; so we have to SAMPLE.
Well sampling is something we have come to understand at WUWT. You walk up to a tree, with something like a laboratory cork borer; and you pick a spot about chest high, where you can work conveniently, and we drive the cork borer into the tree; hopefully pointed about in the direction of the very center of the roughly circular outline; and we extract a nice core of layered samples of each of the annual growth rings; that we happen to hit on.
Now there’s that old Murphy’s Law joke that says that that single core that we just bored from the tree, is a PERFECT sample of everything that is inside that tree. Murphy’s law says that is so; and we can prove it, by cutting down the whole tree, and running it through a sausage slicer; we could call it an Ultramicrotome instead; and then cutting the slices into narrow segments or sectors; and when we examine each and every single element of each of the rings of that entire tree we will find that every piece is absolutely identical to the pices in the original cork borer sample. That is what Murphy’s law says; but it also adds that the trick is to stop when you extract the borer core; and not assassinate the entire tree.
Well as we know too well from actual full slices of murdered trees, our bored core is anything but a representative sample of that complete former glorious edifice that was a tree.
Well that is the problem of SAMPLED DATA SYSTEMS. How the hell do we know that the sample(s) we take of some continuous function (of maybe several variables), is a truly representative sample of the entire function.
Well fortunately we do know the answer to that question; and the entire modern world of high density high bandwidth communications, is totally dependent on the sure knowledge that we know when enough is enough.
If we have a continuous let’s say time varying function (of whatever); in general that function cannot change its value in zero time; it takes some finite time to change from one value to another. We say the function is “Band Limited”. The continuous time function can be represented by some Fourier series or integral of functions with different frequencies, the simplest being a sum of sinusoidal components; having some maximum upper frequency; beyond which no material signal components exist; that maximum frquency being the bandwidth or band limit of the signal; and remember that instead of time; it could be a function of ANY variable or set of variables.
In the case of the time varying function; it is known that we can represent the band limited signal COMPLETELY by a regular sequence of INSTANTANEOUS samples of the continuous signal; measured at discrete times. And the theory tells us that we can exactly recover the band limited continuous signal from just those instantaneous samples; IF WE DO IT ALL CORRECTLY.
So why is that useful ? Well if we have a voice telephone signal that contains NO signals varying at higher than say 3 kHerz; and we have a communications “channel” that has a maximum signal frequency capability of say 500 kHerz; we know that it is possible to transmit a signal pulse of some amplitude and just one microsecond pulsewidth on that channel of communication; so we could sample our voice signal, at say 10kHz; every 100 microseconds; and create a one micorsecond long pulse from our sample, and send them all down the line. Well hell; we have another 99 microseconds in between our samples that isn’t doing anything useful. We coudl take another 99 telephone calls, and do the same thing to those; and then interleave all the 100 sets of samples, and send them all down the same wire. With a little bit of overhead catalog information; we can sort out the pulses down the other end of the wire, and send them to 100 different telephones; and then we can reconstruct all those messages perfectly (in theory).
So we do know for absolute certainty that sampled data theory works; and a hell of a lot better than I have described it here.
So what does this have to do with measuring the earth’s temperature. Well the theory says that we don’t have to be as well endowed as Mother Gaia is. We don’t need a thermometer in every molecule.
Sampled data theory tells us that if we take our Temperature samples properly; that is by sampling CORRECTLY both in TIME and SPATIAL POSITION; then from just those samples we can reconstruct the continuous two dimensional data map that was our global temperature map in time and space; and then from that reconstructed data we are free to calculate any averages or other information about the set that we want; the information that was in the original signal is all there in the properly gathered samples.
Well the principal theorem of sampled data theory, is the Nyquist Theorem; which says that any band limited continuous function can be fully represented by a set of instantaneous samples of the function’s value; provided we take at least one sample during each half cycle of the highest frequency signal presnt in out band limited function. In the cae of our 3 kHz voice phone call, we needed to sample at at least 7 kHz, to satisfy the requirement. We chose to do it at 10 kHz instead to have some margin.
So what happens if our signal was not band limited like we assumed. Supposing there is some screaming child in the background of our phone call; and he is putting out some 6 kHz harmonics on his wailing. which is 1 kHz beyond our band limit assumption of 5.0 kHz. We will find in practice, and in theroy, that upon reconstruction of the signal from the samples, we now have a spurious (aliassing noise) signal, that wa sNOT presnet in the original, and since it started at 1kHz beyond our band limit; the reconstructed signal will contain a component that is 1kHz BELOW our band limit; which is at 4 kHz. We originally had NO signals at 4 kHz; now we do; and since 4 kHz is less than 5 kHz, that spurious noise signal is INSIDE the signal band of 5 kHz; so now it is inherently impossible to filter it out, and get rid of it.
If our child was shrieking with some 10 kHz harmonics; the same as or higher than our sample rate; and just twice our signal band limit, the aliassed noise will be at 5.0 -5 kHz or zero frequency. We have another word for the zero frequency component of ANY signal. We call it the AVERAGE VALUE OF THE SIGNAL.
Hey isn’t that what it was we were trying to calculate in the first place; was the AVERAGE Temperature over time and space of our planet.
Well sadly; we only have to undersample our signal by a factor of two in either time or space; from what the Nyquist Theorem tells us we need, and we end up with a non-removable noise corruption of the very thing we were trying to determine; the average value of our continuous function.
Min max daily time measurement already violates Nyquist by at least two since the diurnal temeprature cycle is not s single frequency sinusoidal signal; so it has at least a second harmonic 12 hour periodic component; which is right at our sample rate of twice daily measurment.
And of course; with 7000 odd global position sampling stations; it is a total joke to talk about sampling strategy.
Sampling theory does not requirte regularly spaced samples; they can be at difefrent intervals; but the minimum separation cannot exceed the separation required by Nyquist of one half the period of the highest signal frequency; and any non-uniform sampling regimen requires an even higher number of samples; not a lower number.
So no; I do not place any confidence in whatever results the experts come up with for the Mean Global Temperature; their problem is not with statistical methodology; the central limit theorem is not a cure for a Nyquist headache.
Now I’m not going to say that global anomaly sampling is worthless; very little information is totally worthless.
But I wish they would stop making claims for it that are unsupportable.
And Anna; back to my original point; I’m happy to read that you are perhaps somewhat like minded. Wish I knew all the particle Physics of your experience.
Just a brief note here for Steven Mosher, and Zeke Hausfather.
Please be advised; that I AM NOT here knocking your work or the paper/essay that you presented here; and I have not fully digested it yet; and Steve; I did see and appreciate your clarification of just what the grid sampling process that you (or others) use. You did essentially clear up my confusion.
When you were talking grids; I had visons of Peter Humbug, and his Playstation models with the gridded computer modelling of whatever it is that he models. So I just wanted to be sure that what you were talking about, was not the same thing.
As I said, I haven’t yet fully digested the detail of your extensive paper here, that looks like you and Zeke have put a lot of effort into.
I usually start to grasp the essence of some of those papers ; and Steve Goddard’s too about the time; they disappear off the bottom of Anthony’s very busy menu. So I am often reading and sometimes posting a full two pages of posts below the current page.
So hopefully, by the time I digest your essay, I will have some idea of what the different methodologies do in the way of changing the apparent results.
My rant about the whole concept of anomalies is NOT addressed to you or anyone else who posts information or analysies such as the two of you presented here.
And make no mistake; I am constantly learning from the efforts of posters who present information like this.
Sometimes I think Willis thinks I’m a bloody pest; which I am; but I do appreciate his efforts as intensely.
Thanks again; for your analysis; and yes I did see your response to my questions about the gridding Steven.
George
RomanM (July 13, 2010 at 4:23 pm):
Having worked extensively with LSE methods, I’m confident of the results when there is a homogeneous spatial field and a consistent datum-level is maintained by the available data. Certainly, it is an attractive approach in principle. As a practical matter, however, I’m not so confident that offsets in station-records are effectively treated when the data from neighboring stations is spatially inhomogeneous and has a nondescript temporal bias introduced by intensifying UHI.
As a concrete example, consider the two stretches of record (through 1950 and later) for Gibraltar, whose station was moved to a more-inland location 7km away from the original site. The closest staion whose record overlaps the break is a very short segment from across the strait at Tanger. Long overlapping records are available only from Casablanca, Lisboa, and Marseilles–all of which show pronounced, but non-uniform, UHI warming. Perhaps you could allay the skepticism of Sherrington and others by demonstrating how your algorithm estimates the offset in the post-1950 stretch of Gibraltar record.
“”” Steven Mosher says:
July 14, 2010 at 1:00 pm
Steve where is to be found, the definitive paper; presumably some peer reviewed recognised climate journal; that proves that the concept of “climate sensitivity” is valid; which is to say that T2-T1 = cs.log (CO22/CO21) ; of course base 2 logarithms.
Why do people who refuse to read the primary texts, argue that they dont exist. “””
Well Steve; I don’t recall ever saying these texts don’t exist. I’ve been Googling Stephen H. Schneider till I am blue in the face; trying to find whatever paper it was in which he reputedly coined the term “Climate Sensitivity” and pointed out (presumably) its logarithmic relationship; which is quite inherent in the very concept of a fixed Temperature increase for any doubling of the atmospheric CO2 abundance. So far; it hasn’t popped out; so maybe my starting assumption that he is the father of “Climate Sensitivity” is all wrong; and sombody else “discovered” the logarithmic relationship.
Now I’ve read all kinds of folk tale descriptions that talk about the avaiable CO2 “trapping sites” getting removed (as busy with another capture) so that the “effect” “Tapers off” with additional CO2.
But “Tapers off” or “expansion of the wings” of some absorption spectrum is not what I usually construe the term logarithmic to mean.
I understand exactly what the logarithmic/exponetial relationship is; and constantly use it over maybe six orders of magniude (of current) in the case of the forward Voltage of Semi-conductor diodes; so that is what I envision when somebody tells me that in climatology there is this constant global surface temperature offset that accompanies each doubling of CO2.
That is a little bit surprising given that the Mauna Loa Data set; has so far not observed even 1/3rd of one doubling of the CO2 yet; and that presumably along with the post 1958 global Temperature data, would seem to be the information that shows this logarithmic relationship.
And for small increments, the logarithmic function is simply linear with the variable; which begs the question of how one tells the difference given that the slope of the purported straight line T/Log CO2 is only known within a 3:1 uncertainty range.
So the issue is not one of my making; I have simply accepted the IPCC and other authoritative sources fo information and followed where that leads to.
So I don’t say the information isn’t there; or doesn’t exist; and I have read countless hours of the “primary texts” most of which in no way addresses that single issue.
I have even considered writing to Professor Schneider and asking him for a copy of the original defining paper (if he was indeed the originator) but I suspect he simply wouldn’t respond. I don’t even get responses from acknoledged skeptical authors; when I query them about their work.
John Christy was singularly forthcoming in responding to a query of mine; about his paper on the ocean buoy measurements.
Either I am asking the question incorrectly or for some reason; I am not making it clear what I am trying to learn.
But asserting that there is no such information has not been one of my approaches.
It’s sad to see that what started out as a methodological discussion of how the “global temperature” is calculated, has morphed into position statements about what the results obtained from a highly incomplete and patently urban-biased GHCN data set show or don’t show about AGW. The fact of the matter is that we have consistent, truly global measurements only since the dawn of the satellite era. Prior to that, nothing resembling a credible time-series is available for the great bulk of the oceans and for vast regions of the land masses as well.
Outside of well-traveled sealanes, rarely does one find enough observations in a 5-degree Marsden square to construct a truly reliable climatological summary over the past century, even under the assumption of homogeneity within the square. (Since SMOs are made 4 times a day, a credible time-series of monthly averages would require at least 4x365x100= 146,000 observations, assuming continuous coverage.) Nor does substituting even more sparsely available SSTs for air temperatures, as is is done by Hadcley Centre and others, provide anything beyond a stopgap measure to fill the void. We simply don’t know what the actual time-history of dry-bulb temperature over the world’s oceans looks like for most of the prior century.
And even if we did, it would still leave us with a physically incomplete specification of total energy transfer from surface through the atmosphere to space. Contrary to what Steve Mosher avers here, that energy transfer is not by radiative means alone! Only those who are learning their physics from the mistake-riddled science-of-doom website can ignore the central role of evaporation from the oceans in the global energy budget. With ~1.5 m of the oceans evaporating each year, the latent heat transport far exceeds the net backradiation from the moistened atmosphere.
w00t!
Steve cleaned the kitchen today!
Charles,
Two words for you two: paper… plates. ☺
There are no urban heat islands in the troposphere (that I am aware of).
Yes, George, our two different approaches lead to the same conclusion.
Mine is never to lose sight of the constants of motion, ( energy, momentum, angular momentum + some esoteric ones) whose conservation is absolutely independent of the type of solutions of the differential equations entering the problem.
Thanks for your statistical/information-theory approach that teaches the same lesson. You give a very concise and clear summary.
BillN says:
July 14, 2010 at 10:38 am
can someone please point me to a cogent discussion of the right way to “average” the energy of two different locations. Is it the energy of the air? What about radiation, thermal “inertia,” etc.?
Read George’s very clear outline of what not to do when extracting information statistically.
Here is my physics view point for the specific problem, sun, earth, atmosphere:
The energy mainly ( gravitational ignored because much smaller, eve more magnetic and electric) comes from the sun’s radiation impinging on the top of the atmosphere.
It goes out in various forms again finally as mainly low energy electromagnetic radiation with various mechanisms.
.
Input output is not in equilibrium and that is what is being discussed with AGW that CO2 changes the equilibrium and allows higher temperatures to manifest, than would exist without the excess.
The problem is not in averaging energy. Energy once measured can be averaged easily since it is a scalar.
The problem is in the measurement. The real answer to “is the earth warming” would come if we stood outside the stratosphere and measured all incoming and all outgoing radiation. The sign of the difference would give us a clue whether we are heating or cooling. In principle one could do it systematically with satellites and obey fully Nyquist’s criterion . I have not seen such an analysis. What we get from satellites are temperatures over the globe at various heights, which are then extrapolated mathematically to the surface.
What is being done is using the temperatures of sampled locations on the surface of the earth as proxies of the energy , and anomalies of temperatures as proxies of temperatures. One convolution to go from energy to temperature, and another convolution ( i.e. integration over large numbers of variables) from temperature to anomalies of temperatures.
Suppose we do manage to measure temperatures well, ( Nyquist satisfied), how to we go to energy?
It is the black body radiation formula j=c*T^4, j radiation flow, c constant for black body. The amount of change of c due to not being a black body is called “emissivity” and is different for different materials. This is for solids, liquids. For air the radiation formula is something else.
So the question “which temperature” is important. In principle, the skin surface temperature ( first mm of ground,water, ice) is what should enter the formula. What one sees in all the climate essays is the temperature at 2 meters, which is what meteorological stations measure. Logical for gauging the weather, man lives on the first 2 meters of ground. Illogical for measuring radiated energy, because it is the ground that radiates it.
Then the second level of distortions enters, that of using anomalies. Anomalies have a meaning when one knows the base, for specific locations. Averaging over the globe can have no quantitative meaning. It can only say : energy retained is increasing/decreasing.
In my opinion this climate change experiment is designed all wrong and of course will come up with distorted results, as did Chicken Little when the drop of a feather was amplified to the drop of the sky.
Why all the praise? Would be much more thorough if Min/Max/Avg trends were each calculated and displayed as that is where the real mystery lies. C’mon, if we are going to dig into it, then let’s dig into it. Min/max/2 anomalies hide important data IMO. Please add to the reconstructions, maybe delineated by season, Spring/Summer/Fall/Winter, separate min and max and avg anomalies, for both Urban and Rural, and now we are talking. Otherwise, it’s all integrated noise…Until you take that step, you are just validating the integrated mess that we currently have to swallow. The mystery is in the mins no? and the urban vs rural? and the adjustments…dissect please…then put it back together.
Steven or Zeke
Why do you believe that the global sea temperatures back to 1880 have any merit as a scientific measure when clearly they don’t?
Having met someone who actually took the bucket measurements, haphazard would be too small a word to describe what went on. This haphazard activity took place along a tiny fraction of the Worlds oceans meaning the records become even more pointless.
Similarly the periapetic nature of land based stations and the way they are influenced by encroaching cities, the change in their micro climate due to moves, poor siting, the differences in the manner in which temperatures were taken- especially before the advent of min/max/thermometers- also means that individually they have some merit, but collectively they are just a jumble of material.
To believe we have a reliable global record that can be parsed to fractions of a degree is to defy reality. (It was still a great piece of work though)
tonyb