BoM's bomb on station temperature trend fiddling

Yeah. What I said too. And even discarding outliers when you THINK you know a reason is deadly dangerous, unless you are gifted with perfect prior knowledge.
The bete noire of empirical human reason is confirmation bias. There are some truly (in)famous examples of confirmation bias (and its statistical companions, data dredging, cherrypicking, etc) producing horrendous conclusions that were — eventually soundly rejected when somebody sane and unbiased re-examined the problem.
Rejecting outliers — especially outliers that fail to conform to a belief about the way the data should behave — can easily be data dredging and cherrypicking both in disguise. The only “safe” way to do it is via an unbiased algorithm that works completely automatically and that can be shown in application to simulated data to actually work without bias (not just theoretically be without bias). Mann’s hockey stick was built via a selection process that rejected data as “unsuitable” or an “outlier” if it failed to conform to his prior beliefs about what tree rings were supposed to proxy. Again, there was a huge Bayesian assumption built right into the algorithm, one that took simulated random noise and turned it into hockey sticks and that would have been soundly rejected as a posterior probability if anybody had bothered to do this sort of analysis ahead of time.
It also — literally — overrides the very power of statistics that one hopes to exploit when forming averages in the first place. The default assumption in most experimental science is that when one makes measurement errors, they are as likely to be too large as too small. Hence if one samples many times, one homes in on the true mean. In some cases, one can even support this assumption via the central limit theorem if the measurements themselves are in some sense an “average” of some microscopic process and hence are already likely to be normally distributed according to the CLT.
There are, of course, examples of cases where a measurement has a systematic bias. Somebody always rounds down, never up. A piece of dynamic measurement apparatus is “sticky” and never reads as high as it should. The problem even there is that it is supremely dangerous to assume that you know how to correct it. Perhaps another person always rounds up, never down! Perhaps a different apparatus in a different location is sticky the other way and never reads as low as it should.
The classic example of this in climate science is UHI. Note well: HADCRUT4, IIRC, does not correct for the UHI effect at all. GISS — from what I’ve read — has invented a “UHI correction” that actually warms the urban present compared to the rural past more often than it works the other way, which makes absolutely no sense, since the UHI should nearly always introduce a warming bias compared to non-urban (but nearby) sites. UHI is a specific example of an entire class of systematic biases that can result from weather station siting — another one is the horrendous placement of official weather stations at airports, often right next to concrete runways and underneath air that contains many times the average concentrations of CO_2 and H_2O simply because enormous jets burn thousands of gallons of kerosene every few minutes directly overhead as they take off and land. Again this is almost invariably a warming effect — active hot greenhouse gas production concentration plus solar heated runways a meter or so thick surrounded by shops, parking lots and car-filled expressways in no way compare to evaporatively cooled grassland surrounded by evaporatively cooled trees. All ignored in HADCRUT4 and turned into more warming of the present compared to the past in GISS.
This warming bias can be seen at a glance on e.g. Weather Underground’s own personal weather station maps, in spite of the mediocre precision/accuracy of over-the-counter personal weather stations. I see it every day — the predicted weather (predicted to conform to the area “official” readings at RDU airport) is invariably 1-3 F warmer than the weather in my own back yard, or the temperatures reported by the many PWS in the surrounding rural countryside. The PWS temperatures in town are similarly warmer by a degree or two. You can even clearly identify hot side outliers — one sits only a mile or so from my house — where a PWS is systematically 4 or 5 F warmer than anyplace else, usually even RDU. I’m guessing that the PWS sits square over a south-facing driveway or that the thermometer is directly exposed to the sun. A second PWS, less than a mile from that one, reads numbers that are in reasonable agreement with my backyard and the general field of readings.
Now consider — suppose one has four “official” weather stations that one wants to keep in the official record (perhaps because they have long-running records). Two of them are located at airports one in a town, and the fourth is in an area park in the comparatively rural country. The two cities have, over the lifetime of the temperature record, gone from populations of a few thousand people riding horses to a few hundred thousand people driving cars to and from the houses that have covered the landscape over 100 square miles around. The airports have gone from handling a propeller-driven flight or two a day on a single runway to becoming local hubs servicing hundreds of flights with hundreds of acres of tarmac and runway where maybe ten or twenty acres was all that was paved at the beginning.
Along comes the automated “data homogenizer”. It notes that three of these sites have experienced substantial warming, and the fourth is neutral or maybe even cooled a bit. Aha! it thinks. An outlier! It rejects it, or worse, corrects it, and concludes that ten thousand square miles of surrounding countryside warmed like the cities simply because the thermometric record has a really, really significant urban bias today, and the major “official” anomaly computations correct the wrong way by both ignoring the UHI altogether and by “homogenizing” the record so that dissident sites — which might be the only ones reading the correct average temperature in spite of being outnumbered — are rejected in favor of the sites that outnumber them but have a common systematic bias.
As I said way up at the beginning, somewhere (or perhaps on a different thread) — it is almost impossible to tell what the temperature is today compared to the temperature thirty, fifty, a hundred, two hundred years ago, on the basis of the thermometric record. We (apparently) cannot agree on how to handle the dominant source of systematic error in this record — the relentless urbanization of the places where thermometers, including/especially the most “official” government run thermometers with the longest running records, are located. The breaks visible when (some) of those thermometers are moved are pure evidence that they were not consistently reliable on either side of the breaks, nothing more. Many of the changes that affect their consistency are simply not visible in “metadata” — they aren’t discrete changes that come from resiting or poor siting, they are changes that come about because of gradual changes in the entire surroundings of the siting, slow changes that one cannot observe, measure, or correct for other than to note that we “expect” most of those changes to result in warming from many things — including, BTW, increased atmospheric CO_2. But some of that might be local increases. Some of it might come from alteration of groundwater retention as forestland is converted to farmland is converted to suburban backyards surrounded by shopping and business centers and expressways with acre after acre of pure asphalt. A nontrivial fraction of the land surface area of the U.S. (for example) is pure pavement, especially in selected urban zones.
I read HADCRUT4 and GISS, as being “the temperature anomaly, relative to an arbitrary set point evaluated in the present and accurate to no more than 1C either way, as computed by a biased model that ignores or enhances UHI warming by some unknown amount and presented without any error bars as if it is a simple fact, a fait accompli, beyond question or doubt”. I then mentally subtract a guestimate for the ill-compensate UHI trend of a few tenths of a degree per century, add error bars that start at the HADCRUT4 acknowledged, modern error of 0.15C and scale up smoothly with time into the past to end up close to maybe 0.5 to 1.0C by the mid-19th century, where the error starts being constrained by independent proxy measurements and a string of plausible but possibly mistaken assumptions as much as by the lack of thermometers. Remember, over well over half of HADCRUT4 we basically knew next to nothing about the SS temperature of the oceans and much of the land area of the major continents.
By the time you properly dress the corrected curves with error bars, it is actually rather difficult to be certain it has warmed at all on a century timescale. Lief has indicated how they have systematically fixed systematic biases in the sunspot record. This latter record was made by responsible, competent scientists! Metadata was insufficient to make the correction, and even the corrected sunspot record doesn’t correct reflect the absolute state of the sun! They were only able to manage this because they had four independent ways to measure solar state — not four sets of sunspot measurements, four independent methods — which was enough to use two to check the third and correct the sunspots. Suddenly solar activity no longer has a grand maximum.
How improbable is it that we will eventually manage the same thing with the thermal record? How much of the “grand maximum” of temperature in the latter 20th century is a mix of systematic relative bias in the high frequency, highly accurate modern measurements compared to the less accurate and more sparse measurements made in the past, the uncorrected UHI effect corrupting the station data, the utter neglect and mistreatment of the SST component responsible for 70% of the Earth’s surface and the cavalier assumption that we have any knowledge at all about the temperatures in, say, Antarctica prior to the very recent past, if indeed we know them now?
It won’t come easy, and it won’t come soon. Lief comments on how hard it is to get “grand maximum” solar researchers to quit because their funding depends on it now, even after the notion is pretty thoroughly rejected. GISS was under the tutelage of James Hansen for most of its evolution, and neglect of UHI is the least of its sins — GISS’s funding today is almost entirely a result of the predicted progressive grand maximum in global temperature — it just didn’t exist until he talked the US congress into it after addressing them in a meeting with the capitol building air conditioning deliberately turned off, until an unknown person named Michael Mann wrote his own “special” version of PCA code that could turn a single series of bristlecone pine records from one part of the US into an international multicentury hockey stick. At this point, the record is so muddled that the only thing that could motivate an objective re-treatment of the data by (newly) objective researchers is a stretch of blatent and inexplicable global cooling, cooling too large to be “corrected away” in GISS or HADCRUT, cooling that is openly constrained by the harder-to-futz satellite LTT measurements and ARGO. Or perhaps enough years without much warming.
In the meantime, who really knows what the temperature anomaly is compared to 1850? We see assertions that it is 0.7C or thereabouts (sometimes even higher) but that neglects the error both systematic and not. Throw in the error and it might not have warmed much at all or it might have warmed by far more, 1.5 C, say. Arguably, the systematic error would make less warming more likely than more warming, especially in the recent past compared to the intermediate past. Was the latter half of the 20th century really a lot warmer than the first half? It’s hard to say. Just about exactly 1/2 of the state high temperature records were set in a single decade, the 1930s, in the first half of the century, with considerably less of a contribution from the UHI. Even with “global warming”, those records still stand well into the 21st century. Arctic ice was documented as almost disappearing during that same general period (although not documented as well given the lack of satellites or airplanes capable of overflying the Arctic).
In my opinion, we will not have a reliable picture of global temperatures prior to maybe 1950 or 1960 ever. A more conservative person might even push it up to the mid-to-late 1970s and the beginning of satellite measurements and make the reliable record only 30-40 years long. We are decades away from having enough data to say much of anything about the climate that might not be neglected or mistreated bias, measurement error and plain old statistical sampling error in the data and models used to estimate past temperatures, pardon me, past temperature anomalies since we know we cannot accurately estimate the global average temperature within more than about a degree now, in the modern era. This greatly complicates the science, the modeling, the politics, the economics. We very probably are trying to build models that cannot possibly work (based on what we know of chaotic nonlinear fluid dynamic systems and the limits on the scale of integration over the globe, limits on our knowledge of initial state, limits on our knowledge of many of the actual input numbers to those models) that are nevertheless being tuned to try to reproduce the output of other models that are being tuned to reflect each other and hence the collective bias of the people that are paid to build and maintain them because they show warming that the other models predict will eventually have a catastrophic impact.
Somewhere in there lies reality, but where? How would we even know? The bulk of “climate papers” are based on the predictions of the models that cannot even predict the past or present data models for the temperature outside of the tiny time period used as a reference/training set. The best thing to do is just wait. Time will tell. It usually does.
And in the meantime, it might be nice to take the infinitude of thumbs off of the scales, and not assume that we know the answer better than the data itself, well enough to krige, interpolate, homogenize, infill, backfill, and what the heck, just shift the data around wholesale for everything but a sane implementation of the UHI.
rgb