While BEST is making a public relations train wreck for themselves by touting preliminary conclusions with no data analysis paper in place yet to support their announcements, there have been other things going on in the world of surface data integrity and uncertainty. This particular paper is something I have been assisting with (by providing metadata and images, plus being part of the review team) since May of last year. Regular WUWT readers may recognize the photo on the cover as being the USHCN station sited over a tombstone in Hanksville, UT photographed by volunteer Juan Slayton.
What is interesting about it is that the author is the president of a private company, WeatherSource who describes themselves as:
Weather Source is the premier provider of high quality historical and real-time digital weather information for 10,000’s of locations across the US and around the world.
What is unique about this company is that they provide weather data to high profile, high risk clients that use weather data in determining risk, processes, and likely outcomes. These clients turn to this company because they provide a superior product; surface data cleaned and deburred of many of the problems that exist in the NCDC data set.
I saw a presentation from them last year, and it was the first time we had ever met. It was quite some meeting, because for the first time, after seeing our station photographs, they had visual confirmation of the problems they surmised from examining data. It was a meeting of the minds, data and metadata.
Here’s what they say about their cleaned weather data:
Weather Source maintains a one-of-a-kind database of weather information built from a combination of multiple datasets merged into one comprehensive “super” database of weather information unlike anything else on the market today. By merging together numerous different datasets, voids in one dataset are often filled by one of the other datasets, resulting in database that has higher data densities and is more complete than any of the original datasets. In addition all candidate data are quality controlled and corrected where possible before being accepted into the Weather Source database. The end result is a “super” database that is more complete and more accurate than any other weather database in existence – period!
It is this “super” database that is at the core of all Weather Service products and services. You can rest assured knowing that the products and services obtained through Weather Source are the best available.
They say this about historical data:
Comprehensive and robust are the two words which best describe the Weather Source Historical Observations Database.
Comprehensive because our database is composed of base data from numerous sources. From these we create merged datasets that are more complete than any single source. This allows us to provide you with virtually any weather information you may need.
Robust because our system of data quality control and cleaning methods ensure that the data you need is accurate and of the highest quality.
As an example, here is a difference comparison of their cleaned data set with a GHCN station:
The processes they use are not unlike that of BEST; the emphasis is on solving problems with the historical data gathered by the NOAA volunteer operated Cooperative Observer Network (COOP). The COOP network also has a smaller subset, the U.S. Historical Climatology Network (USHCN) which is a hand picked set of stations from researchers at the National Climatic Data Center (NCDC) based on station record length and moves:
USHCN stations were chosen using a number of criteria including length of record, percent of missing data, number of station moves and other station changes that may affect data homogeneity, and resulting network spatial coverage.
The problem is that until the surfacestations.org project came along, they never actually looked at the measurement environment of the stations, nor did they even bother to tell the volunteer operators of those stations that they were special, so that they would perform an extra measure of due diligence in data gathering and ensuring that the stations met the most basic of siting rules, such as the NOAA 100 foot rule:
Temperature sensor siting: The sensor should be mounted 5 feet +/- 1 foot above the ground. The ground over which the shelter [radiation] is located should be typical of the surrounding area. A level, open clearing is desirable so the thermometers are freely ventilated by air flow. Do not install the sensor on a steep slope or in a sheltered hollow unless it is typical of the area or unless data from that type of site are desired. When possible, the shelter should be no closer than four times the height of any obstruction (tree, fence, building, etc.). The sensor should be at least 100 feet from any paved or concrete surface.
As many know and NOAA concedes in their own publications on the issue, about 1 in 10 weather stations in the USHCN meet the 100 foot rule. That is the basis of the criticisms I have been making for sometime – that siting has introduced unaccounted for biases in the data.
WeatherSource is a private company catering to business using a special data cleaning process that removes many, but not all of the data integrity issues and biases that have been noted. I’ve seen a detailed flowchart and input/output comparisons on their processes and have confidence that they are addressing issues in a way that provides a superior data set. However, being a business, they have to keep their advantage close, and I have agreed not to disclose their processes which are their legal intellectual property.
The data cleaning techniques of the BEST team made known to me when I met with them in February 2011 were similar, but they promised to use publicly available data and to make the process fully transparent and replicable, as shown on their web page:
So, knowing that BEST had independently identified surface data issues that needed attention, and because there was some overlap in the ideas pioneered by the privately held intellectual properties of WeatherSource with the open source promise of BEST, you can imagine my excitement at the prospect.
Unfortunately, on March 31st in full public view before Congress BEST fumbled the ball, and in football parlance there was a turnover to the opposing “team”. BEST released some potentially buggy and admittedly incomplete preliminary results (which I’ve seen the basis for) with no transparency or replicability whatsoever. BEST doesn’t even have a paper submitted yet.
However, the WeatherSource team has produced a paper, and they have it online today. While it has not peer reviewed by a journal, it has been reviewed by a number of professional people, and of course with this introduction on WUWT, is about to be reviewed worldwide in the largest review venue it could possibly get.
It should be noted, that the data analysis was done entirely by Mr. Gibbas of WeatherSource. My role was limited to providing metadata, photographs from our surfacestation project, and editing support.
Here’s the full paper:
An Investigation of Temperature Trends from weather station observations representing various locations across Utah (21.5mb PDF) by Mark Gibbas
Bear in mind, because this is a private company, and because small companies must hold their IP rights close to maintain competitive advantages, the data cleaning method can be discussed in general terms, but they cannot provide the code for review without compromising their IP rights. However, the source data used in this paper can be made available on request, as WeatherSource has tentatively offered access to it provided that the end user is doing research, does not use it for commercial gain, and does not republish it in its original form. Use the WUWT contact page if you would like to request a copy of the WeatherSource cleaned data after first agreeing to the caveats listed. Anonymous requests will be ignored. The metadata list of stations is available here: Utah-stations-metadata (PDF) Utah-stations-metadata (Excel .xls)
Here are some excerpts:
BACKGROUND
However, in this study done on a much smaller scale, our goal was to examine the long-term observational temperature records for weather stations scattered across the state of Utah, and attempt to identify local influences acting upon them. For instance, it is well known that urban areas can exhibit a sharp warming trend due to the influence of expanding urban infrastructure. Less well known are other influences upon temperature such as those associated with modern agriculture.
It is not until these local-scale influences are properly quantified that one can begin to infer fluctuations or trends that may be occurring on larger-scales, such as state, region, country, or global. This investigation is an important step towards that ultimate goal—but more importantly, shows that this goal is still difficult to divine given the uncertainty and methodology currently employed to extract a climate change signal from the surface temperature record.
…
APPROACH
To better understand the various influences on the observed temperature record, this study identified and examined numerous weather stations with long observational histories from urban, agricultural and natural wilderness or low impact locations (a low impact location is one that has experienced minimal alterations to its natural state) from around the state of Utah.
Initially, weather data from the National Climatic Data Center (NCDC) U.S. Historical Climatology Network version 2 (USHCNv2 2009)6 were to be used for this study. However, numerous data issues were discovered within the USHCN and as a result this data set was abandoned. Instead, the study used data from the original NCDC U.S. Cooperative Summary of the Day Data (referred to as the DSI-3200 dataset), that was cleaned via a comprehensive regression based data cleaning methodology. A brief explanation of the issues found within the USHCN will follow.
Within the cleaned DSI-3200 dataset, the study identified weather stations within Utah that had at least 50 years of data. Using tools such as Google Earth and demographic information based on the US census, these weather station locations were
ranked in terms of how urban and how agricultural they were. The ranking was done on a scale of 1 to 5 with 5 being the highest. Low impact stations were identified as locations that had low urban and low agricultural rankings.
Based on the urban and agricultural indices, the weather stations were divided into three groups representing urban, agricultural and low impact locations.
…
CAUSES OF TEMPERATURE OBSERVATION VARIANCE
One would assume that if you had two weather stations that were about 30 miles apart in nearly identical elevation and terrain and both recorded temperature consistently for fifty years, the time-series of these stations should correlate nearly perfectly and that both stations would show the same general trend. However, in practice we find that such stations often have significant differences in their observations even though they were both subject to the same large-scale weather patterns. But how could this be? The answer most often is related to changes that occur at the weather stations sites that influence the temperatures that are recorded.
…
Having established that station changes can artificially compromise the capture of real
weather trends, an obvious question has to be: So what good are all the station data if each station is subject to false trends due to station changes? When considering very small numbers of stations this is indeed a problem, but when more stations are averaged as a group, the effects of random station changes largely cancel out. The reason for this is related to the statistical principal that random actions (station changes) average out to zero whereas non-random actions (urban growth) do not.
Thus, to more accurately determine the trend of the urban, agricultural, and low impact
groups, we must average the members into a composite trend for each group. Chart 5 provides a view of the average trends from each group. JJA refers to the summer season months of June, July and August. DJF refers to the winter season months of December, January, and February. TMax refers to the trend of the daily maximum temperature and TMin refers to the same for the daily minimum temperature. The vertical axis is the trend in degrees per year.
In studying chart 5, we note the following:
- The trend in the minimum temperature is more significant than that of the maximum temperature. A thorough discussion as to why this is the case would be quite involved, so for now we’ll simply state that the dominance of the minimum temperature trend is as expected.
- The urban group of stations exhibits the strongest trend, followed by the agricultural group, then the low impact group.
- The difference between the agricultural group to that of the low impact group is more distinct in the summer than in the winter.
- The trend of the low impact minimum temperature is nearly the same in both the summer and winter seasons, whereas the trend of minimum temperature for the urban and agricultural groups is stronger in the summer.
Chart 6 provides a view of the aggregated summer and winter trend for the three groups in degrees Fahrenheit per decade. The urban value of 0.42F is on par with the published estimates of 0.36F to 0.44F per decade for current global warming.
The net trend for the agricultural group is 0.27F per decade which is 37% less than the urban group. The low impact group has trend of 0.20F per decade which is 52% lower than the urban group.
CONCLUSIONS
This study has established that there are significant differences in temperature trends
between weather stations sited at urban, agricultural, and low impact locations. In addition, it has been shown that urban stations exhibit significantly higher trends, which are created by numerous urban factors that are not related to global CO2 levels. Evidence has also been provided that agricultural locations experience a warming trend related to increases in humidity. Given that both urban and agricultural environments likely have warming trend factors that are unrelated to CO2 based warming, it makes sense that to get a more accurate determination of CO2 based global warming, station observations from low impact locations need to be given much greater weight. Currently that is not the case since the vast majority of weather observations originate at airports, which by design are near urban areas.
…
DAILY TEMPERATURE CLEAN METHOD
The method to clean the DSI-3200 data used in this study utilizes regression estimates from neighboring stations to validate, correct and fill missing values as needed to produce a high quality continuous time-series of daily temperature data. The cleaning method does not attempt to correct for urban warming, discontinuities or any other external influences upon the recorded temperature. Key features of the cleaning method include:
- Adjustment of morning, afternoon and midnight reporting times to produce working time-series arrays that are consistent across all stations used in the cleaning process.
- All regressions are based on seasonal time windows.
- Two types of regressions are done for each neighbor to target pair; a seasonal regression based on three years of data centered on a date of interest, and a seasonal regression based on all data. If the three year regression is stable, this value is preferred; otherwise the long term seasonal regression value is used.
- Regression estimates from all neighbors are collect into a distribution, which is analyzed and compared to the original value from the target station. If the target station original value is confirmed based on statistical tests, it is retained. Otherwise if it is found to be suspect it is replaced with the regression estimate from the station with the greatest correlation to the target station. In all cases, missing values within the target time-series are filled using the estimates from the highest correlating neighboring station.
The method outlined above has been found to produce values that are most consistent with the original time-series. This has been determined by qualifying accurate values then artificially removing these values and running the degraded time-series through the clean method. The resulting filled values are then compared to the original qualified values that were removed. These tests have routinely produced replacement values that when compared to the original values, have an R2 or 0.985 or greater.
– end of excerpts
One issue that was noted was how a formerly remote and rural USHCN station, the Snake Creek Powerhouse, had become surrounded by an Olympic sports center and later an irrigated golf course. It was made a special case study #2.
From case#2
The next plot provides the addition of the NCDC USHCN adjusted data. Unfortunately, the adjustment does not correct the issues with the Snake Creek site and in fact degrades the quality of the data further. While the trend for the Snake Creek station is suspect to begin with, the Clean DSI-3200 data set shows an upward trend of 0.246 or roughly a quarter degree Fahrenheit per decade. In contrast the USHCN adjusted data show a cooling trend of -0.05 or roughly -0.5 degrees Fahrenheit per decade.
Clearly this adjustment is wrong; otherwise this location in UT would be experiencing some of the most extreme cooling on the planet.
The adjustment applied to Snake Creek can only be regarded as alarming, and results in the following conclusions:
1. It is suspected that the adjustments are the result of a simple automated method
that applies a linear adjustment to the data such that the end portions of the linear
adjustment can have results that are unrealistic.
2. It is apparent that there is a lack of review and quality checking of the results.
…
end excerpt
While Utah is most certainly not the world, this paper does demonstrate what can be learned by doing a detailed look at station metadata in concert with temperature data cleaning and analysis. As noted in the paper, like BEST’s early presentation, it has not tried to correct for UHI or for humidity (moist enthalpy) which tends to effect night time temperatures the most, particularly in irrigated agricultural areas. Human experience knows this effect in southwest USA dry deserts, which can be very cold at night due to low humidity versus the Southeast USA, where humidity makes for warm summer nights. The question is how much are night-time temperatures being elevated by the addition of humidity from irrigation?
A study of California’s central valley in 2006 by Dr. John Christy of UAH did in fact do a small scale study in the California Central Valley where he took extreme care to hand code in reams of metadata in order to discover this effect. See:
Christy on irrigation and regional temperature effects
Until we get a method in place to look at issues like this and UHI, the surface record is still filled with uncertainties.
However, I find it encouraging that a private company that provides weather data where risk management is involved, where being right provides economic reward and being wrong means economic failure, has seen and dealt with the issues of NOAA COOP station biases, at least partially. Much more is needed.
The mere fact that company exists for the express purpose to create and sell corrected and cleaned data from our government surface temperature record, and they they have clients that need it, speaks volumes.
Details on this paper from Mark Gibbas is here at SPPI with a full download available
Given the move in NASA to private delivery of launch vehicle service, why not have private industry take a greater role in weather and climate forecasting?
It could be argued that government run and funded weather and climate services have little or nothing to lose when they get things wrong. Over time this will lead to poor services at high cost.
Private companies however have a lot to lose if they deliver poor forecasts. Over time this will weed out poor forecasters or high cost forecasters. The result will be better forecasts for less cost.
Hello Anthony Watts:
Can you say when the data here: “http://www.surfacestations.org/USHCN_stationlist.htm”, will be updated to reflect the latest station entries with their associated quality ratings? Or if there is another more up to date station list elsewhere (link?) with the quality ratings for each station?
Thanks,
REPLY: When our paper is accepted for publication (it is in late stage peer review now, shouldn’t be too much longer) that data and much more will be posted on the website – Anthony
It is wonderful what can be done by researchers financed by selling their results on the basis of the track record of success of their work.
It is demoralising what can be done by researchers financed by their academic positions who get funded irrespective of the track record of failure of their work.
Richard
It is great to see this published. Well done Mark!
I’ve only had a very quick look (it is late here) and look forward to reading it properly tomorrow.
Hey, I’m not trying to be difficult here but the word ‘robust’ makes me feal uneasy. I wonder why? ;O)
REPLY: Well I’ve met the man, and when he says “robust”, there’s no swagger with it. – Anthony
Why must they always assume that any temperature increase has to be due to CO2?
I don’t know whether editing suggestions are welcome are not. I certainly cannot improve you science.
“it has not tried to correct for UHI or for humidity (moist enthalpy) which tends to effect night time temperatures the most” (Effect should be affect?)
“where being right provides economic reward and being wrong mean economic failure” (Mean should be means.)
REPLY: Fixed thanks – A
I think I agree with them. The problem isn’t we don’t have enough data –the problem is we aren’t focusing on the best data we have available to us to tell us what it shows about the past, and to produce more of that kind of data for the future (i.e. “low impact” siting).
Great article. But the “cold desert night” myth always bugs me:
“Human experience knows this effect in southwest USA dry deserts, which can be very cold at night due to low humidity versus the Southeast USA, where humidity makes for warm summer nights. The question is how much are night-time temperatures being elevated by the addition of humidity from irrigation?”
As is true with much of “human experience,” one needs to be careful about conclusions. If you compare Southeast USA locations with Southwest desert locations AT THE SAME LATITUDE AND ELEVATION, you will discover that it is actually MUCH HOTTER at night (as well as daytime, of course) in the desert in summer! (Try Phoenix and Atlanta, e.g.). This is one of the reasons why I wonder about the atmospheric GHE.
The “cold desert night” myth lives on because most deserts and arid areas in general are higher in elevation than most humid areas and therefore cool much more at night. Alamosa, CO (7,534 ft. elev.) is certainly not a desert, but it gets very hot during a summer day and very cold at night.
Data: http://rredc.nrel.gov/solar/old_data/nsrdb/redbook/sum2/state.html
So, is it fair to say that the present state of temperature data makes it basically unsuitable for detecting supposed climate-forcing factors? It would seem that many serious decisions and interpretations are based on nothing more than guesses at this point!
Very nice work – thank you.
Anthony, my guess is that the website server will tank when your paper is published. No…matter…how…big…it…is. My respect always.
The money shot………………………..
Clearly this adjustment is wrong; otherwise this location in UT would be experiencing some of the most extreme cooling on the planet.
The adjustment applied to Snake Creek can only be regarded as alarming, and results in the following conclusions:
1. It is suspected that the adjustments are the result of a simple automated method
that applies a linear adjustment to the data such that the end portions of the linear
adjustment can have results that are unrealistic.
2. It is apparent that there is a lack of review and quality checking of the results.
Minor edit suggestion,
“…had become surrounded by AND Olympic swim center …”
Should be AN instead?
Interesting article. It’s becoming clear that a new well sited system will be needed to tease out the 1/100th degree of purported warming per year without corrections, and to cross check the current network. A tough task.
Hmm. The BEST quote of the week? 🙂
I guess the numbers in these charts are not to be taken seriously? Chart 6 shows a low trend of two tenths degree Fahrenheit per decade and a high trend that is twice that. That would mean a minimum of two degrees per century and a maximum of four. Do they mean to say that?
Can’t wait for the full up paper. Who would have ever thought the phrase “Hide the decline” had nothing to do with temperatures but everything to do with scientific integrity, basic morality, and data quality?
I find this interesting. If the trend was merely due to urbanisation, the winter minimums would be higher. This tells me (without knowing much at all about it) that these factors have been successfully compensated for. What has not been, however, is the additional heat retention of the surrounding environments, such as tarmac.
So the oft-heard statement from the team that UHI effects have been accounted for may be true, but there is another more obvious effect, the immediate environment, that has not. All that takes is actually going out there and checking, something that obviously has not been done by the team.
JAE says:
April 4, 2011 at 5:07 pm
Can we look at Australia for a moment?
In my experience the desert is very cold at night. I must admit, that is compared with the coast, where you have the sea to keep things warm. The elevation is Oz is certainly not a factor, however. Some of it is below sea level, for example.
Is it just the sea keeping the coast warm, or is the humidity also a strong factor?
JAE, Phoenix has a lot of UHI. How many A/C units are blowing hot air all day and into the night? How much energy has the pavement absorbed and is radiating at night?
Gettum’ Anthony!
How stupid do these people think we are?
How long do they think we are going to tolerate such complete perversions of the scientific method?
They rush stuff to Congress when it is expedient for them to do so….and then they stall peer review [take Lindzen’s paper which has been sitting on ice for months at Science], when it isn’t!!
Grrrrr. Time to light the torches.
Chris
Norfolk, VA, USA
Same reason mining geologists criticise the data aggregation methods etc used by academia and state sponsored science to come up with the global temperature metric – if we get it wrong the mining engineers, not least the shareholders – tend to do serious things to us – apart from the humility of explaining to the corporate watch dogs that we got the ore -reserve slightly out. It’s all about QAQC which this post is about, and comparing physical reality with the collected data. In the real world your models have to work, otherwise its the unemployment queue or a university appointment.
The CAGW camp are really not that much different to the speculators behing BrEx some years back.
The graph of the difference between GHLC and Weather Source shows a distinct upward trend most reminiscent of the warming reported by alarmists. It is not clear which is the minuend and which the subtrahend (look ’em up; I forgot my first grade arithmetic too), but if GHCN is the “top number,” then we have an indication tha the whole warming scare is due to bad data collection. Which is what I have thought all along.
Otherwise, it really is “worse than we thought.” Except that warming is good for plants and animals.
You would almost think that professional statisticians are employed by private companies today. This research is not really open to scientific debate but in the end they may be more trustworthy than a project providing us with a random sample from a data mess.
It seems that the main argument about the “dodgy” temp records has always been answered with “well, if you don’t like what the temp record says, gather your own data, make your own record, publish a paper, and then we’ll talk”.
Their turn…
Mr. Gibbas,
Congratulations on the release of your paper. It will take me a while to read it, but I’m down to page 14 and I’d like to comment on Cedar City, which seems to me to be an even better example of step changes than a casual reader might grasp.
For better than a year, I puzzled over the many MMS reported station locations where there clearly has never been, and perhaps never could be, a weather station. Lost a lot of time checking them out, too. Last year I found out why, at least in part.
It appears the weather people have been updating the station locations, but rather than correct the old, incorrect coordinates, they record the new corrected figures as a station location change. I discovered this first at Tejon Ranch, (California) where the observers insisted that the station has not been moved for many years, but MMS reported a location change. But if you look on the ‘update’ tab you see that on the date of the purported relocation, what was actually done was a coordinate correction. The station itself did not move–so say the observers, and I believe them.
A few days later, I found the same thing at Riddle (Oregon). Again, MMS showed a location change in the location tab, that coincided with a coordindate correction in the ‘updates.’ And again, the observer assured me that the station had not moved.
So I look at Cedar City with a suspicious mind. The reported May 28, 1998 move does not appear on the update tab. What does appear there is a coordinate correction (effective 8/25/98) and an earlier GPS correction (effective 12/11/97). The only location change shown on the update tab is in the period 6/1/94-10/1/95 (no effective date shown).
I would suspect that the 5/28/98 move shown on the location tab is fictional, except for one thing: the temperatures went sharply up in ‘95, then down in ‘98. Which pretty much illustrates your point about the effects of relatively short location changes.
This is a VERY, VERY, VERY good paper.
This is clearly done by a group of people that have to work for a living by providing good reports that are clear, concise and useful (ok, not all work papers are useful).
This is the best (no pun intended) article I have seen on this topic. It clearly shows the types of problems that show up with stations. I would not have considered agriculture to have such and impact, but it is clear that it does. Their idea of increasing the weighting the low impact stations would be helpful.
I am very impressed by the work these people have done.
Hey, don’t diss all university staff. In normal scientific and engineering fields, if you get it wrong there goes your reputation. We aren’t all climate scientists! 🙂
If you venture into the desert regardless of elevation you will find a larger temperature swing. When ever I venture into the Desert no mater what the season is I will carry a jacket. Here seventy degrees does not feel all that warm in fact I consider it some what chilly, I consider the eighties and nineties to be comfortable, I was sealing my roof on Sunday and it was in the nineties it was not all that bad. In the past when I was young in northern Minnesota, I have done roof work and a temperature like that would have been brutal.
As for day and night temperature swings even within the development where I live in the temperature can vary quit a bit. On our walk tonight near the center of the development there was a chill in the air so my wife put on a jacket, it was in the eighties today had by then cooled down to the low seventies or high sixty’s. As we walk out to the edge along the airport, a six lane road and a block wall it was rather warm, the coat my wife had on became very unnecessary. All of this was within a half of mile from our house. Yes building materials. lack of vegetation and moisture all can make a difference.
Just one last note, I have only lived here a very short time. Where I came from and where my sons and siblings still live it snowed this weekend, yes I do know cold weather. I was back there in early December and did get to experience below zero weather for a short while, the below zero temps were not considering wind chill.
Mark Luhman
Mesa AZ
For those looking for a reference for this. I think the general form is:
Gibbas M. (2011) An investigation of USHCN station siting issues using a cleaned dataset. In A.Watts, ed. Watts Up With That. April 4, 2011. [ONLINE] Available at: http://wattsupwiththat.com/2011/04/04/an-investigation-of-ushcn-station-siting-issues-using-a-cleaned-dataset/.
@Mark Gibbas:
Well done, Sir! Remarkably well done.
Perhaps we can suggest that as a budget reduction measure the folks at NASA, NCDC et. al. can be “retired” and the “data cleaning” work simply outsourced to a private company at far lower cost and with far better quality?
Ray says:
April 4, 2011 at 4:08 pm
Why must they always assume that any temperature increase has to be due to CO2?
Ray I think you may have misunderstood what was written and I’m sure you realise that by now. Their assumption wasn’t an assumption. It was, in part, a statempent of the opposite colour to the one you express.
As Joe Bastardi and Piers Corbyn say, when you living depends on your work you tend to get it right or starve. IT IS TIME TO STARVE NASA, BEST, NOAA, NCDC and all the other governement workers. RIGHT NOW.
If this new private team have yet to go gobal I hope they can do so soon.
It looks like they could really shake a few trees.
Thank you, thank you, Anthony!
You amaze me how you can pull up papers so fast that hit on the nose what was being discussed in the London UHI thread. Great paper.
@AusieDan
See, look at that Utah chart of the stations in seasons, Tmins and Tmaxs. Isn’t that what you were speaking of? Those relations seem to sum up Oklahoma’s stations anyway. It’s in the Tmins and in the rural stations where the difference is.
Of that 0.2 degrees C residual, it only takes a four degrees C change in the mean solar surface temperature to account for that, all of it. Does anyone not think that the sun’s mean surface temperature could not change by a mere four degrees, let’s say 5778 to 5782 K? Mars’s polar caps seem to think so, even though NASA’s satellites fail to record it. It is understandable though, those older instruments measuring TSI have an absolute accuracy of about 1% even though their relative repeatable precision is great, something like 0.01%, I saw that in the user manuals in the specifications near the last page of the PDFs. The older satellites were recording ~1367 while the current one is reading a bit over 1361, that I have seen. Makes you wonder since those evidentally have rather large error bars. That so much resembles the same thing seen in station temperature baseline adjustment, assumptions.
Mr Gibbas, excellent work and a very good example of why Capitalism in general and business in the particular can be superior to anything that a state enterprise can do – get it right and you (and your clients) prosper, get it wrong and…
State enterprises can be very good when they are managed and staffed by good men and women with a proper sense of mission and service, but when the sense of mission and service is lost or subverted by strange ideologies, so too is honour and the central purpose of the entire enterprise.
” surface data cleaned and deburred of many of the problems that exist in the NCDC data set.”
Deburred. Anthony, bravo. It is a delight to read a paragraph with an infrequently applied term which so precisely conveys a meaning.
Do you, too, have metalworking in your past?
REPLY: My dad was a machinist and welder, I could weld, braze, run a lathe, and fabricate with steel at a young age. – Anthony
but… UHI and other land-use doesn’t modify the trend in any way!! I read that someplace. These people must be wrong!
Ok I’ve read it. Like the format, like the method and the comments relate to the data and it all makes sense. What a difference to gov agencies.
Anthony,
One thing that is becoming increasingly evident in Australia at the moment is increasing low-level cloud cover. We haven’t experienced this type of of cloud cover since the 1960s. Indeed, it is often reminiscent of the cloud banks that we experienced in Alaska in June 2009. I suspect that that Svensmark is correct and that the current weakness in sunspot activity and increased cosmic radiation penetrating the earth’s troposphere are the triggers behind the sudden increase in low-level cloud cover.
In Sydney, daytime temperatures in the CBD and at Sydney Airport are still above average – primarily a consequence of UHI and overdevelopment of the CBD and surrounding suburbs . However, as one moves away from the city, to regions beyond the Sydney basin, the daytime temperatures are starting to drop in relation to the long-term averages, whilst the night time temps remain close to the long-term averages. I suspect that nocturnal cloud cover in such locations is the primary reason for the maintenance of nighttime temps close to the long-term minimum temps outside of the basin. If, however, the current “cooling” trend continues (in response to a weak solar cycle), we are likely to see nighttime temperatures follow the daytime temps in a downwards direction.
Greg B.
The fact that there are companies willing to pay money to buy from this company, the same “data” that the govt provides for free, shows us how much confidence these companies have in the quality of the govt provided data.
I was under the impression that corrections for UHI were applied to the raw data.
How else could the satellite data agree with the land based temperature data if the land based measurements are so bad?
Anthony, you still don’t get it. Muller can make preliminary announcements about the results because he knew what the results were going to be before they even started looking at the data. What they’re doing isn’t science unless you want to call what actuaries do “actuarial science”. I call it “pencil pushing” and making the numbers say what you want them to say I call “pencil whipping”. As far as I know these phrases are engineering slang that predates me.
The AGW crowd is not going to like a bunch of upstart, dirty capitalists poking around in the attic.
I wonder if during your conversation with Mr. Gibbas whether or not you discussed the cases where stations have been physically moved into or out of one group or another, and how he deals with those cases. I think of Crawfordsville, IN, when I ask that. Recall a few years back we discussed that station as having spent many years within the downtown area next to the college and then the power plant, then being moved well outside the city to a farm. Because it is “rural” in the present day, GISS uses all of that station’s data to adjust nearby urban stations with “less pristine” data. The record for Burlington, VT, also was affected by similar moves, originating in downtown, moving up toward the University, spending a period of time in farmland, and then finally landing at the airport.
Muller clearly implied that UHI was a non-critical factor, and that urban stations were similar to rural ones, based on a few highly selective reports.
° This flies in the face of the satellite infrared, that shows the urban hot spots clearly.
° This flies in the face of already established SOPs for sites, that Anthony, you have shown to be in violations of protocol more than 80% of the stations. Muller implies that instrumentation does not have to follow established SOPs, since there is inherently no difference among good stations and bad, they give the same mean result. So I can go out and add my outside plastic duck thermometer to the database with no qualms, since SOPs be damned!
Imagine, if in my dealings with the FDA, that I would arrogantly assert that my electronic balance did not need servicing, ever, and that my beam balance was suitable for weighings! And that I did not need a QC department or a QA department to assure that the measurements were recorded properly and with a witness. Trust me, I know what I am doing! Do you think they would accept the data, if I showed that the mean was close to the proper SOP set performed under the guidelines much of the time, at least in a selected dataset that suits my hypothesis?
I think not. SOPs for instrumentation are why we have ISO (international guidelines). Such guidances set forth strict rules are necessary to obviate the corruption we have all seen happening with the surface record. I don’t know what kind of scientist would accept that. A political scientist?
JerOme: You say:
“Can we look at Australia for a moment?
In my experience the desert is very cold at night. I must admit, that is compared with the coast, where you have the sea to keep things warm. The elevation is Oz is certainly not a factor, however. Some of it is below sea level, for example.
Is it just the sea keeping the coast warm, or is the humidity also a strong factor?”
Without some data and metadata, I can’t say much. What latitude are you talking about (Australia is a very big place)?
How cold is “very cold?” What time of year?
I don’t see how you can say that elevation is not a factor. Do you mean that all of the desert is at sea level or below??
It’s been my experience that if you have an onshore wind from the sea, the air temperature will be very close to the temperature of the sea, for quite some distance from the sea, and little else matters.
There is always more diurnal variation in desert regions. This is due in part to the fact that convection rate is proportional to temperature (high daytime temps mean faster convection rate). (Newton’s Law of Cooling). Part may be due to humidity–the higher the humidity, the more heat is stored in the air overnight, which decreases diurnal variation. Part may even be due to the “atmospheric greenhouse effects” of water vapor 🙂
After reading over the article a couple of concerns come to mind.
Regional weather stations were never designed to account for anything other than local conditions and are frequently sited (Airports for instance) to account for runway temperature not regional or local weather conditions.
The idea of cobbling local weather station data together to reflect regional or global temperature trends seems fundamentally flawed.
I also find it curious that NASA is publishing a 1.5F change from 1880 to the present when NASA instrumentation and data only accounts for a few recent decades. What in the world are they using to account for the other decades of temperature to arrive at this trend and with what degree of accuracy?
The mere fact that company exists for the express purpose to create and sell corrected and cleaned data from our government surface temperature record, and they they have clients that need it, speaks volumes.
I agree but I hope they aren’t attempting to use the data for global climate projections and keep it regionalized and in context to ENSO known patterns.
Dave Springer says:
April 5, 2011 at 7:39 am
Muller’s testimony did get the paranoid conspiracy part of my cerebrum working. I imagined a conversation taking place, on a throwaway cell phone of course, that went something like this:
Caller: Richard?
Muller: Yes, who is this?
Caller: Who I am is not important. Let’s just say I play on the good Team.
Muller: What do you want?
Caller: I hear you are testifying before Congress.
Muller: Yes, tomorrow.
Caller: I hope you will be careful in what you say. I would not want to see any negative fallout from your testimony.
Muller: What do you mean? I will tell the truth.
Caller: The truth? They can’t handle the truth.
Muller: I can’t lie.
Caller: Just be very confusing and contradictory. That’s all we ask.
Muller: I don’t know if I can do that.
Caller: We have faith in you, Dick. Besides, we want you to know it is a dangerous world out there, and we would not want you to put yourself in jeopardy.
Muller: I don’t follow you.
Caller: Yes, you do. Just be careful.
[dial tone]
I’m sure this is just a fantasy caused by many contradictions out there. Of course this never happened…
A wink is as good as a nudge to a blind man, after all.
@Mark Gibbas – Thanks, Mark, for taking the time to check things out. Looks like we need to take more care of how we do climate observation and spend less money on the fictional climate models.
Without good data science becomes nothing more than a dogmatised belief system.
Looking at Fig. 5, the smallest change in the three areas looked at (urban, agricultural and low impact) is in Dif-Tmax. This ,too me, is an indication that Tmax is the best signal for warming/cooling. If we attempting to measure changes to the incoming signal (solar heat gain) then Tmax seems to be the best metric. Late afternoon signals and Tmin are a measure of heat retention with or without the benefit of moisture and are just an after affect of heat gain.
Tmax only is a much clearer signal.
Reply to
JAE said:
April 4, 2011 at 5:07 pm
Great article. But the “cold desert night” myth always bugs me:
I have lived in North Carolina, and now live in Idaho. I haven’t been here long, but I am convinced that desert nights are in fact much colder. To check, I found two stations at about the same elevation, both rural, in NC and ID. I averaged the day high and night lows and found the difference. For Meridian, Idaho, the average delta for August 2010 was 38.35 Deg. F. For the same month, in Little Switzerland, NC, it was 21.1 Deg. F. Quite a difference. Elevations were 2572 (ID) and 2560 (NC).
If you were comparing day and night temps in city environments, like Atlanta and Phoenix, you were probably only observing the importance of UHI.
Remember that, as a rule, night lows cannot be lower than the dew point. If the humidity is high, the temp cannot drop as much as it would in a dry area. The enthalpy of the moisture in the air going from gas to liquid is too much to easily overcome.
So the cold desert night is a physical fact, not a myth.
Agree w/others, Tmax is the best measurement for trends. Inversions (or lack of) make nighttime lows problematic & very dependent on local winds, elevation of the sensor, surrounding impediments, etc. Pielke Sr has documented this. Sunlight during the day causes thermals & winds that disrupt the inversions & reliably mix the air near the ground (except in the polar regions where sunlight is absent).
Whatever analysis Anthony eventually comes up with should separate out the daytime highs/nighttime lows & compare the resulting trends. I realize this could be very difficult from a data viewpoint, but otherwise, the combined trends are not really useful.
The suggestions to cut NOAA’s budgets for the next decade are very appropriate. Reorganization and downsizing (RIF – Reduction In Force) should also procede without regard to seniority. Every federal department should also go through the same process. We’re NOT broke, we’ve just got too many boot-licking in-laws on the payroll.
Re: Cold desert nights
Depends on the desert. The low deserts of California, Arizona & Sonora, for example, stay pretty damn hot late into the summer nights, urban OR rural. By low, I mean elevations of 1,000 ft (300 m.) or less.
But hotter longer in bigger cities, most notably Phoenix, with midnight temps of 100F pretty common in the hotter months. IMS this is also true in the Persian Gulf, and also Bagdad & vicinity.
RE: UHI stuff
An interesting research project for amateur (or professional) climatologists would be to study urban microclimates using private weather stations (PWS). Here, for exmple, is <a href="“>WUG’s map of Phoenix, showing (by eyeball) 35 or more weather stations in the metro area, including some out in (maybe) open desert, for comparison. Maybe Anthony can speak to the accuracy and reliability of these little stations, but I’d guess they’re no worse than many official stations. WUG keeps archives, indefinitely?
Anyway, interesting project for someone. I’m sure most PWS owners would be happy to have their instruments calibration checked.
Thanks to all for the supportive comments. Also a BIG THANKS to SPPI (http://scienceandpublicpolicy.org/) for providing funding for much of this work.
A little bit about me…
I am neither a global warming advocate or global warming denier. My only bias (so-to-speak) is I have an unquenchable passion for truth. Also I’m an avid outdoorsman, I enjoy skiing, hiking, kayaking and everything else where I can be in the wilderness. Consistent with this lifestyle, I am a fierce advocate for protecting our environment. And I am a meteorologist/mathematician/programmer. I guess this grouping of characteristics is a bit odd, but I believe they serve me well, at least I hope.
A few short years ago, after having reviewed the many studies on global warming from NASA, IPCC, and others, I did indeed lean towards the idea that carbon based global warming (CBGW) theory was valid. From my perspective back then, the evidence was compelling. There was indeed a documented rise in temperatures coinciding with a documented rise in CO2. This together with the fact that CO2 is a ‘greenhouse’ gas seemed to sway the evidence in favor of CBGW theory.
However sometimes it turns out that things are not so simple, and in recent years I have viewed other evidence that suggests CBGW theory may not be entirely accurate. Specifically there are four pieces of evidence:
1) It has come to light that the oceans are a (hugely) significant sink and source of CO2. Warm ocean waters emit CO2 into the atmosphere while cold ocean water absorbs CO2 from the atmosphere. Over the past many years we have all seen the charts showing the correlation of CO2 and temperature. CO2 goes up and temperatures rise. CO2 goes down and temperatures fall. In all this, it has been implied that CO2 is what is driving temperature trends. But the fact that warm oceans emit CO2 forces us to challenge this assertion. Correlation is not causation after all, so it is quite possible (and perhaps likely) that increases in ocean temperature have been the cause of increased CO2. In other words, global warming may cause CO2 increase.
2) Studies by NASA have suggested that the net radiative balance between CO2 warming and the affect of subsequent increased cloud cover limits or negates CO2 warming. In other words the net effect of increased clouds serves to mitigate any warming CO2 may produce.
3) It can be argued logically that the rise in temperature and atmospheric CO2 over the past 60 years was caused (to a large degree) by a lengthy cycle within the Pacific ocean where there occurred a high frequency of El Nino and positive PDO conditions. These combined conditions worked to increase global temperatures and increase CO2. Over the past decade, the Pacific has been in a state where La Nina and negative PDO condition persist and during this time global temperatures have declined.
4) Lastly, it has become obvious to me that the treatment of global weather observations, whether intentional or by happenstance, has lead to a view that the global temperature is warming at an incredible rate. The truth is that many specific locations are warming rapidly to due to other factors such as urbanization and agriculture, where as other low impact areas are not warming at all. One argument that this is happenstance is the fact that the vast majority of temperature observing sites tend to be near developing areas, and thus the aggregate of these stations vastly over shadows the near zero trend exhibited by low impact stations.
So, being a person who seeks truth, I had to dig into the details of global warming myself. Today after having reviewed lots of science on both sides, and having done my own research, I am currently of the opinion that CBGW is, at a minimum, being over hyped. Is it possible that CBGW is valid to some degree? Perhaps. At this point I can’t rule out that carbon emissions may have some influence on warming temperatures – but to answer this with any confidence we simply have to learn more (and unfortunately much of the research done up to now is questionable for various reasons). That said I do have considerable doubts that CO2 emissions will result in extreme global warming scenarios that have been proposed by some groups (at least with respect to current theory)
Recommendations for proving/disproving CBGW:
1) We need a more quantitative understanding of how the oceans affect global temperature and CO2 levels.
2) We need to quantify, as objectively as possible, the affect clouds play in CBGW theory.
3) We need a much more sound and robust weather/temperature dataset from which we can objectively track global temperature trends.
These three items really wouldn’t be that hard to address if sufficient (yet modest) funds were provided to a team of objective, qualified, non-political scientists.
Lastly, over the past many years, my knowledge of CBGW has evolved and along with this my stance has changed as well. But what is my stance? What action do I recommend? Well, here it is… For many reasons (other than CBGW) I believe we should be working aggressively to wean ourselves off of carbon based fuels. Decreased emissions, energy security and improved energy efficiency are just some of the reasons. In the long run, we’ll need to have alternative energy sources as carbon based fuels will eventually run out. And something about a world that is powered by simple things such as the wind, the sun, the tides, and hydrogen just sounds like the right way to go. For me it’s almost an occam’s razor kind of thing… The simplest solution is the best.
And to all my friends who see my ‘telling the truth’ as giving CBGW and Big Oil the green light, well, I’m sorry if I may have caused you to be concerned or angry. But the telling the truth and keeping science honest is VERY important. Otherwise we can be lead into a false future with a lie. If CBGW is real, the advocates of this theory are going to have to prove it using fair data that does not conveniently, yet falsely, bolster the case for CBGW.
Peace,
Mark
Mike Bromley says:
@Mike Bromley: Is it fair to say the data is unsuitable? The difficulty I have with that answering that question is the answer is not a simple yes or no. The data certainly has issues, but it is not totally useless. In addition, there are effective methods for correcting the data. Lastly, even with data that is known to have issues, statistical measures of uncertainty can be helpful in quantifying the results.
juanslayton says:
@juanslayton: Welcome the weather data head scratchers club! I remember puzzling over the data from Phoenix years ago. It was a frustrating, yet enlightening moment. Little did I know that this research would become central to my activities for the next 10+ years. Anyway, the lesson I learned is that while the station metadata can sometimes be helpful, in many cases station changes are simply not documented. As such, the only way to objectively quantify station changes is through an analysis where a stations time-series is recursively compared to the time-series of other nearby stations. This is confounded by the fact that neighboring stations can also be affected by changes of their own, so this needs to be managed in the analysis. In the end, if done properly, one can obtain a ‘station change signal’ that when applied to the original data, can effectively remove artificial inhomogeneities.
Huub Bakker says:
@Huub Bakker. I’ll second your position. I think it is important to remember that to optimize the advancement our knowledge, we all need to work together in a civil and constructive way.
From a distance, it seems to me that determining the change in the temperature of the atmosphere of the planet earth is much akin to determining the number of angels who may be dancing on the head of a pin, given that angels know how to dance.
A single number representing the change in the temperature of the earth’s atmosphere on an annual basis is a ridiculous number to attempt to generate. What is the real significance in question?
Whether we are a net radiator of solar energy or a net sink of solar energy over time.
Gathering data to record, manipulate, average, mean, median, mumbo-jumbo a bazillion measurements is a waste of time. They are a multitude of measurements, made willy-nilly, at differing times of the day, with different wind conditions, with so many other factors subject to subjective ‘correction’ that it is impossible, IMO, to attempt to reduce those multitudinous measurements to an accurate single number.
Gather by the gigaton, cut with a chainsaw, and measure with a nano-probe.
In actuality, the problem is determining actual solar {‘insolation’ (?), I don’t even attempt to know the terminology, so you can ignore me with full rectitude} radiation that is received, that which is radiated out to space, and that which is absorbed. If you cannot measure that, measuring all the temperature points on the planet is still a waste of time.
You may fire when ready, Gridley…
tom
Interesting point in “When considering very small numbers of stations this is indeed a problem, but when more stations are averaged as a group, the effects of random station changes largely cancel out. The reason for this is related to the statistical principal that random actions (station changes) average out to zero whereas non-random actions (urban growth) do not.”
I’d want to review the station changes. If weather measurers were responsible they’d be moving stations to sites less affected by things like A/C unit exhausts, and would ensure there are an adequate number of stations in remote areas such mountainous (where temperatures are cooler because of altitude).
(Noting however that a great many stations are at airports for a reason – airplane operations, so shouldn’t be moved. Instead more should be set up.)
John Goetz says: April 5, 2011 at 9:06 am “Because it is “rural” in the present day, GISS uses all of that station’s data to adjust nearby urban stations with “less pristine” data.”
Why bother using the urban stations as data to look for trends in? The urban stations are no longer independent.
“The record for Burlington, VT, also was affected by similar moves, originating in downtown, moving up toward the University, spending a period of time in farmland, and then finally landing at the airport.”
IMJ that station is useless for trend analysis.
“Same reason mining geologists criticise the data aggregation methods etc used by academia and state sponsored science to come up with the global temperature metric – if we get it wrong the mining engineers, not least the shareholders – tend to do serious things to us – apart from the humility of explaining to the corporate watch dogs that we got the ore -reserve slightly out.”
“Slightly” with tongue in cheek I expect. 😉
And sometimes investors fail to do the most basic checks – as in the Bre-X fraud? (Where an assay report described alluvial gold in samples supposedly crushed from rock.) They lost their investment.
Some of this climate alarmist stuff fails simple checks, gullible humans don’t check, so they will force the rest of us to reduce our quality of life.
Global warming is real. Just ask the the people of Tuvalu. Simply proofed. Whats worse though is how the atmosphere is becoming even dirtier. In fact this kind of thinking does not consider the current respiratory diseases that comes from open burning (including combustion engines). Nuclear energy at least keeps the atmosphere temporarily clean (until something bad happens and a Chernobyl occurs). Oh, and don’t rely on Governments to tell you the truth – it is after all the same as big business.
Huub Bakker says: April 5, 2011 at 12:20 am
“Hey, don’t diss all university staff. In normal scientific and engineering fields, if you get it wrong there goes your reputation. We aren’t all climate scientists! :-)”
Good point, but I ask if you are working to get university hierarchy to understand how much bad work or worse goes on in academia. (“Worse” is for example the academically incompetent racist demagogue employed by the University of British Columbia.)
Think how much more money might be available for your work if funding for such was eliminated.)