Readers may recall some months ago that the Met Office planned to “do over” the surface temperature data sets:
Met office pushes a surface temperature data “do over”
The effort has started, and a website has been setup at http://www.surfacetemperatures.org
They write:
Surface temperature datasets for the 21st Century
To meet 21st Century requirements it is necessary to reconsider our analyses of historical land surface temperature changes. This is about much more than simply re-engineering existing datasets. These datasets were adequate for assessing whether climate was changing at the global scale. This current exercise should not be interpreted as a fundamental questioning of these previous efforts. But these pre-existing datasets cannot answer all the questions that society is now quite rightly asking. They do not constitute a sufficiently large sample to truly understand our uncertainty at regional scales. At monthly resolution they are also of limited utility in characterising extremes in climate and their changes.
Regardless of the causes, climate is not and never has been stable. Changes in climate impact all of society. But it is not changes in the monthly mean at scales of hundreds of Km that impact us all. It is changes at much more local scales and that last a few hours or days that have a major impact upon society. Whilst the long-term changes characterised by the current datasets can ameliorate or exacerbate the effects these datasets are fundamentally ill-suited to meet most of our needs as a global society.
The challenge set out in the proposal from the Met Office to the Commission for Climatology (CCl, see background link) is to produce a new suite of datasets capable of answering these questions. This requires more than the work of a single institution to do at all. It certainly requires very many partners to do properly. Following the positive outcome of the CCl deliberations an organising committee has been convened and the Met Office will be hosting a workshop in September in Exeter, UK. The organising committee, with substantial international representation, includes a broad range of expertise and perspectives and is undertaking planning activities. As yet planning is at too early a stage to publish details. More specific information will be posted by mid-June. However, the expected outcome of this meeting will be an in-depth plan with multi-institution sign on as to how to proceed. Broad aspects to be covered will be:
- Data recovery, digitisation and provision;
- data homogenisation and homogenisation system performance benchmarking;
- and communication, engagement and auditability
The plan is to solicit white papers on the range of topics to be discussed in advance of the meeting and post these on a (moderated) blog for broad input so that non-participants in the meeting can still have some meaningful input. The invitee list includes representatives from a number of relevant disciplines including a number that need to be engaged if the project is to be a success: climatologists; metrologists (measurement scientists); and statisticians amongst others. To be effective the meeting will have to be relatively small but, as stated above, stringent efforts will be made to entrain input from non-attendees in advance. And, of course, participation in the work will not be limited to attendees of this initial planning meeting only – to be successful it needs lots of participants, many more than will be at the meeting.
Update: 7/26 A revised version of the agenda is now available with only minor changes from the original. The white papers have begun to be posted at http://sites.google.com/a/surfacetemperatures.org/home/whitepapers (all hopefully to be posted within a week) and a moderated blog for public comments is available from http://surfacetemperatures.blogspot.com/ until August 23rd.
Update: 8/20 New version of agenda and comment period will remain open until September 1st.
==============================================
Here’s my take on it:
1) The effort, while noble, is a reaction to a series of data transparency blunders rather than a proactive approach to open replication. In the original Fox News article I cited on Feb 23rd, 2010 they write:
At a meeting on Monday of about 150 climate scientists, representatives of Britain’s weather office quietly proposed that the world’s climatologists start all over again to produce a new trove of global temperature data that is open to public scrutiny and “rigorous” peer review.
After the firestorm of criticism called Climate-gate, the British government’s official Meteorological Office apparently has decided to wave a white flag and surrender.
While this effort is a step forward, it is unfortunate that it took Climategate to break free the idea of open and transparent data, and of surface data that has gone through rigorous quality control procedures. As we’ve seen recently, Canada’s own surface weather data is in such a mess that Environment Canada squelched their own embarrassing internal report and it took a freedom of information request to pry it loose. They called the state of the network “disturbing’.
2) This statement from the white paper 3, item 8 discussed here says:
A parallel effort as an integral part of establishing the databank is required to create an adjunct metadata databank that as comprehensively as feasible describes known changes in instrumentation, observing practices and siting at each site over time. This may include photographic evidence, digital images and archive materials but the essential elements should be in machine-readable form.
Is essentially a stamp of approval of my surfacestations.org project. Without knowing the changes in measurement conditions surrounding the century long experiment in climate monitoring, it is impossible to know the true quality of the data. I see this as a positive step forward.
3) Making this effort known to the climate community has apparently not been a strong suite of the Met Office, for example, I only found out about it a couple of days ago via a reporter asking questions about my views on it.
The Met office needs to be far more proactive in communications.
I encourage readers to make submissions before the Sept 1 deadline, as only a few days remain.
Tony Armstrong. ‘Fraid not! That doesn’t apply, as the site is a website within its own right – because it has a separate domain name. There is no ‘Contact us’ on the surfacetemperatures.org site! It’s breaking UK law as it stands.
David, the CET record has been corrected for UHI (by certain Phil Jones btw ;)). Compared with truly rural Armagh Observatory in Ireland, there is no trend in their mutual difference so it is quite good representation of European/NH climate.
Strangely enough it appears that my local paint store is out of whitewash. Again!
The “do-over” of the surface temperature record has been in the making for quite a few month. I wonder for how long the Met Office have been at work to craft the announcement of the conference at which the constraints and specifications for the “do-over” will be hammered out. Moreover, I am concerned about the short time interval remaining between the release of the announcement now and the start of the conference on the first of September. The extremely short interval cannot possibly have been an outcome of an accident or because the Met Office were pressed for time.
Every single sentence in the write-up was put together with great care to express exactly what it is intended to bring across. The trick in reading the write-up is to determine not so much what is being said but what is not being said. Taking just the sentences in the first paragraph:
“To meet 21st Century requirements it is necessary to reconsider our analyses of historical land surface temperature changes.”
That means not changing meteorological data records but reconsidering how they analyzed them before and whether the conclusions reached before will still be promoted in the future.
“This is about much more than simply re-engineering existing datasets.”
That means the opposite of the first sentence, namely that the data sets will not only be changed, they will be *re-engineered.*
“These datasets were adequate for assessing whether climate was changing at the global scale.”
Aha! There was truly nothing wrong with the data they used before. The data did prove that there was climate change, namely global warming.
“This current exercise should not be interpreted as a fundamental questioning of these previous efforts.”
Naturally not, after all, it would not do to admit that anything was wrong with what the Met Office previously did.
“But these pre-existing datasets cannot answer all the questions that society is now quite rightly asking.”
Some ‘value’ needs to be added, perhaps extensive ‘re-engineering of the data’ probably cannot be ruled out, but let there be no mistake. The Met Office will not necessarily address questions society should be asking, it will address the questions “society is now quite rightly asking.” It seems to me that there is a critical distinction to be made between what is logically required and what is necessary to satisfy and perhaps indoctrinate (as before) the unwashed masses.
“They do not constitute a sufficiently large sample to truly understand our uncertainty at regional scales.”
That means that they must 1.) select a much larger sample for analysis from the data records created by weather stations all over the world, or 2.) manufacture data out of thin air (as E. M. Smith pointed out, homogenizing is a red flag, and that is mentioned later in the write-up), or 3.) a combination of 1.), 2.) and perhaps something else.
“At monthly resolution they are also of limited utility in characterising extremes in climate and their changes.”
Beyond insinuating that the resolution will be better than monthly, does that mean that they will produce re-engineered data at weekly, daily or hourly resolution? For instance, will they show daily minima, maxima, averages and standard deviations plus moving averages of temperatures?
The announcement does not inspire confidence and raises a lot of questions. I hope that the people going to the conference will get a chance to ask all of the questions that concern them and be successful in having them all addressed. More importantly, I hope that none of the answers they expect to get will be ruled on account of a preconceived agenda that is being advanced by the forces that steer and control the Met Office. Those things happen. They are called “bargaining in bad faith.”
Lucy Skywalker says:
August 25, 2010 at 1:07 am
There are still two huge things missing that I want to see before I can believe them
(1) Sorry.
(2) Thank you Anthony for your Surface Stations work that brought all these problems to our notice.
It is shameful that professionals should need to be brought to heel by amateurs. And what I don’t understand is the reticence in apologizing.
=================================
I could not agree more, Lucy.
What is even doubly shameful is that these people are “public servants” that is, not only are they not saying “sorry”, but they are doing so with taxpayer, public money.
Public servants have NO right…NOT to say sorry. They are not their own. They are owned by the public.
What sheer utter, and unfounded arrogance.
Again…as I said before….no wonder they are up for the sale chopping block.
Maybe, once they are sold and they are in private hands and have to depend on PRIVATE money like in the real world, will they become a little more real-world scientific and darwinian, in their approach!
Maybe the US should sell NOAA and NASA to the highest American bidder.
Maybe Canada should sell Environment Canada to the highest Canadian bidder, and so on.
Chris
Norfolk, VA, USA
I agree with Galane (August 25, 2010 at 12:52 am)
“To be credible, they should have at least two other parties involved in transcribing the existing historical data into electronic form, to ensure there’s no “value adding”, “homogenizing” or any other mucking about.
Simply enter the date, time and temperature etc from the logbooks as written.”
Nothing the MET office can do today ALONE can be trusted. If they do not join with others who retain scientific credibility, their new (taxpayer/government supported?) efforts cannot be respected.
Where to begin? With the last Ice Age, which we are still recovering from? Or the fact that atmospheric temperature changes are 1000 times less relevant than oceanic temperatures? Ignoring of enthalpy and humidity in the temperature record? The systematic destruction of Science–correlation taken as causation, consensus elevated to god-hood, and ad hominem argument endorsed as logic? Or the suppression of dissent, and incestuous peer review? Or the whitewashes that have established governmental units as full participants in Climategate after the fact? The system is rotten to the core; why should we expect this part of it will be any different? Without full public, front-page MEDIA REPORTED apologies, the credibility of all these governmental, journalistic, and climatological bedfellows must remain exactly zero.
The met office has blown any credibility it had IMHO, too many ridiculous assertions based on a political narrative and too many false predictions based on flawed data and methods.
The met office led and directed by by possibly the worst leader ever placed in such an office and the reliance on a very expensive super computer running corrupted models has destroyed a magnificent national institution and sullied its reputation as the father and mother of the art of meteorology.
When I think of hundreds of dedicated staff stretching back to the dawn of the industrial age and how they would feel now if they could but regard the wreckage that is the met office of today, what a tragedy.
It will take years of effort and a detailed reformation to rebuild the met office to a shadow of its former self, if they ever do.
I just threw some back of the envelope figures together about the costs of a world temperature database. For background, I’ve been a database administrator, architect and analyst since 1998, working on enterprise databases for a very large firm, so I know a little bit about that of which I speak.
First I took an impossibly dense network of temperature reporting stations: starting with one exactly on the North Pole, and extending southward on concentric latitudinal great circles 5km apart. That is, one at the pole, then four along a circle 5km radius from the pole, then seven along a circle 10km radius from the pole, etc, down to the equator. Unless I dropped a decimal point somewhere or went way off the tracks somewhere else, I got roughly 50 million stations to cover the entire earth with reporting stations within 5km of each other — each station covering an area of about 78.5 square km.
If you built a database with a 200 byte record length for each station to record that station’s daily highs and lows — along with identifying information on each station — you’d have 50 million records/day X 200bytes/record = 10 gigabytes/day for storage. Mutiply that times 365 days a year and you have 3.65 TB of data/year of raw temperature records for the Earth.
A two terabyte hard drive online is priced at $110, so it would cost about $220/year to store the data.
There’s a lot of other database considerations to take into account — but remember, that’s using a completely unachievable level of data collection covering the entire surface of the Earth. What we actually can do would use a lot less data storage.
In the field, we use that hoary old expression “He knows just enough about databases to be dangerous.” I’d bet every field uses a variation on that theme. And that’s just what the state of world temperature data looks like to me. My heart went out to that poor database guy in the emails who bemoaned the state of the data he was trying to make sense of — I see that sort of mess all the time as well.
To end this tome, I’ll just add that what needs to be done is to let some actual database professionals design and implement the next generation of temperature and climate records. This data is too important to be left to the amateurs.
Why does the presence of the word “homogenisation” make the hairs on the back of my neck stand up?
Because pasteurization is never far behind.
“This current exercise should not be interpreted as a fundamental questioning of
these previous efforts.”
Our “engineered” data has been shown to be flawed and inadequate, but the conclusions we drew from that data is not to be questioned.
As if, “we KNOW that the sun revolves around the earth, and those who question this are heretics. We simply are having a little trouble explaining the motions of the planets. We will re-engineer our measurements till we get it right.”
Thanks Juraj it’s useful to know that the Central England Temperatures as featured on the Met Office website are both adjusted to take account of the urban heat island affect and that the record is representative of Northern Hemisphere/Europe temperatures generally. But, in which case how does the CO2/AGW theory explain the past decade’s rather rapid and sustained cooling in the UK and across the NH?
PS. does anyone know where we can view the unadjusted CET data?
I still don’t know why the Wet Office et al insists Solar effects are not responsible for any apparent warming. As I have said before, the illogical attitude of their arguements when the IPCC reports openly state in the tables of all forcings, that there is a very low (UNIPCC 2001) to low (UNIPCC 2007) level of scientific understanding. This of course does give them a back door out of the room when the elephant crashes in! However, (not “But”), it’s a case of petitio principii, if you have a low to very low level of scientific understanding about something or anything for that matter, why play the ostrich & display an utterly cavalier disregard for its potential effects on climate? (The Wet Office, not the ostrich!) From personal experience when talking with my Wet Office fellow chorister earlier this year, he told me that the Wet Office does have top astrophysicists looking at possible solar effects all the time. Well, they’re either rather slow learners, or aren’t too bright, or somebody doesn’t want them to find any potential causal link! The exact nature of the Sun’s effects on climate may never be known, but to deny that there is any effect is just stupid.
It will be interesting to see if Britain gets another chilly blast from here there is no ice, if a similar situation occurs re salt-grit stocks. Already this year people were injured & some killed in road accidents due to treacherous conditions & roads had not been gritted. The opporunity to sue will not be far away once the blame game starts in ernest!
I missed the w in where! Apologies.
I posted a tip on this a couple days back.
The white papers are a must read.
Especially on the historical data portion.
we can hope that more historical data will be brought into the system. There never has been any “dropping” of data or stations. Just an inglorious task of historical research that stopped for GHCN in the 90s.
I think its better if we stop refering the change in number of stations as a “drop” as if they had the data in the database and dropped it.
As long as they are going to reconstruct the data, how about assigning a confidence level to each temperature data item this time? That way climate models using the data can report how much of their model result is reliant on interpolated, guessed, corrupted, or non-existant historical temperature measurements.
Aren’t there pristine, rural sites with long, continuous records throughout the world which haven’t been contaminated by UHI effects, and don’t the climate record centers have that data in their possession? Can’t we just see the raw data from these pristine stations, and then perhaps we will know how much temperatures have risen. It seems to me that most of the climate scientists working in the public domain, and receiving their funding and salaries from the public, should not have a problem revealing their raw data, as:
1. The public paid for it, and should have access to the data.
2. IMO, these folks are in violation of the law.
3. It is the ethical thing to do.
4. What are they trying to hide?
Of course, I know what the real problem is. Data from pristine stations are very likely to show only a slight warming, which is entirely consistent with what I and most others on this blog beleive. I have no doubt the world is warming slightly, as it has been pretty consistently since the LIA, and has been in a general long-term warming since the last glacial period.
But to turn the world upside down because of a slight warming which is consistent with what has happened in the past is for me not justified. But I do have an open mind to people who would engage me in a respectful, decent manner, exactly the way Dr. Curry did on two threads right here on WUWT last week. She made her case for her hypothesis in a forthright, professional manner. Are you listening, Phil?
“The head of the Met Office centre for climate change research explains why the momentum on emissions targets must not be lost.”
http://www.guardian.co.uk/environment/2008/oct/01/climatechange.carbonemissions
I think Walter Schneider 6.59 25/08 is absolutely correct in his analysis of the press release. We in the UK have had a decade and more of this type of carefully crafted spin from our former NewLiebor masters and their diktocrats. This double speak jargon is horribly like the outcome of the reports into the Iraq war (Hutton/Butler) for instance or the recent climate gate whitewash.
Every time you read one of these statements you must ask “what are they NOT saying” rather than “what are they saying”. I only hope that Anthony and the other sceptic heavy hitters can influence events enough to at least make the data/process transparent as the Met Office have already decided upon their conclusions.
Interesting comment that more, as well as better, data is needed. What does this say about the GISS/NOAA data in general, and the massive dropout of data in the last few years? Hansen has said that the dropout is not important after a stastistical review. Presumably a properly selected single station could show the trend. The English Met Office seems to have a different view.
They are wrong from the getgo. There is no mention of revisiting the papers that established the adjustments including UHI and homogenisation. Without do overs of this most important basis for the temperature record they have no credibility and the data they produce will be contentious and unsupported except by warmers and fanatics.
In other words they simply stated they would create more corrupt data sets.
IT’S WORSE THAN WE THOUGHT!!!
I applaud you vigorously, Anthony. Your surface stations project is turning out to be a truly great service to mankind. It has long been there, in their faces, begging much deserved attention. But now it is actually forcing accountability on this tightly controlled mass movement.
Their ultimate paradise:
History will tell, perhaps in less than a hundred years: First our forefathers started with the homogenization of temperatures, then, as time passed and they realized that there were inconvenient attitudes among some people then called “deniers”, they wisely decided to homogenize, also, the genus homo-sapiens into three well differentiated classes: The higher class, that of the leaders, called the most highly and venerable ALPHA CLASS, was formed by the direct descendants of our forefathers without any genetical manipulation at all. The lower second class, the class destined for the honorable service of the higher class, were the BETA CLASS, genetically managed so as to make them, at the same time, faithful servants and believers. The third class, that of slaves, called the GAMMA class, was engineered as not to think or discourse anything whatsoever, so they were born tongueless and genetically modified as to have strong and resistant bodies …..
To be successful this project needs buy-in from ALL interested parties. One crucial aspect is to assemble a group of such parties (listed with their credentials) – from professional societies to bloggers – and give them an opportunity to attach their own Seal-of-Approval (or quality rating) on individual datasets. This would be done after examination and testing using whatever documented criteria they like. There will be datasets everbody can agree on and datasets nobody puts much confidence in. Most will have a range of disagreement, but at least a preponderance of support or rejection. Evaluations might be more specific than a simple approval or rating. Categories of evaluation might include provenance of the data, metadata, proportion of missing or uncorrected data, etc.
Sure, it’s a consensus evaluation, but for basic data that’s mostly a good thing. Researchers then can decide which datasets to use and what criticisms will need to be addressed right from the beginning.
Lucy:
“It is shameful that professionals should need to be brought to heel by amateurs. ”
A strong element of truth, but also a trap. I am an amateur. Watts, Pielke, Wegener, Linzen, Spencer, Grey–It’s a very long list– are not. They bring the skill of their respective disciplines to the discussion. Don’t let the alarmist crowd dismiss dissenting views as the work of mere amateurs.
John