Berkeley Earth releases new version of the BEST dataset

oh, if folks want source to version 1.1 to the package they can just write me. Its been posted to CRAN, in line for building… sometimes that happens in a day or two.

Thanks to Mosh for all his efforts on this.
Thx also to Greg, from Spokane for the link.

cui bono says:
February 17, 2012 at 4:08 pm
Thanks Mr. Mosher, Mr. Watts.
This ain’t going to work on my pesky ZX81! What’s the best way (if any) to edit down the data set to the last n years? [Sorry if stupid question.]
=========================================================================
My Commodore 64 is eagerly awaiting 22nd BEST. It can handle “42.” What are you crying about? ;o)

wmsc.
Currently the package is focused on just reading in the various files into R friendly data structures.
Given the large number of files, that’s going to be a big task that requires a bunch of testing. You have to start somewhere. After that, I’m sure people will want to understand every step between the various files. So that will be added to the package. Next, people will want to undestand how the scalpal works, so thats on the table. Of course a matlab version will aslo be made available, And I will continue to emulate the entire system in R. Its a huge task to put this code into a FREE format.
For additional functionality I’ve coded it to work with my other packages. So you can read in best data and run nick stokes method, or crus method, maybe Gisstemp if I get a break, taminos method, jeffid/Romans method..
So, baby steps, Now my time is consumed with trying to make this beast fit in small memory.
I could say screw you all go buy a bigger machine, but I figured that some guys might appreciate the effort, plus I get to learn some new R packages. that makes me happy and peaceful.
V1.0 is posted, V 1.1 is in the build que, and 1.2 is going to be done tonight

At face value this BEST exercise looks like an excellent resource. They have several different categories of data such as “Quality Controlled”, and “Seasonality Removed”.
To be fair, we sceptics cannot moan about both rogue data (we yell, “hah! Faulty instruments skew the data – GIGO!”) and massaged data (we yell, “They’ve been fiddling the figures! Warm bias! Just give us the pure unadulterated source data!”). This immense dataset would appear to make the whole lot available. I say such transparency is to be welcomed.
Now, who among us has the stamina to wade through it all and find the fallacies behind the warmist claim that the Arctic is roasting?

Maybe some commentators here have never tried to reconstruct and analyse temperatute/time series. Just for Australia alone, with about 1,00 stations, Simon Torok noted in this 1996 PhD thesis that –
“Station history documentation was investigated for each long-term temperature station. The files typically contained 250-300 items of correspondence per volume, and each station had one to three volumes. The earliest correspondence in the Station history files was generally from 1908, when the colonial meteorological responsibilities were passed over to the Federal Meteorological Bureau, although earlier information was available from ROs. However, most files commenced correspondence in the 1920s. Station history documentation for WA was very detailed in the early sections of the files (prior to 1930) and inspectors were particularly meticulous.
A summary of notes made regarding of changes that may have affected the temperature record is given in Appendix Al. The summary focuses on changes in conditions rather than ongoing details. Notes such as site location, painting of screens, changes of observer were recorded but are not listed in the Appendix. Further summarised details relating to temperature measurement may be obtained in hand-written summary form on request from the author. Full documentation is available in NCC. The date of the first correspondence archived at the BoM in Melbourne is listed, however earlier information was often found through other sources, such as ROs and State archives”.
It should be obvious that there is a huge task to bring data up to date. There are thousands of pages of data. They were never collected in the early days with the knowledge that they would be subjected to this scrutiny for this purpose.
………………………
Here, in a ligh-hearted vein, are a few of the problems that Torok found. Why don’t you insert yourself in the shoes of Steven Mosher or the BEST team and tell us how you’d deal with them, quantitatively?
Thermometer exposure.
Hanging under gum tree facing west under galvanised iron verandah against mud or stone walls.
Thermometers originally placed in screens on a suitable site were, at whim of observer, taken on to a balcony twenty feet above the ground.
Also brought close to or inside the house, so as to be more convenient for the reading.
Attached thermometer on the barometer inside the house read by one observer.
Screen’s exposure and condition.
Beer case used temporarily
Stevenson Screen painted cream. Or brown, or green, or silver, or not
Football found inside screen
Birds enter screen, to drink from the wet-bulb thermometer well.
Pumpkins growing all over yard, including beneath screen.
Cows, goats and other stock gather around screen and other instruments.
Observer skill and attitude.
Observers hard to replace as they were sent to gaol or to the war.
Prisoners make observations in gaol.
Observer lacking in height asked to make the observations while standing on a box, to avoid parallax error.
Observer not able to make observations at the correct time due to work commitments, so helpfully estimated the temperatures later in the day.
Observer described as good, “provided his enthusiasm does not wane on cold, wet, or windy days.”
Observers known to send in their monthly climate returns early, temporarily forgetting the number of days in a particular month.
Observer sent an entire month of entries to the BoM before they had been made, in order to have a month’s holiday.
Site closed by BoM as “the readings were taken by girls.”
Observations of temperature during a heat wave suspected to be exaggerated, as employees in area paid more when the mercury topped 100°F (37.8°C) .
During an outback feud, regarding who was to have the privilege of making the climate measurements, the telegraph lines were cut to prevent the efficient provision of observations.
Other problems.
• Postal employee caught willfully smashing the thermometers.
• Irate wife, perhaps tired of being woken every morning as her husband made the 3 am observations, took to the valuable screen with an axe, turning it into a pile of firewood.
• Eagle destroyed Stevenson Screen by flying into the side of it.
• Horses known to knock down Stevenson Screens and, more recently, cars and trucks have taken on this task.
• Termites wreak havoc on wooden screens.
• A dingo once stole a thermometer which had been read following the slaughter of farm animals (the observer was advised to wash his hands in future).
• Therrmometer broken on nose of observer’s dog.
• Laundry torn when it caught on the Stevenson Screen after being hung above it to dry.
Remediation suggestions can be sent to me, but a civil response should not be assumed.

Philip Bradley says:
February 18, 2012 at 1:45 am
This is the same nasty data we have ALWAYS had access to more or less. We have a dodgy series generally of one minimum and one maximum temperature value per day
AFAIK it is exclusively one minimum and one maximum temperature value per day.
##############
wrong

Philip Bradley says:
February 18, 2012 at 1:45 am
This is the same nasty data we have ALWAYS had access to more or less. We have a dodgy series generally of one minimum and one maximum temperature value per day
AFAIK it is exclusively one minimum and one maximum temperature value per day.
I’d like to see someone compare the BEST min/max derived average to the average derived from hourly measurements available for quite a number of sites.
###############
there is literature on this. read that before you suggest that someone else do busy work.

Until you get past the drudgery of getting the raw numbers in the best form possible, you can use exquisitely sophisticated mathematics to derive little more than a big argument.
Please, stop the argument and turn your minds to materially assist with data quality validation. Major errors reside in some countries. Please help fix them.