I’m not sure what the motivation for this is, none was given in the announcement.
The historical data contained in the 3200 directory will remain on ftp3 for a limited time.
This mirrors the removal of other datasets, such as USHCN1, from NCDC’s servers. Why not leave it in place? Older data has value, particularly when one wants to compare what has changed over time. Disk storage is cheap, particularly when dealing with numeric data. The datasets could fit on any hard drive with room to spare, so cost of storage isn’t an issue. My advice? Grab it now.
From NCDC’s whats new page:
Summary of the Day/3200FTP Subscription (data being replaced by GHCN-Daily)
March 23, 2011
Effective March 11, 2011 the Summary of the Day, 3200/FTP Subscription product has been discontinued.
There are major changes being implemented to the Summary of the Day/3200 dataset. Beginning with the January 2011 data month, the existing 3200/FTP product data (containing 3200, 3201, 3202, 3206) in the ftp3 area is being replaced with the GHCN (Global Historical Climatology Network)-Daily data. All users will have access to directories for ftp/ghcn/ and ftp3/3200 & 3210/ data at no charge. For more detailed information on the transition of the 3200 FTP Subscription data please see ftp://ftp3.ncdc.noaa.gov/pub/data/3200/IMPORTANTINFO.pdf. The GHCN-D web site at http://www.ncdc.noaa.gov/oa/climate/ghcn-daily/ provides detailed information on the dataset.
Also coming in the next few months: the Surface Data, Daily (3200/3210/3205/3206) accessed via the Climate Data Online (CDO) system, will be replaced with the GHCN-Daily dataset. Also a new interface for the Climate Data Online (CDO) system will include new Dynamic maps and many more updates and features… More information will be coming in the next few weeks.
Here is what is in the PDF
Important info for ALL who access this data:
There are major changes being implemented to the DS3200 data. Beginning with the January 2011 data month, the existing 3200 data (containing 3200, 3201, 3202, 3206) in the ftp3 area is being discontinued and being replaced with the GHCN (Global Historical Climatology Network)-Daily data. The 3210 data will continue to be updated. The historical data contained in the 3200 directory will remain on ftp3 for a limited time.
GHCN-D is a database that addresses the critical need for historical daily temperature, precipitation, and snow records over global land areas. Like its monthly counterpart, GHCN-Daily is a composite of climate records from numerous sources that were merged and then subjected to a suite of quality assurance reviews. The archive includes the following meteorological elements:
- Daily maximum temperature
- Daily minimum temperature
- Temperature at the time of observation
- Precipitation (i.e., rainfall and snow water equivalent)
- Snowfall Snow depth
- Other elements are reported by selected stations
The format of the GHCN-D data is different from the 3200 dataset and may necessitate some changes for users who are downloading data electronically. The format documentation and list of country codes can be found in the GHCN-Daily ReadMe file on the ftp server (http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt) The data files are located on our ftp server at ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/
The format of the ghcn-d file is arranged similar to the 3200 format with each row containing the station identifier, the year and month, the observation type, followed by 31 days of observations and flags. There is NO “Time of Observation” (TOBS) contained in this file. The GHCN-D data are currently available to ALL users at no charge. (Note: the current files are entire period of record for each station, updated daily, and are extremely large files).
We will be creating directories on the ftp server, within the /ghcn/ directory, that will contain subsets of the GHCN-D data in a year/month file and will be updated daily. The year/month file will be formatted so that every observation is on a new row, so its arrangement will be station identifier, date, observation type, observation, and time of observation. These files should be available in the near future. Separate documentation for this file format doesn’t exist; however, the definitions of element, flags, etc will be the same with the exception of time of obs.
All users will have access to directories for ftp/ghcn/ and ftp3/3200 & 3210/ data at no charge.
For detailed information on this new dataset visit the GHCN-Daily web page at http://www.ncdc.noaa.gov/oa/climate/ghcn-daily/
Please email questions/concerns to nndc.weborder@noaa.gov
Like I posted in the ‘The smallest sunspot cycle in two hundred years’
thread, once data has been adjusted to a new format, the original data has a very nasty habit of getting lost.
No exceptions here.
Just ask Dr. Phil Jones how that worked out for him.
And NCDC still has observers listed for my USHCN 040490 that it has provided no data for, and they still have not come up with it.
Regional Climates are everything, escpecially since we all live in one.
“Older data has value, particularly when one wants to compare what has changed over time.”
Or is about to.
Another “oops, we lost it” disguised as FOI? Um….
Maybe we can store the about to be removed data here on the WUWT pages along with already removed data that may be in the possession of readers. More integrity.
Why not leave it in place? Older data has value, particularly when one wants to compare what has changed over time.
I think the answer is in the question!
When you’re no longer the darling of the organisation with a “money no object – just get us the (good) publicity” budget, and they start putting the squeeze on….
Don’t you think you would also cut back on “free” services and deny access to the key information which you hope will form the core of your “consultancy” service on climate information?
gees, how much does a few terabytes of storage cost nowadays, or is there more than that?
Surely they should have an accessable archive for this data.
Consider the amount spent on a single inefficient wind turbine, then imagine the storage space you could buy !
Something very odd is going on here.
So, they say:
“There are major changes being implemented to the DS3200 data. Beginning with the January 2011 data month, the existing 3200 data (containing 3200, 3201, 3202, 3206) in the ftp3 area is being discontinued and being replaced with the GHCN (Global Historical Climatology Network)-Daily data.”
The quotation is a clear statement and admission that “the DS3200 data” is wrong so needs to be “replaced”.
Many of us knew the data was wrong before this admission.
What we now want to be told is whether the data which is to “replace” it will be sufficiently right for it to not need to be “replaced” in future.
Richard
Our friend Winston is at it again. Down the memory hole it goes.
“This mirrors the removal of other datasets, such as USHCN1, from NCDC’s servers. Why not leave it in place? Older data has value, particularly when one wants to compare what has changed over time.”
Older unmodified data tend to be inconvient facts when they get compared to “new” “adjusted” data. Just another way to “hide the decline”.
Hmm… 20 gigs of data… 1/150th of a 3 TB disk drive. A Hitachi Deskstar of that capacity is $180 at Amazon. We should thank NCDC for this vital, cost-saving measure. The deletion of this data will save over… $1… ($2 if RAID-1 is being used.)
In this era of government belt-tightening and financial prudence, NCDC should be applauded for reducing the deficit by 1/10,000,000,000,000th… (1/20,000,000,000,000th if you include social security and other unaccounted liabilities.) Great work, guys, only $19,999,999,999,999 to go! May I suggest you discontinue all weather observations and make weather predictions solely from computer models?
make your minds up – are you for cutting NOAA climate services or against it?
FYI, Amazon has the 3200 data stored as a “snapshot” on their EC2 facility. I stumbled upon it a week ago.
If you are familiar with their cloud offerings, this will make sense:
I started a new instance of a “micro” linux server.
Then, I created a volume and used snapshot “snap-ac47f4c5” as a basis for a new 20GB volume.
I logged onto the instance and mounted the volume manually.
A simple rsync command got the data down to me in a relatively short amount of time (text compresses really nice).
Once done, I deleted the volume and the instance.
Total cost was something like $0.09 for the server, $0.15 for the volume, and $0.50 for the data transfer.
If anyone is interested, I might be able to find some time to put together a document on the process.
The disappearance of USHCN v.1 data from the web is highly regrettable. I hope that someone has devoted disk space to preserving that largely unadjusted data set, because USHCN v.2 is grossly adulterated and unfit for scientific use. The coherence that neighboring stations once showed has been greatly degraded.
Dropping or blinding the citizens to a National historical database they fisically & to a large degree physically maintained without the permission of Congress ( the people ) is disgusting and completely in line with the arrogance & unproffessional conduct for which NOAA/ NCDC is now becoming well known.
At a time when politics & globalism are invading science & displacing measurement with retoric & illusion, the citizens must become sentinel… how convenient to blind them – and with their own money no less…
Speaking of which… What is this “provided FREE” language… given NOAA’s annual budget, the enormous slice recieved through the Job Stimulous Bill & the newest state of the art facility located ocean front Hawaii inhabited in January – all paid for by the taxpaying citizens… in return are we expected to kiss NOAA’s feet for allowing us to see a few temperatures? To expect some ethics? How has the master become the slave in this “prepaid” structure?
What happened to the 1961-1990 average? I don’t mind the centenial mean, but what is NOAA doing – trolling for the most radical positive anomaly?
PSU-EMS-Alum
Please do
Regards