GISS Hockey-Stick Adjustments

Dear Mr. Dnes,
I discovered in March 2012 that NASA-GISS changed retro-actively their temperature data offered in March 2010. I had downloaded the 2010-data and put into my archives. Between March 2012 and January 2013 I evaluated the discrepancies between 2010-data and 2012-data for 120 stations. The results are dealt with in a comprehensive paper. Meanwhile a German version has been successfully peer-reviewed and will be published still this year. I prepared also an English Version and send it, among others, also to Mr. Watts but got no answer. I would like to send you that paper – perhaps we can cooperate – if I had your e-mail address. Please send it. Many thanks in advance and best regards
Friedrich-Karl Ewert

Peter says:
July 5, 2014 at 1:44 am
I was actually referring to all the temp dataset holders – which have been shown elsewhere to be erroneous (can’t remember the thread, but I had to point out that BEST di use adjusted data despite Mosh and others protestations to the contrary)
Returning to GISS in particular, I have no idea if GISS do adjustments ‘on’ adjustments themselves (but we know others do!) but my primary point was to say that with ALL the datasets, and remember that many of which are cross correlated to some degree as they share station data, etc – I have never YET seen a raw dataset, followed by an adjusted dataset COMPLETE with description of each and every ‘wave’ of adjustments (and I don;t mean a generic type description – I mean a station by station algorithmic and then manual QC check followed by careful checking of details (e.g. does the station still exist even though there is missing data! see recent wuwt thread! LOL) and recording of said findings and finally explanation of the adjustment made and why. Simply put – without such detail – any data made available or used in any scientific capacity (such as say, ;proving’ CAGW – lol again!) is scientifically invalidated as the scientific method is clearly being ignored.
When the harryreadme txt file was released it clearly shows that no one knows the situation with crutemp or hadcrut (cant remember which one) and hence the data is unverified – or more specifically UNVERIFIABLE in any way shape or form! (this of course, bearing in mind that Jones stated the raw data was not available anymore!).
I have no problem with adjustments – or more specifically ‘corrections’ being made – but it you cannot produce a full traceability trail for the ‘data’ – and I mean FULL – forget it…..without such traceability, the data and the data pushers are putting themselves up for ridicule!

Kev-in-UK said “However, when said data is used and upheld as scientifically valid – that changes the game – it must be reproducible and proven as valid. Frankly, to this day, I don’t think we have reliable data – and certainly not without questions as to its history or validity! Ergo, in almost any other scientific endeavour, the ‘results’ or ‘conclusions’ based on such data would be thrown out or at best held in very low esteem (think wagonload of salt here!)”
Kev,
Sure we’d all like perfect historical data with proof that is is valid, but for the historical readings we have got only what we have got, so have to make the best of it – you can’t go back in time to do individual station equipment quality checks. All you can do is to quality check your current equipment and then post-process and adjust the historical observations to see what you can get out of them.
“Throwing out” is not an option – we have no better data, and “esteem” should be expressed in more statistical terms.
Specifically, with post processing and adjustment of station readings you should not only emerge with a best estimate as to what the temperature actually was at that time, but also with a statistical calculation of the expected error in the readings. That error estimate can be used to tell you how much confidence to place in any results you get when you use the figures, and what the likely bounds of the results should be.
So maybe a scientist does a sound piece of research using the adjusted data, including a detailed error calculation, and produces a well-written paper containing, maybe a trend with error bars. What happens then? Well it probably gets published in a reputable peer-reviewed journal which only makes articles available online behind a paywall. Scientists and academics have access because their institutions subscribe, but general members of the public (who have often paid for the research in their taxes) would have to pay $20-30 dollars per throw to get at such stuff, and who can afford it, particularly since, despite reading the abstract, you don’t necessarily know until after you have bought a paper whether it really was worth it. Such is life!

RichardSCourtney said ‘The asserted “total waste of time” provides unquantified error to every analysis which uses the data set. No scientist can find that acceptable.
So, it IS “important to know what errors were present in every single one of the past versions of data”. And that can only be determined in falsifiable manner by knowing “exactly how correction of these errors has changed every single station reading”‘
My reading of your statement is that you are saying that you wish to know about every single station reading in every single version of a temperature data set because somehow it affects the errors present in the most recent version of the data set. If that is not what you are saying then you had better correct me.
Each new version of GISTEMP or UKHCN stands alone. The metadata documents all errors anomalies and identifiable inaccuracies which have every been discovered with the raw or source data and the code release for each version handles all of these. They are documented and handled not as fixes to a single weather station and/or single reading, but as a fix to a situation which is going to occur for multiple weather stations and/or readings. Further there will be an accompanying set of error data with each release of “best estimate” temperature data.
So we have a current list of situations handled by the current code. It is difficult to see how the subset of situations handled by previous versions of the code has any bearing at all on the best estimate temperature data or the error estimates for a current data set. Typically an old version would handle only a subset of the anomaly situations, or an inferior algorithm for a particular situation.
In other words once BEST (or whatever process you are using to validate a data set) had done comparisons with GISTEMP (or some other dataset) at a particular version and validated the processing, then there is no more to be said about any previous versions and no point in making comparisons between older versions pre-validation and newer versions post-validation.
Even in there is no validation process the presence or absence of differences between the current and past versions of a data set (particularly ones from 20 years ago) does not change the validity or error estimates for the current version, though they might indeed change the error estimates for the older versions.

Richard said “Scientific standards require publication of each and every change to each and every datum because the data has been changed and somebody who wants to use it may need to know the change; n.b. each and every change to each and every datum.”
The computer run to produce a specific version of an adjusted dataset always starts with the raw data and not with a previous version of the dataset. Thus there is no scientific or statistical requirement to document changes by station from previous versions of the dataset.
The documentation required is the following :
1) Raw input data
2) Specification of the processing which will be done on the raw data.
This will typically use language describing common conditions. For instance, it might describe how to handle missing readings from stations which would normally report. This is a generic process – there is no requirement to list in the specification all the days missing for each station and new data may come to light at some point which fixes the problem in later runs.
3) The computer code which does the processing.
Someone wishing to validate the process can read this code to look for problems and may wish to amend it to make it easier to check how the anomalies relating to certain stations were handled if it does not already write out interim files after each adjustment step. The code should clearly handle all the situations documented in the specification correctly. Another method of checking the code would be to write an independent set of code from scratch (which has been done for GISS adjustment code) and then just compare the two sets of output which should then be substantially the same, though not necessarily absolutely identical. It is very unlikely that the same bug will be present in two independently written versions of code although they might handle the same condition in very slightly different ways.
4) The adjusted output data
All this stuff is available for GISS temperature data.
No-one in their right mind is going to write a spec which includes a detailed set of processing requirements for every individual station – it is a total and utter waste of time as the information can be extracted from a computer run if necessary. It is possible that some conditions are in the spec and apply to only a single station and are documented as such, but this is likely to be a very small number.
The data on precisely what changes are made to each station reading during the different processing steps within a specific (normally the current) version of code are available from the computer run, if they are needed.
Now this is a definitive and complete set of information to tell an interested party exactly how the data adjustments have been done in a particular version for each an every station if necessary.
You seem to be suggesting that such validation should include looking at all previous data set versions , but clearly from the above this is not necessary in the process of validation in order to meet the requirements of the users, the science and the statistics.