The NCDC press office sent an official response to Politifact, which is below.
The NCDC has not responded to me personally, I only got this by asking around.
I’ve provided it without comment.
=====================================================
Are the examples in Texas and Kansas prompting a deeper look at how the algorithms change the raw data?
No – our algorithm is working as designed. NCDC provides estimates for temperature values when:
1) data were originally missing, and
2) when a shift (error) is detected for a period that is too short to reliably correct. These estimates are used in applications that require a complete set of data values.
Watts wrote that NCDC and USHCN are looking into this and will issue some sort of statement. Is that accurate?
Although all estimated values are identified in the USHCN dataset, NCDC’s intent was to use a flagging system that distinguishes between the two types of estimates mentioned above. NCDC intends to fix this issue in the near future.
Did the point Heller raised, and the examples provided for Texas and Kansas, suggest that the problems are larger than government scientists expected?
No, refer to question 1.
==================================================
Wow! Never explain, never excuse.
Business as usual.
Who goes there?
What we are seeing here is entropy in data. My undergraduate P. Chem professor explained entropy this way. “If you add a millileter of wine to a liter of sewage, you get 1001 ml of sewage. If you add a milliliter of sewage to a liter of wine, you get 1001 ml of sewage. That is entropy.”
The same concept applies to mixing real data values with estimates or interpolations. If it is not ALL 100% real data, then it is not real data.
For my money, the best computer representations of climate and weather are produced by Bethesda Softworks for the Elder Scrolls series.
The response from NCDC does not reflect any insight in the challenge at hand. My hypothesis is that average value provided by NCDC is not, and cannot possibly be, suitable for its intended purpose. I would like to elaborate my view on this:
From Wikipedia:
“Metrology is defined by the International Bureau of Weights and Measures (BIPM) as “the science of measurement, embracing both experimental and theoretical determinations at any level of uncertainty in any field of science and technology.”
“A core concept in metrology is metrological traceability, defined by the Joint Committee for Guides in Metrology as “property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the measurement uncertainty”. Metrological traceability permits comparison of measurements, whether the result is compared to the previous result in the same laboratory, a measurement result a year ago …..”
From Guide to the expression of uncertainty in measurement:
“This Guide is primarily concerned with the expression of uncertainty in the measurement of a well-defined physical quantity — the measurand — that can be characterized by an essentially unique value. If the phenomenon of interest can be represented only as a distribution of values or is dependent on one or more parameters, such as time, then the measurands required for its description are the set of quantities describing that distribution or that dependence.”
How the temperature measurements could have been used in this perspective:
Each of the temperature measurements can be well defined (e.g. in terms of location, time of day and equipment). Each of the measurements can be traceable to international standards through a documented unbroken chain of calibrations.
If each of the measurands are well defined, the average over a set of measurements, each performed at defined times at defined locations will also be well defined. Further, the average can be calculated again and again and will provide the same result every time. It is also possible to use statistics to calculate standard deviations and it is possible to calculate the uncertainty of the measurement from standard deviations and uncertainty estimate for the individual measurements.
Hence the average value produced in this way can have metrological traceability with a stated level of uncertainty.
How about the average temperature values from NCDC in this respect?
The temperature value produced by NCDC (National Climatic Data Center) seems to fail to meet the requirements to metrological traceability in several ways. Most seriously:
The measurement is not repeatable. It produces a different result for a given period back in time every time the measurement is performed (the model is run). Furthermore the lack of repeatability seems to be very significant in relation to the intended used of the measurement.
Another fundamental flaw is that the measurement result is not traceable, it cannot be calibrated. The measurement result is not traceable primarily because the measurand “the average temperature” is not well defined, and consequently you cannot find a traceable reference to calibrate it towards. The model, algorithms and software integrated in the measurement system complicates the definition of the measurand to such a degree that it cannot possibly be calibrated in a way that will provide traceability towards a defined reference.
Further, the uncertainty of the measurement result, estimated in accordance with an acceptable standard, is not stated for the measurement result (see: Guide to the expression of Uncertainty in Measurement.”
Hence, it is no doubt that the model that is intended to produce an average temperature of some kind keeps a lot of people enthusiastically engaged. However, the model does not meet the fundamental requirements for it to be an acceptable measurement. That is, if the intended use is to identify a trend or change in order of magnitude 0,01 K/year.
All standards and principles within measurement have been made so that people can trust and agree on measurement results. The temperature output from the model fails to meet these standards and fails to produce a useful measurement result, primarily because the model, algorithms and or software implementation does not provide a metrological traceable and repeatable result with stated uncertainty.
JN says:
July 1, 2014 at 9:53 pm
“our algorithm is working as designed” = “There was some bone-headed decisions… Not even a smidgeon of corruption.”
So INS, IRS, VA, EPA, NSA, NLRB, Federal Reserve, policies on Iraq, Syria, Russia, Ukraine, etc. — everything is “working as designed.”
—————————————————–
You left out health care (HHS) and Homeland Security.
This is all kind of scary.
cn
Steven Mosher says:
July 1, 2014 at 10:38 pm
“However, this can all be avoided by using the method suggested by skeptics:
oh wait, that would be the berkeley method.”
Do the Berkeley method provide a traceable, repeatable measurement result with stated uncertainty for a well defined measurand?
“Their conclusion:
“It is noteworthy that while the various time series are highly correlated, the adjustments to the RAW record result in a significant warming signal in the record that approximates the widely-publicized 0.50°C increase in global temperatures over the past century.”
At least now Nick Stokes know this so why is he so anxious to promote CAGW rather than taking on the Lintzen attitude that a doubling of CO2 will lead to a +/- 1C increase in temperatures. Considering that CO2 has a logarithmic effect, an addition of 120ppm has caused 0.5C increase so a doubling giving 1C is in the “ballpark”. That is discounting any natural variation. What is all the fuss about other than making money from CAGW by some people?
Isn’t it just a great pity that so many skeptics pooh-poohed Goddard’s claims without reading them and understanding them properly. The toothpaste is now out of the tube and no amount of post-facto blogging is going to put it back. Politifact has denounced the allegations as “pants on fire” and nothing will change that. When will people learn that concerning AGW claims, if they are skeptics, they only get one chance to get it right and maybe not even one?
Latitude says:
July 1, 2014 at 6:22 pm
It’s just their press office…….The NCDC press office
____________________________________________
And that press office works “as designed” with the purpose of bumfuzzling the rest of the world.
Reads like FYTW.
Non Nomen says:
July 2, 2014 at 6:45 am
Latitude says:
July 1, 2014 at 6:22 pm
“It’s just their press office…….The NCDC press office”
I cannot imagine that a press office will issue anything that has not been internally approved by a director.
Their response may not work as designed.
“4.2 OBTAINING ARCHIVED VERSIONS
At this time, the National Climatic Data Center does not maintain an
online ftp archive of daily processed USHCN versions. The latest
version always overwrites the previous version and thus represents the latest data, quality control, etc. ”
Does this mean that the data, like emails, is no longer available at all, or just they have it but won’t make it available?
Why would a gov. entity charged with providing accurate data…change the data daily and not keep public records as to the changes?
How can a government agency charged with providing accurate data….keep changing the data and decide the latest is now definitely accurate?
How can a gov. agency that produces the base data forming the foundation for their own belief that the world is in for a climate catastrophy…credibly change the data daily and not make the changes publicly available on an issue so important to mankind?
The fact they create data…and they do create data. whether they admit it or not…would alone be a valid reason to throw out an expert opinion in court based on it. The fact they keep changing the data constantly for past years would make it less likely to survive scrutiny in court But the admitted fact their data is regularly wrong and changed by nearly a degree, while claiming they have proven the world is warming by a degree, is almost laughable. Would love to cross examine the NOAA on this. DNA science would never be accepted in court as reliable, on data that constantly changes and is constantly manipulated like this
If ‘all scientists agree the data needs adjustment”…that might work for a one time adjustment…maybe, if clearly explained but I’m not even there yet..
But to admit to changing historical data on a daily basis , then toss yesterdays data as if it is meaningless, and claim that ”today we got it right, yesterdays was wrong”…very troubling.
B
Anthony,
You made a big deal of saying that these folks are all professionals and that they would quickly review and fix these sorts of obvious errors …
Do you still stand by that assessment ?
REPLY: That is an excellent question. From the delay so far I can surmise that the problem is real and probably is more of a problem to fix than they may have originally thought. The problem itself might be simple but the effect of the problem might be very very inconvenient. As a result I think they are trying to figure out how to diplomatically deal with it without looking like boneheads.
If the problem wasn’t real, we’d have gotten some sort of govspeak issued almost immediately that basically says “Watts and Goddard are idiots”. But instead, we are getting delays. If they try to stretch it into the holiday weekend, we’ll know they have been circling the wagons, and then the questions of credibility will be valid to ask. – Anthony
amongst other points ,cooling the past makes no sense.the past WAS cooler already. surely with the onset of uhi the later measurements should have been cooled,not the earlier ones ?
No – our algorithm is working as designed.
Ah, it’s “Hide the Decline II” time, is it?
Their statement is factual. It is working as designed. The question is, what is the value of that design. It certainly has _some_ value as statistical tools and algorithms always do, but it’s the context that matters. Does the general public understand that at least ~40% of that representation is manipulated by the algorithm? I doubt it.
“…working as designed.”
I have run into this response before. Quite a long time ago now I was conducting a great deal of literature research where the authors frequently included 2X2 contingency tables and then proceeded to “explain” the tabled data however seemed reasonable to them with no quantitative examination at all. I at first would load up a stat program to run a Fisher’s Exact Test, which is a handy first tool for examining categorical data in a contingency table. I hated the wait for the stat package to boot and decided to write my own utility (in TurboPascal). The routine is simple enough that with a small N, the results can be checked by hand.
In doing this I discovered just how stupid computers really are and how careful and really, really paranoid a programmer has to be about results. Since the marginal totals are fixed, there should be no difference for a given set of cell values in results that is dependent on mirrored arrangement of the numbers in the cells. A routine had to be included that double-checked the arrangement to be sure that results were not the inverse (1-P, rather than P) for instance. As a double, double check I checked my results against commercial stat software Fisher’s Exact routines. Three out of three commercial packages I had access to were sensitive to cell arrangement and would yield erroneous results to arrangements with mirrored symmetry – 2,2|2,1 for instance would give a different result than 1,2|2,2. I contacted the individual publishers and two of three responded that the error was “not possible” and their software worked as expected. the third said they would look in to it and later sent me a “thank you note.” All three packages yielded correct results in the next release.
The idea that software works “as designed” in any complex system is likely to be illusory. If results look “reasonable” there is a high chance that an error will not be detected. The “zombie stations” problem exmplifies the entire problem with in-filling and estimating data.
I wonder how many hours they spent concocting that statement, I can imagine the conference room strewn with discarded pizza boxes, empty coffee containers and redolent with the odor of unwashed bodies.
Defensive “spinning” at it’s finest.
“Our algorithms are working as designed”
——————
That was a CYA response to the question asked,….. but still an absolutely true statement.
The same as …. all computer programs that contain “bugs” resulting in mistakes, error or crashes …… “are working as coded”.
So the creation of zombie stations is part of the design?
Let’s go back to the original Goddard issue.
Zombie station might have been part of the design as a simplifying assumption when there is one zombie in 20 stations.
The population of zombie stations is growing grotesquely large to where the zombie hoard is almost half the total population.
Zombiegate indeed.
No, Mr Mosher. That won’t wash.
As I said, there are TWO approaches to constructing an average
1. Create, manufacture, find, LONG STATIONS : GISS and CRU
2. Use the raw data AS IS and estimate the field.
Option 2 was first proposed and implemented by SKEPTICS
http://noconsensus.wordpress.com/2010/03/25/thermal-hammer-part-deux/
The man who invented this is RomanM.
When we started at BerkeleyEarth our head statistician consulted with Romanm to understand
what he had done.
RomanM used a simple but effect least squares approach.
We improved that by using a refinement called Krigging
That idea was first proposed at Climate audit.
So there you have it.
two approachs:
A) find or create LONG STATIONS and average them CRU AND GISS
B) use all the raw data to estimate a field, an approach first implemented by skeptics.
Oh ya, when skeptics first did this, they showed a warming trend that was HIGHER
ya’ll forget that
“Do the Berkeley method provide a traceable, repeatable measurement result with stated uncertainty for a well defined measurand?”
we are not doing measurments. we are creating a prediction ( an expected value)
given
A) the raw data
B) a geostatistical model of climate.
So the right questions are
A) have you tested your prediction? Yes, it validates
B) do you have uncertainties– yes they are required to do #1
At the web site http://www.berekleyearth.org it is stated:
“The Berkeley Earth averaging process generates a variety of Output data including a set of gridded temperature fields, regional averages, and bias-corrected station data.”
Here is the Wikipedia explanation of the terms prediction and Measurement:
“A prediction or forecast is a statement about the way things will happen in the future, often but not always based on experience or knowledge.”
“Measurement is the assignment of numbers to objects or events. It is a cornerstone of most natural sciences, technology, economics, and quantitative research in other social sciences.” ….. .. “The science of measurement is called metrology.”
“Metrology is defined by the International Bureau of Weights and Measures (BIPM) as “the science of measurement, embracing both experimental and theoretical determinations at any level of uncertainty in any field of science and technology.”
I would say that you are providing a measurement result. Hence, you should adhere the standards for measurement.
What? No Hitler hissy fit video yet?
Steven Mosher:
I write to request a clarification of your post at July 2, 2014 at 12:57 pm which includes this
You “not doing measurments (sic)” does not absolve you from a need for an independent calibration standard when have “tested your prediction” to determine “it validates”. Your “prediction” needs to be compared to something if its validity is to be established.
Please say
(a) What are you predicting?
and
(b) How do you “test” the result of your “prediction”?
and
(c) How does that “test” determine “it validates”?
Thanking you in anticipation of your answers
Richard
“..then the questions of credibility will be valid to ask. – Anthony”
Anthony,
you may have noticed Nick Stokes running around the web making this claim –
“TOBS is an adjustment made when a station changes its time of observation. The amount depends on the change.”
– about TOB adjustments in the USHCN “Temperaturette” product. (I can’t believe it’s not temperature!)
When pressed at “Not a Lot of People Know That”, Nick linked to the USHCN V2 methods, which in turn links, guess where? Tom Karl’s pet rat TOBy –
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/papers/karl-etal1986.pdf
From the abstract –
“A self-contained computer program has been developed which allows a user to estimate the time of observation bias anywhere in the contiguous United States without the costly exercise of accessing 24-hourly observations at first-order stations.”
The Feb 1986 paper starts well, dealing with solar and human time zone issues, but then switches to estimating TOB without metadata. The conclusion claims as much as 2C corrections may need to be applied and specifically mentions “climate change”. This paper was submitted in 1985.
Nick quotes on this thread –
“An important feature of the USHCN is an extensive metadata file aiding in adjustments to the temperature data associated with station moves, instrument changes, microclimatic changes near the station, urbanization, and/or time of observation biases.”
“and/or”? If TOB adjustments have been applied to stations that only ever reported hourly or “zombie” stations, that can only be TOBy’s nibbling, not properly referenced metadata.
Tom’s pet rat TOBy may have been nibbling on temperatures and leaving his dropping in Stevenson screens from one side of the US to the other for quite some time. The question now is whether little TOBy has a cardboard box with air holes and a little passport. Is TOBy an international traveller?