A question for Zeke Hausfather

Zeke is upset that I made this statement in a story at Fox news:

Is history malleable? Can temperature data of the past be molded to fit a purpose? It certainly seems to be the case here, where the temperature for July 1936 reported … changes with the moment. In the business and trading world, people go to jail for such manipulations of data.

he says:

In the spirit of civility, I would ask Anthony to retract his remarks. He may well disagree with NCDC’s approach and results, but accusing them of fraud is one step too far.

I’d point out that Zeke has his interpretation but nowhere did I say “fraud”. He’s mad, and people don’t often think clearly when they are mad. That’s OK.

Without getting into semantics, I’d like to ask Zeke these simple questions:

  1. What is the CONUS average temperature for July 1936 today?
  2. What was it a year ago?
  3. What was it ten years ago? Twenty years ago?
  4. What was it in late 1936, when all the data had been first compiled?

We already know the answers to questions 1 and 2 from my posting here, and they are 76.43°F and  77.4°F respectively, so Zeke really only needs to answer questions 3 and 4.

The answers to these questions will be telling, and I welcome them. We don’t need broad analyses or justifications for processes, just the simple numbers in Fahrenheit will do.

Advertisements

  Subscribe  
newest oldest most voted
Notify of

Anthony, come on, I can already hear the crickets warming up…. LOL

Absolutely! It’s well past time to stop the history revisionism.

Doug Proctor

The warmists don’t understand that the amount of warming the “consensus” of the IPCC believe is human-caused is of the 1.0F difference between CONUS and other versions. That the possible UHIE inaccurate correction is 40% of the “unprecedented” warming. That an unknown but if going only from 28% to 26%% in global cloud cover you can account for 100% of the warming. Or that a regional change, say a 28% to 18% change over only 20% of the planet, could do the same.
Global warming is a process of fractions made out to be thing of large, whole numbers. As if multitudes of “extreme weather” events were the way the world is getting warmer, so you could see a wall of heat moving across the landscape. A reversal of cause and effect.

Isn’t it said that “History is written by the vanquished?” no no wait … that can’t be correct …
.

Doug UK

How can current “warming” be proved by making yesterdays temperatures cooler?
Was that what Global Cooling was all about in the 1970’s? (sarc)
Keep your eye on that pea now!

jeanparisot

Eventually, CAGW will be a text book case of stock manipulation.

Anthony:
As you know, all the climatological temperature time series data sets are changed with time, and it is a rare month when at least one of them does not alter past data. This is not only true for the US temperature compilations: it is also true for the hemispheric and global temperature time series.
It seems appropriate to remind people of how GISS global temperature data sets have been adjusted, and this picture is worth a thousand words
http://jonova.s3.amazonaws.com/graphs/giss/hansen-giss-1940-1980.gif
Richard

Brent Hargreaves

I’d like to ask Zeke why his organization reduced the recorded Jan 1900 temperature at Teigarhorn, Iceland from 0.7C to -0.2C. http://endisnighnot.blogspot.co.uk/2012/03/giss-strange-anomalies.html
If historical temperatures are adjusted downward (they did this from 1900 to 1962) and more recent temperatures are bumped up then….. oh, you finish the sentence Zeke. You use the word “fraud”; I use the word “fiddling”. Tell ya what, Zeke, let’s agree on this phrasing: “Distorting the historical record in order to create an artificial warming trend in furtherance of a political or financial agenda”.

Joe Grappa

“I’d point out that Zeke has his interpretation but nowhere did I say “fraud”
Well, you did say that in the business world people go to jail for such manipulations of data, so you were implying the error, if there was one, was a lot worse than an honest mistake.

Accusing someone of manipulating data, suggesting that they should go to jail, but not uttering the word “fraud” is a trick. “Climate skeptics are like those people who refuse to admit the holocaust happened.” See? I never used the d word but i said the same thing.
REPLY: Sort of like suggesting some ex NASA scientists skeptical today caused the space shuttle to blow up?
Can you answer the questions? – Anthony

David, UK

“In the business and trading world, people go to jail for such manipulations of data.”
To be fair, that does sound like an example of “fraud” even if the word was not used. Unless “manipulations of data” for which “people go to jail” has an alternative definition.
Not disagreeing with the comparison, considering how climate data is indeed adjusted and manipulated. Just confused as to why the comparison with illegal “manipulations of data” in the business world doesn’t constitute an accusation of fraud. Is a spade not a spade?

Huh, not only did it change, it also gained an extra significant digit.
Wait till I tell my 7th grade science teacher he was wrong.

Andrew

Anthony – this must be akin to handing the microphone to someone who develops incontinence when handed the microphone in a karaoke bar.
Let the bed wetting commence.

Steven Mosher:
At January 23, 2013 at 3:22 pm you say

Accusing someone of manipulating data, suggesting that they should go to jail, but not uttering the word “fraud” is a trick. “Climate skeptics are like those people who refuse to admit the holocaust happened.” See? I never used the d word but i said the same thing.

Words fail!
The holocaust happened. Claiming it did not is a lie.
The historic temperature data are often altered. Saying they are is the truth.
Why people change the historic temperature data can be debated. But it is NOT ascribing any motivation to point out that it is done and to ask if it is acceptable .
Richard

Robert of Ottawa

Richard Courtney, why should a measurement be re-adjusted later? Why, also, in the case of Crimatology, alll the adjustments have the same effect of cooling the past. This is a neat strategy as it enables the Crims to show that current temperatures are what they say they are so obviously no malarky there.

Go look at the US HCN data from the GHCN site for 2012 and then tell me how they got anything useful out of it. The last 4 days of December are missing for virtually all sites and there are huge swaths of missing data across the nation for October and other months. A good exercise would be to see if the missing data was cold. I suspect it was but have been too busy in the last couple of weeks to write the code to analyze that hypothesis.
I do have all of the data pulled and archived though if anyone wants the mid-January dataset. 2012 is a freakin’ mess. This from the best data source in the world, from a government that thinks they can manage money. They can’t even record data from an instrument. Yeah, I trust that record.

Terry Bixler

Mr. Mosher you avoid the point at all costs. Why are the temperatures for items 1-4 not the same. If they are the same then they have not been manipulated. If they have been manipulated all of the points of manipulation need to be described. Who, What, Where, Why and When. Certainly modern computer databases can handle those issues. Of course you know this but choose to not address the issue of professional data keeping.

Ed Caryl

I may be proved wrong, but a series of thousands of small occasions of rounding up, confirmation bias, in filling missing data just slightly too high, siting problems, mis-estimation of UHIE, and other errors that just happen do go in the direction wanted, can account for all of it. Intentional fraud? That may never be found.

Pompous Git

Steven Mosher said @ January 23, 2013 at 3:22 pm

Accusing someone of manipulating data, suggesting that they should go to jail, but not uttering the word “fraud” is a trick. “Climate skeptics are like those people who refuse to admit the holocaust happened.” See? I never used the d word but i said the same thing.

I fail to see where Anthony suggested that the people who are clearly manipulating the data “should go to jail”. He merely pointed out that people engaged in business are put in jail for manipulating data. Anthony might well have been arguing that business data manipulators deserve the same treatment that climate data manipulators receive.
Anthony said:

“[Zeke’s] mad, and people don’t often think clearly when they are mad. That’s OK.”

Yes, the CAGWers are all quite mad and unable to think clearly. What’s not OK is that they expect those of us that are able to think clearly to share their delusions.

Robert of Ottawa:
At January 23, 2013 at 3:31 pm you ask me

Richard Courtney, why should a measurement be re-adjusted later? Why, also, in the case of Crimatology, alll the adjustments have the same effect of cooling the past. This is a neat strategy as it enables the Crims to show that current temperatures are what they say they are so obviously no malarky there.

I have been objecting to these changes for strong personal reasons for many years. Clearly, you have not read this
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc0102.htm
Richard

Manfred

Would such OVERWRITING of meta data “to correct the uncertainty” have happened in any other branch of science ?
http://climateaudit.org/2011/07/12/hadsst3/

Zeke Hausfather

Hi Anthony,
Unfortunately, calculating the absolute temperature for the entire contiguous U.S. in July of 1936 is a non-trivial matter. I could download the raw absolute average temperatures from all stations available July of 1936. However, many of those stations were located on the rooftops of city buildings (this was the pre-airport age, after all). These stations also used liquid-in-glass thermometers which produce notably higher maximum temperature readings than modern electronic instruments. All of these things mean that a simple average of instruments available at that time would tell us something interesting about the conditions at the locations of those instruments, but not necessarily produce an unbiased estimate of CONUS temperatures.
Instead, I will start with the Climate Reference Network, and assume that is represents a true unbiased estimate. I will estimate the actual U.S. temperature from 2004 to 2012 by assigning each CRN station to a 2.5×3.5 lat/lon gridcell (using the standard NCDC gridding approach), average the temperature readings in each gridcell, and average the grid cells based on their land area to get a CONUS average.
It turns out that during the period of overlap (e.g. 2004-2012), USCRN and USHCN largely agree on U.S. temperatures, at least once they have been converted into temperature anomalies (absolute temperatures will differ due to siting, elevation, wind exposure, and other factors). See this figure: http://rankexploits.com/musings/wp-content/uploads/2013/01/Screen-Shot-2013-01-16-at-10.37.51-AM.png
If I add in the average absolute temperature obtained from CRN over that period to the HCN anomalies, I can get a good estimate of the actual temperatures for CONUS in the past. This yields an estimate of 75.44 F for July 1936. This is slightly higher than the 2012 July temperature of 75.34 F. However, the result will differ a tad based on the grid size chosen and the baseline years used. Even in the homogenized data the July temperatures in the 1930s are about as warm as today; for annual average temperatures, not so much.
This does raise the point that NCDC probably erred in hyping July 2012 as the “hottest ever” when the difference (0.2 C) between it and July of 1936 is within the range of error introduced by methodological choices.
For folks still confused about the use of anomalies, and why they are important when estimating regional averages of temperatures (especially when individual stations are subject to biases), this post should serve as a useful primer: http://rankexploits.com/musings/2010/the-pure-anomaly-method-aka-a-spherical-cow/
REPLY: Thanks for pointing out the 0.2C error band issue. Note that I’m not asking you to calculate it, just show what the reported averages were. The Climate Reference Network is irrelevant to the July 1936 issue, as it did not exist prior to 2001. That said, it all becomes an issue of reporting.
While the work products of those at NCDC may be fully steeped in the belief they are doing things correctly, comparing things than have been changed through history is most certainly an incorrect way to do it. NCDC is charged with knowing these figures, yet, they change depending on who you ask.
Again, what are the temperatures for July 1936? Which one is the “real” one? – Anthony

As far as the fraud question goes, while you didn’t use that word, accusations of data manipulation and remarks that folks would be jailed if they engaged in the same behavior in the business or financial realms amounts to effectively the same thing in my reading at least. Perhaps I should have used the term unethical behavior instead of fraud, and I apologize if I put words in your mouth.
REPLY: that’s OK. To the point though, the historical data has in fact been changed (manipulated as you describe), and the question is: when reporting record temperature comparisons to the public, is this ethical?
Really, what is the correct temperature for July 1936? Nobody seems to know for certain, and that’s my point. – Anthony

RockyRoad

Freudian slip by Zeke, it appears.
By the way, I always see questions asked of Steven Mosher, but he never answers them. I’m going to apply the nickname “Crickets” to Steven.
Sorry, Steven, er…., Crickets, but it fits.

Zeke Hausfather:
I have personal interest in the frequent “adjustment” of the data sets. You can see my interest from my answer – especially its link – to Robert of Ottawa at January 23, 2013 at 3:37 pm.
Hence, I appreciate your having made your post at January 23, 2013 at 3:45 pm.
However, your post says different methods give different results. It does not say why each of the methods is often changed, and that is the only answer which matters, isn’t it?
Richard

Zeke Hausfather says:
January 23, 2013 at 3:45 pm

Hi Anthony,
Unfortunately, calculating the absolute temperature for the entire contiguous U.S. in July of 1936 is a non-trivial matter. I could download the raw absolute average temperatures from all stations available July of 1936. However, many of those stations were located on the rooftops of city buildings (this was the pre-airport age, after all). These stations also used liquid-in-glass thermometers which produce notably higher maximum temperature readings than modern electronic instruments. All of these things mean that a simple average of instruments available at that time would tell us something interesting about the conditions at the locations of those instruments, but not necessarily produce an unbiased estimate of CONUS temperatures.

I notice that you failed to answer any question Anthony actually asked, but let me throw in a few of my own:
Where did the extra significant digit come from?
Why is the adjustment in the same order of magnitude as the claimed signal?

tz

“Mad”, as in “angry” or “insane”?
REPLY: “angry” – Anthony

Zeke Hausfather

Whelp, I made an error. This does illustrate the danger of trying to do things too quickly. My code was accidentally still set to use the raw data rather than the homogenized data.
After homogenization, we get:
July 1936: 74.59
July 2012: 75.35

Anthony,
What exactly do you want to know about July 1963? The simple average of the temperature readings of all stations? The best estimate of the full CONUS temperature field? Its not a simple question with a simple answer.

Stark Dickflüssig,
I gave my best estimate of the CONUS average absolute temperature for that month and year. Given that we don’t have a measurement of ever inch of the U.S., but rather a specific set of locations, any attempt at creating an average temperatures will require at least a few assumptions.
Anthony,
I think CRN is somewhat germane. We need a good unbiased estimate of the absolute temperature field, and I would argue that it provides a rather good one, or at least a better estimate than a weighted average of absolute temperatures in 1936.
REPLY: I think you’ve been overthinking – Anthony

Sean Houlihane

I still read it as fraud, only not trying to hide behind more questions which we all know are irrelevant to the intention.

Latitude

I still find it amazing that glorified weathermen can clearly see the past, and correct it where it’s wrong…
….and they still can’t pick lotto numbers

gnomish

who fiddles the data? who diddles the chart?
who plays tickle-monster with the temps?
who has soapy showers with statistics?
would the consensus be ‘climate molesters’?

RockyRoad

Zeke Hausfather says:
January 23, 2013 at 3:47 pm

As far as the fraud question goes, while you didn’t use that word, accusations of data manipulation and remarks that folks would be jailed if they engaged in the same behavior in the business or financial realms amounts to effectively the same thing in my reading at least. Perhaps I should have used the term unethical behavior instead of fraud, and I apologize if I put words in your mouth.

As a practicing mining engineer for 15 years in charge of block model generation and economic analysis to substantiate $Millions to $Billions of investment capital, had I performed the data manipulation on assay values you “climate scientists” do on temperature data, I’d certainly be out of a job and, depending on the severity of the situation, perhaps end up in jail. (Some people call it “salting”; others call it “creative accounting”.)
So continue with your “data manipulation” if you wish, but you’re going to have a hard time getting this engineer to believe your conclusions. (And yes, I did read the article by lucia you linked to in your prior post.)

Lew Skannen

Climate changes over time.
Records of climate change over time.
All under the heading of ‘climate variability’.

Corporate officers have legal responsibilities beyond not commiting fraud.
A corporate officer can certainly go to jail for ‘a mistake’.
Here is an example.
http://online.wsj.com/article/SB10001424052970204443404577052173679627572.html

David A. Evans

I watched the data in GISS figd.txt change in Nov. 2009. (remember that GISS & NCDC are essentially the same.)
2006 came from about 6th to joint warmest with 1998 whilst 1934 went from joint warmest to about 6th.
I’m supposed to believe any of this?
Zeke. Does anyone even know what the RAW data is anymore?
DaveE.

Patrick B

Claiming an error margin of just 0.2 degrees – when calculating the CONUS temperature for a month in 1936 – THAT IS EITHER INCOMPETENCE OR FRAUD, take your pick.

Anthony,
There is a good reason why all the major groups (NCDC, GISS, UAH, RSS, Hadley) primarily report anomalies. Absolute temperatures are tricky things, at least when you aren’t able to sample the full field.
GISS has a rather good explanation: http://data.giss.nasa.gov/gistemp/abs_temp.html

Corporate officers can go to jail for other than fraud. They can go to jail for incompetence or just not exercising due diligence. Here is an example.
http://online.wsj.com/article/SB10001424052970204443404577052173679627572.html
Therefore Anthony’s statement implied one of a range illegal activities for corporate officers. Not just fraud.

RobertInAz

“Accusing someone of manipulating data, suggesting that they should go to jail, but not uttering the word “fraud” is a trick.”
Commentators on business networks frequently make a similar observation with respect to government accounting verses private sector accounting practices. There is no implication that the folks in the government entities should be jailed – the observation is that there are clearly different standards for government practices and the rules the rest of us live by.

David A. Evans,
To access raw data, browse to one of the NCDC’s FTP folders (e.g. ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/) and click on the files labeled .raw
I also show the raw vs. adjusted results for the U.S. in my post over at the Blackboard: http://rankexploits.com/musings/wp-content/uploads/2013/01/USHCN-adjusted-raw.png
Global raw vs adjusted temperatures can be seen here: http://www.skepticalscience.com/pics/GHCN_RawvAdj.jpg

Ray Donahue

” In the business and trading world, people go to jail for such manipulations of data.”
This statement may also be construed to highlight the seriousness of climate data manipulation via comparison with an activity which most would recognize as having consequences, whether legal ones or reputation damaging.

Anthony,
They way that NCDC calculates absolute temperatures is by calculating anomalies and adding in a constant absolute temperature for each month, based (I believe) on modeled outputs of a U.S. temperature field for the climate normals period. I’m well versed in the details, however, as I rarely if ever deal with absolute temperatures.
My approach was to try and replicate what NCDC does, but to use the CRN to determine the “true” absolute temperatures for the period in which CRN and HCN overlap. I realize this is somewhat complicated (and perhaps overthinking), but the point is that there is no “simple” way to get an absolute temperature value.
As far as anomalies go, you can see (homogenized) July anomalies here: http://i81.photobucket.com/albums/j237/hausfath/ScreenShot2013-01-23at43154PM_zps3d02490c.png
REPLY: and this exactly illustrates my concern, nobody seems to be able to reproduce the CONUS average temperature for July 1936. Therefore, making claims about “hottest ever” etc are indeed, nothing but hype. NCDC knows full well what they are doing here.
The historical accounts should be in the Monthly Weather Review for 1936. Since I illustrated here how a simple average of the CRN stations in the CONUS came close to the COOP values, I see the point your are trying to make. That said, today’s data is not yesterday’s data. My interest is in the true data from 1936, not a derivative comparison of what it “might” be.
– Anthony

Zeke, The difference between NOAA and USCRN can be as much as plus or minus 4.7F for individual states for a month.
Yet you argue the “trend” is close enough. It isn’t.
For example, in Sept 2012, NOAA was 4.78F warmer than USCRN for Tennessee.
http://sunshinehours.wordpress.com/2012/10/09/uscrn-vs-noaa-september-2012/
Don’t try and argue that NOAA and USCRN know what the temperature in Tennessee was for September 2012.

steven haney

Zeke, Anthony and Richard, Why is there no longer any discussion of temp. data prior to 1850? Why would the end of “the little Ice age” be the starting point for data in this discussion, when there are plenty of temp. records form around the globe available from the 1770’s on… when thermometers were simple, uniform and accurate. It seems to me that using a data set starting at the low point would be like reading one heart beat of of an EKG starting with a flat line and ignoring the previous beat. Every heart beat is a hockey stick when analyzed this way!

MrE

Fudge the data? Where on earth would scientists learn to do that and get rewarded? Answ. First year students in University science lab classes.

Leon0112

Zeke – I agree that there is a fair amount of measurement error in these calculations due to siting and other issues. This is probably even more serious for global temperature averages than the US alone. The measurement errors are sufficiently large that trying to determine causes and effects of various variables would be very trying. Indeed, it makes it quite difficult for the science to be “settled”. It makes it quite difficult to justify raising the prices of food and energy to the poor and middle class, just in case.
I know I did nothing “just in case” the Mayans were right. Dooming poor people to a lower standard of living, based on theories with such high levels of measurement error is bad policy.

Zeke Hausfather:
Following my post addressed to you at January 23, 2013 at 3:56 pm you have answered each of the posts addressed to you from others. It seems that you may have overlooked my question.
I would be grateful if you were to answer my question. To save you scrolling to find it, and for hope of clarification, I rephrase it and expand it here.
Temperatures were measured decades ago so their combined values should be a constant in the absence of altered homogenisation and consolidation methods. Why does the method used to homogenise and consolidate historic temperatures (regional, hemispheric and global) change with time? Which of the used methods can be considered to be correct and why?
Please note that I am not asking about differences between the methods of different teams (e.g. HadCRU and GISS). I am asking about the sequence of different methods used by your organisation.
Richard

Owen in GA

Zeke Hausfather,
When you were an undergraduate taking a lab course, what would have happened to your academic career if you had gone back to your lab book and “adjusted” the recorded data so your conclusion matched expectation? I suggest that you would not be working in science now if you had done such a thing. Experimental data is sacrosanct! It can never be altered. If you measure an item as a meter, it is a meter forever (even if your professor set the lab up with altered meter sticks to make sure everyone was staying honest – and yes I would do that!) There can be no adjustment of the past, only traceable, explained in metadata, correction of the present.
Smoothing algorithms should only change current analytical data and should always be traceable to the original data. Too many of the changes have no documentation of the individual adjustments. Most of us understand adjustments are needed to make apples to apples comparisons for sites that have had significant land use changes around them, but that doesn’t seem to be what is happening. The homogenization process seems to just smear the worst errors or site problems across whole regions and time periods rather than to correct for poor or missing data.
Quite frankly, the whole UHI adjustment process as it now happens seems to be counter-intuitive. If the problem is urban encroachment of the site, the correct adjustments would do one of two things to make an apples to apples comparison of the same site at different points in time and development: adjust the current readings down by an amount equivalent to an experimentally determined value of UHI, or adjust the past up by the same amount, though I’d rather the first than the second, and would rather no adjustment at all until a valid experimental valuation of UHI for different types of sites can be done. The way the current adjustment process is done implies that UHI has a cooling effect on the sensors which is just plain crazy. (See the experiment being done at Oak Ridge on siting issues for ideas on how to get at this value!)
Most of us here understand that science is hard work. I suspect that most people working these data sets are trying to do the right thing for the science with the best information they have available. Thus you will rarely hear the fraud word from this site, but you will hear things like “obviously mistaken”, “out of their gourds”, “too full of themselves to see their error”, or even “suffering from mushroom syndrome” and very rarely for some of the most arrogant practitioners of data manipulation and scientific information suppression, “dishonest”. For the most part, I am looking for data that passes the smell test and so far that has been in short supply. Some of the problem is the ever-popular “science by press release” that the MSM picks up and runs with out of all proportion to the underlying data/paper, but then again, some of the abstracts are written with the obvious intent to engender that response. (I’ve been told my abstracts are too dry, so I am probably not one to ask how to correct that problem.)
The problem with this site is not that people don’t understand science, the problem is that they do, and that makes it hard to put “science in the dark” by us. (not implying that NCDC does this, but there is still the little matter of all the little adjustments of the past – especially those that happen between major dataset methodology revisions!)