Climategate: hide the decline – codified

Nice Tricks!!

It just gets better and better (worse for them, that is). The question remains: is this enough to overcome the momentum acquired by the AGW side since 2005? Senator Inhofe’s commitment to the skeptic’s cause and his ranking minority seat in the Environment and Public Works Committee looms large. But why won’t the mainstream media report on the matter?

Hi Anthony,
last dinner at Copenhagen!
I love it!
http://i47.tinypic.com/mrxszt.png

The last two lines stand out. I wonder what was shown in the data between the year 440 and 1400? Medieval Warm Period maybe??????

ouch… it’s worse than I thought!

Simplified code for CRU to use (from an average programmer with no climatology expertise):
For intYear = 1400 to 2009;
floatGlobalMeanTemerature = floatGlobalMeanTemperature + WHATEVER_THE_HELL_YOU_WANT_IT_TO_BE;
intYear++
next
Print “Holy Crap! We’re all going to die!”

hmmm (09:14:27) :
It’s almost as if whoever wrote these notes wanted us to find them.
Positively Freudian.I know enough about programming though no expert,
just from my own limited experience the commands and data input are
to quote:” Crap crap ‘..

Get better results by oursourcing the coding to medieval Timbuktu

Oh dear, poor chap. I hope this doesn’t turn into a CLM for him (no, that’s not yet another climate acronym, it’s a programmer one: http://www.jargon.net/jargonfile/c/CLM.html)

Can there be any doubt that this trail of crumbs was laid deliberately? I think the identity of the whistleblower may be coming into focus.
However, it is curious that the coder makes no reference to brining the problems to the attention of Jones et al

The pot of gold in Climategate. While we knew via Hansen’s scripts and Fortran77 something beyond incompetence, probably fraud, was afoot, Mann never released anything but McIntyre’s reconstructions are a strong indication of fraud.
This project file of commentary is the sort of file, only much larger, I’ve often used, maintaining old code I’d not developed originally. Usually it’s to establish hierarchies, patterns of attack, this file includes comments better placed in the files themselves, but gives every indication, to me, of authenticity.
The weight of the evidence following Climategate is inescapable.

Totally amazing. I’m no programming expert; but I do routinely construct Optical system models for ray trace simulations; and even they require detailed comment of every element that is in the system, or else even months later, I cannot keep track of why the hell, I included some element. Now this is orders of magnitude more straight forward, than actually writing the code; but I certainly understand the concept that the comments better explain in plain English (and evidently include also some Australian terminology) so anyone can tell what it is supposed to do.
Well I have always believed that the failure to observe Nyquist in sampling strategies, was at the heart of this climate data inadequacy, and reading about all their ad hoc inclusions and exclusions of data sites; clearly shows what total BS they are cooking up.
And If I was paying the bills for that stuff; I would really like my money back, and also see some jail (gaol) time included in the correction process for the varmints purveying this rubbish as science data gathering.

And yet the lap-dog press will continue to embargo the controversy, Obama will “get one more step closer to climate change legislation”, and the scientists (sic) will offer lame but “scientific” excuses.
Maybe the way to nail the bastards will be the old fashioned way, mail-fraud and IRS transgressions. I would imagine there has been snail-mail crimes of some sort , and I’m positive such large amounts of money led to some malfeasance on Mann’s and other’s part. This type of personality would never be clean in their persoanal lives while perpetrating such monumental fraud in academia.

Was the data that the code uses also released?
With the data, one could remove the fudge factors from the code and see what really comes out.
The tone of the comments do raise the question as to whether their author helped release the information.

Having worked as a research assistant at Columbia University, the only thing that surprises me is the childish language and the tone of all the emails. Scholarship is corrupt to the core and that’s why I decided never to get a PHD.
The number one corruption is that no thesis is approved for research unless the research passes a political litmus test. Translation: Unless you already have a conclusion in mind that does not seriously detract from that of your colleagues, you have no hope of being published.
Now, I am published (only one paper), but it is in the field of Philosophical History, it did not require any grant money, and it passed the political litmus test among Orientalists detracting from those whose background ais Occidental. Translation: the paper is perceived as “progressive” so it is good to go.
Of course it’s a bad idea to base policy upon modern scholarship in highly controversial areas. Unless proposed research is designed to meet, substantiate, or reflect progressive ideals, you haven’t got a chance.
Mainstream scholarship in recent history has got some very obvious things dead wrong because of progressive politics. Just as we heard all the lies about imminent O-Zone disaster to racial integration increasing property value (believe what you will about the social merit of integration, the economic is quite obvious), they are wrong again about yet another one.
However, their arrogance is astonishing in the email and programming notes. It should be no surprise they fudge evidence and conclusions–that’s 95% of scholarship, where position is more important than the truth. The surprise is that they all seem to be knowingly complicit in deception and not care. Everyone whom I worked with believed their crap!

Oh, good lord!…. and these mob were given a grant of 13 million pounds of British taxpayers money to do climate science?
…. I think the only thing “done” here, was the British taxpayer. They were well and truly “done” over.:-(

“Somebody should Slashdot this, it is exactly the sort of thing the programming geeks there will understand and appreciate. They may also be able to provide valuable insight.”
I’ve been watching Slashdot for this story to come up, obviously this is just the sort of thing that SHOULD show up there.
It won’t. Slashdot is filled with pro AGW types, and controlled by them. I wouldn’t be surprised if it never show up there at all. Even the “science” section is glaringly silent on the issue.
Yes they are the type that would understand this, but for many it would mean exposing their preconceptions to themselves, and that won’t happen.
There is a reason why people will defend the crappy deal they got from a used car salesman.

BOTO (09:17:11) :
Hi Anthony,
last dinner at Copenhagen!
I love it!
http://i47.tinypic.com/mrxszt.png
Great picture, but I have a question. Which one is Judas?

I know why Warmists deny global warming is not man-made. They are living in a Hollywood dream world and they can’t wake up.

If this isn’t enough to get MainStreamMedia to actually do their work – then I’ve lost any hope for humanity!

Reed Coray (09:35:23) :
in my knewing, the one at the very right (your president…)

One should just run this code on some totally random data series and plot the output.

If a physicist were to submit a paper without showing the math, that paper would (I assume) be rightly ridiculed and sent back with a “show your work” rebuke. It doesn’t seem right that one can hide one’s work in software, and then casually dismiss the absence of documented code upon submitting a paper as these yahoos have done. And yet, that seems exactly the way mainstream climatology works. Do any other sciences permit one to hide calculations in a program and then not publish said program with the paper?

Is this real??????? hot stuff !!!!!!

It reminds me of Jones public statement:
“So apart from the removal of a few troublesome editors, blocking some FOI requests, deleting emails, blocking a few contrarians from publication, peer reviewing each others papers, cherry picking trees and modifying code to hide the decline: please tell me – exactly what is wrong with our science?”

Do we know what CRU “software” we are talking about here? The first comments seem to be for ten year old proxy reconstructions, and some of the later ones the temperature “product(s)”
Is there any relationship to, or does this provide any insight into GISS software? It seems that Phil Jones made some winking statement about the CRU temperature anomalies being remarkably similar to GISS, “as they should be” or something similar.
The programming is clearly bad and any output from the software described here would have to be questionable. However, it is very difficult to assess risk and estimate error without knowing which software has problems and for what it is currently used (or which AGM dogmas it supports.)

http://politics.slashdot.org/comments.pl?threshold=5&mode=nested&commentsort=0&op=Change&sid=1451926

Chris (09:36:01) :
My question is how the HARRY_READ_ME file fits into the greater scope of CRU’s data analysis. It’s not clear to me whether this was some side project for the programmer to analyze a problematic data subset or if this represents his frustrations with the main CRU data set. There is certainly a story here just don’t know if there are more chapters in this novel.
—————————————————–
Well that’s a simple problem to solve then….. Just send CRU a FOI request for the data and meta data and check it out;-)

James Hastings-Trew (09:16:45) :
If you have looked at the massive differences between the timespans quoted in HARRY_READ_ME.txt stations like Elko, NV / Charleston, SC / Red Bluff, CA (and dozens of other sites) and what exists today, it’s going to take years to recover, providing that original submitted paper forms have NOT been destroyed.
It’s going to have to be done. Too much damage has occured, and the digital databases are not to be trusted.
Do you (or anyone else for that matter) personally know where the original paper forms submitted are kept?
And where to obtain a legitimate copy of Tom Karl’s 1990 USHCN database?

SIR, American Thinker tied these events to ACORN in their article today. http://www.americanthinker.com/2009/11/acorning_the_climate_change_mo.html

All I can say it, WOW. The way in which this programmer commented his work makes me think that he was somewhat frustraded with the fraud that he was being ask to commit. Really, we are now past the point of scientific debate. The skeptics need to have some good lawyers in their camp to help resolve some of these issues. Without taking these people to court, they are simply not going to come clean about what they have been doing.

I used to sincerely believe in the global warming theory. But one thing that’s troubled me in the past few years were the shrill responses to skepticism regarding it.
Now, I’m not a scientist by profession–but shouldn’t there be room for debate in discussing the following counter arguments:
1. Correlation does not prove causation
2. A true independent, critical peer review
From a brief cursory reading of these emails, it sounds to be like the head of research wants to make the collected data fit the hypothesis.
It’s human nature to be biased and partisan. I wonder if it could be possible to collect the raw data and have it examined by qualified people with no ideological axe to grind–and let the chips fall where they may?
What the global warming proponents want to do is redesign our entire economic system and create layers and layers of bureaucracy in order to regulate almost every aspect of our lives.
I believe in being good custodians of our planet and the environment of course. It is a good thing to eliminate pollutants that are proven to be harmful. As far as CO2 goes, nothing has been proven yet.

Been looking into the code and data held in the \documents\cru-code\f77\mnew directory. Here is what I have found:
The data sets are master.dat.com, master.src.com. master.src.com is the important file. Don’t open these without changing the extension to .txt otherwise Windows interprets them as executables, and you won’t be able to view them properly anyway. Could send copies capable of being opened in Windows. These contain monthly weather station data with one row per year. I don’t know the exact nature of these files, but some of the data does relate to sunlight duration. A site in Finland suggests master.src.com is temperature related, but there’s a lot of speculation flying around the Internet regarding the leaked files at the moment, so can’t be certain.
There are 3526 stations in all and 2578488 monthly observations. -9999 in a field means the observation for that month is absent. There are 269172 (10%) missing observations in master.dat.com and 14226 complete missing years. The programs are designed to completely ignore any years with no observations. In total there are 200649 rows (years) of observations which should equate to 2407788 months, however due to some years having up to 11 missing months there are 2309316 monthly observations used. Now what’s interesting is how these missing months are processed. Programs such as split2.f, where a year contains one or more missing months actually invents the figures using the following heuristic
If a month is missing try to infill using duplicate. if duplicates both have data, then takes a weighted average, with weights defined according to inverse of length of record (1/N)
That’s from the comment at the start of split2.f
What this really means is more than 4% of the data is being completely fabricated by at least some of the Fortran data processing programs. If this were done in other disciplines this would be extremely questionable.
Also did notice quite a few programs, especially in the documents\cru-code\idl\pro directory are designed to process data deemed anomalous, though this isn’t necessarily suspicious.
This is the header comment from documents\cru-code\idl\pro\quick_interp_tdm2.pro
; runs Idl trigrid interpolation of anomaly files plus synthetic anomaly grids
; first reads in anomaly file data
; the adds dummy gridpoints that are
; further than distance (dist)
; from any of the observed data
; TDM: the dummy grid points default to zero, but if the synth_prefix files are present in call,
; the synthetic data from these grids are read in and used instead
What is ‘synthetic data’ and why might it be applied to dummy gridpoints away from genuine observation points? This could be a recognised statistical procedure, or data massaging, or creating more observations out of thin air to skew certainty levels, just can’t tell and don’t have time to look at anything else in depth right now. Like it says, e-mails can be open to interpretation but it’s the code and what it does to the raw data which really matters. The comment in the Mann code described in the link below is a work-around to a recognised issue with dendrochronology data. During 1960s the correlation coefficient between tree growth rate and temperature altered.
The recent ERBE results are really significant, the discrepancy between IPCC modelled values and the real world figures is quite something.
http://wattsupwiththat.com/2009/11/22/cru-emails-may-be-open-to-interpretation-but-commented-code-by-the-programmer-tells-the-real-story/

“Judas” is at the very right…
http://i49.tinypic.com/2gy8w9v.jpg

In electrical engineering, any unique code used is included in the methods portion.
Because engineers expect to be able to investigate each others’ claims, instead of letting them be hidden in software black boxes.
Of course, that is engineering, not “science”.

Will there be a Congressional Investigation?
Comment by JamesD
Wednesday, 25 Nov 09 @ 12:24 PM
There will be no investigations with a Democrat congress. No committee chairman will allow it. I hope some of the heavyweight scientists will come to the defense of scientific integrity but I’m not holding my breath. For a long time this naked emperor has been parading down the street. The press and, particularly, the educated public all agree, he’s wearing a fine suit of clothes.
http://carboneconomy.economist.com/
http://carboneconomy.economist.com/content/programme

Eric (09:40:29) :
In fairness that is too general a statement. It is important to be precise and specific, otherwise folks at RealClimate who actually really know their stuff will simply rip you to shreds. Certain critical pieces of station data have been requested. Certain pieces of code have been requested. More generally, authors of cliamte science research papers have been asked to post their raw data and their code in a way that will allow a complete replication of their results by interested third parties. It is the Institutional and individual refusal to do these simple things that has caused the questioning of the motives of climate scientists in general.

How do know that the HARRY_READ_ME.txt file and the data he’s working on represents the official temperature ouput of CRU?
Maybe he is put to work on a special dataset tha has been corrupted somehow?
Surly there is more than only 1 guy doing this stuff at CRU??

Here is the output of some code from a Briffa related file. If you run a flat temperature graph through it if gives a Hockey Stick. If you run an inconveniently divergent tree ring graph through it, it acts as a trick to hide the decline.
http://i49.tinypic.com/m9vcxv.jpg

Sounds like they have some real PEBKAC errors over there. That is Problem Exists Between Keyboard and Chair.

Hilarious:
‘discovered that a sum-of-squared variable is becoming very, very negative!’
Don’t they know ‘ i ‘ ??? imaginary as some theories?
For Europe the subject is less that hilarious!
Schellnhuber is the adviser of Kanzlerin Merkel and of the President of the European Commission Barroso!
A position with extreme power.
Now look at his writings, with Mann, Schneider,Rahmstorf and others, very recent:
http://www.copenhagendiagnosis.com/
and on the reports in the media:
http://www.n-tv.de/politik/dossier/Noch-gibt-es-Hoffnung-article79835.html
http://www.spiegel.de/wissenschaft/natur/0,1518,663045,00.html

I did some numerical coding for my MSc-equivalent thesis, and in theory is possible to obtain a negative number by the addition of positive ones if the total is large enough to overflow, and the most significative bit of the mantissa is the sign bit, with the value of 1 for negative numbers. With those conditions, an untreated overflow would cause negative numbers.
If the code does not treat that, the results are no good. The solution is the use of data types with more bytes, and some studies about the numerical stability of the algorithms employed is a must. However, it’s amazing that such a newbie error could be made in one of the most prestigious research institutions, so another explanation could be possible. We did not find this problem in our work because we used algorithms that we knew stable with the sets of data employed. Normalization of data also helped 🙂
However, it amazes me that a organization with so much at stake in numbers did not get some basic reference texts* and had to resort to search for algorithms to calculate distances across Great Circles in Wikipedia!
* Such as this, edited in 1992: http://www.amazon.com/Numerical-Recipes-FORTRAN-Scientific-Computing/dp/052143064X

No rob it’s not hot stuff. No one cares. Outside this narrow little circle of people who care about truth nobody give a f—
AGW plays to some sort of primitive pre-rational impuse or mythic structures that some people have … and that’s it. There’s no science here. Just stories.
Expect green taxes. No nuclear power stations. Shutting down the coal fired stations we do have.
“My three main goals would be to reduce human population to
about 100 million worldwide, destroy the industrial infrastructure
and see wilderness, with it’s full complement of species,
returning throughout the world.”
-Dave Foreman,
co-founder of Earth First!
More here The Green Agenda
The warmists are essentially the forces of the counter-enlightenment – quite literally the people who want to put the lights out. Do you think they’ll be swayed by truth?

Having been a developer for 25 years (I’ve written a fair amount of Fortran code), I can sympathize with the programmer. He’s been asked to produce consistent output that appears plausible from a train wreck of ratty source data collections. The collections don’t agree where they overlap, so he’s been told which ones to use over which time periods using what appears (to him) to be a completely arbitrary basis; to make it all fit together he has to “fudge” the numbers and write code to handle each set differently. It’s as if he’s been asked to generate the data for a bank’s corporate tax return using checkbook register copies obtained from customers and the bank’s stock price for the last year, along with general ledger entries from three different internal accounting systems. The results of his efforts are then to be used at the annual shareholder’s meeting in two week’s time, and if they’re wrong, the CEO will make him the fall guy.
Programmers are normally a meticulous bunch, and this is a programmer complaining bitterly about the crap he’s been handed. Further, he’s upset that he’s been asked to produce plausible results in short order by people who as scientists are supposed to care about accuracy as much as he does, but clearly don’t care in this situation so long as the results come out looking like what they want to see. His notes clearly indicate that he wants everyone to know what he was facing and what sort of shenanigans had to be done to make it all look good.
What the public is finally getting to see is just how inconsistent and error-prone all the source data is and just how much manipulation is going on behind the scenes. Everyone has been led to believe that the measurement data sets are pristine, accurate and from data sources shown to be reliable. It’s assumed they are good enough to be used to make decisions that affect millions of people and cost billions of dollars. What public is now seeing is a real shock, and people are coming to a common sense conclusion: The data is crap, and the results can’t be trusted at all.
I feel for the programmer….I would not want to be in his shoes.

The ‘science’ behind the AAM narrative can now be seen clearly for what it is a cloak to conceal the true intentions of the political classes.
I suppose the false cloak of science was only meant to cover the political aims until the point had been reached when the political narrative no longer needed a cloak to hide behind.
The BBC.
The university of East Anglia.
The climatic research unit.
The UK meteorological office(met office).
The Hadley climate research centre.
The US link being the Goddard institute of space sciences(GISS).
All of these institutions are involved in the scandal to at least some extent in the scandal and by some strange coincidence all of them are fanatical believers in and supporters of the AAM narrative, they report the science as settled and beyond doubt, they indulge in attacking sceptical scientists and flout the rules when it suits them, they feel they are above reproach and audit perhaps because they have powerful friends?
Of all the world reach media platforms the BBC has been perhaps the most fanatical in peddling the anti CO2 anti capitalist anti industrial anti free market stories of the ‘eco industry’ it has used its nearly unlimited resources to provide a tsunami of supposed evidence however thin and patchy using state of the art visual manipulation and using the propaganda arts to full effect. The link to all of them is money and/or political affiliation, they are a closed loop of interconnected people with a common agenda showing the arrogance of those who know they have friends in high places, powerful and influential allies.
In attacking the fake cloak of science we are in fact not directly attacking the puppet masters driving the entire narrative, like a bull attacking the matadors cloak? The political forces need this sea change in our civilisation and they feel that they cannot be open and honest with us about the reasons for the required changes so they cover it in lies to hide it. Whatever the real reasons for the massive reordering of our entire civilisation are I suspect that AAM is not one of them.
To get at the truth we must expose the entire chain and the links in that chain, the political classes are moving fast now, faster than perhaps they expected to move because the foundations of their scam ie a warming planet is not happening, without the cloak of fake science to cover it the real movers are exposed for the world to see, if the political classes refuse to alter their policies when the science is exposed as fraudulent we will know who is behind the cloak.

OMG! The sum of squares parameter going negative? This isn’t possible! Even if all computed values from the model deviate negatively from the actual data, the squares of these numbers are positive values and thus the sum of squares is always positive.
I had a similar problem fitting titration data back in graduate school where the fitting statistics weren’t working out. It turned out I had coded in an incorrect equation for a derivative, neglecting to multiply by ln(2) in one line out of a thousand lines of code. It was plainly obvious something was wrong with my model from output plots, but wasn’t obvious in the code.
I feel for the programmer, but it’s his job to straighten out glaring problems, particularly when the output is a mathematical impossibility.

ralph (09:51:07) : writes:
O.T. but:
Did you know that the BBC held a secret meeting that decided NOT to give balanced time to GW and anti-GW viewpoints. Apparently, the communique said:
“The BBC has held a high-level seminar with some of the best scientific experts, and has come to the view that the weight of evidence no longer justifies equal space being given to the opponents of the consensus.”
This is only partly correct, my friend.
For over 3 years I have been trying to elicit answers from both Mark Thompson (Director General) and Sir Michael Lyons (Trust Chairman). All I had received was sophistry and obfuscation, until I engaged the help of my MP.
Recently it came to light that a report had been commissioned in June 2007 jointly by the Trust and BBC Board of Management entitled “From Seesaw to Wagon Wheel-Safeguarding Impartiality in the 21st Century”. It concluded: ‘There may be now a broad scientific consensus that climate change is definitely happening and that it is at least predominantly man-made… the weight of evidence no longer justifies equal space being given to the opponents of the consensus’.
(SO THEY HAVEN’T EVEN TRIED TO MAKE A SECRET OF THIS…JUST SHOWS THEIR ARROGANCE!)
Despite this damning evidence from their own report, they steadfastly cling to the belief that their impartiality is intact as required by the BBC Charter. Such is their state of denial that Sir Michael Lyons has even tried to deliberately mislead my MP despite evidence I have to the contrary.
In light of this I have posed the question, through my MP: “On whose authority did the BBC cease to be an impartial Public Service Broadcaster, as required by its Charter, and become the judge, jury and sponsor of such dangerously specious political dogma so eloquently described as ‘…the consensus…’?
Answer comes there none! I believe it is time for the BBC to be subjected to an enquiry on this matter.

The blog post should distinguish between the various bits of code.
The stuff in folders like osborn-tree6\mann and harris-tree\recon1.pro and osborn-tree6\mann are most likely programs used to generate things for peer-reviewed articles.
This is quite different and distinct from the code used to produce the HADCRUT3 temperature series. I am not postive, but it does appear that the HARRY_READ_ME.txt file is about the “CRU Code” as most of us would interpret it — the code used to produce the HADCRUT temperature record.

Re. the “sum-of-squared variable is becoming very, very negative”.
I’d imagine that was due to an overflow.
For the non-programmer folks; there are various data types that can be used to represent numbers (signed integer, unsigned integer, float, double etc.) but they aren’t capable of representing arbitrarily large numbers. If you exceed the limit, you generally end up with a negative number of the same magnitude.
i.e. MAX_NUMBER + 1 -> -(MAX_NUMBER + 1)
The fix is simply to use a type that allows larger numbers.

Just watched ‘The Cloud Mystery’ on Youtube about Svenmark’s work on Cosmic Rays and their creation of the aerosols on which clouds form. Interesting that his experiment was conducted in Copenhagen? – You don’t think December’s meeting is a distraction ploy while a certain group smash his lab up??

The Pennsylvania State University has a prepared statement you may request by calling the Office of Public Information at 814-865-7517. I have it if you can’t get through.

Now I am convinced it was an inside job. Seeing the exasperation in the comments paints for me a very convincing picture of who the leaker was. Some code monkey, who was probably also doing double duty as an IT tech. He (or she) finally got fed up with the constant demands of the heads to do things that fly in the face of both ethical scientific procedure, and worse still, best practice computer programming.
I guess they worked him one sunday too many, or gave him black marks on his review because he couldn’t get the computer to say what they wanted. The programmer went off the deep end and decided to start compiling a file of some of the more incriminating skeletons in the CRU’s closet.

I have worked as a professional programmer for more than 20 years, and I think that the language in these comments is strange, to say the least. I mean – I have often been swearing over poorly documented spaghetti code – (almost) as bad as this one – but I have NEVER put the swearing into writing. In my opinion, this stinks. It seems that the author of these comments WANTED the world to see them. He is certainly writing to another audience than his fellow programmers. So there are two possibilities: 1) Either, the programmer (Harry?) is the whistleblower, or 2) This is a trap.

Great reporting. With any luck CERN will finally nail it all with their CLOUD experiments.

Hysteria, I’m not so sure. O’Reilly has, I believe, said that Global Warming is real and something needs to be done about it. So did McCain during his bid for the Presidency. That indicates a fraction of the conservative base had been convinced this was a real issue that needed to be addressed.
This revelation of subterfuge and skulduggery has certainly made a bunch of conservatives re-think this position, and I’m sure libertarians and a handful of liberals as well. Beat this drum loud enough, often enough, and the support base for global warming hysteria will once again return to little more than tree-hugging alarmists. But the time to act is NOW, before any more talk of ‘cap and trade’ or Copenhagen concessions make their way through Congress.

New internet meme: “Harry_Read_ME”
Examples:
HRM: You may have the chart upside down.
It was a HARRY_READ_ME job.
After an industrial accident: Their control code was still waiting for Harry to read it.
If only Harry were here to read this.
Etc.

Over at Connolley’s blog I posted:
“The Emails show that Jones and Mann can’t be trusted. HARRY_READ_ME shows that the code is incompetent and the code itself shows manual adjustments that have no scientific basis. This is sufficient evidence to call for a third party review of the entire CRU methodology. ”
To which I got two replies:
——
PR Guy – what papers was the HARRY_READ_ME code used on? Have you any evidence it was used at all?
Posted by: Chris S. | November 25, 2009 12:07 PM
—–
PR Guy,do you even know which dataset/product the HARRY_READ_ME code is dealing with?
Posted by: Adam | November 25, 2009 12:46 PM
—-
To which I responded:
“Chris S and Adam, these are very reasonable questions. Perhaps you should submit a FOIA to find out. I’m sure we all agree that answers to these sorts of questions are vital and should not be obstructed.”
This last comment was deleted by William (or maybe the moderator, if there is a moderator). The Team never lets points get scored against them on their court.

I’ve done some programming myself. I used to put in swear words in the code all the time.
In fact, I would even use the F word and variations thereof for variable and object names, LOL.

I guess this proves their point that Global Warming truly is man-made. Totally made up, in fact, by a few.

I had a really scary thought…
What if the problem isn’t with the tree rings, but with the temperature series?
What if the temperature series is actually off by 2.6C (high) since 1940?
Does that mean we’re actually 2C or more below the 1940s temps?
Are we screwed?

JCS
“Steve Gavin at RealClimate has refused more than 6 times to post the following message, are you willing to present this very important point for me?”
LOL. JCS, welcome to the club of thousands who Gavin has moderated out because he finds their comments inconvenient. Gavin is a part of the AGW cabal. He is on the distribution list for many of the e-mails from CRU gate. Some of those comments make it clear that Jones and others consider Gavin as the guy that runs interference for them. I have written a small piece about how debates are orchestrated at RC here:
http://reallyrealclimate.blogspot.com/

Run these numbers, see if that makes a hockey stick…………
http://spreadsheets.google.com/ccc?key=0Ah4XLQCleuUYdFIxMnhMNnlXb2JQcDZUendjUXpWWUE&hl=en

;mknormal,yyy,timey,refperiod=[1881,1940]
;
“; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
(…)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj”
************************************
Hmm. Are there edited versions out there? My file:
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
yearlyadj=interpol(valadj,yrloc,timey)
;
;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20
;
filter_cru,5.,/nan,tsin=yyy,tslow=tslow
oplot,timey,tslow,thick=5,color=21
;
oplot,!x.crange,[0.,0.],linestyle=1
;
plot,[0,1],/nodata,xstyle=4,ystyle=4
;legend,[‘Northern Hemisphere April-September instrumental temperature’,$
; ‘Northern Hemisphere MXD’,$
; ‘Northern Hemisphere MXD corrected for decline’],$
; colors=[22,21,20],thick=[3,3,3],margin=0.6,spacing=1.5
legend,[‘Northern Hemisphere April-September instrumental temperature’,$
‘Northern Hemisphere MXD’],$
colors=[22,21],thick=[3,3],margin=0.6,spacing=1.5
;
end

If I produced work like that I would want to delete it also rather than give it to an auditor.
Short of a detailed explanation and the full production of a model establishing this as nothing more than an interim piece of work; Jones should be fired for this alone.
He probably should also be held to account for the fraud that this is. This is beyond incompetence for somebody in Jones position if he can’t bring something forward to mitigate this!
This would also appear to establish Jones as a liar in suggesting the “to hide the decline” behavior only dealt with modifying a graphic.

I’ve done some programming myself. I used to put in swear words in the code all the time.
In fact, I would even use the F word and variations thereof for variable and object names, LOL.

That isn’t considered very professional in my business, to be honest.

If we don’t fix the science and clean out the “climate caves” we will have more of this bogus in the future:
http://heliogenic.blogspot.com/2009/11/can-it-get-any-more-hysterical.html

documents\cru-code\linux\cruts
This code is used to convert new data into the new CRU 2.0 data format (.cts files).
There is another version of this code in cru-code\alpha which is, per comment in the readme file, intended for running on the “Alphas”
Data can come in from text files or Excel spreadsheet files (or actually Excel spreadsheets written out to text files from Excel). These programs are designed to read multiple climate data file formats include
GHCNv2
CLIMAT (Phil Jones formt)
MCDW
CLIMAT (original)
CLIMAT (AOPC-Offenbach)
Jian’s Chinese data from Excel (appears to be text output from Excel)
CRU time-series file format – with the comment “(but not quite right)”
Data files for running these code files are not available in this archive.
Software engineering comment – this collection of programs – very large source code files – is implementing a crude database management system. Most of the source code is uncommented and undocumented. From a s/w engineering perspective, it would have seemed wise to have used an existing DBMS that had been extensively tested and verified. Instead, the approach chosen results in extremely large amounts of custom code being written. There is no evidence provided of software quality assurance (SQA) procedures applied, such as a test plan, test scenarios, unit testing, test driven development and so forth. It would most likely have been quicker and more reliable to use existing software tools like DBMS.
The goal of the software is to eventually calculate the anomalies of the temperature series from the 1961-1990 mean.
Because station reporting data is often missing, the code works to find nearby stations and then substitute those values directly or through a weighting procedure. In effect, the code is estimating a value for missing data. Station data will be used as long as at least 75% of the reporting periods are present (or stated the other way, up to 25% of the data can be missing and missing data will be estimated).
The linux\_READ_ME.txt file contains an extensive description. Of interest, stations
within 8km of each other are considered “duplicates” and the data between the stations is “merged”. I have a question about this which may not really matter – but there is no attempt to determine if the nearby stations are correlated with one another. It is possible, for example, that one station is near a body of water (and less volatile) and another is on the roof of a fire station (see surfacestations.org). Or the stations could be at different elevations. In my town, the official weather reporting station moved 4 times over the past century – from downtown in a river valley, to eventually up on a plateau next to a windy airport. These locations would today fall within the 8km bounding area. My concern is that this could skew results in an unpredictable way. Then again, it could be that the situations like I describe are rare and would have negligable impact on the calculations.

Hiding the Decline: Part 1 – The Adventure Begins (Eric S. Raymond)
http://esr.ibiblio.org/?p=1447

documents\cru-code\f77\mnew\sh2sp_m.for
This program, sunh2sunp, converts the “sun hours monthly time series to sun percent (n/N)”. I do not have access to the cited reference used for calculation so have not yet determined if the code is implemented correctly.
However, in the odd situation where the calculation exceeds 100%, the code, surprisingly, checks for this but then leaves the incorrect value in place:
c decide what to do when % > 100
if(sunp(im).gt.100)sunp(im)=sunp(im)
For non programmers this says, in simplified form
if x > 100, then let x = x
Normally, if a value is incorrect, the error is either flagged or perhaps in this case it
could be due to round off error – in which case, we might expect something like:
if x > 100, then let x = 100
which would force the value of x to never exceed 100.
The purpose of this program and how it fits into any analysis is not yet understood. The program appears to date back to 1997 (and probably went out of usage by 2003) and it may no longer be in use. It is entirely possible that the above error condition never occurred – and consequently, this defect in the software would have no impact on the results.
In a separate code file (sp2cld_m.for), the above test is implemented correctly:
IF(CLD(im).GT.80.0) CLD(im)=80.0
The file exhibits poor Fortran coding standards such as:
ratio=(REAL(sunp(im))/1000)
IF(RATIO.GE.0.95) CLD(im)=0
Note the lower case ‘ratio’ and the upper case ‘RATIO’ variable names.
The variables XLAT and RATIO are not declared. Similarly for iy, iy1, iy2. Fortran 90 permitted this practice and would automatically define the value based on the first letter of the variable name: A through H and O to Z are set to type ‘real’. Use of this feature is discouraged because the compiler is then unable to flag typographical errors – instead of warning of using an undeclared variable, it just defines a new one. This can result in erroneous program operation – if that occurs. Note – this is a software engineering issue and is not the source of any identified execution errors in this program. This is to note that this is poor programming practice. It does not appear to have resulted in an implementation or execution error.
Note – the issues I cite do not mean the program’s executed incorrectly. They are more indications of poor programming practices. And I believe we the people deserve the utmost care and professionalism in a matter as important as this.

Amazing !!!
I can’t believe that this is all real !

Forgive me for jumping in here, as its a bit OT.
Richard A. said:
‘But if we’re talking about a study in biology on the contents of the feces of some frog in the far corners of the jungle, it’s likely the journal it’s submitted to wouldn’t enforce their policy, and it’s likely no one would care.’
Not so! To the contrary. Without documenting how many frogs you were studying, and where (not forgetting a control group, heh!) and at what time (dd/mm/yy), you’d get your paper sent back.
All this comes under ‘Material and Methods’.
Next you have ‘Results’.
Thats where all your numbers go, and the stats.
The point is, especially in biology, that anybody must be able to go where you went, do exactly as you’ve done, and come up with the same results (given a dead frog here and there …)
Then you can talk about what you’ve done and what your results mean.
That is why I, a retired zoologist, find these revelations so utterly distressing. If you don’t provide the data on which you’ve built your hypothesis, how can it ever be replicated? How can it be confirmed or refuted?
Science is about replication of that what you discovered – its not about secret knowledge which only the select are allowed to share.
I am dismayed at the huge disservice these people have done to science.

This all may be a distraction, to keep our eye off the Copenhagen ball.
Make no mistake, they have not given up. Quite the contrary.
“Obama says ‘step closer’ to climate deal”
http://news.ninemsn.com.au/article.aspx?id=975599
They have given us the sacrificial goat, which has served it’s purpose. And, while we have a feeding frenzy, the real beast walks by unhindered, and barely noticed.

this is so bad.
the global financial ramifications of this fraud are incomprehensibly huge. even when one considers damages incurred to date, let alone the future damages of pending legislation, there are at least hundreds of billions of dollars that have been bilked from taxpayers worldwide to fund this insidious mess. it must be the largest single fraud ever perpetrated.
anybody an expert on class action lawsuits to recover damages and put the fraud that is global climate change on trial?
at least in that case the world may be able to subpoena these groups to get at the truth once and for all…
makes one sick to think what a handful of politicians and “scientists” have been able to do to all of us and nearly every industry.
as a builder, the entire LEED certification process is nearly fully predicated on data and conclusions put forth by these groups, which has resulted in enormous additional costs on nearly every public construction project. what a sham.
i’m all for sustainability, but fraud is fraud.
-patternbuilder

Seems to me that “calibration” of tree ring data is a bit of a joke.
For the modern periods where we have the highest quality records available we see the “recalibration” where the numbers get “fudged” (the scientific term used in the code is “fudge factor”) in order to make a “calibration” work that makes the pre 1900 data reflect cooler temperatures than if the data had been calibrated to post 1900 data.
Not only do we have the “hide the decline at the end” (which can be argued as a plausible treatment if young tree rings are somehow not “ripe”), but the pre 1900 manipulation is just the sort of thing you need to do in order the straighten the handle of the hockey stick, isn’t it?

Ironically, the University of East Anglia has a Computer Science department:

Welcome to the School of Computing Sciences, CMP. With about 35 staff and 500 students we conduct research and teach in the fields of Computer Science, Business and Electronics. We have a consultancy company, SYSCO, and many of our staff have close links with industry.
Computing science graduates are well paid and our graduates are among the best paid from UEA and can be found in almost every employment sector all over the world.

The fact is that they have the expertise on campus to engineer a decent piece of software. I’m not saying it would neccessarily work that way of course. Most of the good practice in design and implementation I have learnt since leaving University, not while I was an undergraduate. In industry you literally won’t have a paycheck if things don’t work.
The problem is that nobody outside of a small circle of users was asked to audit the software, or, I suspect, to contribute to its design or development. It’s quite stunning that its output is being used as “evidence” (cast iron!) forming the basis of trillion dollar government programmes.
But anyway, we don’t know that the program is broken. It probably produces the desired output ;).

“Pieter F (09:16:40) :
… why won’t the mainstream media report on the matter?”
…Because they’re all waiting for each other to be the first to break the story properly, which will take a huge investment and a massive gamble with their credibility to pull off.
When the Telegraph exposed our MPs’ expenses they were assigning up to 60 journalists to cover it, and check all the facts before each publication. That’s why all the main papers are still only putting the CRU fraud in sidebars and in non-staffers’ opinion pieces, and not in their headlines.

Wow.
*** MUST ALTER FUNCT_DECLINE.PRO TO MATCH THE COORDINATES OF THE
; START OF THE DECLINE *** ALTER THIS EVERY TIME YOU CHANGE ANYTHING ***
Just…wow….

Swiss Bob (10:38:34) :
For those of you who don’t know, the Cloud experiment at CERN is meant to test the connection between cosmic rays (from the sun and galactic centers) and cloud nucleation and hence cloud formation. The rate of cosmic rays entering the atmosphere is controlled by the strength of the earths magnetic field. During strong sunspot activity the earths magnetic field strengthens, limiting the amount of cosmic rays. When the sun is quiet (as it is now) the earths magnetic field weakens allow in more cosmic rays. To summarize the theory:
Quite Sun -> more cosmic rays -> more clouds
Active Sun -> fewer cosmic rays -> fewer clouds

Got to nail these guys, otherwise, to use an apt quote “For if we fail, the whole world will sink into a new Dark Age, made more sinister, and perhaps more prolonged, by the light of a perverted science.”

Increddible.
Scientists defends climate data he knows is tempered with and calls for an investigation at the same time. That’s what I call “turning on a dime”.
“Scam artist” gets a whole new meaning.
http://climaterealists.com/index.php?id=4457&utm_source=feedburner&utm_medium=feedutm_campaign=Feed%3A+co2sceptics%2Fnews+%28CO2sceptics+News+Blog%29

Only in their world that tree-thermometers are better than real thermometers. Amazing!

*************************************
Robinson (11:58:42) :
The fact is that they have the expertise on campus to engineer a decent piece of software. I’m not saying it would neccessarily work that way of course. Most of the good practice in design and implementation I have learnt since leaving University, not while I was an undergraduate. In industry you literally won’t have a paycheck if things don’t work.
**************************************
Judging from the comments, it probably isn’t the programmer that’s bad, it’s the data. It appears he was being pushed to achieve a certain outcome. If he was disgusted and frustrated enough, he just might do something a little rash like push the code and other info into the wild.

Connected to the station count problem, this was one of my jaw-dropping moments so far in HARRY. It confirms the problem I diagnosed in GHCN concerning station “sphere of influence” effects.
“Worked out an algorithm from scratch. It seems to give better answers than the others, so we’ll go
with that. Also decided that the approach I was taking (pick a gridline of latitude and reverse-
engineer the GCD algorithm so the unknown is the second lon) was overcomplicated, when we don’t
need to know where it hits, just that it does. Since for any cell the nearest point to the station
will be a vertex, we can test candidate cells for the distance from the appropriate vertex to the
station. Program is stncounts.for, but is causing immense problems.
The problem is, really, the huge numbers of cells potentially involved in one station, particularly
at high latitudes. Working out the possible bounding box when you’re within cdd of a pole (ie, for
tmean with a cdd of 1200, the N-S extent is over 20 cells (10 degs) in each direction. Maybe not a
serious problem for the current datasets but an example of the complexity. Also, deciding on the
potential bounding box is nontrivial, because of cell ‘width’ changes at high latitudes (at 61 degs
North, the half-degree cells are only 27km wide! With a precip cdd of 450 km this means the
bounding box is dozens of cells wide – and will be wider at the Northern edge!
Clearly a large number of cells are being marked as covered by each station. So in densely-stationed
areas there will be considerable smoothing, and in sparsely-stationed (or empty) areas, there will be
possibly untypical data. I might suggest two station counts – one of actual stations contributing from
within the cell, one for stations contributing from within the cdd. The former being a subset of the
latter, so the latter could be used as the previous release was used.
Well, got stncounts.for working, finally. And, out of malicious interest, I dumped the first station’s
coverage to a text file and counted up how many cells it ‘influenced’. The station was at 10.6E, 61.0N.
The total number of cells covered was a staggering 476! Or, if you prefer, 475 indirect and one direct.”

As a software engineer I ask: where are the test cases that prove the proper functioning of all this code? Surely there could be test data sets that could be fed in and check that the output is as expected, before applying it to real data? Test handling missing stations, duplicated stations, wildly varying Tmin/Tmax to generate alerts during generation.
If this is not done then a simple programming bug introdiced when modifying code will remain hidden and be very difficult to discover. Only code reviews/blind luck would catch this, without test cases.

Finally, someone mentioned Nyquist! It is about time.

I starting to think “climategate” is going to turn into a $100,000,000 (or so) grant to CRU for software engineers to clean up the mess. Jones will be the sacrificial lamb, but he’ll walk with a huge severance package since he brought the huge grant in.
Never let a good crisis go to waste, you know.

Mark (11:33:11) :
“I’d like to know what the “decline” is. Is it temperature?”
I’ll take a stab at your question with an offer to have anyone else correct it. The decline is not in the temperature record, but in the proxy data(tree rings, etc.). During a “calibration period”, the temp data is matched up to proxy data for good correlation. The CRU problem is the well known “divergence” that occurred between the proxy data and the temperature record, i.e., the temperatures went up, but the proxy data went down. To hide that decline in the proxy data, the temperature was spliced onto the end of the proxies at a convenient point (1940-1960). Voila, no more divergence
Which begs the question: if the proxies diverged after this period, why couldn’t they have diverged before the era of instrumental temp records. Any takers?

Slashdot comments has an interesting link to a Finnish TV documentary where the reporters discovered the ‘climate scientists’ had flipped a temperature chart that was showing cooling to show warming instead.

How do you say Gotcha! in Finnish?

We ‘programming geeks’ try to comment our code so we can understand it when we go back to revise it, sometimes years later. Since we expect that we are the only ones who will ever read it, we tend to be very honest in our comments, especially “why” we include/exclude/fudge something.
Fudge factors are normal when trying to account for ‘real world’ data with an imperfect model. Anyone involved in modelling knows this. You have to have some way to account for factors that are not understood. The idea is to find a way to bring the results of the model into line with real world’s data. It is never suppposed to be used to twist the data to create an artifical world, which is what this nightmare is trying to do.

Averaging day and night temperatures has been reported as enhancing perceived warming in England as daytime alone shows less, or no, warming.
This was highlighted this summer, which was dull and miserable. Our esteemed Met Office( see list of usual warmist suspects) put out some spin that it was warmer than usual when they included the night.
As trees grow in the sunlight while eating CO2, perhaps the tree ring proxies reflect only daytime conditions. This would make them different from the “real” temperatures if these included the night. This might hide any decline in daytime growth of trees if spliced on after 1960.