The Smoking Code, part 2

That’s the virus!
For a satirical look at the climategate computer programming:
Anthropogenic Global Warming Virus Alert.
http://www.thespoof.com/news/spoof.cfm?headline=s5i64103

@Andrew…
LOL!! That is some quality satire. 🙂

Very nice work! I think the code is where the real damage lays.

The “fudge factor” part is just to scale the ‘artificial’ array. I’m pretty sure looking at the code that is just to help line things up. That isn’t a pass — the programmer is a total amateur on the comments alone, but that part doesn’t mean they were fudging the big picture, just they were wanting to use a single variable (err, constant in this case) to help line things up.
The bigger picture is the inevitable ‘blade part’ of of every graph these guys produce is based on a different cocktail of data than the stick part. They go on and on how this is ‘ok’, and how this is ‘known’, but it isn’t up to climatologists to decide what is ok with math and statistics. It is obviously trying to squeeze desired results from data.
The world seems happy to ignore this — whatever, I guess they are tired of oil or voting or something. Regardless, this is shit science. Actually it isn’t even that, it is tainted before it even gets past the math, and probably before it even gets past writing down the data. The ‘science’ never had a chance.
Ahhh, its depressing, I’m just going to go to bed (umm, not with Shell though).

There’s some interesting stuff in cru-code/linux/mod/ghcnrefiter.f90 too.

Finally, news about the solar minimum.
“The sunspot count continues to be at the lowest level in a century. Looking back in the past, there appears to be a strong link between an inactive sun and a cooler earth. The sun is the source of incoming heat on this planet. It has cycles that we haven’t been around long enough to study and understand clearly.”
“Climategate” Exposes Global Warming Hoax
http://www.courierpress.com/news/2009/dec/06/quotclimategatequot-exposes-global-warming-hoax/

Are the EPA mad? How on earth can CO2 be a Public Danger.
OK – got it! We breathe out and the CO2 content is around 40,000 parts per million. So…….. as we are panicking about Mauna Lou CO2 levels climbing and at present 387.75 parts per million then why not issue draconian laws when the magic figure of 40,000 parts per million can be quoted.
Solution is we all [6+ Billion of us] stop breathing! Problem solved! Danger over!
EPA – Environmental Panic Attackers

Kiss off the next three years. The current administration isn’t interested in facts. They’re on a mission to ram an agenda down America’s throat while they can. We need to look at how we can recover after 2012 (if we’re lucky)

Phillip Bratby (23:23:43)
Yeah, but given the apparent mess that is the CRUs’ electronic data, even with complete access to all the files, it doesn’t look good. e.g. what’s in
crucode/idl/pro/README_GRIDDING.txt
“Then use quick_interp_tdm2.pro
on the secondary variable, with synth_prefix supplied, to create the new grids.
Bear in mind that there is no working synthetic method for cloud, because Mark New
lost the coefficients file and never found it again (despite searching on tape
archives at UEA) and never recreated it. This hasn’t mattered too much, because
the synthetic cloud grids had not been discarded for 1901-95, and after 1995
sunshine data is used instead of cloud data anyway.”
Wonder what else is on or gone from the archives…

http://strata-sphere.com/blog/index.php/archives/11630
Sorry loink to AJstrata’s work.

The smartest kid in the class (CRU) just got caught cheating. Why are his frat brothers (GISS, NOAA) claiming that everything is ok because they all put down the same answers on the test? If CRU takes the real data and adds fudge.dat, then aren’t they all just selling different flavors of fudge?

“Emalis that rocked climate change campaign leaked from Siberian ‘closed city’ university built by KGB
An investigation by The Mail on Sunday has discovered that the explosive hacked emails from the University of East Anglia were leaked via a small web server in the formerly closed city of Tomsk in Siberia.
Computer hackers in Tomsk have been used in the past by the Russian secret service (FSB) to shut websites which promote views disliked by Moscow.
In 2002, Tomsk students were said to have launched a ‘denial of service’ attack at the Kavkaz-Tsentr portal, a site whose reports about Chechnya angered Russian officials.
The FSB office in Tomsk put out a special Press release saying that what the students had done was a legitimate ‘expression of their position as citizens, one worthy of respect’.”
http://www.dailymail.co.uk/news/article-1233562/Emails-rocked-climate-change-campaign-leaked-Siberian-closed-city-university-built-KGB.html

jorgekafkazar, please may I have your permission to use this bit for my quote of next week?
“CRU has been caught with weapons of math destruction in their code.”
Thanks in advance.

Brian Johnson uk (23:27:41) :
Are the EPA mad? How on earth can CO2 be a Public Danger.
The EPA is composed of bureaucrats, with a very few genuine scientists tossed in to give the appearance of a science-oriented gummint agency.
Some time ago, I was monitoring a methane burn at a landfill and had an EPA inspector claim that the burner was inoperative because he couldn’t see the flames. When I told him that pure methane burned at such a high temperature the flames were only visible in the infrared spectrum, he harrumphed and thrust his hand into the plume to *prove* I was lying.
I drove him to the hospital to have his second-degree burns treated.

CRU has been caught with weapons of math destruction in their code.

This is a quotable quote! 😉
Larry

Wow, I really think you guys missed the boat with all that code up there. I’ve come up with a simple equation that even a 12 year old can understand that explains EVERYTHING:
AGW = (OT + (DFU/$)RG + AGF) / CAPBSIF
For those who need help sorting this out, this shows that AGW is the sum of:
OT – Original Temperature Findings
DFU/$ – Tells us how much money Mann, Jones, et al, get per each Degree they can Fudge Upward (And subsequently this leads to RG, Reasearch Grants.
AGF – Al Gore’s Carbon Footprint
And this is very important, so that we get the numbers right – After we sum all of that up, you MUST divide by CAPBSIF, that is, the # of computer animated polar bears you can show stranded on ice flows.

As the article points out, even if this code wasn’t used, the mere fact that it was there points to a need to “adjust” data. This is not interpolation.
And even though this code seems to have been for the treemometer data, doesn’t mean there wasn’t other shenanigans going on with the temperature dataset.
AJStrata’s work seems to point out that the raw data doesn’t confirm the warming that the GISS and CRU “adjustments” shows, but no doubt, if cornered, they’ll tell us that a “fudge factor” has to be added in to cover some inadequacy or other in the GHCN dataset.
This whole thing is too big for people to be going off on their own tangents in analysing the data. What we need is to coordinate our efforts, standardise how the data is treated, and come up with a single alternative analysis.
We can’t wait three years for the Met Office to announce their re-analysis, and anyway, one visit to their site shows where their prejudices lie.
We already have a large number of capable people posting to this site, and their email addresses are known to this site, so I’d suggest that a section of the site or another effort like the surfacestations.org work be got up and running.
I have skills in website and application programming, and am currently learning R, so if there was anything I could do to help with this effort, I’d be glad to.

From previous post:
Climate Heretic (16:18:03): I disagree with you.
woodentop (16:01:48): Sorry pal, but that’s rubbish.
The need for source code readability, maintainability and quality certainly depend on the context in software engineering. I addition this is not the main point here, let us assume that the source code is poor quality as you suggest, does that help us in any way? NO! I think we need a much more PRAGMATIC point of view here, KILL YOUR DARLINGS (in software), we need to COMPILE the code, EXECUTE the code to determine WHAT it is doing!
Complaining that the source code is poor quality is a dead end – please stop doing that!

OT. BBC explanation of climate change http://news.bbc.co.uk/1/hi/sci/tech/8394168.stm But am I right in thinking that you will get a temp rise no matter what gas you put it to the ‘other’ bottle?

osborn-tree6\mann\mxdgrid2ascii.pro 10/03/2008
endif else begin
ml=where(finite(mxdfd) eq 0,nmiss)
if nmiss gt 0 then mxdfd(ml)=-9.99
printf,1,’Osborn et al. (2004) gridded reconstruction of warm-season’
printf,1,'(April-September) temperature anomalies (from the 1961-1990 mean).’
printf,1,’Reconstruction is based on tree-ring density records.’
printf,1
printf,1,’NOTE: recent decline in tree-ring density has been ARTIFICIALLY’
printf,1,’REMOVED to facilitate calibration. THEREFORE, post-1960 values’
printf,1,’will be much closer to observed temperatures then they should be,’
printf,1,’which will incorrectly imply the reconstruction is more skilful’
printf,1,’than it actually is. See Osborn et al. (2004).’
printf,1
printf,1,’Osborn TJ, Briffa KR, Schweingruber FH and Jones PD (2004)’
printf,1,’Annually resolved patterns of summer temperature over the Northern’
printf,1,’Hemisphere since AD 1400 from a tree-ring-density network.’
printf,1,’Submitted to Global and Planetary Change.’
;
printf,1,’Grid resolution is 5 by 5 degrees, for the Northern Hemisphere only,’
printf,1,’with the first value at ‘,g.x[0],’ W’,g.y[0],’ N,’
printf,1,’then moving eastwards to ‘,g.x[g.nx-1],’ and then southwards to ‘,g.y[g.ny-1],’.’
printf,1
printf,1,’Missing value is -9.99′
printf,1
for iyr = 0 , mxdnyr-1 do begin
print,mxdyear[iyr],format='($,I5)’
printf,1,mxdyear[iyr],format='(I4)’
printf,1,mxdfd[*,*,iyr],format='(10F8.2)’
endfor
print
endelse

Heard on the Andrew Marr show on the BBC this morning from the US Ambassador to the UK:
‘Climategate’ has not been picked up on in the US. There you have it. Nothing to see here. Move along please. Have I been in a dream for the past 2 weeks or so? I am sure there are Americans who have been blogging like mad. I know Fox News has. Oh I know. He was talking about most of the American mainstream media wasnt he????

From previous post:
Carsten Arnholm, Norway (02:57:28) : It really does not matter whether the people writing this code were “smart” or not. What matters is that the result is of very poor quality.
No. That’s complete nonsense! We need to COMPILE the code, EXECUTE the code, REVERSE ENGINEER how it is supposed to work together with the RAW data files in the dump in order to CONCLUDE exactly what it is doing.
Complaining that the source code is poor quality does not help us in any possible way, that’s certainly a dead end a RED HERRING that draws attention away from the central issue which is whether they have ADAPTED the code to the AGW hypothesis. Imagine that we manage to find accurate digital proof that the code reveals CONVENIENT ADJUSTMENTS – that would really be something.

Robert (OP), the point you haven’t dealt with, that several people pointed out in the previous posting, is that this code was either used as “thought experiment” test of the calibration procedure for the Briffa tree ring data (the filename indicates this), or as a way of bootstrapping a correlation process, both of which are perfectly reasonable things to do.
Here’s Gavin of RC on the subject (which was quoted by “Norman” in comments on your previous posting):
“It was an artificial correction to check some calibration statistics to see whether they would vary if the divergence was an artifact of some extra anthropogenic impact. It has never been used in a published paper (though something similar was explained in detail in this draft paper by Osborn). It has nothing to do with any reconstruction used in the IPCC reports.”
And indeed, in the same set of comments, “Morgan” pointed out that the Osborn et al. paper explicitly describes this step:
“To overcome these problems, the decline is artificially removed from the calibrated tree-ring density series, for the purpose of making a final calibration. The removal is only temporary, because the final calibration is then applied to the unadjusted data set (i.e., without the decline artificially removed). Though this is rather an ad hoc approach, it does allow us to test the sensitivity of the calibration to time scale, and it also yields a reconstruction whose mean level is much less sensitive to the choice of calibration period.”
I’m not sure which one of these your particular code snippet is doing, but either seem perfectly reasonable explanations to me – and both require the code to be added and them removed again. The lazy programmer’s way of doing this is by commenting and uncommenting.
Let me give you an analogy: My professional sphere is in real-time video streaming software. In one of my bits of server code there is a mode (switchable by configuration, not comments, but never mind…) where it can corrupt every Nth byte of every Mth packet; we use this to test the resilience of the client device to network errors.
Now let’s imagine some ‘hacker’ breaks in and steals this code, and for some reason the probity of our software was internationally contentious (not likely, but still…), and people start picking over the code looking for “juicy bits” without understanding the full context. Maybe they find something like this:
// Corrupt every ‘spacing’ bytes
void Buffer::corrupt(unsigned int spacing)
{
for(unsigned int i=0; i<length; i+=spacing)
*((unsigned char *)start+i) ^= 0xA5;
}
Imagine the furore! Deliberate corruption/falsification of video streams! Programmer uses hacky C-style casts in C++! What is the significance of this mysterious A5 value?!
Let me be clear: I think there are issues about scientific openness here, especially given the importance of the output, but really, this isn't one of them.

My question would be… what kind of configuration management system or scheme was the code stored in at UEA, and can the history of changes to the code be tracked from release to release?
Certainly, if trillions of dollars are at stake, along with the birth of a new industry (to dwarf all other industries), they’d have at least tried at a minimum, to proceed using a “bare minimum entry-level” of code configuration and documentation to cover their hineys, wouldn’t they?
If this is not the case, is it acceptable practice in the scientific community to not have past code versions stored/documented, which produced specific charts and data?
All our code is under CM where I work but I’ve always chuckled about the fact nobody has ever had to use it to quantify something from the past – we’ve never had to rely on our CM system although not much is at stake making it seem like a waste. However, this particular exercise makes quite clear to me, that despite my arrogance and my laughing about the usefulness of it, I clearly and completely now get it; I understand why, at a minimum, we do attempt to practice this “bare minimum” of due diligence; this is my newly self imposed personal professional biaach slap and it is a good one. My face is red but it is smiling 🙂
So, in summary, to what degree of CM did UEA keep this seemingly important code under? I mean it must have one hefty history eh??

Ok so after all this they are not in fact Climate Scientist but very very bad statisticians?

I’ve been programming in IDL to produce plots for scientific papers for about 10 years and this pro clearly isn’t producing a plot for publication. The pro just plots to screen – if it were for publication then it write the graph out as a postscript file to submit to the publisher. To me, this looks like someone experimenting with the data, which doesn’t really mean anything.

Suzanne,
Since when is permission needed to quote someone?
Quoting is everyone’s right.

😀
http://blogs.telegraph.co.uk/news/jamesdelingpole/100019018/climategate-how-the-copenhagen-grinches-stole-christmas/

Invariant (03:31:01) :
From previous post:
Carsten Arnholm, Norway (02:57:28) : It really does not matter whether the people writing this code were “smart” or not. What matters is that the result is of very poor quality.
No. That’s complete nonsense! We need to COMPILE the code, EXECUTE the code, REVERSE ENGINEER how it is supposed to work together with the RAW data files in the dump in order to CONCLUDE exactly what it is doing.

No need to shout. I was commenting on your assertion that “smart” people wrote poor code, which indeed is wrong and a distraction.
Of course we need to figure out what the code is doing, nobody suggested anything else.

[Completely off topic. ~dbs, mod.]

It seems (paranoia perhaps), that all the major search engines are trying to play the skeptics (sites, news etc..) down.. Has anybody else noticed? Delaying news stories, typing climategate etc..?

Marc Sheppard explains THE TRICK
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html
This is where the focus has to be, and thr FOIA violations.

IDL_chap:

To me, this looks like someone experimenting with the data, which doesn’t really mean anything.

It speaks volumes about the mindset of those involved.

Anna V: Does not
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
amount to much the same caveat? Certainly if I wanted to actually fudge something without anyone knowing, a 15-star comment wouldn’t be my first thought.

Yahoo may be a better choice for climategate news me thinks

Carsten Arnholm, Norway (04:18:08) : I was commenting on your assertion that “smart” people wrote poor code, which indeed is wrong and a distraction.
OK. Then we agree. Indeed not all smart developers write spaghetti code, but my personal experience is that it is sometimes difficult to understand the source code written by the “smart” people that “manage to juggle 40 balls (functions, variables)” without doing mistakes. Another point is, of course, that if you intend to do something illegal, then you would possibly try to hide it in a mess…

And, slightly OT, here’s “Scientists behaving badly, Part 3”:
The editorial in the current issue of the journal, “Nature”, talks about ‘denialists’.
When one of the premier scientific journals starts using language like that, it’s clear that the age of reason is well and truly over.
http://www.nature.com/nature/journal/v462/n7273/full/462545a.html

See Al Gore’s response to Climategate here
[snip – we don’t run the Hitler parody video here – think “deniers” – A]

woodfortrees (Paul Clark) (04:34:04) :
Anna V: Does not
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
amount to much the same caveat? Certainly if I wanted to actually fudge something without anyone knowing, a 15-star comment wouldn’t be my first thought.
Not so, in my opinion of course. I do not think these people were really aware that what they were doing was unscientific. They had the market cornered and they were sure they were right.
Anthony does not like religious analogies, but, in my opinion, the sociology applies to these “scientists”. Were the people counting the number of angels that could fit on a pinhead aware of the preposterousness of their case? When you start by believing that angels exist and can be counted, then anything goes. When you start believing that there is unprecedented twentieth century warming then it is obvious that the data is wrong and has to be corrected for. Simple.

P Gosselin (04:24:36) :
Marc Sheppard explains THE TRICK
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html
This is where the focus has to be, and thr FOIA violations.
Totally agree. This is an excellent summary of exactly who Gordon Browns and Ed Millipeds real flat-earthers and climate sabouteurs are. No other than themselves!
IMHO, this article would be a perfect thread here.

Another interesting quote from the UK Telegraph this morning:
“The other is that the ugly, drum-like concrete building at the University of East Anglia which houses the CRU is named after its founder, the late Hubert Lamb, the doyen of historical climate experts. It was Professor Lamb whose most famous contribution to climatology was his documenting and naming of what he called the Medieval Warm Epoch, that glaring contradiction of modern global warming theory which his successors have devoted untold efforts to demolishing. If only they had looked at the evidence of those Siberian trees in the spirit of true science, they might have told us that all their efforts to show otherwise were in vain, and that their very much more distinguished predecessor was right after all.”
A bit ironic you might say.

[snip – thanks but we have this video prominently on the main page – A]

anna v (05:37:25) : When you start believing that there is unprecedented twentieth century warming then it is obvious that the data is wrong and has to be corrected for. Simple.
Excellent! That’s the correct focus. The BBC coverage “CRU’s programming ‘way below expected standards’” is clearly a red herring.
http://news.bbc.co.uk/2/hi/programmes/newsnight/8395514.stm

Peter (04:53:56) :
And, slightly OT, here’s “Scientists behaving badly, Part 3″:
The editorial in the current issue of the journal, “Nature”, talks about ‘denialists’.
When one of the premier scientific journals starts using language like that, it’s clear that the age of reason is well and truly over.
http://www.nature.com/nature/journal/v462/n7273/full/462545a.html
No, Peter, I don’t think so. Only the age of this co-conspirator journal — and Science Mag is soon to follow. End your subscriptions and subscribe to the valid science “publications” on the internet like WUWT, CA TAV, Chiefio, and others. Give them the same subscription fee that you would have — before they went with THE FIX –given to every print mainline science publication. This would be one huge shift in economic power.
Add to that decision, another one to either take back or end your membership in every professional scientific society that approved THE FIX under the guise (masquerade) of AGW, global warming, climate change.
THE FIX I refer to is that Obama (probably) is going to Copenhagen to revel in his “citizen of the world, not the USA” self-aggrandizing title as the agreement re our CO2-pollution fees (in whatever form the EPA decides) are sent to an unelected UN bureaucracy (unaccountable to any sovereign nation) that is ready to rule the world. For what purposes? This is as important a “scientific” topic as is the rotten code and the traitorous pseudo-scientists. May they all rot in jail before their time in hell. They have been agents of this treason.

Deliberate,calculated,and possibly incompetent while doing so.
This is indeed the smoking gun,er, code…

Here’s another take on some of the code bugs, etc.:
http://bishophill.squarespace.com/blog/2009/12/6/good-code-analysis.html

AlanG (01:01:07) :
harrfn (harry filename?) is an array of data filenames found here:
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
My guess is they correspond as follows:
nwcan = North West Canada
wnam = Western North America
cecan = Central Canada
nweur = North West Europe
sweur = South West Europe
cs and es could be ISO country codes but ns and ti are not.
nsib = Northern Siberia?
csib = Central Siberia?
tib = ??
esib = Eastern Siberia?
tib = Tibet?

Regardless of what the raw data might show, hiding it IS foul play. They are a government agency and subject to FOIA requests legally which they have ignored. I say prosecute them and lock them up. They are asking us to spend fortunes without providing the raw data!!!

Al Gore heckled on Youtube…

It may actually take this kind of protest to wake up the sheeple.
What I see here is some young frustrated skeptics resorting to making a protest on amateur video and publishing it on Youtube in response to the blatant fascist propaganda being pumped out by mainstream western media.
Make no mistake about it, when deniers are vilified and attacked, when scientists are prevented from publishing opposing data in journals, when the media ruthlessly shuts down and covers up all opposing views, when the media continues publishing pro-anthropogenic stories daily, when skeptics are called a*ssholes on national television (and the BBC allows it to go on the air …oops a mistake…sure, and I was born yesterday):
We are dealing with rampant FASCISM.
It would all be so funny if it was on a Monty Python show but the sad thing is that this is REALLY happening.
The proof is – like all FASCISTS – the people swept up in this movement are totally absolutely convinced – they have no compunction about following the “end justifies the means” path. What is amazing is that these are mostly “liberals” who are orchestrating mass media manipulation and suppression of free speech (yes when I find countless posts MODERATED out of existence and a complete refusal to debate Climate Change, I think we can call this suppression of free speech). Since when did “liberals” turn to the dark side of tyranny and dictatorship – we know what is best for you – so “Shutup” as Dr Watson at the CRU put it!

Wood for trees
Thank you for you level headedness where all else is hype.

This quote from the BBC is worth keeping:
”The IPCC relied on three documents to arrive at 2035 as the “outer year” for shrinkage of glaciers.
They are: a 2005 World Wide Fund for Nature report on glaciers; a 1996 Unesco document on hydrology; and a 1999 news report in New Scientist.
Incidentally, none of these documents have been reviewed by peer professionals, which is what the IPCC is mandated to be doing…..
But in a joint statement some the world’s leading glaciologists who are also participants to the IPCC have said: “This catalogue of errors in Himalayan glaciology… has caused much confusion that could have been avoided had the norms of scientific publication, including peer review and concentration upon peer-reviewed work, been respected.”…
“Under strict consideration of the IPCC rules, it should actually not have been published as it is not based on a sound scientific reference.
http://news.bbc.co.uk/2/hi/south_asia/8387737.stm
Lets see they screwed up the reports of melting glaciers, the polar bear population is increasing not decreasing, and the CRU temperature data is extremely suspect and certainly does not meet “strict consideration of the” scientific method.
Yup, the science is settled, it is settling in to a pool of quick sand, or is that a peat bog?

OK. Then we agree. Indeed not all smart developers write spaghetti code, but my personal experience is that it is sometimes difficult to understand the source code written by the “smart” people that “manage to juggle 40 balls (functions, variables)” without doing mistakes. Another point is, of course, that if you intend to do something illegal, then you would possibly try to hide it in a mess…

Indeed, as a Software Developer that is my experience too. I’ve seen the most amazingly tortuous (in terms of cyclometric complexity) code come from the smartest people.

wood for trees:
Mann’s nature trick of “hiding the decline” signifies the appropriate method to cover up the declining temperature data.
Its no less hard to imagine than calibrating satellite data in line with ground stations when they didn’t show a warming trend. If anything they should have been calibrated downwards – if satellites drift, they go closer to earth and read higher temperatures. The official explanation is exactly the opposite of the truth.

The way these closed circle of statisitcs manipulators (I can’t call them climatologists) corrupt and delete/manipulate data then foment excuses about it is like those accounting procedures at Enron.

Who cares about code, emails, etc. when a really good time is to be had in Copenhagen? Biggest block party ever. Open bar, limo’s, free hookers, hash, caviar, etc. 1200+ limos, of which 5 are electric and hybrid. Can you spell hypocrisy?
http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6736517/Copenhagen-climate-summit-1200-limos-140-private-planes-and-caviar-wedges.html

Debunking Briffa’s Version of the Hockey Stick
Jim Lindgren • December 6, 2009 12:18 am
Among the most ethically challenged of the scientists at the Climate Research Unit at the University of East Anglia is Keith Briffa. A couple of months ago, the Bishop Hill blog retold in detail the sad story about Briffa’s own version of the Hockey Stick, which he was able to keep alive by his attempts to prevent other scientists from discovering what he had done with the data, a practice facilitated by biased journals that refused to apply their own rules.
http://volokh.com/

http://www.woodfortrees.org/plot/hadcrut3vgl/from:1979/to/plot/hadcrut3vgl/from:1979/to/trend/plot/uah/from:1979/to/plot/uah/from:1979/to/trend/plot/rss/from:1979/to/plot/rss/from:1979/to/trend
Why would CRU need to fudge lines of code if their temperature record since 1979 shows warming that is fairly consistent with RSS and UAH. Certainly skeptics Spencer and Christy are not using CRU code.
Here is a 9 minute video that shows how ridiculous the Climate Gate emails really are:
http://www.desmogblog.com/another-look-stolen-emails
As I said when this story first broke, the wolves were given a fake piece of meat.

I’m a skeptic, but I agree with John Graham-Cunning on this issue:
http://www.jgc.org/blog/2009/11/very-artificial-correction-flap-looks.html

Knowing what you now know about CO2 and about the politics behind this entire cluster*, what does this action mean to you? link: http://www.wvmetronews.com/index.cfm?func=displayfullstory&storyid=33948

Correction:
That should be NOEL Sheppard (not Marc).
THE TRICK
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html

Robinson (07:48:06) : I’ve seen the most amazingly tortuous (in terms of cyclometric complexity) code come from the smartest people.
Thanks. I try to write clear code myself though. Actually a nice analogy is that many of the developers that really care about maintanability, readability and code quality really has more in common with marketing people than most hard core scientists and engineers…

(These days, I find “Pravda” more truthful than say “The Economist.”
Whence this madness?)
Very good question. Perhaps it is the human need to have a uniting religion to help feel “connected” through “fellowship”.(the so called God gene). This can be easily exploited by politicians like Al Gore, who is a theology major.
Each to his own I like to say, but in this case, this AGW religion would destroy the US economy by driving up energy prices for no good reason, and spending the money on useless CO2 sequestration.
I fear for our futures. Will we end up having street fights between eco freaks and the millions of unemployed that surely will result from economic socialism? Will we see guns come out to greet the eco freaks, and not just hostile words? I don’t know, but I think we are going to find out how “patriotic” Americans react to the destruction of thier freedom. Maybe that is why the USA currently is arming itself like crazy with guns and ammunition. Throw in a racial bias against the President, and all hell could break loose. Tea parties indeed!
Can Mann et al live with what they are producing through thier lies??? Have they NO sense of responsibility??? Is there NO attorney general in the USA that sees the need to charge Mann et al with fraud? The clock is ticking.

geo (00:08:53) :
>The smartest kid in the class (CRU) just got caught cheating. Why are his frat brothers (GISS, NOAA) claiming that everything is ok because they all put down the same answers on the test? If CRU takes the real data and adds fudge.dat, then aren’t they all just selling different flavors of fudge?<
An excellent analogy.

Heck – I don’t know his name anymore – How about just Mr Sheppard…
(see link above )

If the CRU lost their original data then what are they working with now? You would think that this would shut down all research history matching.

Robinson (07:48:06) :
“Indeed, as a Software Developer that is my experience too. I’ve seen the most amazingly tortuous (in terms of cyclometric complexity) code come from the smartest people.”
In my working life, before I retired and regained my sanity, I wrote many time-critical hardware drivers (mostly assembly, as C-callable procedures) for various production machinery and telescopes. We had a standing rule that no mathematician, statistician, scientist, or astronomer was allowed to write ANY hardware code! This followed a rather spectacular crash of a telescope that attempted to simultaneously rotate the RA and Dec axises through 360 degrees! Cables??!! We don’t need no stinking cables!!!

Everybody who has a computer laughes about Harry, poor chap. And he was only trying to understand the genius from CRU.
So let us offer him space in this blog to explain himself; maybe he is smarter that the rest of them together.

Nick Stokes (11:01:38) : The term “fudge factor” doesn’t indicating cheating, as some commenters imply.
In physics and simulation science, fudge factor is the factor you need multiply your equation with to get a perfect match with experiments.
After I wrote the sentence above I checked with Wikpedia.
Fudge factors are invented variables whose purpose is to force a calculated result to give a better match to what happens in the real world.
http://en.wikipedia.org/wiki/Fudge_factor

woodfortrees (Paul Clark) (04:34:04) :
Anna V: Does not
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
amount to much the same caveat? Certainly if I wanted to actually fudge something without anyone knowing, a 15-star comment wouldn’t be my first thought.
I have a question for woodfortrees. Why do you assume the author of the comment didn’t want anyone to know about it? When in a den of thieves you don’t try to hide criminal exploits, you brag about them.

Scott A. Mandia (08:29:35) :
If you read the emails, or as many as you can, such as Briffa’s severe scepticism, his ambivalence to Mann, it is odd that such a conclusion is drawn. Its clear to me that Briffa gave into the project, or was browbeaten into it. (That might be why he isn’t being implicated in the fraud.)
they are interesting to read.
Anyhow, shouldn’t this be a court case in the manner an Inconvenient Truth was taken to court?

Reed Coray (12:08:13) :
.
.
.
When in a den of thieves you don’t try to hide criminal exploits, you brag about them.
I see a different possibility: An honest programmer forced to do dirty work but who protests about it in the comments.
And also this, truth is invaluable, even to liars.

“we need to COMPILE the code, EXECUTE the code to determine WHAT it is doing!”
Indeed compilation and execution with a dummy dataset would be an initial step. After that one would need to use a known dataset and matching published CRU results to check reproducibility.
In the majority of the discussion over the code (its evident poor quality) and version control (there appears to be none) one should not forget that the data appears to be in a similar state.
I would hazard a guess that CRU do not have any precise record of the code and data version used to obtain any of their published output. Its all indicative of an exceptionally sloppy working methodology. Was their scientific methodology equally sloppy?
“Complaining that the source code is poor quality is a dead end – please stop doing that!”
Agreed but we also need to be alert to the fact that some of the coding errors already identified may well have not insignificant effects on the computed results.

Anna V: Does not
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
amount to much the same caveat? Certainly if I wanted to actually fudge something without anyone knowing, a 15-star comment wouldn’t be my first thought.
————
No, but if you might if your boss instructed you to write code that fudged data and you were unhappy about it.

On the VERY ARTIFICAL CORRECTION topic:
Back2Bat (13:56:27) : I see a different possibility: An honest programmer forced to do dirty work but who protests about it in the comments.
D (16:36:18) : No, but if you might if your boss instructed you to write code that fudged data and you were unhappy about it.
Been there, done that.
Sometimes the only fix available is a horrible one, and most programmers don’t want sloppy work to reflect on them. So you do it but leave a big enough trail so those that follow don’t think you’re a moron. Which is what I think “Harry” did.
The question here is whether it was a fix or something more sinister.
On to the topic of the poor quality of the source code:
Invariant (02:02:29) :
Complaining that the source code is poor quality is a dead end – please stop doing that
I’m not sure I agree. I think the quality of the code does have some bearing. With the number of files involved, basically doing their own DBMS instead of using a commercial one, leaves lots of room for errors and unintended consequences. But yes, it would be good to run it and see what it does. Where’s that data?
Back in my mainframe days in the 80s you couldn’t put a report program into production without a design review and operations review. I guess no such standards exist in the scientific world.

“Past divergence could have been up, or down. Correct?”
Correct.
Although the only divergence that we have documentation of is down …
If the effect that is causing the current downward divergence is also operating anywhere in the past data, then previous warm periods would be clipped as well. Such a divergence could be caused by some real effect (such as a temp limit in the trees response to temp), or it could be an effect of the Team’s analysis methods. Given that their obvious intent is to clip pre-modern warm peaks, the divergence that requires ‘hiding the decline’ could be an artifact of ‘hiding the incline’ of the past.

“This just in:
EPA Set to Declare Carbon Dioxide a Public Danger
http://www.foxnews.com/politics/2009/12/06/epa-set-delare-carbon-dioxide-public-danger/
Only benefit of Cap-and-trade is that someone makes money on the deal.”
This is no surprise. I do believe I indicated months ago the public comments would have no effect. The policy was already decided.

Quote from Fox News Sunday panel discussion:
“With regard to this Email issue … If you come back to the reality, what we find here is scientists trying to change the language, at times, to make it less controversial in terms of the political discussion, but not to change the facts on the ground; the facts on the ground being that, you know, we continue to emit too many carbon dioxides, and CO2, and other gasses and as a result we are capturing the heat from the sun and global warming is a reality….” Juan Williams, 6 Dec 2009.
I believe this is a good example of the honest opinion that most of the current ‘elite’ media have of the hypothetical Carbon Dioxide Crisis and Climategate in particular.
[REPLY – Maybe honest. But likely in error. (I like Juan.) It seems CRU may well have changed the “facts on the ground”, namely 1the 1930s. And whether or not we emit too much CO2, which may have very little effect on temperature, has yet to be determined. ~ Evan]

‘Hide the decline’ with precorrected data files?
if you grep ‘artifi’ the document- and subfolders, you get 32 files.
Hardcoded (factors in source) correction for ‘decline’ is only found in briffa_sep98_d.pro (dated 03/99) and briffa_sep98_e.pro (dated 09/98).
It seems, later on CRU used another technique for correction values, these are precalculated in the data files itself.
In more recent sources like data4alps.pro (dated 08/2008) you find:
;
; Writes an ASCII file with data (gridded, yes/no extended, corrected,
; yes/no ABD-adjusted, calibrated) for input to the Arctic synthesis update.
;
doinfill=0 ; use PCR-infilled data or not?
doabd=1 ; use ABD-adjusted data or not?
docorr=1 ; use corrected version or not? (uncorrected only available
; for doinfill=doabd=0)
missval=-9.99
;
; Get the calibrated data
;
win
;
print,’Reading reconstructions’
if doabd eq 0 then begin
if doinfill eq 0 then begin
restore,’calibmxd5.idlsave’
; Gets: g,mxdyear,mxdnyr,fdcalibu,fdcalibc,mxdfd2,timey,fdseas
if docorr eq 0 then fdcalibc=fdcalibu <—- look here
endif else begin
restore,'calibmxd5_pcr.idlsave'
; Gets: g,mxdyear,mxdnyr,fdcalibc,timey,fdseas
endelse
endif else begin
if doinfill eq 0 then begin
restore,'../mann/calibmxd5_abdlow.idlsave'
; Gets: g,mxdyear,mxdnyr,fdcalibu
print,'PROBABLY WANT THIS ONE'
if docorr eq 0 then fdcalibc=fdcalibu <—– look here
endif else begin
restore,'../mann/calibmxd5_abdlow_pcr.idlsave'
; Gets: g,mxdyear,mxdnyr,fdcalibu
endelse
endelse
The statement
if docorr eq 0 then fdcalibc=fdcalibu
seems to overwrite precorrected values (fdcalibc) with uncorrected ones (fdcalibu) only when docorr = 0
Further processing here and in many other sourcefiles uses only fdcalibc.
if you grep 'calibmxd5.idlsave' (one datafile), you get 34 matches. In only one file this data file is written, look at pl_calibmxd4.pro (dated 09/99):
;
; Now compute the calibrated values now that all boxes have coefficients
;
for iyr = 0 , mxdnyr-1 do begin
fdcalibu(*,*,iyr)=fdalph(*,*)+fdbeta(*,*)*fdcalib(*,*,iyr)
fdcalibc(*,*,iyr)=fdalph(*,*)+fdbeta(*,*)*fdcorrect(*,*,iyr)
endfor
;
; Now save the data for later analysis
;
save,filename='calibmxd5.idlsave',$
g,mxdyear,mxdnyr,fdcalibu,fdcalibc,mxdfd2,timey,fdseas
;
end
Variable fdcorrect is computed in only two sourcefiles with same name pl_decline.pro, the more recent one is dated 03/2004
I would like to examine these sources files precise, only my IDL-knowledge is restricted, it's a travesty…

Phil Jones’ famous E-Mail (‘Mike’s nature trick’, ‘to hide the decline’ is time-stamped “Date: Tue, 16 Nov 1999 13:31:15 +0000”
Source pl_decline.pro from FOIA\documents\osborn-tree6\mann\oldprog is dated 05-01-2000 dd-mm-yyyy
There might be a relation between the E-Mail and the source file.

J. Bob (08:31:36) : I find it unimaginable a supposedly 1st rate research facility would tolerate this code. […] The only time we would ever keep commented code in the source stream, is that it would be readily available to re-insert for whatever reason at a later date. Otherwise it was deleted. PERIOD.
Many scientist and engineers have no interest in the extra effort to generate readable code; they are more interested in the results of the calculations than keeping the source code elegant and beautiful. As long as they understand the code themselves they are happy, they simply ignore the readability for the next guy. This behavior is nasty but true…
But that’s not the major point. The major point is that poor source code is completely irrelevant here, imagine New York Times headlines “Copenhagen cancelled due to unreadable source code”… Well, that’s not so likely is it? 🙂

Richard & others,
I think the “thought experiment” was something along the lines of “what if the tree data didn’t suffer from the decline – how would that affect the correlation of the whole dataset to actual temperature?”. That doesn’t mean in this case that they ignored the decline, only that they wanted to know what the its effect on their process was.
The “bootstrapping” process is described in that Osborn paper quotation I posted above, but here’s how I understand what they did (as through a glass, darkly):
1) Recognise (as is well known) that the post 1960 tree data doesn’t track real temperatures, and that if you include them in your analysis it’s going to screw things up. Assuming you just don’t want to give up and go home at this point, you have to do something about it.
2) Then there are two ways of working out a correlation and a calibration between the tree ring widths/densities and real temperatures even though you know the post-1960 results are dodgy:
a) Chop them off and only use the ones before 1960 (there is code that does this elsewhere)
b) Fudge the post-1960 temperatures so they track real temperatures better.
I’ll discuss the benefits and dangers of each of those later…
3) This gives you a calibrated mapping – that is to say, it allows you to work out a temperature for a given ring width or density.
4) Then you apply this mapping to the original *unmodified* data to give you your temperature reconstruction. You know the output of this is going to diverge post 1960, and needs to be replaced with real temperatures (or “hide the decline”, if you must), but at least it gives you something. Obviously you would document the process you did, as Osborn et al. did in their paper – which apparently was never published, so maybe the reviewers thought it was all a bit too dodgy – but it was all out in the open.
OK, so lets come back to the 2(a) and (b) – the choice of whether to chop (“bobbit”, as we say in programming 😉 or fudge.
2(a), bobbiting, gives you a more accurate view of the correlation/mapping for the period you leave in, but because the period is shorter it will be more sensitive to that particular period – as Osborn”s comment indicates.
2(b), fudging, is really pretty horrible, granted, but it does leave the *short-term* post-1960 variations in place to correlate with, even though you’ve messed with the long-term trend – so it gives you a little less sensitivity to the time period.
So when I say it’s “reasonable”, what I mean is, it’s a reasonable way to make the best of some data which has known, documented problems in recent years. Extracting information from imperfect data is what science (and signal processing) is all about. As long as it was done in the open, it’s fine. Now granted the code wasn’t published, and I agree it should be, but Osborn did summarise what it did in the paper.
(BTW, I don’t know if the bit about programming video software in C (actually it’s C++) was meant as a jibe about my qualification to comment – if so, fair enough, if that’s all there was. But I have also spent the last two years building and maintaining woodfortrees.org, which gives me some insight into some of the concepts, here, I think. But I don’t claim to be a Real Statistician!)

… and (as Gary points out in another thread), I should have said “dendroclimatology”.

This was not convincing to me as I don´t understand it. Could you please expain what you have found in more simple terms, understandable to somebody who doesn´t understand a shit about programming? And please use simple words. I´m a stupid Swedish journalist, you know.

Tenuc (00:11:04) :
To carry your analogy a bit further; we’ve got emails where they discuss disposing of the body (raw data), and now the body is missing.
We’ve not only got method, motive and opportunity, we’ve got a missing body and emails where they openly discuss their intent to dispose of it.
I don’t know how the science community works, but I know several prosecutor’s who would love to have this much to go on in the court room. If this were a criminal case (at least in the U.S.) we’d be talking about pleading guilty and turning state’s evidence in exchange for a life sentence. Somehow in the scientific community, a bunch of the co-conspirators bear character witness, with the understanding that that will be sufficient to render a verdict of not-guilty.
Finally, after having read the comments here, it seems this code is potentially a bit more innocuous than what I first interpreted, per se. However, in light of the missing raw data, emails that display a clear sense of contempt for the scientific method and actual criminal intent with respect to FOI, this code should be just another stick in the funeral pyre. Let us hope.

JJ: I think if you find the tree data does track temperatures pretty well for 110 years, but doesn’t for the last 40, you could argue it’s not Game Over and try to do the best you can. But I agree, it certainly demands a rain check , and I haven’t yet seen anything that really satisfies me to explain it, either (but I’ve only been looking at this particular issue for a couple of weeks).
My real point is just this: this whole issue with divergence was already well known to those in the field, and all that’s been “discovered” here is the code that implements techniques (dubious or not) that were already described by the authors in various papers. It’s interesting to discuss the finer points of dendroclimatology as to whether what they did was valid; but to claim this is a SMOKING GUN THAT RENDERS THE WHOLE OF AGW A FARCE (paraphrasing) is just overkill.

woodfortrees (Paul Clark) (08:55:48) :
JJ: I think if you find the tree data does track temperatures pretty well for 110 years, but doesn’t for the last 40, you could argue it’s not Game Over and try to do the best you can.
—
I am not a scientist but this seems like nonsense to me because I do not see how data for 110 years can tell us anything definitive about conditions on a planet that has been around for an estimated 4.5 billion years. Am I wrong? Do we have any data that goes back 4.5 billion years?

Just a note about the adjustment factor array. It sure looks similar to the correction factors “tobs” that is in use by our US temperature data massagers.
Has anyone checked this out?

“To carry your analogy a bit further; we’ve got emails where they discuss disposing of the body (raw data), and now the body is missing.”
I see many claims to that effect…but where’s the PROOF?!
It’s completely dishonest to make a subjective interpretation of an email that is taken out of context and then brandy that about as the truth.
I see many claiming “We have emails showing they are hiding data!”
No, you don’t… You have emails that SUGGEST they might…but the fact is they could also suggest many other things.
I’m not sure exactly what data you speak of, but much of the data that was supposedly being “suppressed” is in fact available from the sources it originally came from!
CRU collected data from various sources…so they might very well have been justified in saying it was not their place to give it out …..or there are many many legit reasons for them to “suppress” it.
My biggest problem in this whole scandal is so called “skeptics” taking bits and pieces out of context and bellowing “THIS IS ABSOLUTE PROOF OF XXXX” ….based on a SUBJECTIVE INTERPRETATION.
We need a thorough investigation and ALL The facts to come out before we can make any absolute claims.

Another thing:
So many are loudly proclaiming the fact that the scientists have motivation to fake the data…
Where’s the proof of this?
Where’s the proof that they are making more money than they would if they didn’t support the “conspiracy”?
Also I’m surprised that some still don’t get it:
The point is NOT “Was the data processed (or output or whatever) by this code published or used”
the point is “Was this output data used WITHOUT people being told that it was processed by this function”

WFT,
“They were only trying to go back a 1000 years or so, so 110 years of data is a fair proportion of the total.”
Which remains wholey irrelevant, given that 40-60 years dont track temps at all. If 25-30% of the instrumented period bears zero or negative relationship to the tree rings, then the premise of a positive tracking relationship is falsified. Tree rings as a proxie are invalidated unless and until that is sorted out appropriately.
TKI,
“I agree on your statement. But one argument was: “the corrected values are not used”.”
That was not my argument. My arguement is that we do not know what, if anything, they were used for, let alone if they were ever used for anything published.
“A little survey of the sources shows that corrected values are used all over the place, not only in briffa_sep98_e.pro.”
Which tells us precisely nothing WRT the above.
“Usually there is a hint not to plot beyond 1960, but for what reason “adjust” this values, if you do not want to use them in any context?”
That is a question. Before making claims, one needs answers.
JJ

Nice analysis! This analysis should be used along with this analysis to produce a complete report:
http://69.84.25.250/blogger/post/ClimateGate-Data-Series-Part-5-A-break-down-of-large-data-file-for-manipulating-global-temperature-trends-from-2006-2009.aspx