Climategate Code Analysis Part 2
There are three common issues that have been raised in my previous post that I would like to officially address concerning the CRU’s source code.
If you only get one thing from this post, please get this. I am only making a statement about the research methods of the CRU and trying to show proof that they had the means and intent to falsify data. And, until the CRU’s research results can be verified by a 3rd party, they cannot be trusted.
Here are the four most frequent concerns dealing with the CRU’s source code:
- The source code that actually printed the graph was commented out and, therefore, is not valid proof.
- No proof exists that shows this code was used in publishing results.
- Interpolation is a normal part of dealing with large data sets, this is no different.
- You need the raw climate data to prove that foul play occurred.
If anyone can think of something I missed, please let me know.
The source code that actually printed the graph was commented out and, therefore, is not valid proof.
Had I done a better job with my source analysis, I would have found a later revision of the briffa_sep98_d.pro source file (linked to in my previous post) contained in a different working tree which shows the fudge-factor array playing a direct result in the (uncommented) plotting of the data.
Snippit from: harris-tree/briffa_sep98_e.pro (see the end of the post for the full source listing)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
filter_cru,20,tsin=densall,tslow=tslow,/nan
cpl_barts,x,densall,title='Age-banded MXD from all sites',$
xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
zeroline=tslow,yrange=[-7,3]
oplot,x,tslow,thick=3
oplot,!x.crange,[0.,0.],linestyle=1
;
Now, we can finally put this concern to rest.
Interpolation is a normal part of dealing with large data sets, this is no different.
This is partially true, the issue doesn’t lie in the fact that the CRU researchers used interpolation. The issue is the weight of the valadj array with respect to the raw data. valadj simply introduces too large of an influence to the original data to do anything productive with it.
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
No proof exists that shows this code was used in publishing results.
Correct! That’s why I am (and always have) taken the following stand: Enough proof exists that the CRU had both the means and intent to intentionally falsify data. This means that all of their research results cannot be trusted until they are verified. Period.
The fact that the “fudge-factor” source code exists in the first place is reason enough for alarm. Hopefully, they didn’t use fudged results in the CRU research results, but the truth is, we just don’t know.
You need the raw climate data to prove that foul play occurred.
This is assuming the raw data are valid, which I maintain that it probably is. Several people question the validity of the climate data gathering methods used by the different climate research institutions, but I am not enough of a climate expert to have an opinion one way or the other. Furthermore, It simply doesn’t matter if the raw climate data are correct or not to demonstrate the extreme bias the valadj array forces on the raw data.
So, the raw data could actually be temperature data or corporate sales figures, the result is the same; a severe manipulation of data.
Full Source Listing
As promised, here is the entire source listing for: harris-tree/briffa_sep98_e.pro
[sourcecode language=”text”]
1. ;
2. ; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
3. ; standardised datasets.
4. ; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
5. ; with missing values set appropriately. Uses mxd, and just the
6. ; "all band" timeseries
7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
8. ;
9. yrloc=[1400,findgen(19)*5.+1904]
10. valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
11. 2.6,2.6,2.6]*0.75 ; fudge factor
12. if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
13. ;
14. loadct,39
15. def_1color,20,color=’red’
16. plot,[0,1]
17. multi_plot,nrow=4,layout=’large’
18. if !d.name eq ‘X’ then begin
19. window, ysize=800
20. !p.font=-1
21. endif else begin
22. !p.font=0
23. device,/helvetica,/bold,font_size=18
24. endelse
25. ;
26. ; Get regional tree lists and rbar
27. ;
28. restore,filename=’reglists.idlsave’
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
31. ;
32. rawdat=fltarr(4,2000)
33. for i = nreg-1 , nreg-1 do begin
34. fn=’mxd.’+harryfn(i)+’.pa.mean.dat’
35. print,fn
36. openr,1,fn
37. readf,1,rawdat
38. close,1
39. ;
40. densadj=reform(rawdat(2:3,*))
41. ml=where(densadj eq -99.999,nmiss)
42. densadj(ml)=!values.f_nan
43. ;
44. x=reform(rawdat(0,*))
45. kl=where((x ge 1400) and (x le 1992))
46. x=x(kl)
47. densall=densadj(1,kl) ; all bands
48. densadj=densadj(0,kl) ; 2-6 bands
49. ;
50. ; Now normalise w.r.t. 1881-1960
51. ;
52. mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
53. mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
54. ;
55. ; APPLY ARTIFICIAL CORRECTION
56. ;
57. yearlyadj=interpol(valadj,yrloc,x)
58. densall=densall+yearlyadj
59. ;
60. ; Now plot them
61. ;
62. filter_cru,20,tsin=densall,tslow=tslow,/nan
63. cpl_barts,x,densall,title=’Age-banded MXD from all sites’,$
64. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
65. zeroline=tslow,yrange=[-7,3]
66. oplot,x,tslow,thick=3
67. oplot,!x.crange,[0.,0.],linestyle=1
68. ;
69. endfor
70. ;
71. ; Restore the Hugershoff NHD1 (see Nature paper 2)
72. ;
73. xband=x
74. restore,filename=’../tree5/densadj_MEAN.idlsave’
75. ; gets: x,densadj,n,neff
76. ;
77. ; Extract the post 1600 part
78. ;
79. kl=where(x ge 1400)
80. x=x(kl)
81. densadj=densadj(kl)
82. ;
83. ; APPLY ARTIFICIAL CORRECTION
84. ;
85. yearlyadj=interpol(valadj,yrloc,x)
86. densadj=densadj+yearlyadj
87. ;
88. ; Now plot it too
89. ;
90. filter_cru,20,tsin=densadj,tslow=tshug,/nan
91. cpl_barts,x,densadj,title=’Hugershoff-standardised MXD from all sites’,$
92. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
93. zeroline=tshug,yrange=[-7,3],bar_color=20
94. oplot,x,tshug,thick=3,color=20
95. oplot,!x.crange,[0.,0.],linestyle=1
96. ;
97. ; Now overplot their bidecadal components
98. ;
99. plot,xband,tslow,$
100. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
101. yrange=[-6,2],thick=3,title=’Low-pass (20-yr) filtered comparison’
102. oplot,x,tshug,thick=3,color=20
103. oplot,!x.crange,[0.,0.],linestyle=1
104. ;
105. ; Now overplot their 50-yr components
106. ;
107. filter_cru,50,tsin=densadj,tslow=tshug,/nan
108. filter_cru,50,tsin=densall,tslow=tslow,/nan
109. plot,xband,tslow,$
110. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
111. yrange=[-6,2],thick=3,title=’Low-pass (50-yr) filtered comparison’
112. oplot,x,tshug,thick=3,color=20
113. oplot,!x.crange,[0.,0.],linestyle=1
114. ;
115. ; Now compute the full, high and low pass correlations between the two
116. ; series
117. ;
118. perst=1400.
119. peren=1992.
120. ;
121. openw,1,’corr_age2hug.out’
122. thalf=[10.,30.,50.,100.]
123. ntry=n_elements(thalf)
124. printf,1,’Correlations between timeseries’
125. printf,1,’Age-banded vs. Hugershoff-standardised’
126. printf,1,’ Region Full <10 >10 >30 >50 >100′
127. ;
128. kla=where((xband ge perst) and (xband le peren))
129. klh=where((x ge perst) and (x le peren))
130. ts1=densadj(klh)
131. ts2=densall(kla)
132. ;
133. r1=correlate(ts1,ts2)
134. rall=fltarr(ntry)
135. for i = 0 , ntry-1 do begin
136. filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
137. filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
138. if i eq 0 then r2=correlate(tshi1,tshi2)
139. rall(i)=correlate(tslow1,tslow2)
140. endfor
141. ;
142. printf,1,’ALL SITES’,r1,r2,rall,$
143. format='(A11,2X,6F6.2)’
144. ;
145. printf,1,’ ‘
146. printf,1,’Correlations carried out over the period ‘,perst,peren
147. ;
148. close,1
149. ;
150. end
[/sourcecode]
geo (00:08:53) :
“The smartest kid in the class (CRU) just got caught cheating. Why are his frat brothers (GISS, NOAA) claiming that everything is ok because they all put down the same answers on the test?”
LOL!
Yahoo may be a better choice for climategate news me thinks
The term Weapons of Maths Destruction is excellent! Hats off to its creator.
There are many parallels between the “Dodgy Dossier” of Iraqi WMDs and the the reports of the IPCC.
At the time of the “Sexed Up” campaign that led to the last Gulf war, I like many others was convinced of the validity of the claims made.
Hindsight brought different perceptions!
We were cynically misled then and the evidence is mounting that we are being manipulated again.
To be fooled once is unfortunate. To be fooled twice is just plain stupid!
Let’s pray that this episode does not through up another Dr Kelly.
Carsten Arnholm, Norway (04:18:08) : I was commenting on your assertion that “smart” people wrote poor code, which indeed is wrong and a distraction.
OK. Then we agree. Indeed not all smart developers write spaghetti code, but my personal experience is that it is sometimes difficult to understand the source code written by the “smart” people that “manage to juggle 40 balls (functions, variables)” without doing mistakes. Another point is, of course, that if you intend to do something illegal, then you would possibly try to hide it in a mess…
YOU’ALL GOT TO READ MARC SHEPPARDS PIECE I LINKED ABOVE!
And, slightly OT, here’s “Scientists behaving badly, Part 3”:
The editorial in the current issue of the journal, “Nature”, talks about ‘denialists’.
When one of the premier scientific journals starts using language like that, it’s clear that the age of reason is well and truly over.
http://www.nature.com/nature/journal/v462/n7273/full/462545a.html
@ur momisugly Ripper (23:51:23) ,
the link to quote mining code was interesting.
But as a defence it has some holes.
“This doesn’t seem to be a smoking gun so much as a gun that hasn’t been fired.”
Why write code that isn’t intended to be used? (or not destroy it or very clearly mark it not to be used).
There is obviously a lot of very poor file management, file naming systems leave a lot to be desired and so on, so it is quite feasible to suggest a piece of code written for no good reason might not be clearly marked and I get the impression that some of the original code wasn’t exactly well written any way.
But why have the team that wrote the code or had it written not been asked specifically what this piece of code was all about? Why have they volunteered no information?
OK, maybe we have some sub judice issues.
In the end we can have all sorts of possible excuses for why such code has been written but unless the raw data is available to test we can’t sort out whether it has been used or not.
It is a valid caution that it may or may not have been used maliciously. It is possible to postulate that it cold have been used to illustrate something or other.
So, how can we know wether it has been used or not?
Simple, run the raw data through it and see if it matches published work. (which shouldn’t be necessary, the science ought to have required the papers were presented with the raw data and the code….. ) except we are told we don’t have all the raw data (though some suspect that this might not be the whole truth either).
We could take published data and run it backwards through this code to see what sort of “raw” data sets we come up with but we always have that problem. We don’t know that what we get is raw data because we don’t have the raw data.
About the only chance I see is if we have work available we can run backwards through the code available for that data to generate some re-constituted “raw” data.
What we are looking for is a sequence of reconstituted data that overlaps with later data that was so “fixed” i.e. some data from the 1960’s forward that was part of published data that did not have the decline fixed in it that is also used in later work where the fix was in. Or not, as the case may be.
It seems to me that had this code been applied only to new data we might be in a problem but if the fix has a retroactive element to it, as suggested, then this might indeed be a possible.
See Al Gore’s response to Climategate here
[snip – we don’t run the Hitler parody video here – think “deniers” – A]
Carlo (02:49:45) :
> osborn-tree6\mann\mxdgrid2ascii.pro 10/03/2008
What was you point in posting this? Just that it has the word ARTIFICIALLY in it?
All the code does is print out a couple arrays to different destinations in different formats.
woodfortrees (Paul Clark) (04:34:04) :
Anna V: Does not
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
amount to much the same caveat? Certainly if I wanted to actually fudge something without anyone knowing, a 15-star comment wouldn’t be my first thought.
Not so, in my opinion of course. I do not think these people were really aware that what they were doing was unscientific. They had the market cornered and they were sure they were right.
Anthony does not like religious analogies, but, in my opinion, the sociology applies to these “scientists”. Were the people counting the number of angels that could fit on a pinhead aware of the preposterousness of their case? When you start by believing that angels exist and can be counted, then anything goes. When you start believing that there is unprecedented twentieth century warming then it is obvious that the data is wrong and has to be corrected for. Simple.
I do not even know what code was used for what purpose, To me the damnation lies in Harry’s RE#ADME text file and the quality of the code, which is not up to standard.
Robert notes:
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
He’s assuming that the adjustments are in units of °C, but the code makes no references to temperature at all. It does make several references to tree rings and density, but I have no idea what sort of units or range of values those include.
I fear that Robert is too interested in proving his suspicions and has forgotten that good scientists are as skeptical of their own work as they are of others.
It would be nice to see some actual plots produced by this code.
It would also be nice to have time to read up on the “divergence problem” in order to understand reasons why it appears to be only a recent phenomenon, or if similar issues are lost in past noise.
-Ric
— New pages! See http://home.comcast.net/~ewerme/wuwt/toc-2009-11.html —
What we are looking for is a sequence of reconstituted data that overlaps with later data that was so “fixed” i.e. some data from the 1960’s forward that was part of published data that did not have the decline fixed in it that is also used in later work where the fix was in.
“The fix was in” — in underworld parlance — describes it pretty succinctly…
P Gosselin (04:24:36) :
Marc Sheppard explains THE TRICK
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html
This is where the focus has to be, and thr FOIA violations.
Totally agree. This is an excellent summary of exactly who Gordon Browns and Ed Millipeds real flat-earthers and climate sabouteurs are. No other than themselves!
IMHO, this article would be a perfect thread here.
Paul Clark,
I think you raise a reasonable point. Or, it had occurred to me, that an easier answer from CRU is that, “…this was a test version, not the version of the program that we actually used…”
But, then the next logical step is to release the actual version of the software AND the raw data and prove they work with no fudging.
Without the release, we are free to think the worst.
Another interesting quote from the UK Telegraph this morning:
“The other is that the ugly, drum-like concrete building at the University of East Anglia which houses the CRU is named after its founder, the late Hubert Lamb, the doyen of historical climate experts. It was Professor Lamb whose most famous contribution to climatology was his documenting and naming of what he called the Medieval Warm Epoch, that glaring contradiction of modern global warming theory which his successors have devoted untold efforts to demolishing. If only they had looked at the evidence of those Siberian trees in the spirit of true science, they might have told us that all their efforts to show otherwise were in vain, and that their very much more distinguished predecessor was right after all.”
A bit ironic you might say.
As we all know CRU will be very open from now on. So i suggest that the first thing they tell us is
WHO IS HARRY ?
P Gosselin (04:42:20) :
> [Shouting toned down] You’all got to read Marc Sheppard’s piece I linked above!
Normally, I hate posts like this, however, given the traffic of late, I confess I skipped over your first comment and this convinced me to check it out. I’m impressed. I’m impressed so much that I’m compelled to say
ME TOO!
Ordinarily I hate “Me too” posts, apologies to those more pure than I am. 🙂
Seriously, this is so much better than Robert’s analysis, (though it spends less time looking at code) that future discussion should be driven by Sheppard’s claims.
One thing that programmers (who have time) could do is look for places in the code where the proxy data is switched to instrumental data, that’s at 1980 for Mann’s proxy, and 1960 for Briffa’s.
The link bears repeating.
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html
That references (through a Google search!?)
http://www.americanthinker.com/2009/11/crus_source_code_climategate_r.html
I am probably the last to notice but have you seen the article on BBC web site
“The arguments made by climate change sceptics”
– even has a link to WUWT at the bottom, never thought I would see that.
http://news.bbc.co.uk/1/hi/sci/tech/8376286.stm
[snip – thanks but we have this video prominently on the main page – A]
I’m lost…
The author said:
“The source code that actually printed the graph was commented out and, therefore, is not valid proof.”
and
“Now, we can finally put this concern to rest.”
So what’s the conclusion? The code is valid or not?
anna v (05:37:25) : When you start believing that there is unprecedented twentieth century warming then it is obvious that the data is wrong and has to be corrected for. Simple.
Excellent! That’s the correct focus. The BBC coverage “CRU’s programming ‘way below expected standards’” is clearly a red herring.
http://news.bbc.co.uk/2/hi/programmes/newsnight/8395514.stm
No, when the very election of a (liberal/socialist/”green-focused” environmental presidential) administration are going to be based on whether or not you can claim that there are “very dangerous” angels dancing on that pin, and how many trillion dollars in taxes can be charged based on how many angels are dancing on that pin, THEN you need to do anything you can to claim as many angels as possible.
Because, to these ecotheists, the “end” justifies ANY means.
Peter (04:53:56) :
And, slightly OT, here’s “Scientists behaving badly, Part 3″:
The editorial in the current issue of the journal, “Nature”, talks about ‘denialists’.
When one of the premier scientific journals starts using language like that, it’s clear that the age of reason is well and truly over.
http://www.nature.com/nature/journal/v462/n7273/full/462545a.html
No, Peter, I don’t think so. Only the age of this co-conspirator journal — and Science Mag is soon to follow. End your subscriptions and subscribe to the valid science “publications” on the internet like WUWT, CA TAV, Chiefio, and others. Give them the same subscription fee that you would have — before they went with THE FIX –given to every print mainline science publication. This would be one huge shift in economic power.
Add to that decision, another one to either take back or end your membership in every professional scientific society that approved THE FIX under the guise (masquerade) of AGW, global warming, climate change.
THE FIX I refer to is that Obama (probably) is going to Copenhagen to revel in his “citizen of the world, not the USA” self-aggrandizing title as the agreement re our CO2-pollution fees (in whatever form the EPA decides) are sent to an unelected UN bureaucracy (unaccountable to any sovereign nation) that is ready to rule the world. For what purposes? This is as important a “scientific” topic as is the rotten code and the traitorous pseudo-scientists. May they all rot in jail before their time in hell. They have been agents of this treason.
I was perusing the Bishop Hill blog when I came across mention of another blog where the blogger was performing an analysis of the code. See:
http://www.jgc.org/blog/
There are several posts on the subject of the CRU code that are most fascinating, including references to several buggy bits of code. Easy to read for someone not into IDL. Goes beyond the usual references to the Harry_Read_Me.txt file.