Climategate Code Analysis Part 2
There are three common issues that have been raised in my previous post that I would like to officially address concerning the CRU’s source code.
If you only get one thing from this post, please get this. I am only making a statement about the research methods of the CRU and trying to show proof that they had the means and intent to falsify data. And, until the CRU’s research results can be verified by a 3rd party, they cannot be trusted.
Here are the four most frequent concerns dealing with the CRU’s source code:
- The source code that actually printed the graph was commented out and, therefore, is not valid proof.
- No proof exists that shows this code was used in publishing results.
- Interpolation is a normal part of dealing with large data sets, this is no different.
- You need the raw climate data to prove that foul play occurred.
If anyone can think of something I missed, please let me know.
The source code that actually printed the graph was commented out and, therefore, is not valid proof.
Had I done a better job with my source analysis, I would have found a later revision of the briffa_sep98_d.pro source file (linked to in my previous post) contained in a different working tree which shows the fudge-factor array playing a direct result in the (uncommented) plotting of the data.
Snippit from: harris-tree/briffa_sep98_e.pro (see the end of the post for the full source listing)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
filter_cru,20,tsin=densall,tslow=tslow,/nan
cpl_barts,x,densall,title='Age-banded MXD from all sites',$
xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
zeroline=tslow,yrange=[-7,3]
oplot,x,tslow,thick=3
oplot,!x.crange,[0.,0.],linestyle=1
;
Now, we can finally put this concern to rest.
Interpolation is a normal part of dealing with large data sets, this is no different.
This is partially true, the issue doesn’t lie in the fact that the CRU researchers used interpolation. The issue is the weight of the valadj array with respect to the raw data. valadj simply introduces too large of an influence to the original data to do anything productive with it.
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
No proof exists that shows this code was used in publishing results.
Correct! That’s why I am (and always have) taken the following stand: Enough proof exists that the CRU had both the means and intent to intentionally falsify data. This means that all of their research results cannot be trusted until they are verified. Period.
The fact that the “fudge-factor” source code exists in the first place is reason enough for alarm. Hopefully, they didn’t use fudged results in the CRU research results, but the truth is, we just don’t know.
You need the raw climate data to prove that foul play occurred.
This is assuming the raw data are valid, which I maintain that it probably is. Several people question the validity of the climate data gathering methods used by the different climate research institutions, but I am not enough of a climate expert to have an opinion one way or the other. Furthermore, It simply doesn’t matter if the raw climate data are correct or not to demonstrate the extreme bias the valadj array forces on the raw data.
So, the raw data could actually be temperature data or corporate sales figures, the result is the same; a severe manipulation of data.
Full Source Listing
As promised, here is the entire source listing for: harris-tree/briffa_sep98_e.pro
[sourcecode language=”text”]
1. ;
2. ; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
3. ; standardised datasets.
4. ; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
5. ; with missing values set appropriately. Uses mxd, and just the
6. ; "all band" timeseries
7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
8. ;
9. yrloc=[1400,findgen(19)*5.+1904]
10. valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
11. 2.6,2.6,2.6]*0.75 ; fudge factor
12. if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
13. ;
14. loadct,39
15. def_1color,20,color=’red’
16. plot,[0,1]
17. multi_plot,nrow=4,layout=’large’
18. if !d.name eq ‘X’ then begin
19. window, ysize=800
20. !p.font=-1
21. endif else begin
22. !p.font=0
23. device,/helvetica,/bold,font_size=18
24. endelse
25. ;
26. ; Get regional tree lists and rbar
27. ;
28. restore,filename=’reglists.idlsave’
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
31. ;
32. rawdat=fltarr(4,2000)
33. for i = nreg-1 , nreg-1 do begin
34. fn=’mxd.’+harryfn(i)+’.pa.mean.dat’
35. print,fn
36. openr,1,fn
37. readf,1,rawdat
38. close,1
39. ;
40. densadj=reform(rawdat(2:3,*))
41. ml=where(densadj eq -99.999,nmiss)
42. densadj(ml)=!values.f_nan
43. ;
44. x=reform(rawdat(0,*))
45. kl=where((x ge 1400) and (x le 1992))
46. x=x(kl)
47. densall=densadj(1,kl) ; all bands
48. densadj=densadj(0,kl) ; 2-6 bands
49. ;
50. ; Now normalise w.r.t. 1881-1960
51. ;
52. mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
53. mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
54. ;
55. ; APPLY ARTIFICIAL CORRECTION
56. ;
57. yearlyadj=interpol(valadj,yrloc,x)
58. densall=densall+yearlyadj
59. ;
60. ; Now plot them
61. ;
62. filter_cru,20,tsin=densall,tslow=tslow,/nan
63. cpl_barts,x,densall,title=’Age-banded MXD from all sites’,$
64. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
65. zeroline=tslow,yrange=[-7,3]
66. oplot,x,tslow,thick=3
67. oplot,!x.crange,[0.,0.],linestyle=1
68. ;
69. endfor
70. ;
71. ; Restore the Hugershoff NHD1 (see Nature paper 2)
72. ;
73. xband=x
74. restore,filename=’../tree5/densadj_MEAN.idlsave’
75. ; gets: x,densadj,n,neff
76. ;
77. ; Extract the post 1600 part
78. ;
79. kl=where(x ge 1400)
80. x=x(kl)
81. densadj=densadj(kl)
82. ;
83. ; APPLY ARTIFICIAL CORRECTION
84. ;
85. yearlyadj=interpol(valadj,yrloc,x)
86. densadj=densadj+yearlyadj
87. ;
88. ; Now plot it too
89. ;
90. filter_cru,20,tsin=densadj,tslow=tshug,/nan
91. cpl_barts,x,densadj,title=’Hugershoff-standardised MXD from all sites’,$
92. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
93. zeroline=tshug,yrange=[-7,3],bar_color=20
94. oplot,x,tshug,thick=3,color=20
95. oplot,!x.crange,[0.,0.],linestyle=1
96. ;
97. ; Now overplot their bidecadal components
98. ;
99. plot,xband,tslow,$
100. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
101. yrange=[-6,2],thick=3,title=’Low-pass (20-yr) filtered comparison’
102. oplot,x,tshug,thick=3,color=20
103. oplot,!x.crange,[0.,0.],linestyle=1
104. ;
105. ; Now overplot their 50-yr components
106. ;
107. filter_cru,50,tsin=densadj,tslow=tshug,/nan
108. filter_cru,50,tsin=densall,tslow=tslow,/nan
109. plot,xband,tslow,$
110. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
111. yrange=[-6,2],thick=3,title=’Low-pass (50-yr) filtered comparison’
112. oplot,x,tshug,thick=3,color=20
113. oplot,!x.crange,[0.,0.],linestyle=1
114. ;
115. ; Now compute the full, high and low pass correlations between the two
116. ; series
117. ;
118. perst=1400.
119. peren=1992.
120. ;
121. openw,1,’corr_age2hug.out’
122. thalf=[10.,30.,50.,100.]
123. ntry=n_elements(thalf)
124. printf,1,’Correlations between timeseries’
125. printf,1,’Age-banded vs. Hugershoff-standardised’
126. printf,1,’ Region Full <10 >10 >30 >50 >100′
127. ;
128. kla=where((xband ge perst) and (xband le peren))
129. klh=where((x ge perst) and (x le peren))
130. ts1=densadj(klh)
131. ts2=densall(kla)
132. ;
133. r1=correlate(ts1,ts2)
134. rall=fltarr(ntry)
135. for i = 0 , ntry-1 do begin
136. filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
137. filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
138. if i eq 0 then r2=correlate(tshi1,tshi2)
139. rall(i)=correlate(tslow1,tslow2)
140. endfor
141. ;
142. printf,1,’ALL SITES’,r1,r2,rall,$
143. format='(A11,2X,6F6.2)’
144. ;
145. printf,1,’ ‘
146. printf,1,’Correlations carried out over the period ‘,perst,peren
147. ;
148. close,1
149. ;
150. end
[/sourcecode]
Having been involved programming, software design and systems analysis for the better part of 5 decades, I find it unimaginable a supposedly 1st rate research facility would tolerate this code. We were using software design procedures in the 60’s to keep track of the software design. This was to verify what we were doing was right, reduce errors in code changes, and maintenance of the software.
The only time we would ever keep commented code in the source stream, is that it would be readily available to re-insert for whatever reason at a later date. Otherwise it was deleted. PERIOD.
I’m a skeptic, but I agree with John Graham-Cunning on this issue:
http://www.jgc.org/blog/2009/11/very-artificial-correction-flap-looks.html
A STEP BACK
I am not a computer type but the impression I get from this whole discussion about the computer codes is:
#1. It is very messy, badly documented and badly programed.
#2 It is a very poor unprofessional job.
#3 Poor Harry could not untangle it.
If this is an example of sophisticated topflight leading edge programing done to model climate, I would not let these people program a simple game for a three year old much less trust their programing skills enough to base policies on that change the world.
The whole messy code is the smoking gun. It looks like these guys could not program their way out of a paper bag.
I had originally thought that climate modeling was done by some of the best computer programmers in the world. Are these the people actually doing the climate modeling programs?? If so the bad programing alone should make any computer savy person horrified.
Knowing what you now know about CO2 and about the politics behind this entire cluster*, what does this action mean to you? link: http://www.wvmetronews.com/index.cfm?func=displayfullstory&storyid=33948
From the Sunday Times in London
“Almost a month before they were posted on a website popular with climate-change sceptics, the hacked data was sent to a BBC weatherman who had previously expressed his doubts about climate science on his blog.”
“The BBC has now confirmed that Paul Hudson received some of the documents on October 12. But no story was broadcast or printed by Mr Hudson and the corporation”
Great to see our national broadcaster on the case, left to them this story would never have seen the light of day.
Correction:
That should be NOEL Sheppard (not Marc).
THE TRICK
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html
woodfortrees (Paul Clark) (03:38:54) : Seconded again
stevemcintyre (07:10:48) : sounds like a good idea.
Robinson (07:48:06) : I’ve seen the most amazingly tortuous (in terms of cyclometric complexity) code come from the smartest people.
Thanks. I try to write clear code myself though. Actually a nice analogy is that many of the developers that really care about maintanability, readability and code quality really has more in common with marketing people than most hard core scientists and engineers…
I’m suspicious of any manipulation of the data. I can see where simple correction, such as the kind done when moving a station (running the new and old station for a period to adjust the old data to the new records) and I can see similar corrections for UHI where their is comparative data from just outside the island to measure the creep in temp. But, if there is a statistically significant warming occurring over a century and a half of recorded temps, it will show through without bending and coaxing. If not, it is reasonable to conclude that there is no significant trend. It would appear that most agree that there has been somewhere around a degree of warming over a century and a half but the question is: a) is it merely from climbing out of the LIA or is it caused by mans activities. Lets see and plot the historical record, warts and all and see what we are likely dealing with.
(These days, I find “Pravda” more truthful than say “The Economist.”
Whence this madness?)
Very good question. Perhaps it is the human need to have a uniting religion to help feel “connected” through “fellowship”.(the so called God gene). This can be easily exploited by politicians like Al Gore, who is a theology major.
Each to his own I like to say, but in this case, this AGW religion would destroy the US economy by driving up energy prices for no good reason, and spending the money on useless CO2 sequestration.
I fear for our futures. Will we end up having street fights between eco freaks and the millions of unemployed that surely will result from economic socialism? Will we see guns come out to greet the eco freaks, and not just hostile words? I don’t know, but I think we are going to find out how “patriotic” Americans react to the destruction of thier freedom. Maybe that is why the USA currently is arming itself like crazy with guns and ammunition. Throw in a racial bias against the President, and all hell could break loose. Tea parties indeed!
Can Mann et al live with what they are producing through thier lies??? Have they NO sense of responsibility??? Is there NO attorney general in the USA that sees the need to charge Mann et al with fraud? The clock is ticking.
Layne wrote:
Layne Blanchard (23:50:50) :
“Kiss off the next three years. The current administration isn’t interested in facts. They’re on a mission to ram an agenda down America’s throat while they can. We need to look at how we can recover after 2012 (if we’re lucky)”
The only thing that can stop the progressive elites in the US is a veto proof overhaul of the US congress in 2010. Such an event is not likely. We are the metaphorical live frog being slow-boiled for dinner.
Most human beings do not learn from history; therefore, we are doomed to re-live it. (Did I steal that? Is a dead Spanish guy calling me an Ar’se right about now?)
Progressivism is another dangerous religion.
markm
geo (00:08:53) :
>The smartest kid in the class (CRU) just got caught cheating. Why are his frat brothers (GISS, NOAA) claiming that everything is ok because they all put down the same answers on the test? If CRU takes the real data and adds fudge.dat, then aren’t they all just selling different flavors of fudge?<
An excellent analogy.
Robert
Thanks for the clarity.
On “fudge factor” of 2.5, may I recommend adding that since this is interpolated with the temperature, the units are in degrees (C or K).
ie interpolating the raw data with a 2.5 degree fudge factor.
Keep us posted for when someone is able to show by back calculations or against the raw data that this fudge factor was actually used in published work or data.
Heck – I don’t know his name anymore – How about just Mr Sheppard…
(see link above )
Gumby (03:48:06) : My question would be… what kind of configuration management system…
Andrew (03:56:56) : … they would have some sort of control on versions of their code…
I have the same question. Given that harry is doing this work 2006-2009, one would think that some type of version control would be used. Having worked in the 90s on commercial database training courses with a team of less than 10, we used version control software to keep track of everything (courses, instructor notes, code examples, etc.). It is really the lazy person’s approach, since once you get it set up your life is immensely easier.
Perhaps they have version control and we just don’t know it. The Harry read me file is a strong indicator that they do not.
The short version
002 2. ; PLOTS ‘ALL’ REGION MXD timeseries
007 7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
011 11. fudge factor
012 12.’Oooops!’
026 26. ; Get regional tree lists and rbar
050 50. ; Now normalise w.r.t. 1881-1960
055 55. ; APPLY ARTIFICIAL CORRECTION
060 60. ; Now plot them
077 77. ; Extract the post 1600 part
083 83. ; APPLY ARTIFICIAL CORRECTION
088 88. ; Now plot it too
097 97. ; Now overplot their bidecadal components
105 105. ; Now overplot their 50-yr components
115 115. ; Now compute the full, high and low pass correlations between the two
145 145. printf,1,’ ‘
148 148. close,1
150 150. end
mkurbo,
Yes it seems I am not the only liberal leaning socialist who is offended by the way extremist fascists are taking over the left.
Left or Right – to my mind – both become evil and fascist when taken to the extreme. A healthy society is one where we balance Right and Left and therefore a healthy debate should ALWAYS be encouraged. No matter what side of a position I find myself leaning towards, I would still want to hear ALL the arguments from the other side in case it changes my mind or in case we can find the best middle ground. Nevertheless, in the case of Climategate, it matters not what side of the fence you sit on; the emails have been recognized by CRU as being authenticate – they are factual, fact is fact and conspiracy to commit fraud is fraud, IMHO.
Scott A. Mandia (08:29:35),
Thanks a lot for posting that insufferable video… NOT.
The only thing that didn’t make it a complete waste of time is the fact that the apologist who made it obviously cherry-picked a few comments that were arguably arguable.
But I notice that he never once mentioned things like the CRUcial statements by these scientific fraudsters, who refused to cooperate with legal FOIA requests [at least 40 of them!]; their open admissions of gaming the peer review system, conspiring to ruin journals and individuals that didn’t play their AGW game, the finagling of the code, etc.
The jamoke who made your video is far less credible than any of the folks he tries to discredit. The reason you can’t do any better than that lame video is because it is clear to even the most casual observer that the tens of millions of dollars funneled into the pockets of these globaloney fabricators, on both sides of the Atlantic, and the constant, all expense paid trips around the world to hobnob with others of their ilk [with no skeptical scientists invited] was more than enough to thoroughly corrupt them.
What would it take to make the scales fall from your blinkered eyes, and let you see what was going on? The original, raw temperature data? …oh, right. That’s the part that’s missing. Convenient, eh?
Scott A. Mandia (08:29:35) : “Why would CRU need to fudge lines of code if their temperature record since 1979 shows warming that is fairly consistent with RSS and UAH”.
Your right for this period of time, but the code here is for the period 1904 to 1999 (or something?), and the adjustment of station data is before 1979.
I’m not sure why this code is implemented? I can’t exclude that it “hide the decline” between the 1940s and the 1970s if it is implemented for a subset of stations. However the temperature record after 1979 is irrelevant.
Do you remember this USHCN station data adjustment:
http://www.coyoteblog.com/coyote_blog/2007/07/an-interesting-.html
This is the distribution of adjustment between rural areas and major cities:
http://www.climateaudit.org/?p=1859
The post is confusing for the average person who reads it.
Robert Greiner “The source code that actually printed the graph was commented out and, therefore, is not valid proof.”
Actually there was a later source code which printed the garph which was NOT commented out Here: http://di2.nu/foia/harris-tree/briffa_sep98_e.pro
So that makes Robert Greiner’s comment above WRONG.
What he is now saying is. Ok the code that prints the graph, which fudges a value by 2.5, AND IS NOT COMMENTED OUT IN A LATER VERSION OF THE CODE, would have a significant impact on the data set. (Impact shown in the hockey stick graph above)
But we do not know whether it was actually used until we have Briffa’s (of Yamal fame) raw data.
Have I got that right?
The code was named (had a heading) “APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE”
Hiding the decline? Very ARTIFICIALLY?
I thought that some raw data of Briffa’s has already been obtained by Steve McIntyre? Or are we talking about different raw data?
Robert,
Once again, this is the same sort of overreaching [snip] that the Team has been doing for years.
You have not proved anything untoward until you have demonstrated that the effect of this code is incorrect and that it was used to produce published results that are wrong. You have not done that.
It would appear that the only thing nearly as pathetic as a biased climate scientist writing computer programs, is a biased computer programmer dissecting climate science.
What you have presented thus far is context free conjecture – exactly what the Team accuses. This crap plays into their hands.
If the CRU lost their original data then what are they working with now? You would think that this would shut down all research history matching.
The one graph shown on this post is completely mis-interpreted. Robert Greiner seems to think they are temperatures, but they are not. They are in MXD units (a tree-ring density index) and are added to a MXD series. So the comments on their magnitude are completely wrong. Furthermore the whole later part of the series was not used at all (that’s the point of some of the code comments).
The term “fudge factor” doesn’t indicating cheating, as some commenters imply. It is a term commonly used in an experimental context. You’re trying something out – set the FF to 1 for full effect, to zero for no effect, and then intermediate values. It helps pick up errors, avoid instabilities etc. These people do research, which means they write lots of code to just try out ideas. Most of the ideas don’t pan out.
That’s science.
Robinson (07:48:06) :
“Indeed, as a Software Developer that is my experience too. I’ve seen the most amazingly tortuous (in terms of cyclometric complexity) code come from the smartest people.”
In my working life, before I retired and regained my sanity, I wrote many time-critical hardware drivers (mostly assembly, as C-callable procedures) for various production machinery and telescopes. We had a standing rule that no mathematician, statistician, scientist, or astronomer was allowed to write ANY hardware code! This followed a rather spectacular crash of a telescope that attempted to simultaneously rotate the RA and Dec axises through 360 degrees! Cables??!! We don’t need no stinking cables!!!
All CRU must do is just release everything. If emails are out of context, just show us the context. If this isn’t the production source code, show us the production source code.
Just obey the FOIA laws and release the documents.