Climategate Code Analysis Part 2
There are three common issues that have been raised in my previous post that I would like to officially address concerning the CRU’s source code.
If you only get one thing from this post, please get this. I am only making a statement about the research methods of the CRU and trying to show proof that they had the means and intent to falsify data. And, until the CRU’s research results can be verified by a 3rd party, they cannot be trusted.
Here are the four most frequent concerns dealing with the CRU’s source code:
- The source code that actually printed the graph was commented out and, therefore, is not valid proof.
- No proof exists that shows this code was used in publishing results.
- Interpolation is a normal part of dealing with large data sets, this is no different.
- You need the raw climate data to prove that foul play occurred.
If anyone can think of something I missed, please let me know.
The source code that actually printed the graph was commented out and, therefore, is not valid proof.
Had I done a better job with my source analysis, I would have found a later revision of the briffa_sep98_d.pro source file (linked to in my previous post) contained in a different working tree which shows the fudge-factor array playing a direct result in the (uncommented) plotting of the data.
Snippit from: harris-tree/briffa_sep98_e.pro (see the end of the post for the full source listing)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
filter_cru,20,tsin=densall,tslow=tslow,/nan
cpl_barts,x,densall,title='Age-banded MXD from all sites',$
xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
zeroline=tslow,yrange=[-7,3]
oplot,x,tslow,thick=3
oplot,!x.crange,[0.,0.],linestyle=1
;
Now, we can finally put this concern to rest.
Interpolation is a normal part of dealing with large data sets, this is no different.
This is partially true, the issue doesn’t lie in the fact that the CRU researchers used interpolation. The issue is the weight of the valadj array with respect to the raw data. valadj simply introduces too large of an influence to the original data to do anything productive with it.
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
No proof exists that shows this code was used in publishing results.
Correct! That’s why I am (and always have) taken the following stand: Enough proof exists that the CRU had both the means and intent to intentionally falsify data. This means that all of their research results cannot be trusted until they are verified. Period.
The fact that the “fudge-factor” source code exists in the first place is reason enough for alarm. Hopefully, they didn’t use fudged results in the CRU research results, but the truth is, we just don’t know.
You need the raw climate data to prove that foul play occurred.
This is assuming the raw data are valid, which I maintain that it probably is. Several people question the validity of the climate data gathering methods used by the different climate research institutions, but I am not enough of a climate expert to have an opinion one way or the other. Furthermore, It simply doesn’t matter if the raw climate data are correct or not to demonstrate the extreme bias the valadj array forces on the raw data.
So, the raw data could actually be temperature data or corporate sales figures, the result is the same; a severe manipulation of data.
Full Source Listing
As promised, here is the entire source listing for: harris-tree/briffa_sep98_e.pro
[sourcecode language=”text”]
1. ;
2. ; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
3. ; standardised datasets.
4. ; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
5. ; with missing values set appropriately. Uses mxd, and just the
6. ; "all band" timeseries
7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
8. ;
9. yrloc=[1400,findgen(19)*5.+1904]
10. valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
11. 2.6,2.6,2.6]*0.75 ; fudge factor
12. if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
13. ;
14. loadct,39
15. def_1color,20,color=’red’
16. plot,[0,1]
17. multi_plot,nrow=4,layout=’large’
18. if !d.name eq ‘X’ then begin
19. window, ysize=800
20. !p.font=-1
21. endif else begin
22. !p.font=0
23. device,/helvetica,/bold,font_size=18
24. endelse
25. ;
26. ; Get regional tree lists and rbar
27. ;
28. restore,filename=’reglists.idlsave’
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
31. ;
32. rawdat=fltarr(4,2000)
33. for i = nreg-1 , nreg-1 do begin
34. fn=’mxd.’+harryfn(i)+’.pa.mean.dat’
35. print,fn
36. openr,1,fn
37. readf,1,rawdat
38. close,1
39. ;
40. densadj=reform(rawdat(2:3,*))
41. ml=where(densadj eq -99.999,nmiss)
42. densadj(ml)=!values.f_nan
43. ;
44. x=reform(rawdat(0,*))
45. kl=where((x ge 1400) and (x le 1992))
46. x=x(kl)
47. densall=densadj(1,kl) ; all bands
48. densadj=densadj(0,kl) ; 2-6 bands
49. ;
50. ; Now normalise w.r.t. 1881-1960
51. ;
52. mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
53. mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
54. ;
55. ; APPLY ARTIFICIAL CORRECTION
56. ;
57. yearlyadj=interpol(valadj,yrloc,x)
58. densall=densall+yearlyadj
59. ;
60. ; Now plot them
61. ;
62. filter_cru,20,tsin=densall,tslow=tslow,/nan
63. cpl_barts,x,densall,title=’Age-banded MXD from all sites’,$
64. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
65. zeroline=tslow,yrange=[-7,3]
66. oplot,x,tslow,thick=3
67. oplot,!x.crange,[0.,0.],linestyle=1
68. ;
69. endfor
70. ;
71. ; Restore the Hugershoff NHD1 (see Nature paper 2)
72. ;
73. xband=x
74. restore,filename=’../tree5/densadj_MEAN.idlsave’
75. ; gets: x,densadj,n,neff
76. ;
77. ; Extract the post 1600 part
78. ;
79. kl=where(x ge 1400)
80. x=x(kl)
81. densadj=densadj(kl)
82. ;
83. ; APPLY ARTIFICIAL CORRECTION
84. ;
85. yearlyadj=interpol(valadj,yrloc,x)
86. densadj=densadj+yearlyadj
87. ;
88. ; Now plot it too
89. ;
90. filter_cru,20,tsin=densadj,tslow=tshug,/nan
91. cpl_barts,x,densadj,title=’Hugershoff-standardised MXD from all sites’,$
92. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
93. zeroline=tshug,yrange=[-7,3],bar_color=20
94. oplot,x,tshug,thick=3,color=20
95. oplot,!x.crange,[0.,0.],linestyle=1
96. ;
97. ; Now overplot their bidecadal components
98. ;
99. plot,xband,tslow,$
100. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
101. yrange=[-6,2],thick=3,title=’Low-pass (20-yr) filtered comparison’
102. oplot,x,tshug,thick=3,color=20
103. oplot,!x.crange,[0.,0.],linestyle=1
104. ;
105. ; Now overplot their 50-yr components
106. ;
107. filter_cru,50,tsin=densadj,tslow=tshug,/nan
108. filter_cru,50,tsin=densall,tslow=tslow,/nan
109. plot,xband,tslow,$
110. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
111. yrange=[-6,2],thick=3,title=’Low-pass (50-yr) filtered comparison’
112. oplot,x,tshug,thick=3,color=20
113. oplot,!x.crange,[0.,0.],linestyle=1
114. ;
115. ; Now compute the full, high and low pass correlations between the two
116. ; series
117. ;
118. perst=1400.
119. peren=1992.
120. ;
121. openw,1,’corr_age2hug.out’
122. thalf=[10.,30.,50.,100.]
123. ntry=n_elements(thalf)
124. printf,1,’Correlations between timeseries’
125. printf,1,’Age-banded vs. Hugershoff-standardised’
126. printf,1,’ Region Full <10 >10 >30 >50 >100′
127. ;
128. kla=where((xband ge perst) and (xband le peren))
129. klh=where((x ge perst) and (x le peren))
130. ts1=densadj(klh)
131. ts2=densall(kla)
132. ;
133. r1=correlate(ts1,ts2)
134. rall=fltarr(ntry)
135. for i = 0 , ntry-1 do begin
136. filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
137. filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
138. if i eq 0 then r2=correlate(tshi1,tshi2)
139. rall(i)=correlate(tslow1,tslow2)
140. endfor
141. ;
142. printf,1,’ALL SITES’,r1,r2,rall,$
143. format='(A11,2X,6F6.2)’
144. ;
145. printf,1,’ ‘
146. printf,1,’Correlations carried out over the period ‘,perst,peren
147. ;
148. close,1
149. ;
150. end
[/sourcecode]
The BBC’s Environment Correspondent has told commentators on his blog to stop posting comments about ClimateGate! Apparently that’s because the alleged falsification of research is boring and no-one wants to read about it – so stop nit-picking. Seriously:
I’ve another request, too – if you can restrain yourselves from plastering this thread with stuff about ClimateGate, please do.
There are more than 700 comments on the previous thread, the vast majority related to it. I know from e-mails that some readers find endless picking over of climate science repetitive and boring – and when they do, they don’t read through the comments. Fresh, pertinent and interesting are my suggestions.”
BBC – Richard Black’s Earth Watch
http://www.bbc.co.uk/blogs/thereporters/richardblack/2009/12/copenhagen_countdown_3_days.html
jorgekafkazar, please may I have your permission to use this bit for my quote of next week?
“CRU has been caught with weapons of math destruction in their code.”
Thanks in advance.
http://www2.macleans.ca/category/opinion/mark-steyn-opinion/
Back in the summer, I wrote in a column south of the border:
…“If you’re 29, there has been no global warming for your entire adult life. If you’re graduating high school, there has been no global warming since you entered first grade. There has been no global warming this century. None. Admittedly the 21st century is only one century out of the many centuries of planetary existence, but it happens to be the one you’re stuck living in.”
In response to that, the shrieking pansies of the eco-left had a fit. The general tenor of my mail was summed up by one correspondent: “How can you live with your lies, dumbf–k?” George Soros’s stenographers at Media Matters confidently pronounced it a “false claim.” Well, take it up with Phil Jones. He agrees with me. The only difference is he won’t say so in public.
Which is a bit odd, don’t you think?…
Brian Johnson uk (23:27:41) :
Are the EPA mad? How on earth can CO2 be a Public Danger.
The EPA is composed of bureaucrats, with a very few genuine scientists tossed in to give the appearance of a science-oriented gummint agency.
Some time ago, I was monitoring a methane burn at a landfill and had an EPA inspector claim that the burner was inoperative because he couldn’t see the flames. When I told him that pure methane burned at such a high temperature the flames were only visible in the infrared spectrum, he harrumphed and thrust his hand into the plume to *prove* I was lying.
I drove him to the hospital to have his second-degree burns treated.
Good analysis so far Robert. I agree with you on the Modus Operandi side of things – motive, means and opportunity.
The next thing is to try to run the program with and without the valadj ‘adjustments’ using made up test data. Then see what the difference is. The data values can be the same for all the dates but different for each file because you’re testing the code not trying to replicate a published chart.
After that maybe the next possible step is to use real data and see if the produced chart corresponds to anything published. The real data should be obtainable via FOI. One for Steve M perhaps!
harrfn (harry filename?) is an array of data filenames found here:
29. harryfn=['nwcan','wnam','cecan','nweur','sweur','nsib','csib','tib',$
30. 'esib','allsites']
My guess is they correspond as follows:
nwcan = North West Canada
wnam = Western North America
cecan = Central Canada
nweur = North West Europe
sweur = South West Europe
cs and es could be ISO country codes but ns and ti are not.
This is a quotable quote! 😉
Larry
This is not the first post about a code for fudging.
Indeed if it is claimed it was never used, then why was it written to begin with?
I think we should continue demanding the code and raw data under the FOIA.
That’s the only way this is going to get investigated properly.
Wow, I really think you guys missed the boat with all that code up there. I’ve come up with a simple equation that even a 12 year old can understand that explains EVERYTHING:
AGW = (OT + (DFU/$)RG + AGF) / CAPBSIF
For those who need help sorting this out, this shows that AGW is the sum of:
OT – Original Temperature Findings
DFU/$ – Tells us how much money Mann, Jones, et al, get per each Degree they can Fudge Upward (And subsequently this leads to RG, Reasearch Grants.
AGF – Al Gore’s Carbon Footprint
And this is very important, so that we get the numbers right – After we sum all of that up, you MUST divide by CAPBSIF, that is, the # of computer animated polar bears you can show stranded on ice flows.
How to save the planet:
http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6736517/Copenhagen-climate-summit-1200-limos-140-private-planes-and-caviar-wedges.html
Fudge Factors have always had me worried.
Once picked up some real-time safety critical code from a major defence player, now, very thankfully, no longer with us, which had fudge_factor_1 to fudge_factor_10 declared and used throughout.
Very, very scary in a plane, but this is on a global scale B)
OT
Just seen in the Daily Mail – “Emalis that rocked climate change campaign leaked from Siberian ‘closed city’ university built by KGB”
Read more:
http://www.dailymail.co.uk/news/article-1233562/Emails-rocked-climate-change-campaign-leaked-Siberian-closed-city-university-built-KGB.html
Not the most credible article I’ve ever read, but it’s an interesting hypothesis anyway.
Perhaps part 3 could address a couple of issues? First, you need to redraw the graph of valadj with the 0.75 multiplier included – so 2.6 goes down to a mere 1.95! Second, I should like to see a mathematical description of the interpolation code, as I don’t understand exactly what it does.
Thanks,
Rich.
I fear this climategate is noting more than a fart in a bottle ! The blogosphere can yell wathever they wont…in Europe the Kopenhagen conference has to come out with a drastic reduction of CO2 to save the climate. On the official newschannels there is almost no mention of the hacked e-mails or of climategate. I also have the feeling that nobody is interested in what those emails could reveal.
In Europe the whole antropogenic warming circus simply goes on with a never seen energy. And as cherry on the top,…more and more European people start to believe them. It has been an very warm autumn, just as the year before, and the year before that and….The official weathermen are promoting the idea this is not coïncidential anymore. Not normal. Global warming is overwhelming, they say !
So in Europe the AGW Kopenhagen show can benefit from the reality in nature. And no ripple could disturb the already now achieved succes of the summit. Whatever the source code of whatever climate model contains or not.
As the article points out, even if this code wasn’t used, the mere fact that it was there points to a need to “adjust” data. This is not interpolation.
And even though this code seems to have been for the treemometer data, doesn’t mean there wasn’t other shenanigans going on with the temperature dataset.
AJStrata’s work seems to point out that the raw data doesn’t confirm the warming that the GISS and CRU “adjustments” shows, but no doubt, if cornered, they’ll tell us that a “fudge factor” has to be added in to cover some inadequacy or other in the GHCN dataset.
This whole thing is too big for people to be going off on their own tangents in analysing the data. What we need is to coordinate our efforts, standardise how the data is treated, and come up with a single alternative analysis.
We can’t wait three years for the Met Office to announce their re-analysis, and anyway, one visit to their site shows where their prejudices lie.
We already have a large number of capable people posting to this site, and their email addresses are known to this site, so I’d suggest that a section of the site or another effort like the surfacestations.org work be got up and running.
I have skills in website and application programming, and am currently learning R, so if there was anything I could do to help with this effort, I’d be glad to.
Brian Johnson UK:
“Solution is we all [6+ Billion of us] stop breathing! Problem solved! Danger over!”
By your command:
http://www.georgiaguidestones.com
From previous post:
Climate Heretic (16:18:03): I disagree with you.
woodentop (16:01:48): Sorry pal, but that’s rubbish.
The need for source code readability, maintainability and quality certainly depend on the context in software engineering. I addition this is not the main point here, let us assume that the source code is poor quality as you suggest, does that help us in any way? NO! I think we need a much more PRAGMATIC point of view here, KILL YOUR DARLINGS (in software), we need to COMPILE the code, EXECUTE the code to determine WHAT it is doing!
Complaining that the source code is poor quality is a dead end – please stop doing that!
No wonder the CRU are delaying the publishing of the FULL raw station data, and the specific computer code. If this is a sample of the real code then the coding is a “complete mess” and I suspect that the CRU are embarrassed by the poor quality of their work.
Poor Harry has been given the impossible task of re-writing some kind of coding that can be applied to the raw data to recreate a temperature record that bears some resemblance to their “value added” data.
IT’S HILARIOUS!
To use a chess term: ZUGZWANG.
They have been forced to make a move but whatever they do will cause their doom.
By the way Jonathan Porrit (a very active AGW campaigner) has just been on the BBC accepting that natural variation may have a large influence on the climate. You wouldn’t have heard that a few weeks ago.
OT. BBC explanation of climate change http://news.bbc.co.uk/1/hi/sci/tech/8394168.stm But am I right in thinking that you will get a temp rise no matter what gas you put it to the ‘other’ bottle?
I’ve not gone through the code in detail but if that’s what passes for clear modern programming then I’m in the wrong business (I’m a programmer). Looks more like a BASIC program for a ZX81.
mikey (07:38:46) : on a different story posted this link
http://news.bbc.co.uk/1/hi/world/south_asia/8387737.stm
IPCC – out by 300 years and using non-peer reviewed information!
Is this the first time the BBC has been so critical of anything done by the IPCC?
I note Wikipedia (unlike Google) recognises that ClimateGate happened and has a page dedicated to it:
http://en.wikipedia.org/wiki/Climatic_Research_Unit_e-mail_hacking_incident
I am sure it will come as a huge shock to readers here that they downplay the incident…
Two points –
Be careful – if the adjustments are in hundredths of a degree, you will look stupid when that comes out.
Its obvious to me that the Met Office has said their data rework will take 3 years so they can see what happens to the climate (sorry – weather) in that time. So they dont know and they are waiting to see what happens next -like all of us.
osborn-tree6\mann\mxdgrid2ascii.pro 10/03/2008
endif else begin
ml=where(finite(mxdfd) eq 0,nmiss)
if nmiss gt 0 then mxdfd(ml)=-9.99
printf,1,’Osborn et al. (2004) gridded reconstruction of warm-season’
printf,1,'(April-September) temperature anomalies (from the 1961-1990 mean).’
printf,1,’Reconstruction is based on tree-ring density records.’
printf,1
printf,1,’NOTE: recent decline in tree-ring density has been ARTIFICIALLY’
printf,1,’REMOVED to facilitate calibration. THEREFORE, post-1960 values’
printf,1,’will be much closer to observed temperatures then they should be,’
printf,1,’which will incorrectly imply the reconstruction is more skilful’
printf,1,’than it actually is. See Osborn et al. (2004).’
printf,1
printf,1,’Osborn TJ, Briffa KR, Schweingruber FH and Jones PD (2004)’
printf,1,’Annually resolved patterns of summer temperature over the Northern’
printf,1,’Hemisphere since AD 1400 from a tree-ring-density network.’
printf,1,’Submitted to Global and Planetary Change.’
;
printf,1,’Grid resolution is 5 by 5 degrees, for the Northern Hemisphere only,’
printf,1,’with the first value at ‘,g.x[0],’ W’,g.y[0],’ N,’
printf,1,’then moving eastwards to ‘,g.x[g.nx-1],’ and then southwards to ‘,g.y[g.ny-1],’.’
printf,1
printf,1,’Missing value is -9.99′
printf,1
for iyr = 0 , mxdnyr-1 do begin
print,mxdyear[iyr],format='($,I5)’
printf,1,mxdyear[iyr],format='(I4)’
printf,1,mxdfd[*,*,iyr],format='(10F8.2)’
endfor
print
endelse
Barry Foster (02:15:35)
Complain on the BBC website and be sure to describe it as fraud.
Heard on the Andrew Marr show on the BBC this morning from the US Ambassador to the UK:
‘Climategate’ has not been picked up on in the US. There you have it. Nothing to see here. Move along please. Have I been in a dream for the past 2 weeks or so? I am sure there are Americans who have been blogging like mad. I know Fox News has. Oh I know. He was talking about most of the American mainstream media wasnt he????
These could be:
nsib – northern Siberia
csib – central Siberia
esib – eastern Siberia