Climategate Code Analysis Part 2
There are three common issues that have been raised in my previous post that I would like to officially address concerning the CRU’s source code.
If you only get one thing from this post, please get this. I am only making a statement about the research methods of the CRU and trying to show proof that they had the means and intent to falsify data. And, until the CRU’s research results can be verified by a 3rd party, they cannot be trusted.
Here are the four most frequent concerns dealing with the CRU’s source code:
- The source code that actually printed the graph was commented out and, therefore, is not valid proof.
- No proof exists that shows this code was used in publishing results.
- Interpolation is a normal part of dealing with large data sets, this is no different.
- You need the raw climate data to prove that foul play occurred.
If anyone can think of something I missed, please let me know.
The source code that actually printed the graph was commented out and, therefore, is not valid proof.
Had I done a better job with my source analysis, I would have found a later revision of the briffa_sep98_d.pro source file (linked to in my previous post) contained in a different working tree which shows the fudge-factor array playing a direct result in the (uncommented) plotting of the data.
Snippit from: harris-tree/briffa_sep98_e.pro (see the end of the post for the full source listing)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
filter_cru,20,tsin=densall,tslow=tslow,/nan
cpl_barts,x,densall,title='Age-banded MXD from all sites',$
xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
zeroline=tslow,yrange=[-7,3]
oplot,x,tslow,thick=3
oplot,!x.crange,[0.,0.],linestyle=1
;
Now, we can finally put this concern to rest.
Interpolation is a normal part of dealing with large data sets, this is no different.
This is partially true, the issue doesn’t lie in the fact that the CRU researchers used interpolation. The issue is the weight of the valadj array with respect to the raw data. valadj simply introduces too large of an influence to the original data to do anything productive with it.
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
No proof exists that shows this code was used in publishing results.
Correct! That’s why I am (and always have) taken the following stand: Enough proof exists that the CRU had both the means and intent to intentionally falsify data. This means that all of their research results cannot be trusted until they are verified. Period.
The fact that the “fudge-factor” source code exists in the first place is reason enough for alarm. Hopefully, they didn’t use fudged results in the CRU research results, but the truth is, we just don’t know.
You need the raw climate data to prove that foul play occurred.
This is assuming the raw data are valid, which I maintain that it probably is. Several people question the validity of the climate data gathering methods used by the different climate research institutions, but I am not enough of a climate expert to have an opinion one way or the other. Furthermore, It simply doesn’t matter if the raw climate data are correct or not to demonstrate the extreme bias the valadj array forces on the raw data.
So, the raw data could actually be temperature data or corporate sales figures, the result is the same; a severe manipulation of data.
Full Source Listing
As promised, here is the entire source listing for: harris-tree/briffa_sep98_e.pro
[sourcecode language=”text”]
1. ;
2. ; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
3. ; standardised datasets.
4. ; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
5. ; with missing values set appropriately. Uses mxd, and just the
6. ; "all band" timeseries
7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
8. ;
9. yrloc=[1400,findgen(19)*5.+1904]
10. valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
11. 2.6,2.6,2.6]*0.75 ; fudge factor
12. if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
13. ;
14. loadct,39
15. def_1color,20,color=’red’
16. plot,[0,1]
17. multi_plot,nrow=4,layout=’large’
18. if !d.name eq ‘X’ then begin
19. window, ysize=800
20. !p.font=-1
21. endif else begin
22. !p.font=0
23. device,/helvetica,/bold,font_size=18
24. endelse
25. ;
26. ; Get regional tree lists and rbar
27. ;
28. restore,filename=’reglists.idlsave’
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
31. ;
32. rawdat=fltarr(4,2000)
33. for i = nreg-1 , nreg-1 do begin
34. fn=’mxd.’+harryfn(i)+’.pa.mean.dat’
35. print,fn
36. openr,1,fn
37. readf,1,rawdat
38. close,1
39. ;
40. densadj=reform(rawdat(2:3,*))
41. ml=where(densadj eq -99.999,nmiss)
42. densadj(ml)=!values.f_nan
43. ;
44. x=reform(rawdat(0,*))
45. kl=where((x ge 1400) and (x le 1992))
46. x=x(kl)
47. densall=densadj(1,kl) ; all bands
48. densadj=densadj(0,kl) ; 2-6 bands
49. ;
50. ; Now normalise w.r.t. 1881-1960
51. ;
52. mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
53. mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
54. ;
55. ; APPLY ARTIFICIAL CORRECTION
56. ;
57. yearlyadj=interpol(valadj,yrloc,x)
58. densall=densall+yearlyadj
59. ;
60. ; Now plot them
61. ;
62. filter_cru,20,tsin=densall,tslow=tslow,/nan
63. cpl_barts,x,densall,title=’Age-banded MXD from all sites’,$
64. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
65. zeroline=tslow,yrange=[-7,3]
66. oplot,x,tslow,thick=3
67. oplot,!x.crange,[0.,0.],linestyle=1
68. ;
69. endfor
70. ;
71. ; Restore the Hugershoff NHD1 (see Nature paper 2)
72. ;
73. xband=x
74. restore,filename=’../tree5/densadj_MEAN.idlsave’
75. ; gets: x,densadj,n,neff
76. ;
77. ; Extract the post 1600 part
78. ;
79. kl=where(x ge 1400)
80. x=x(kl)
81. densadj=densadj(kl)
82. ;
83. ; APPLY ARTIFICIAL CORRECTION
84. ;
85. yearlyadj=interpol(valadj,yrloc,x)
86. densadj=densadj+yearlyadj
87. ;
88. ; Now plot it too
89. ;
90. filter_cru,20,tsin=densadj,tslow=tshug,/nan
91. cpl_barts,x,densadj,title=’Hugershoff-standardised MXD from all sites’,$
92. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
93. zeroline=tshug,yrange=[-7,3],bar_color=20
94. oplot,x,tshug,thick=3,color=20
95. oplot,!x.crange,[0.,0.],linestyle=1
96. ;
97. ; Now overplot their bidecadal components
98. ;
99. plot,xband,tslow,$
100. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
101. yrange=[-6,2],thick=3,title=’Low-pass (20-yr) filtered comparison’
102. oplot,x,tshug,thick=3,color=20
103. oplot,!x.crange,[0.,0.],linestyle=1
104. ;
105. ; Now overplot their 50-yr components
106. ;
107. filter_cru,50,tsin=densadj,tslow=tshug,/nan
108. filter_cru,50,tsin=densall,tslow=tslow,/nan
109. plot,xband,tslow,$
110. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
111. yrange=[-6,2],thick=3,title=’Low-pass (50-yr) filtered comparison’
112. oplot,x,tshug,thick=3,color=20
113. oplot,!x.crange,[0.,0.],linestyle=1
114. ;
115. ; Now compute the full, high and low pass correlations between the two
116. ; series
117. ;
118. perst=1400.
119. peren=1992.
120. ;
121. openw,1,’corr_age2hug.out’
122. thalf=[10.,30.,50.,100.]
123. ntry=n_elements(thalf)
124. printf,1,’Correlations between timeseries’
125. printf,1,’Age-banded vs. Hugershoff-standardised’
126. printf,1,’ Region Full <10 >10 >30 >50 >100′
127. ;
128. kla=where((xband ge perst) and (xband le peren))
129. klh=where((x ge perst) and (x le peren))
130. ts1=densadj(klh)
131. ts2=densall(kla)
132. ;
133. r1=correlate(ts1,ts2)
134. rall=fltarr(ntry)
135. for i = 0 , ntry-1 do begin
136. filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
137. filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
138. if i eq 0 then r2=correlate(tshi1,tshi2)
139. rall(i)=correlate(tslow1,tslow2)
140. endfor
141. ;
142. printf,1,’ALL SITES’,r1,r2,rall,$
143. format='(A11,2X,6F6.2)’
144. ;
145. printf,1,’ ‘
146. printf,1,’Correlations carried out over the period ‘,perst,peren
147. ;
148. close,1
149. ;
150. end
[/sourcecode]
That’s the virus!
For a satirical look at the climategate computer programming:
Anthropogenic Global Warming Virus Alert.
http://www.thespoof.com/news/spoof.cfm?headline=s5i64103
MotleyCRU saws off the bottom of the data deck, jacks up decline.
FudgeBoostFactor2.5 sends hockeystick soaring over the Tower of Babel.
Only the Insider knows for sure who did what.
@Andrew…
LOL!! That is some quality satire. 🙂
Very nice work!
Very nice work! I think the code is where the real damage lays.
Yikes! What happened there? I hear an echo…
The “fudge factor” part is just to scale the ‘artificial’ array. I’m pretty sure looking at the code that is just to help line things up. That isn’t a pass — the programmer is a total amateur on the comments alone, but that part doesn’t mean they were fudging the big picture, just they were wanting to use a single variable (err, constant in this case) to help line things up.
The bigger picture is the inevitable ‘blade part’ of of every graph these guys produce is based on a different cocktail of data than the stick part. They go on and on how this is ‘ok’, and how this is ‘known’, but it isn’t up to climatologists to decide what is ok with math and statistics. It is obviously trying to squeeze desired results from data.
The world seems happy to ignore this — whatever, I guess they are tired of oil or voting or something. Regardless, this is shit science. Actually it isn’t even that, it is tainted before it even gets past the math, and probably before it even gets past writing down the data. The ‘science’ never had a chance.
Ahhh, its depressing, I’m just going to go to bed (umm, not with Shell though).
This just in:
EPA Set to Declare Carbon Dioxide a Public Danger
http://www.foxnews.com/politics/2009/12/06/epa-set-delare-carbon-dioxide-public-danger/
Only benefit of Cap-and-trade is that someone makes money on the deal.
excerpt:
Such an “endangerment” decision is necessary for the EPA to move ahead early next year with new emission standards for cars. EPA Administrator Lisa Jackson has said it could also mean large emitters such as power stations, cement kilns, crude-oil refineries and chemical plants would have to curb their greenhouse gas output.
Who is John Galt?
There’s some interesting stuff in cru-code/linux/mod/ghcnrefiter.f90 too.
Nice report on the code, it certainly answers some questions.
Nice link to that spoof, Andrew. Very funny.
John Galt: http://en.wikipedia.org/wiki/John_Galt
Finally, news about the solar minimum.
“The sunspot count continues to be at the lowest level in a century. Looking back in the past, there appears to be a strong link between an inactive sun and a cooler earth. The sun is the source of incoming heat on this planet. It has cycles that we haven’t been around long enough to study and understand clearly.”
“Climategate” Exposes Global Warming Hoax
http://www.courierpress.com/news/2009/dec/06/quotclimategatequot-exposes-global-warming-hoax/
There are so many concerns related to this that the only way to understand what has been going on at CRU is for complete open access to all the data and codes. I wouldn’t be surprised to find that there was (and probably still is) so little control over the codes that all versions previously used no longer exist and would have to be rewritten from paper records (if those still exist). Without a QA system in place that that would have ensured that all code and data were written, documented, maintained, used and archived properly, I suspect it will be impossible to untangle the CRU mess.
Are the EPA mad? How on earth can CO2 be a Public Danger.
OK – got it! We breathe out and the CO2 content is around 40,000 parts per million. So…….. as we are panicking about Mauna Lou CO2 levels climbing and at present 387.75 parts per million then why not issue draconian laws when the magic figure of 40,000 parts per million can be quoted.
Solution is we all [6+ Billion of us] stop breathing! Problem solved! Danger over!
EPA – Environmental Panic Attackers
Robert. It appears you have discovered an alternative “climate forcing” mechanism. Perhaps NATURE should pick it up?
From another thread:
Prosecutor: And where did you find the defendant, Constable Platypus?
Bobby: Be’ind the ware’ouse, standin’ in the shadders by the back gate.
Prosecutor: Was the gate locked when you arrived?
Bobby: Hit was. There was a chain wif a padlock through two o’ the links.
Prosecutor: And what did the defendant have in his hands?
Bobby: ‘E ‘ad that pair o’ bolt cutters, Hex’ibit B. (points at evidence table)
Prosecutor: And how did the defendant explain his presence at the back gate of the warehouse, standing in the dark, with these bolt cutters in his possession?
Bobby: [looking at his notebook] ‘E said, “I were just practisin’, gov’nor. These ‘ere bolt cutters ‘ave been commented out.”
(laughter)
Prosecutor: And had they been commented out?
Bobby: Someone ‘ad put a semicologne on ‘em wif a piece o’ chalk, yerse.
Prosecutor: How long would it take for the defendant to remove the semicolon and cut the chain?
Barrister: Objection, M’lud! PC Platypus is not an expert on chalk.
Judge: Over-ruled. You may answer the question.
Bobby: Habout ‘arf a second.
(laughter)
Prosecutor: Thank you, Constable Platypus.
CRU has been caught with weapons of math destruction in their code. That is sufficient to establish intent, even without showing they were actually used, any more than the defendant’s bolt cutters. Possession of such devices is sufficient to establish guilt.
I am anxious to see more analysis of this code…the problems with it appear to be enormous…a full accounting of those problems is the only way to put to rest any talk of CRU as being innocent of the charge of criminally bad science.
Kiss off the next three years. The current administration isn’t interested in facts. They’re on a mission to ram an agenda down America’s throat while they can. We need to look at how we can recover after 2012 (if we’re lucky)
NH 1910 temp = 1970 temp in 1974
http://www.denisdutton.com/newsweek_coolingworld.pdf
According to http://scienceblogs.com/deltoid/2009/12/quote_mining_code.php
The plot without the correction comes very close to 1910- 1970.
AJstratas work shows 1910 very close to 1970
Here is the NH Hadcrut plot from BOM
http://reg.bom.gov.au/cgi-bin/climate/change/global/timeseries.cgi?graph=global_t®ion=nh&season=0112&ave_yr=0
The correction IMO is to make the proxies match the “adjusted temperature”.
I would just about bet my left nut that if we ran the “virus” backwards though the Hadcrut that we would be very close to the actual raw temperatures and that would correlate well to the individual long rural records.
Phillip Bratby (23:23:43)
Yeah, but given the apparent mess that is the CRUs’ electronic data, even with complete access to all the files, it doesn’t look good. e.g. what’s in
crucode/idl/pro/README_GRIDDING.txt
“Then use quick_interp_tdm2.pro
on the secondary variable, with synth_prefix supplied, to create the new grids.
Bear in mind that there is no working synthetic method for cloud, because Mark New
lost the coefficients file and never found it again (despite searching on tape
archives at UEA) and never recreated it. This hasn’t mattered too much, because
the synthetic cloud grids had not been discarded for 1901-95, and after 1995
sunshine data is used instead of cloud data anyway.”
Wonder what else is on or gone from the archives…
The Ad Hominem Attack Heard Round the World.
I’m thinking about re-upoading this video with this new title, unless someone else beats me to it.
http://strata-sphere.com/blog/index.php/archives/11630
Sorry loink to AJstrata’s work.
I watched that when aired. Have to say it wad train wreck journalism at it’s best (or worst!).
Watson is an arse but I found it quite funny that he got worked up so easily and the other huh fed off this 🙂
mailman
The smartest kid in the class (CRU) just got caught cheating. Why are his frat brothers (GISS, NOAA) claiming that everything is ok because they all put down the same answers on the test? If CRU takes the real data and adds fudge.dat, then aren’t they all just selling different flavors of fudge?
When investigating a conspiracy to commit a crime, motive, method, and opportunity are the main areas to focus on.
This excellent post provides evidence that a method was available to the people involved in the Climategate cabal, and the emails show that they had the motivation and the opportunity to do it.
Now all we need is the ‘body’ – the raw data – to show a crime was actually committed. The Climategate emails show that Jones et al were collusing to stop people finding it, by refusing FOIA requests and even threatening to destroy the data should the worst came to the worst.
Once the raw data is found the IPCC charts can be reconstructed, and comparisons are made to the ones in the IPCC documents. If there is a discrepancy, then a solid case can be built for them to answer.
However, from the context of what we have seen. it is clear that the UEA CRU/GISS/IPCC were not conducting science – they just trying to find evidence to support their belief in CAGW and doing everything in their power to prevent contrary evidence from other scientists being published.
Science is about facts and the truth and the Climategate evidence shows the ‘team’ had little respect for either.
“Emalis that rocked climate change campaign leaked from Siberian ‘closed city’ university built by KGB
An investigation by The Mail on Sunday has discovered that the explosive hacked emails from the University of East Anglia were leaked via a small web server in the formerly closed city of Tomsk in Siberia.
Computer hackers in Tomsk have been used in the past by the Russian secret service (FSB) to shut websites which promote views disliked by Moscow.
In 2002, Tomsk students were said to have launched a ‘denial of service’ attack at the Kavkaz-Tsentr portal, a site whose reports about Chechnya angered Russian officials.
The FSB office in Tomsk put out a special Press release saying that what the students had done was a legitimate ‘expression of their position as citizens, one worthy of respect’.”
http://www.dailymail.co.uk/news/article-1233562/Emails-rocked-climate-change-campaign-leaked-Siberian-closed-city-university-built-KGB.html