Climategate Code Analysis Part 2
There are three common issues that have been raised in my previous post that I would like to officially address concerning the CRU’s source code.
If you only get one thing from this post, please get this. I am only making a statement about the research methods of the CRU and trying to show proof that they had the means and intent to falsify data. And, until the CRU’s research results can be verified by a 3rd party, they cannot be trusted.
Here are the four most frequent concerns dealing with the CRU’s source code:
- The source code that actually printed the graph was commented out and, therefore, is not valid proof.
- No proof exists that shows this code was used in publishing results.
- Interpolation is a normal part of dealing with large data sets, this is no different.
- You need the raw climate data to prove that foul play occurred.
If anyone can think of something I missed, please let me know.
The source code that actually printed the graph was commented out and, therefore, is not valid proof.
Had I done a better job with my source analysis, I would have found a later revision of the briffa_sep98_d.pro source file (linked to in my previous post) contained in a different working tree which shows the fudge-factor array playing a direct result in the (uncommented) plotting of the data.
Snippit from: harris-tree/briffa_sep98_e.pro (see the end of the post for the full source listing)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
filter_cru,20,tsin=densall,tslow=tslow,/nan
cpl_barts,x,densall,title='Age-banded MXD from all sites',$
xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
zeroline=tslow,yrange=[-7,3]
oplot,x,tslow,thick=3
oplot,!x.crange,[0.,0.],linestyle=1
;
Now, we can finally put this concern to rest.
Interpolation is a normal part of dealing with large data sets, this is no different.
This is partially true, the issue doesn’t lie in the fact that the CRU researchers used interpolation. The issue is the weight of the valadj array with respect to the raw data. valadj simply introduces too large of an influence to the original data to do anything productive with it.
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
No proof exists that shows this code was used in publishing results.
Correct! That’s why I am (and always have) taken the following stand: Enough proof exists that the CRU had both the means and intent to intentionally falsify data. This means that all of their research results cannot be trusted until they are verified. Period.
The fact that the “fudge-factor” source code exists in the first place is reason enough for alarm. Hopefully, they didn’t use fudged results in the CRU research results, but the truth is, we just don’t know.
You need the raw climate data to prove that foul play occurred.
This is assuming the raw data are valid, which I maintain that it probably is. Several people question the validity of the climate data gathering methods used by the different climate research institutions, but I am not enough of a climate expert to have an opinion one way or the other. Furthermore, It simply doesn’t matter if the raw climate data are correct or not to demonstrate the extreme bias the valadj array forces on the raw data.
So, the raw data could actually be temperature data or corporate sales figures, the result is the same; a severe manipulation of data.
Full Source Listing
As promised, here is the entire source listing for: harris-tree/briffa_sep98_e.pro
[sourcecode language=”text”]
1. ;
2. ; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
3. ; standardised datasets.
4. ; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
5. ; with missing values set appropriately. Uses mxd, and just the
6. ; "all band" timeseries
7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
8. ;
9. yrloc=[1400,findgen(19)*5.+1904]
10. valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
11. 2.6,2.6,2.6]*0.75 ; fudge factor
12. if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
13. ;
14. loadct,39
15. def_1color,20,color=’red’
16. plot,[0,1]
17. multi_plot,nrow=4,layout=’large’
18. if !d.name eq ‘X’ then begin
19. window, ysize=800
20. !p.font=-1
21. endif else begin
22. !p.font=0
23. device,/helvetica,/bold,font_size=18
24. endelse
25. ;
26. ; Get regional tree lists and rbar
27. ;
28. restore,filename=’reglists.idlsave’
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
31. ;
32. rawdat=fltarr(4,2000)
33. for i = nreg-1 , nreg-1 do begin
34. fn=’mxd.’+harryfn(i)+’.pa.mean.dat’
35. print,fn
36. openr,1,fn
37. readf,1,rawdat
38. close,1
39. ;
40. densadj=reform(rawdat(2:3,*))
41. ml=where(densadj eq -99.999,nmiss)
42. densadj(ml)=!values.f_nan
43. ;
44. x=reform(rawdat(0,*))
45. kl=where((x ge 1400) and (x le 1992))
46. x=x(kl)
47. densall=densadj(1,kl) ; all bands
48. densadj=densadj(0,kl) ; 2-6 bands
49. ;
50. ; Now normalise w.r.t. 1881-1960
51. ;
52. mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
53. mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
54. ;
55. ; APPLY ARTIFICIAL CORRECTION
56. ;
57. yearlyadj=interpol(valadj,yrloc,x)
58. densall=densall+yearlyadj
59. ;
60. ; Now plot them
61. ;
62. filter_cru,20,tsin=densall,tslow=tslow,/nan
63. cpl_barts,x,densall,title=’Age-banded MXD from all sites’,$
64. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
65. zeroline=tslow,yrange=[-7,3]
66. oplot,x,tslow,thick=3
67. oplot,!x.crange,[0.,0.],linestyle=1
68. ;
69. endfor
70. ;
71. ; Restore the Hugershoff NHD1 (see Nature paper 2)
72. ;
73. xband=x
74. restore,filename=’../tree5/densadj_MEAN.idlsave’
75. ; gets: x,densadj,n,neff
76. ;
77. ; Extract the post 1600 part
78. ;
79. kl=where(x ge 1400)
80. x=x(kl)
81. densadj=densadj(kl)
82. ;
83. ; APPLY ARTIFICIAL CORRECTION
84. ;
85. yearlyadj=interpol(valadj,yrloc,x)
86. densadj=densadj+yearlyadj
87. ;
88. ; Now plot it too
89. ;
90. filter_cru,20,tsin=densadj,tslow=tshug,/nan
91. cpl_barts,x,densadj,title=’Hugershoff-standardised MXD from all sites’,$
92. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
93. zeroline=tshug,yrange=[-7,3],bar_color=20
94. oplot,x,tshug,thick=3,color=20
95. oplot,!x.crange,[0.,0.],linestyle=1
96. ;
97. ; Now overplot their bidecadal components
98. ;
99. plot,xband,tslow,$
100. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
101. yrange=[-6,2],thick=3,title=’Low-pass (20-yr) filtered comparison’
102. oplot,x,tshug,thick=3,color=20
103. oplot,!x.crange,[0.,0.],linestyle=1
104. ;
105. ; Now overplot their 50-yr components
106. ;
107. filter_cru,50,tsin=densadj,tslow=tshug,/nan
108. filter_cru,50,tsin=densall,tslow=tslow,/nan
109. plot,xband,tslow,$
110. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
111. yrange=[-6,2],thick=3,title=’Low-pass (50-yr) filtered comparison’
112. oplot,x,tshug,thick=3,color=20
113. oplot,!x.crange,[0.,0.],linestyle=1
114. ;
115. ; Now compute the full, high and low pass correlations between the two
116. ; series
117. ;
118. perst=1400.
119. peren=1992.
120. ;
121. openw,1,’corr_age2hug.out’
122. thalf=[10.,30.,50.,100.]
123. ntry=n_elements(thalf)
124. printf,1,’Correlations between timeseries’
125. printf,1,’Age-banded vs. Hugershoff-standardised’
126. printf,1,’ Region Full <10 >10 >30 >50 >100′
127. ;
128. kla=where((xband ge perst) and (xband le peren))
129. klh=where((x ge perst) and (x le peren))
130. ts1=densadj(klh)
131. ts2=densall(kla)
132. ;
133. r1=correlate(ts1,ts2)
134. rall=fltarr(ntry)
135. for i = 0 , ntry-1 do begin
136. filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
137. filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
138. if i eq 0 then r2=correlate(tshi1,tshi2)
139. rall(i)=correlate(tslow1,tslow2)
140. endfor
141. ;
142. printf,1,’ALL SITES’,r1,r2,rall,$
143. format='(A11,2X,6F6.2)’
144. ;
145. printf,1,’ ‘
146. printf,1,’Correlations carried out over the period ‘,perst,peren
147. ;
148. close,1
149. ;
150. end
[/sourcecode]
WFT,
“They were only trying to go back a 1000 years or so, so 110 years of data is a fair proportion of the total.”
Which remains wholey irrelevant, given that 40-60 years dont track temps at all. If 25-30% of the instrumented period bears zero or negative relationship to the tree rings, then the premise of a positive tracking relationship is falsified. Tree rings as a proxie are invalidated unless and until that is sorted out appropriately.
TKI,
“I agree on your statement. But one argument was: “the corrected values are not used”.”
That was not my argument. My arguement is that we do not know what, if anything, they were used for, let alone if they were ever used for anything published.
“A little survey of the sources shows that corrected values are used all over the place, not only in briffa_sep98_e.pro.”
Which tells us precisely nothing WRT the above.
“Usually there is a hint not to plot beyond 1960, but for what reason “adjust” this values, if you do not want to use them in any context?”
That is a question. Before making claims, one needs answers.
JJ
woodfortrees (Paul Clark) (15:00:37) :
D: They were only trying to go back a 1000 years or so, so 110 years of data is a fair proportion of the total.
—-
11%? I’m sorry. I still thinks it’s nonsense. I’m a programmer and I would never draw any conclusions after analyzing 11% of the data.
And why are we only going back 1000 years? Am I the only one that finds this odd??
IMPORTANT? I think???a few days back I found the old charts and details for the Aussie temperatures for many years back, now it went missing..BOM is tidying up:-) hmmm?
however the page I downloaded now went to this
ftp://ftp2.bom.gov.au/anon/home/bmrc/perm/climate/temperature/annual/
I downloaded the zips in case , but my pc cannot open? some o the others.
this topic seems to dovetail with the similar story above.
so
somebody? anybody!
feel free to save and examine it all:-)
what I saw before was 3? charts with a suspicious lack of rises in it from places like Koolgardie west Aus, and inland Qld
JJ (17:52:10) :
“I agree on your statement. But one argument was: “the corrected values are not used”.”
“That was not my argument. My arguement is that we do not know what, if anything, they were used for, let alone if they were ever used for anything published.”
Misunderstanding here, sorry for that, I did’nt think, that was your argument.
Many of the sources using adjusted values generate graphs, I think it’s likely a few plots were published. This point needs to be examined.
woodfortrees (Paul Clark) (03:31:29) :
(BTW, I don’t know if the bit about programming video software in C (actually it’s C++) was meant as a jibe about my qualification to comment – if so, fair enough, if that’s all there was. But I have also spent the last two years building and maintaining woodfortrees.org, which gives me some insight into some of the concepts, here, I think. But I don’t claim to be a Real Statistician!)
Absolutely Not! I meant no disrespect. I mentioned about your programming video software in C (C++), because that’s what you said you did. I do know about your woodfortrees, which I use occasionally but don’t always understand.
I was trying to sum up the blog and reason things out to the best of my ability. Thank you for the enlightenment.
However I look at this: “So when I say it’s “reasonable”, what I mean is, it’s a reasonable way to make the best of some data which has known, documented problems in recent years.”, differently. And this is the way I look at it from a simple scientific principle perspective: (also through a glass darkly)
We make a hypothesis – say these bristlecone pine tree ring widths are a measure of the temperature. We measure and compile the data and plot the temperature curve and we find that the temperatures of our proxies diverge from 1960 when we have the instrument records, with these records. Our temperatures go down whereas the thermometer readings go up. To me this would indicate it weakens my hypothesis, and truly speaking we should discard it.
The proper scientific attitude would be not to say hey this data has problems in recent years, but for 900 years previously, when we had nothing to check it with, it was spot on. No it doesn’t agree with the temperature records so our hypothesis is probably wrong and we cant be confident that it represents the temperatures of the previous 900 years accurately.
As it happens we do have something to compare these proxies with for the previous 900 years, and they do not agree with that data. And what is that? 771 studies that show that that the Medieval Warm period did exist, from 458 research institutions in 42 countries which say the Medieval Warm Period did exist, it was widespread across the globe and it was on the average about 0.75 C warmer than today.
See here for the graphic on this: http://wattsupwiththat.com/2009/12/04/jo-nova-finds-the-medieval-warm-period/
Against this we have evidence of a few trees in a grove in the Rockies, which does not agree with the temperature records which says no. This is confirmed and corroborated by another lonely tree in Siberia, with its own peculiar history, supposed to stand proxy for the temperatures of the whole world.
Which would you believe? Which evidence is more credible?
You say these emails do not reveal anything new. The “problem” has already been discussed at length in papers. But discussing the “problem” doesnt make it right. The problems with the problems have also been pointed out at length. The emails may not provide ammunition against AGW but it certainly provides vindication for the concerns voiced at length by the sceptics and frustratingly stonewalled.
Another point why would proxy tree ring data from a grove in the Rockies represent Global temperatures? It would be near nigh miraculous if they did. So even if they represented accurately the temperature history of their grove, (neck of the woods, so to speak), which is dubious as other factors also influence tree rings, would it be logical or meaningful to graft global temperature records onto the end of them?
Nice analysis! This analysis should be used along with this analysis to produce a complete report:
http://69.84.25.250/blogger/post/ClimateGate-Data-Series-Part-5-A-break-down-of-large-data-file-for-manipulating-global-temperature-trends-from-2006-2009.aspx
debreuil:
Please do not denegrate “[snip] science”! …. I studied wildlife biology in college and took a very difficult course in scatology and am quite interested in paleoscatology.
We need to all remember where the “science” of Climatology came from — basically it was spawned as an obscure branch of geography — before AGW it was a total backwater of academia, populated mostly by those who could not endure the rigors of hard scinece. The study of geography has always been at odds with the “hard” sciences — mostly because geography tends to inject a distinctly “liberal” or “social science” aspect to the hard science. My observation is that it is easy to see how Climatologists have little regard for the scientific method (they never really learned it!) and how they are easily politicized and radicalized.
This entire thread is an example of the uninformed leading the uninformed.
Consider:
“valadj” is some sort of filter, plotted in the link. “yrloc” is an array of years, from 1400 to 1994. So the output from this filter, is a TIME adjustment, as suggested by the variable name, “yearlyadj”. It isn’t changing the temperatures at all- contrary to the stupid conclusion of the watts-up author and the sheep piling on.
I repeat: the code has no impact on temperature, it is dealing with time, not temperature, and I suspect a gap between 1400 and 1904 in some data set.
The comment::
“valadj, or, the “fudge factor” array as some arrogant programmer likes to call it is the foundation for the manipulated temperature readings. It contains twenty values of seemingly random numbers. We’ll get back to this later.”
is lunacy, it isn’t operating on temperature data!