Climategate Code Analysis Part 2
There are three common issues that have been raised in my previous post that I would like to officially address concerning the CRU’s source code.
If you only get one thing from this post, please get this. I am only making a statement about the research methods of the CRU and trying to show proof that they had the means and intent to falsify data. And, until the CRU’s research results can be verified by a 3rd party, they cannot be trusted.
Here are the four most frequent concerns dealing with the CRU’s source code:
- The source code that actually printed the graph was commented out and, therefore, is not valid proof.
- No proof exists that shows this code was used in publishing results.
- Interpolation is a normal part of dealing with large data sets, this is no different.
- You need the raw climate data to prove that foul play occurred.
If anyone can think of something I missed, please let me know.
The source code that actually printed the graph was commented out and, therefore, is not valid proof.
Had I done a better job with my source analysis, I would have found a later revision of the briffa_sep98_d.pro source file (linked to in my previous post) contained in a different working tree which shows the fudge-factor array playing a direct result in the (uncommented) plotting of the data.
Snippit from: harris-tree/briffa_sep98_e.pro (see the end of the post for the full source listing)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
filter_cru,20,tsin=densall,tslow=tslow,/nan
cpl_barts,x,densall,title='Age-banded MXD from all sites',$
xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
zeroline=tslow,yrange=[-7,3]
oplot,x,tslow,thick=3
oplot,!x.crange,[0.,0.],linestyle=1
;
Now, we can finally put this concern to rest.
Interpolation is a normal part of dealing with large data sets, this is no different.
This is partially true, the issue doesn’t lie in the fact that the CRU researchers used interpolation. The issue is the weight of the valadj array with respect to the raw data. valadj simply introduces too large of an influence to the original data to do anything productive with it.
Here is the graph I plotted of the valadj array. When we’re talking about trying to interpret temperature data that grows on the scale of one-tenths of a degree over a period of time, “fudging” a value by 2.5 is going to have a significant impact on the data set.
No proof exists that shows this code was used in publishing results.
Correct! That’s why I am (and always have) taken the following stand: Enough proof exists that the CRU had both the means and intent to intentionally falsify data. This means that all of their research results cannot be trusted until they are verified. Period.
The fact that the “fudge-factor” source code exists in the first place is reason enough for alarm. Hopefully, they didn’t use fudged results in the CRU research results, but the truth is, we just don’t know.
You need the raw climate data to prove that foul play occurred.
This is assuming the raw data are valid, which I maintain that it probably is. Several people question the validity of the climate data gathering methods used by the different climate research institutions, but I am not enough of a climate expert to have an opinion one way or the other. Furthermore, It simply doesn’t matter if the raw climate data are correct or not to demonstrate the extreme bias the valadj array forces on the raw data.
So, the raw data could actually be temperature data or corporate sales figures, the result is the same; a severe manipulation of data.
Full Source Listing
As promised, here is the entire source listing for: harris-tree/briffa_sep98_e.pro
[sourcecode language=”text”]
1. ;
2. ; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
3. ; standardised datasets.
4. ; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
5. ; with missing values set appropriately. Uses mxd, and just the
6. ; "all band" timeseries
7. ;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
8. ;
9. yrloc=[1400,findgen(19)*5.+1904]
10. valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
11. 2.6,2.6,2.6]*0.75 ; fudge factor
12. if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
13. ;
14. loadct,39
15. def_1color,20,color=’red’
16. plot,[0,1]
17. multi_plot,nrow=4,layout=’large’
18. if !d.name eq ‘X’ then begin
19. window, ysize=800
20. !p.font=-1
21. endif else begin
22. !p.font=0
23. device,/helvetica,/bold,font_size=18
24. endelse
25. ;
26. ; Get regional tree lists and rbar
27. ;
28. restore,filename=’reglists.idlsave’
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
31. ;
32. rawdat=fltarr(4,2000)
33. for i = nreg-1 , nreg-1 do begin
34. fn=’mxd.’+harryfn(i)+’.pa.mean.dat’
35. print,fn
36. openr,1,fn
37. readf,1,rawdat
38. close,1
39. ;
40. densadj=reform(rawdat(2:3,*))
41. ml=where(densadj eq -99.999,nmiss)
42. densadj(ml)=!values.f_nan
43. ;
44. x=reform(rawdat(0,*))
45. kl=where((x ge 1400) and (x le 1992))
46. x=x(kl)
47. densall=densadj(1,kl) ; all bands
48. densadj=densadj(0,kl) ; 2-6 bands
49. ;
50. ; Now normalise w.r.t. 1881-1960
51. ;
52. mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
53. mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
54. ;
55. ; APPLY ARTIFICIAL CORRECTION
56. ;
57. yearlyadj=interpol(valadj,yrloc,x)
58. densall=densall+yearlyadj
59. ;
60. ; Now plot them
61. ;
62. filter_cru,20,tsin=densall,tslow=tslow,/nan
63. cpl_barts,x,densall,title=’Age-banded MXD from all sites’,$
64. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
65. zeroline=tslow,yrange=[-7,3]
66. oplot,x,tslow,thick=3
67. oplot,!x.crange,[0.,0.],linestyle=1
68. ;
69. endfor
70. ;
71. ; Restore the Hugershoff NHD1 (see Nature paper 2)
72. ;
73. xband=x
74. restore,filename=’../tree5/densadj_MEAN.idlsave’
75. ; gets: x,densadj,n,neff
76. ;
77. ; Extract the post 1600 part
78. ;
79. kl=where(x ge 1400)
80. x=x(kl)
81. densadj=densadj(kl)
82. ;
83. ; APPLY ARTIFICIAL CORRECTION
84. ;
85. yearlyadj=interpol(valadj,yrloc,x)
86. densadj=densadj+yearlyadj
87. ;
88. ; Now plot it too
89. ;
90. filter_cru,20,tsin=densadj,tslow=tshug,/nan
91. cpl_barts,x,densadj,title=’Hugershoff-standardised MXD from all sites’,$
92. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
93. zeroline=tshug,yrange=[-7,3],bar_color=20
94. oplot,x,tshug,thick=3,color=20
95. oplot,!x.crange,[0.,0.],linestyle=1
96. ;
97. ; Now overplot their bidecadal components
98. ;
99. plot,xband,tslow,$
100. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
101. yrange=[-6,2],thick=3,title=’Low-pass (20-yr) filtered comparison’
102. oplot,x,tshug,thick=3,color=20
103. oplot,!x.crange,[0.,0.],linestyle=1
104. ;
105. ; Now overplot their 50-yr components
106. ;
107. filter_cru,50,tsin=densadj,tslow=tshug,/nan
108. filter_cru,50,tsin=densall,tslow=tslow,/nan
109. plot,xband,tslow,$
110. xrange=[1399.5,1994.5],xtitle=’Year’,/xstyle,$
111. yrange=[-6,2],thick=3,title=’Low-pass (50-yr) filtered comparison’
112. oplot,x,tshug,thick=3,color=20
113. oplot,!x.crange,[0.,0.],linestyle=1
114. ;
115. ; Now compute the full, high and low pass correlations between the two
116. ; series
117. ;
118. perst=1400.
119. peren=1992.
120. ;
121. openw,1,’corr_age2hug.out’
122. thalf=[10.,30.,50.,100.]
123. ntry=n_elements(thalf)
124. printf,1,’Correlations between timeseries’
125. printf,1,’Age-banded vs. Hugershoff-standardised’
126. printf,1,’ Region Full <10 >10 >30 >50 >100′
127. ;
128. kla=where((xband ge perst) and (xband le peren))
129. klh=where((x ge perst) and (x le peren))
130. ts1=densadj(klh)
131. ts2=densall(kla)
132. ;
133. r1=correlate(ts1,ts2)
134. rall=fltarr(ntry)
135. for i = 0 , ntry-1 do begin
136. filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
137. filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
138. if i eq 0 then r2=correlate(tshi1,tshi2)
139. rall(i)=correlate(tslow1,tslow2)
140. endfor
141. ;
142. printf,1,’ALL SITES’,r1,r2,rall,$
143. format='(A11,2X,6F6.2)’
144. ;
145. printf,1,’ ‘
146. printf,1,’Correlations carried out over the period ‘,perst,peren
147. ;
148. close,1
149. ;
150. end
[/sourcecode]
Deliberate,calculated,and possibly incompetent while doing so.
This is indeed the smoking gun,er, code…
“No proof exists that shows this code was used in publishing results.”
If we run the program and it produces the results of any of their publish graphs, that is proof enough that the program was used.
Here’s another take on some of the code bugs, etc.:
http://bishophill.squarespace.com/blog/2009/12/6/good-code-analysis.html
Robert, can you produce the figures from the contemporary code? I can see if they match the relevant article.
AlanG (01:01:07) :
harrfn (harry filename?) is an array of data filenames found here:
29. harryfn=[‘nwcan’,’wnam’,’cecan’,’nweur’,’sweur’,’nsib’,’csib’,’tib’,$
30. ‘esib’,’allsites’]
My guess is they correspond as follows:
nwcan = North West Canada
wnam = Western North America
cecan = Central Canada
nweur = North West Europe
sweur = South West Europe
cs and es could be ISO country codes but ns and ti are not.
nsib = Northern Siberia?
csib = Central Siberia?
tib = ??
esib = Eastern Siberia?
tib = Tibet?
Al Gore heckled on Youtube…
It may actually take this kind of protest to wake up the sheeple.
What I see here is some young frustrated skeptics resorting to making a protest on amateur video and publishing it on Youtube in response to the blatant fascist propaganda being pumped out by mainstream western media.
Make no mistake about it, when deniers are vilified and attacked, when scientists are prevented from publishing opposing data in journals, when the media ruthlessly shuts down and covers up all opposing views, when the media continues publishing pro-anthropogenic stories daily, when skeptics are called a*ssholes on national television (and the BBC allows it to go on the air …oops a mistake…sure, and I was born yesterday):
We are dealing with rampant FASCISM.
It would all be so funny if it was on a Monty Python show but the sad thing is that this is REALLY happening.
The proof is that like all FASCISTS – the people swept up in this movement are totally absolutely convinced – they have no compunction about folliing the “end justifies the means” path. What is amazing is that these are mostly “liberals” who are orchestrating mass media manipulation and suppression of free speech (yes when I find countless posts MODERATED out of existence and a complete refusal to debate Climate Change, I think we can call this suppression of free speech). Since when did “liberals” turn to the dark side of tyranny and dictatorship – we know what is best for you – so “Shutup” as Dr Watson at the CRU put it!
Regardless of what the raw data might show, hiding it IS foul play. They are a government agency and subject to FOIA requests legally which they have ignored. I say prosecute them and lock them up. They are asking us to spend fortunes without providing the raw data!!!
OT
Just seen in the Daily Mail – “Emalis that rocked climate change campaign leaked from Siberian ‘closed city’ university built by KGB”
Read more:
http://www.dailymail.co.uk/news/article-1233562/Emails-rocked-climate-change-campaign-leaked-Siberian-closed-city-university-built-KGB.html
REPLY:
Now all we need are the street videos showing a furtive figure scurrying from CRU, hopping on his bike and pedal madly the hundred miles to the Russian Embassy in London.
Al Gore heckled on Youtube…
It may actually take this kind of protest to wake up the sheeple.
What I see here is some young frustrated skeptics resorting to making a protest on amateur video and publishing it on Youtube in response to the blatant fascist propaganda being pumped out by mainstream western media.
Make no mistake about it, when deniers are vilified and attacked, when scientists are prevented from publishing opposing data in journals, when the media ruthlessly shuts down and covers up all opposing views, when the media continues publishing pro-anthropogenic stories daily, when skeptics are called a*ssholes on national television (and the BBC allows it to go on the air …oops a mistake…sure, and I was born yesterday):
We are dealing with rampant FASCISM.
It would all be so funny if it was on a Monty Python show but the sad thing is that this is REALLY happening.
The proof is – like all FASCISTS – the people swept up in this movement are totally absolutely convinced – they have no compunction about following the “end justifies the means” path. What is amazing is that these are mostly “liberals” who are orchestrating mass media manipulation and suppression of free speech (yes when I find countless posts MODERATED out of existence and a complete refusal to debate Climate Change, I think we can call this suppression of free speech). Since when did “liberals” turn to the dark side of tyranny and dictatorship – we know what is best for you – so “Shutup” as Dr Watson at the CRU put it!
Carsten Arnholm, Norway (04:08:46)
“………
nsib = Northern Siberia?
csib = Central Siberia?
tib = ??
esib = Eastern Siberia?”
tib = Tibet ?
Wood for trees
Thank you for you level headedness where all else is hype.
I think there should be a heat map displayed on any temperature graph that indicated the % of data that has been “interpolated”, i.e. “made up out of nothing” … a second version of this could show how many station records where used for each years plot …
This would make the clear the amount of uncertainty that truely exists in the study of climate data …
This quote from the BBC is worth keeping:
”The IPCC relied on three documents to arrive at 2035 as the “outer year” for shrinkage of glaciers.
They are: a 2005 World Wide Fund for Nature report on glaciers; a 1996 Unesco document on hydrology; and a 1999 news report in New Scientist.
Incidentally, none of these documents have been reviewed by peer professionals, which is what the IPCC is mandated to be doing…..
But in a joint statement some the world’s leading glaciologists who are also participants to the IPCC have said: “This catalogue of errors in Himalayan glaciology… has caused much confusion that could have been avoided had the norms of scientific publication, including peer review and concentration upon peer-reviewed work, been respected.”…
“Under strict consideration of the IPCC rules, it should actually not have been published as it is not based on a sound scientific reference.
http://news.bbc.co.uk/2/hi/south_asia/8387737.stm
Lets see they screwed up the reports of melting glaciers, the polar bear population is increasing not decreasing, and the CRU temperature data is extremely suspect and certainly does not meet “strict consideration of the” scientific method.
Yup, the science is settled, it is settling in to a pool of quick sand, or is that a peat bog?
Bump:
P Gosselin (04:24:36) :
Marc Sheppard explains THE TRICK
http://www.americanthinker.com/2009/12/understanding_climategates_hid.html
This is where the focus has to be, and thr FOIA violations.
Indeed, as a Software Developer that is my experience too. I’ve seen the most amazingly tortuous (in terms of cyclometric complexity) code come from the smartest people.
When someone claims: “You need the raw climate data to prove that foul play occurred”, they misunderstand science. If something cannot be independently be reproduced, it is mere opinion. Only reproducible results are science. In light of the published source code showing manipulation of the now-missing raw data, honesty demands that any research based on the data published by this group, MUST BE SUSPECT and the burden of proof lays on them to substantiate the basis for any claims or projections they make.
If their work is reproducible, this should not be a problem.
wood for trees:
Mann’s nature trick of “hiding the decline” signifies the appropriate method to cover up the declining temperature data.
Its no less hard to imagine than calibrating satellite data in line with ground stations when they didn’t show a warming trend. If anything they should have been calibrated downwards – if satellites drift, they go closer to earth and read higher temperatures. The official explanation is exactly the opposite of the truth.
Keep it simple.
You are CO2.
You are the danger.
You need to be deleted.
They want you to self delete.
Will you self delete?
The way these closed circle of statisitcs manipulators (I can’t call them climatologists) corrupt and delete/manipulate data then foment excuses about it is like those accounting procedures at Enron.
“REPLY:
Now all we need are the street videos showing a furtive figure scurrying from CRU, hopping on his bike and pedal madly the hundred miles to the Russian Embassy in London.”
These days, I find “Pravda” more truthful than say “The Economist.”
Whence this madness?
Who cares about code, emails, etc. when a really good time is to be had in Copenhagen? Biggest block party ever. Open bar, limo’s, free hookers, hash, caviar, etc. 1200+ limos, of which 5 are electric and hybrid. Can you spell hypocrisy?
http://www.telegraph.co.uk/earth/copenhagen-climate-change-confe/6736517/Copenhagen-climate-summit-1200-limos-140-private-planes-and-caviar-wedges.html
osborn-tree6\README
06/26/1998
MXD and TRW data, optimised regions (best of Harry’s, decline
and volcano regions), for
(1) comparison with T & P
(2) regional volcanic responses
(3) regional decline
rd_allmxd1.pro reads in raw MXD data ready for use
latlon.pro used to convert text to numeric
pl_mxdlocations.pro plots location of MXD sites
mkregions.pro allocates MXD sites into defined regions
Debunking Briffa’s Version of the Hockey Stick
Jim Lindgren • December 6, 2009 12:18 am
Among the most ethically challenged of the scientists at the Climate Research Unit at the University of East Anglia is Keith Briffa. A couple of months ago, the Bishop Hill blog retold in detail the sad story about Briffa’s own version of the Hockey Stick, which he was able to keep alive by his attempts to prevent other scientists from discovering what he had done with the data, a practice facilitated by biased journals that refused to apply their own rules.
http://volokh.com/
Jeremy (07:25:22) :
“ What is amazing is that these are mostly “liberals” who are orchestrating mass media manipulation and suppression of free speech (yes when I find countless posts MODERATED out of existence and a complete refusal to debate Climate Change, I think we can call this suppression of free speech). Since when did “liberals” turn to the dark side of tyranny and dictatorship – we know what is best for you – so “Shutup” as Dr Watson at the CRU put it!”
Thank you Jeremy, this is what I’ve been trying to say for awhile. I’ve caught much grief here on this scientific site for labeling “left or lib” as a political attack. Forget left vs. right, let’s just look around the liberal left side of the spectrum (where this movement was born and nurtured) and ask –
“why suppress the debate on this subject for over a decade?” “why turn your back on the current Climategate issue”
Pamela – it not left vs. right, it’s what the heck happened amongst the lib left to create an atmosphere that condones this type conduct ?
Left, right, green, blue – none of that matters – since when is it ok to turn a blind eye (MSM) on a worldwide trillion dollar scam that will truly hurt people and economies ?
http://www.woodfortrees.org/plot/hadcrut3vgl/from:1979/to/plot/hadcrut3vgl/from:1979/to/trend/plot/uah/from:1979/to/plot/uah/from:1979/to/trend/plot/rss/from:1979/to/plot/rss/from:1979/to/trend
Why would CRU need to fudge lines of code if their temperature record since 1979 shows warming that is fairly consistent with RSS and UAH. Certainly skeptics Spencer and Christy are not using CRU code.
Here is a 9 minute video that shows how ridiculous the Climate Gate emails really are:
http://www.desmogblog.com/another-look-stolen-emails
As I said when this story first broke, the wolves were given a fake piece of meat.