WUWT blogging ally Ecotretas writes in to say that he has made a compendium of programming code segments that show comments by the programmer that suggest places where data may be corrected, modified, adjusted, or busted. Some the HARRY_READ_ME comments are quite revealing. For those that don’t understand computer programming, don’t fret, the comments by the programmer tell the story quite well even if the code itself makes no sense to you.
To say that the CRU code might be “buggy” would be…well I’ll just let CRU’s programmer tell you in his own words.
- FOIA\documents\osborn-tree6\mann\oldprog\maps12.proFOIA\documents\osborn-tree6\mann\oldprog\maps15.proFOIA\documents\osborn-tree6\mann\oldprog\maps24.pro
; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions; of growing season temperatures. Uses "corrected" MXD - but shouldn't usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
- FOIA\documents\harris-tree\recon_esper.pro
; Computes regressions on full, high and low pass Esper et al. (2002) series,; anomalies against full NH temperatures and other series.
; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N
;
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline
- FOIA\documents\harris-tree\calibrate_nhrecon.pro
;; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline that affects tree-ring density records)
;
- FOIA\documents\harris-tree\recon1.pro
FOIA\documents\harris-tree\recon2.proFOIA\documents\harris-tree\recon_jones.pro
;; Specify period over which to compute the regressions (stop in 1940 to avoid
; the decline
;
- FOIA\documents\HARRY_READ_ME.txt
17. Inserted debug statements into anomdtb.f90, discovered thata sum-of-squared variable is becoming very, very negative! Key
output from the debug statements:
(..)
forrtl: error (75): floating point exception
IOT trap (core dumped)
..so the data value is unbfeasibly large, but why does the
sum-of-squares parameter OpTotSq go negative?!!
- FOIA\documents\HARRY_READ_ME.txt
22. Right, time to stop pussyfooting around the niceties of Tim's labyrinthine softwaresuites - let's have a go at producing CRU TS 3.0! since failing to do that will be the
definitive failure of the entire project..
- FOIA\documents\HARRY_READ_ME.txt
getting seriously fed up with the state of the Australian data. so many new stations have beenintroduced, so many false references.. so many changes that aren't documented.
Every time acloud forms I'm presented with a bewildering selection of similar-sounding sites, some with
references, some with WMO codes, and some with both. And if I look up the station metadata with
one of the local references, chances are the WMO code will be wrong (another station will have
it) and the lat/lon will be wrong too.
- FOIA\documents\HARRY_READ_ME.txt
I am very sorry to report that the rest of the databases seem to be in nearly as poor a state asAustralia was. There are hundreds if not thousands of pairs of dummy stations, one with no WMO
and one with, usually overlapping and with the same station name and very similar coordinates. I
know it could be old and new stations, but why such large overlaps if that's the case? Aarrggghhh!
There truly is no end in sight.
- FOIA\documents\HARRY_READ_ME.txt
28. With huge reluctance, I have dived into 'anomdtb' - and already I havethat familiar Twilight Zone sensation.
- FOIA\documents\HARRY_READ_ME.txt
Wrote 'makedtr.for' to tackle the thorny problem of the tmin and tmax databases notbeing kept in step. Sounds familiar, if worrying. am I the first person to attempt
to get the CRU databases in working order?!!
- FOIA\documents\HARRY_READ_ME.txt
Well, dtr2cld is not the world's most complicated program. Wheras cloudreg is, and Iimmediately found a mistake!
Scanning forward to 1951 was done with a loop that, forcompletely unfathomable reasons, didn't include months! So we read 50 grids instead
of 600!!!
That may have had something to do with it. I also noticed, as I was correctingTHAT, that I reopened the DTR and CLD data files when I should have been opening the
bloody station files!!
- FOIA\documents\HARRY_READ_ME.txt
Back to the gridding. I am seriously worried that our flagship gridded data product is produced byDelaunay triangulation - apparently linear as well. As far as I can see, this renders the station
counts totally meaningless. It also means that we cannot say exactly how the gridded data is arrived
at from a statistical perspective - since we're using an off-the-shelf product that isn't documented
sufficiently to say that. Why this wasn't coded up in Fortran I don't know - time pressures perhaps?
Was too much effort expended on homogenisation, that there wasn't enough time to write a gridding
procedure? Of course, it's too late for me to fix it too. Meh.
- FOIA\documents\HARRY_READ_ME.txt
Here, the expected 1990-2003 period is MISSING - so the correlations aren't so hot! Yetthe WMO codes and station names /locations are identical (or close). What the hell is
supposed to happen here? Oh yeah - there is no 'supposed', I can make it up. So I have :-)
- FOIA\documents\HARRY_READ_ME.txt
Well, it's been a real day of revelations, never mind the week. This morning Idiscovered that proper angular weighted interpolation was coded into the IDL
routine, but that its use was discouraged because it was slow! Aaarrrgghh.
There is even an option to tri-grid at 0.1 degree resolution and then 'rebin'
to 720x360 - also deprecated! And now, just before midnight (so it counts!),
having gone back to the tmin/tmax work, I've found that most if not all of the
Australian bulletin stations have been unceremoniously dumped into the files
without the briefest check for existing stations.
- FOIA\documents\HARRY_READ_ME.txt
As we can see, even I'm cocking it up! Though recoverably. DTR, TMN and TMX need to be written as (i7.7)./code> - FOIA\documents\HARRY_READ_ME.txt
OH FUCK THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'mhitting yet another problem that's based on the hopeless state of our databases. There is no uniform
data integrity, it's just a catalogue of issues that continues to grow as they're found.
- FOIA\documents\osborn-tree6\mann\mxdgrid2ascii.pro
printf,1,’Osborn et al. (2004) gridded reconstruction of warm-season’printf,1,’(April-September) temperature anomalies (from the 1961-1990 mean).’
printf,1,’Reconstruction is based on tree-ring density records.’
printf,1
printf,1,’NOTE: recent decline in tree-ring density has been ARTIFICIALLY’
printf,1,’REMOVED to facilitate calibration. THEREFORE, post-1960 values’
printf,1,’will be much closer to observed temperatures then they should be,’
printf,1,’which will incorrectly imply the reconstruction is more skilful’
printf,1,’than it actually is. See Osborn et al. (2004).’
- FOIA\documents\osborn-tree6\summer_modes\data4sweden.pro
FOIA\documents\osborn-tree6\summer_modes\data4sweden.pro
printf,1,'IMPORTANT NOTE:'printf,1,'The data after 1960 should not be used. The tree-ring density'
printf,1,'records tend to show a decline after 1960 relative to the summer'
printf,1,'temperature in many high-latitude locations. In this data set'
printf,1,'this "decline" has been artificially removed in an ad-hoc way, and'
printf,1,'this means that data after 1960 no longer represent tree-ring
printf,1,'density variations, but have been modified to look more like the
printf,1,'observed temperatures.'
- FOIA\documents\osborn-tree6\combined_wavelet_col.pro
;; Remove missing data from start & end (end in 1960 due to decline)
;
kl=where((yrmxd ge 1402) and (yrmxd le 1960),n)
sst=prednh(kl)
- FOIA\documents\osborn-tree6\mann\mxd_pcr_localtemp.pro
; Tries to reconstruct Apr-Sep temperatures, on a box-by-box basis, from the; EOFs of the MXD data set. This is PCR, although PCs are used as predictors
; but not as predictands. This PCR-infilling must be done for a number of
; periods, with different EOFs for each period (due to different spatial
; coverage). *BUT* don’t do special PCR for the modern period (post-1976),
; since they won’t be used due to the decline/correction problem.
; Certain boxes that appear to reconstruct well are “manually” removed because
; they are isolated and away from any trees.
- FOIA\documents\osborn-tree6\briffa_sep98_d.pro
;mknormal,yyy,timey,refperiod=[1881,1940];
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
(...)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
- FOIA\documents\osborn-tree6\summer_modes\pl_decline.pro
;; Plots density ‘decline’ as a time series of the difference between
; temperature and density averaged over the region north of 50N,
; and an associated pattern in the difference field.
; The difference data set is computed using only boxes and years with
; both temperature and density in them – i.e., the grid changes in time.
; The pattern is computed by correlating and regressing the *filtered*
; time series against the unfiltered (or filtered) difference data set.
;
;*** MUST ALTER FUNCT_DECLINE.PRO TO MATCH THE COORDINATES OF THE
; START OF THE DECLINE *** ALTER THIS EVERY TIME YOU CHANGE ANYTHING ***
- FOIA\documents\osborn-tree6\mann\oldprog\maps12.pro
;; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions
; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
;
- FOIA\documents\osborn-tree6\mann\oldprog\calibrate_correctmxd.pro
; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which
; gives a zero mean over 1881-1960) after extending the calibration to boxes
; without temperature data (pl_calibmxd1.pro). We have identified and
; artificially removed (i.e. corrected) the decline in this calibrated
; data set. We now recalibrate this corrected calibrated dataset against
; the unfiltered 1911-1990 temperature data, and apply the same calibration
; to the corrected and uncorrected calibrated MXD data.
- FOIA\documents\osborn-tree6\summer_modes\calibrate_correctmxd.pro
; No need to verify the correct and uncorrected versions, since these; should be identical prior to 1920 or 1930 or whenever the decline
; was corrected onwards from.
- FOIA\documents\osborn-tree5\densplus188119602netcdf.pro
; we know the file starts at yr 440, but we want nothing till 1400, so we; can skill lines (1400-440)/10 + 1 header line
; we now want all lines (10 yr per line) from 1400 to 1980, which is
; (1980-1400)/10 + 1 lines
(...)
; we know the file starts at yr 1070, but we want nothing till 1400, so we
; can skill lines (1400-1070)/10 + 1 header line
; we now want all lines (10 yr per line) from 1400 to 1991, which is
; (1990-1400)/10 + 1 lines (since 1991 is on line beginning 1990)
- FOIA\documents\osborn-tree6\mann\oldprog\maps12.pro
FOIA\documents\osborn-tree6\mann\oldprog\maps15.pro
FOIA\documents\osborn-tree6\mann\oldprog\maps24.pro
; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions; of growing season temperatures. Uses "corrected" MXD - but shouldn't usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
- FOIA\documents\harris-tree\recon_esper.pro
; Computes regressions on full, high and low pass Esper et al. (2002) series,; anomalies against full NH temperatures and other series.
; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N
;
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline
- FOIA\documents\harris-tree\calibrate_nhrecon.pro
;; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline that affects tree-ring density records)
;
- FOIA\documents\harris-tree\recon1.pro
FOIA\documents\harris-tree\recon2.proFOIA\documents\harris-tree\recon_jones.pro
;; Specify period over which to compute the regressions (stop in 1940 to avoid
; the decline
;
- FOIA\documents\HARRY_READ_ME.txt
17. Inserted debug statements into anomdtb.f90, discovered thata sum-of-squared variable is becoming very, very negative! Key
output from the debug statements:
(..)
forrtl: error (75): floating point exception
IOT trap (core dumped)
..so the data value is unbfeasibly large, but why does the
sum-of-squares parameter OpTotSq go negative?!!
- FOIA\documents\HARRY_READ_ME.txt
22. Right, time to stop pussyfooting around the niceties of Tim's labyrinthine softwaresuites - let's have a go at producing CRU TS 3.0! since failing to do that will be the
definitive failure of the entire project..
- FOIA\documents\HARRY_READ_ME.txt
getting seriously fed up with the state of the Australian data. so many new stations have beenintroduced, so many false references.. so many changes that aren't documented. Every time a
cloud forms I'm presented with a bewildering selection of similar-sounding sites, some with
references, some with WMO codes, and some with both. And if I look up the station metadata with
one of the local references, chances are the WMO code will be wrong (another station will have
it) and the lat/lon will be wrong too.
- FOIA\documents\HARRY_READ_ME.txt
I am very sorry to report that the rest of the databases seem to be in nearly as poor a state asAustralia was. There are hundreds if not thousands of pairs of dummy stations, one with no WMO
and one with, usually overlapping and with the same station name and very similar coordinates. I
know it could be old and new stations, but why such large overlaps if that's the case? Aarrggghhh!
There truly is no end in sight.
- FOIA\documents\HARRY_READ_ME.txt
28. With huge reluctance, I have dived into 'anomdtb' - and already I havethat familiar Twilight Zone sensation.
- FOIA\documents\HARRY_READ_ME.txt
Wrote 'makedtr.for' to tackle the thorny problem of the tmin and tmax databases notbeing kept in step. Sounds familiar, if worrying. am I the first person to attempt
to get the CRU databases in working order?!!
- FOIA\documents\HARRY_READ_ME.txt
Well, dtr2cld is not the world's most complicated program. Wheras cloudreg is, and Iimmediately found a mistake! Scanning forward to 1951 was done with a loop that, for
completely unfathomable reasons, didn't include months! So we read 50 grids instead
of 600!!! That may have had something to do with it. I also noticed, as I was correcting
THAT, that I reopened the DTR and CLD data files when I should have been opening the
bloody station files!!
- FOIA\documents\HARRY_READ_ME.txt
Back to the gridding. I am seriously worried that our flagship gridded data product is produced byDelaunay triangulation - apparently linear as well. As far as I can see, this renders the station
counts totally meaningless. It also means that we cannot say exactly how the gridded data is arrived
at from a statistical perspective - since we're using an off-the-shelf product that isn't documented
sufficiently to say that. Why this wasn't coded up in Fortran I don't know - time pressures perhaps?
Was too much effort expended on homogenisation, that there wasn't enough time to write a gridding
procedure? Of course, it's too late for me to fix it too. Meh.
- FOIA\documents\HARRY_READ_ME.txt
Here, the expected 1990-2003 period is MISSING - so the correlations aren't so hot! Yetthe WMO codes and station names /locations are identical (or close). What the hell is
supposed to happen here? Oh yeah - there is no 'supposed', I can make it up. So I have :-)
- FOIA\documents\HARRY_READ_ME.txt
Well, it's been a real day of revelations, never mind the week. This morning Idiscovered that proper angular weighted interpolation was coded into the IDL
routine, but that its use was discouraged because it was slow! Aaarrrgghh.
There is even an option to tri-grid at 0.1 degree resolution and then 'rebin'
to 720x360 - also deprecated! And now, just before midnight (so it counts!),
having gone back to the tmin/tmax work, I've found that most if not all of the
Australian bulletin stations have been unceremoniously dumped into the files
without the briefest check for existing stations.
- FOIA\documents\HARRY_READ_ME.txt
As we can see, even I'm cocking it up! Though recoverably. DTR, TMN and TMX need to be written as (i7.7)./code> - FOIA\documents\HARRY_READ_ME.txt
OH FUCK THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'mhitting yet another problem that's based on the hopeless state of our databases. There is no uniform
data integrity, it's just a catalogue of issues that continues to grow as they're found.
- FOIA\documents\osborn-tree6\mann\mxdgrid2ascii.pro
printf,1,’Osborn et al. (2004) gridded reconstruction of warm-season’printf,1,’(April-September) temperature anomalies (from the 1961-1990 mean).’
printf,1,’Reconstruction is based on tree-ring density records.’
printf,1
printf,1,’NOTE: recent decline in tree-ring density has been ARTIFICIALLY’
printf,1,’REMOVED to facilitate calibration. THEREFORE, post-1960 values’
printf,1,’will be much closer to observed temperatures then they should be,’
printf,1,’which will incorrectly imply the reconstruction is more skilful’
printf,1,’than it actually is. See Osborn et al. (2004).’
- FOIA\documents\osborn-tree6\summer_modes\data4sweden.pro
FOIA\documents\osborn-tree6\summer_modes\data4sweden.pro
printf,1,'IMPORTANT NOTE:'printf,1,'The data after 1960 should not be used. The tree-ring density'
printf,1,'records tend to show a decline after 1960 relative to the summer'
printf,1,'temperature in many high-latitude locations. In this data set'
printf,1,'this "decline" has been artificially removed in an ad-hoc way, and'
printf,1,'this means that data after 1960 no longer represent tree-ring
printf,1,'density variations, but have been modified to look more like the
printf,1,'observed temperatures.'
- FOIA\documents\osborn-tree6\combined_wavelet_col.pro
;; Remove missing data from start & end (end in 1960 due to decline)
;
kl=where((yrmxd ge 1402) and (yrmxd le 1960),n)
sst=prednh(kl)
- FOIA\documents\osborn-tree6\mann\mxd_pcr_localtemp.pro
; Tries to reconstruct Apr-Sep temperatures, on a box-by-box basis, from the; EOFs of the MXD data set. This is PCR, although PCs are used as predictors
; but not as predictands. This PCR-infilling must be done for a number of
; periods, with different EOFs for each period (due to different spatial
; coverage). *BUT* don’t do special PCR for the modern period (post-1976),
; since they won’t be used due to the decline/correction problem.
; Certain boxes that appear to reconstruct well are “manually” removed because
; they are isolated and away from any trees.
- FOIA\documents\osborn-tree6\briffa_sep98_d.pro
;mknormal,yyy,timey,refperiod=[1881,1940];
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
(...)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
- FOIA\documents\osborn-tree6\summer_modes\pl_decline.pro
;; Plots density ‘decline’ as a time series of the difference between
; temperature and density averaged over the region north of 50N,
; and an associated pattern in the difference field.
; The difference data set is computed using only boxes and years with
; both temperature and density in them – i.e., the grid changes in time.
; The pattern is computed by correlating and regressing the *filtered*
; time series against the unfiltered (or filtered) difference data set.
;
;*** MUST ALTER FUNCT_DECLINE.PRO TO MATCH THE COORDINATES OF THE
; START OF THE DECLINE *** ALTER THIS EVERY TIME YOU CHANGE ANYTHING ***
- FOIA\documents\osborn-tree6\mann\oldprog\maps12.pro
;; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions
; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
;
- FOIA\documents\osborn-tree6\mann\oldprog\calibrate_correctmxd.pro
; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which
; gives a zero mean over 1881-1960) after extending the calibration to boxes
; without temperature data (pl_calibmxd1.pro). We have identified and
; artificially removed (i.e. corrected) the decline in this calibrated
; data set. We now recalibrate this corrected calibrated dataset against
; the unfiltered 1911-1990 temperature data, and apply the same calibration
; to the corrected and uncorrected calibrated MXD data.
- FOIA\documents\osborn-tree6\summer_modes\calibrate_correctmxd.pro
; No need to verify the correct and uncorrected versions, since these; should be identical prior to 1920 or 1930 or whenever the decline
; was corrected onwards from.
- FOIA\documents\osborn-tree5\densplus188119602netcdf.pro
; we know the file starts at yr 440, but we want nothing till 1400, so we; can skill lines (1400-440)/10 + 1 header line
; we now want all lines (10 yr per line) from 1400 to 1980, which is
; (1980-1400)/10 + 1 lines
(...)
; we know the file starts at yr 1070, but we want nothing till 1400, so we
; can skill lines (1400-1070)/10 + 1 header line
; we now want all lines (10 yr per line) from 1400 to 1991, which is
; (1990-1400)/10 + 1 lines (since 1991 is on line beginning 1990)
Sponsored IT training links:
Join 70-291 training program to pass 642-446 test plus get free practice files for next 70-643 exam.
Back to the gridding. I am seriously worried that our flagship gridded data product is produced by Delaunay triangulation – apparently linear as well. As far as I can see, this renders the station…
Well, this brings me back to a question I asked here a long time ago: how is average global temperature calculated? Somebody replied, “It’s called gridding.”
This happens to be something I know a bit about through my GIS work. Delauney triangulation (TIN production) would be the last thing I would expect them to use for this sort of a model. Anyway, one must peek into the code to see how the sausage is made, no?
Think about all those attractive colorful thematic maps of the globe, all gridded…
It’s actually quite entertaining to read the programmers notes while playing M4GW’s “Hide the Decline (hide the decline), in the background.
….. it stops you from crying.
Will there be a Congressional Investigation?
Comment by JamesD
Wednesday, 25 Nov 09 @ur momisugly 12:24 PM
There will be no investigations with a Democrat congress. No committee chairman will allow it. I hope some of the heavyweight scientists will come to the defense of scientific integrity but I’m not holding my breath. For a long time this naked emperor has been parading down the street. The press and, particularly, the educated public all agree, he’s wearing a fine suit of clothes.
http://carboneconomy.economist.com/
http://carboneconomy.economist.com/content/programme
Steve Gavin at RealClimate has refused more than 6 times to post the following message, are you willing to present this very important point for me?
JCS says:
Your comment is awaiting moderation.
25 November 2009 at 12:58 PM
Gavin,
I have repeatedly tried to post comments on your website asking you to respond to the following statement:
If the first rule of science is to question everything, and another fundamental rule is that no hypothesis can be proven true, regardless of imposing the precautionary principle, why is the first rule and another fundamental rule being discarded, and any SCIENTIST (SKEPTIC) vilified or or censored for practicing what can only be considered good science.
I am a Climate scientist with a degree in applied science wildlife biology and a masters in climate change and sustainability and I am not convinced by anything I have read, seen, studied or experimented that there is a definitive correlation between CO2/Greenhouse gases and climate variability.
Also I have read much of this information from this hack/screw up/whatever and I clearly see what i would consider malpractice and unethical collusion. Particularly in the case where advice is given in ways to avoid taxes, data has been significantly fudged, code is manipulated and peer review process stacked (more so than usual).
I believe in Climate Variability, I also believe that humans as a whole cause irreparable environmental harm to the planet, however I am skeptical of the hypothesis that is Anthropogenic Climate Change and believe that more research and a more open and debatable approach needs to be undertaken to achieve real results in understanding this. Why is this wrong and why are so many other skeptics with the same opinion vilified and persecuted. Why have you censored more than 6 previous posts I attempted to put up on this topic.
Can you not see how this topic risks the credibility of science as a whole!!!!!!
Eric (09:40:29) :
In fairness that is too general a statement. It is important to be precise and specific, otherwise folks at RealClimate who actually really know their stuff will simply rip you to shreds. Certain critical pieces of station data have been requested. Certain pieces of code have been requested. More generally, authors of cliamte science research papers have been asked to post their raw data and their code in a way that will allow a complete replication of their results by interested third parties. It is the Institutional and individual refusal to do these simple things that has caused the questioning of the motives of climate scientists in general.
I don’t have to be a programmer to understand this:
The tree-ring density’
printf,1,’records tend to show a decline after 1960 relative to the summer’
printf,1,’temperature in many high-latitude locations. In this data set’
printf,1,’this “decline” has been artificially removed in an ad-hoc way, and’
printf,1,’this means that data after 1960 no longer represent tree-ring
printf,1,’density variations, but have been modified to look more like the
printf,1,’observed temperatures.’
they have committed fraud. plain and simple.
meanwhile, I note that Washington Post and WSJ have picked up the story, reporting not “it’s taken out of context,” but rather “it’s starting too look like they manufactured the data.” There is yet hope that MSM will take it up and run with it.
for, even though MSM is firmly in the AGW camp, they will not be able to resist a juicy expose. Juicy makes ratings and sells advertising. They will ignore that they have been made fools.
How do know that the HARRY_READ_ME.txt file and the data he’s working on represents the official temperature ouput of CRU?
Maybe he is put to work on a special dataset tha has been corrupted somehow?
Surly there is more than only 1 guy doing this stuff at CRU??
Completely missed the point; these were entered/typed up AS the code was being written/debugged/maintained/retrofitted.
It is almost obvious you have never ‘coded’.
.
.
Here is the output of some code from a Briffa related file. If you run a flat temperature graph through it if gives a Hockey Stick. If you run an inconveniently divergent tree ring graph through it, it acts as a trick to hide the decline.
http://i49.tinypic.com/m9vcxv.jpg
In reading through this … stuff … I found my mind seemed to be stuck in a goto loop, continuously replaying that YouTube video — “Hide the decline … hide the decline”.
And then I come to this gem —
What the hell is supposed to happen here? Oh yeah – there is no ‘supposed’, I can make it up. So I have 🙂
This is what happens when you approach a problem with a preconceived belief system in place describing what’s happening. Data doesn’t conform to your belief? No problem – fix it so the problem goes away.
Sounds like they have some real PEBKAC errors over there. That is Problem Exists Between Keyboard and Chair.
hmmm (09:14:27) :
> It’s almost as if whoever wrote these notes wanted us to find them.
I disagree, though he is writing for future readers.. Faced with the same morass I likely would have kept something similar to Poor Harry’s Dairy. It would be useful to my manager to help chase after the data providers and document what I had been doing for the next performance review. It would be useful to me as something to refer to when faced with those “I thought I fixed that already” moments (there’s at least one of those comments in his diary). And it would be useful to cram down the original authors’ throats someday.
For a system this complicated, it’s difficult to keep some of the system issues straight in the comments, another reason it’s worthwhile to have a separate document. (Ideally that would be a design specification, but none came with the code. When porting software like this, I generally follow a start with the first things that need to run, end with the things that pull it all together. In the future I’m going add a pass to scan through everything and try to get a sense of what it does. That was my starting point decades ago, but with experience and skill, programmers can develop a “disdain” for the whole and can quickly find their way to the core problems. fortunately I’ve never had to work on something as much a mess as this.)
Hilarious:
‘discovered that a sum-of-squared variable is becoming very, very negative!’
Don’t they know ‘ i ‘ ??? imaginary as some theories?
For Europe the subject is less that hilarious!
Schellnhuber is the adviser of Kanzlerin Merkel and of the President of the European Commission Barroso!
A position with extreme power.
Now look at his writings, with Mann, Schneider,Rahmstorf and others, very recent:
http://www.copenhagendiagnosis.com/
and on the reports in the media:
http://www.n-tv.de/politik/dossier/Noch-gibt-es-Hoffnung-article79835.html
http://www.spiegel.de/wissenschaft/natur/0,1518,663045,00.html
Anthony–
can’t express the gratitude we all owe you for pulling these programmer comments out of the code, so revealing. Question– the ‘FOI’ header, where did they come from? what do they signify?
I did some numerical coding for my MSc-equivalent thesis, and in theory is possible to obtain a negative number by the addition of positive ones if the total is large enough to overflow, and the most significative bit of the mantissa is the sign bit, with the value of 1 for negative numbers. With those conditions, an untreated overflow would cause negative numbers.
If the code does not treat that, the results are no good. The solution is the use of data types with more bytes, and some studies about the numerical stability of the algorithms employed is a must. However, it’s amazing that such a newbie error could be made in one of the most prestigious research institutions, so another explanation could be possible. We did not find this problem in our work because we used algorithms that we knew stable with the sets of data employed. Normalization of data also helped 🙂
However, it amazes me that a organization with so much at stake in numbers did not get some basic reference texts* and had to resort to search for algorithms to calculate distances across Great Circles in Wikipedia!
* Such as this, edited in 1992: http://www.amazon.com/Numerical-Recipes-FORTRAN-Scientific-Computing/dp/052143064X
Maybe it’s time to GPL the codes… Make them a true public venture where everyone can see what’s being done.
No rob it’s not hot stuff. No one cares. Outside this narrow little circle of people who care about truth nobody give a f—
AGW plays to some sort of primitive pre-rational impuse or mythic structures that some people have … and that’s it. There’s no science here. Just stories.
Expect green taxes. No nuclear power stations. Shutting down the coal fired stations we do have.
“My three main goals would be to reduce human population to
about 100 million worldwide, destroy the industrial infrastructure
and see wilderness, with it’s full complement of species,
returning throughout the world.”
-Dave Foreman,
co-founder of Earth First!
More here The Green Agenda
The warmists are essentially the forces of the counter-enlightenment – quite literally the people who want to put the lights out. Do you think they’ll be swayed by truth?
I keep seeing this line in the code;
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
but what does it actually mean?
Observed Temperatures?
Homogenized Temperatures?
Fudge Factor Temperatures?
Not really sure what to make of this statement as yet…
Having been a developer for 25 years (I’ve written a fair amount of Fortran code), I can sympathize with the programmer. He’s been asked to produce consistent output that appears plausible from a train wreck of ratty source data collections. The collections don’t agree where they overlap, so he’s been told which ones to use over which time periods using what appears (to him) to be a completely arbitrary basis; to make it all fit together he has to “fudge” the numbers and write code to handle each set differently. It’s as if he’s been asked to generate the data for a bank’s corporate tax return using checkbook register copies obtained from customers and the bank’s stock price for the last year, along with general ledger entries from three different internal accounting systems. The results of his efforts are then to be used at the annual shareholder’s meeting in two week’s time, and if they’re wrong, the CEO will make him the fall guy.
Programmers are normally a meticulous bunch, and this is a programmer complaining bitterly about the crap he’s been handed. Further, he’s upset that he’s been asked to produce plausible results in short order by people who as scientists are supposed to care about accuracy as much as he does, but clearly don’t care in this situation so long as the results come out looking like what they want to see. His notes clearly indicate that he wants everyone to know what he was facing and what sort of shenanigans had to be done to make it all look good.
What the public is finally getting to see is just how inconsistent and error-prone all the source data is and just how much manipulation is going on behind the scenes. Everyone has been led to believe that the measurement data sets are pristine, accurate and from data sources shown to be reliable. It’s assumed they are good enough to be used to make decisions that affect millions of people and cost billions of dollars. What public is now seeing is a real shock, and people are coming to a common sense conclusion: The data is crap, and the results can’t be trusted at all.
I feel for the programmer….I would not want to be in his shoes.
The story was on Slashdot on Nov 20th. http://politics.slashdot.org/story/09/11/20/1747257/Climatic-Research-Unit-Hacked-Files-Leaked
The ‘science’ behind the AAM narrative can now be seen clearly for what it is a cloak to conceal the true intentions of the political classes.
I suppose the false cloak of science was only meant to cover the political aims until the point had been reached when the political narrative no longer needed a cloak to hide behind.
The BBC.
The university of East Anglia.
The climatic research unit.
The UK meteorological office(met office).
The Hadley climate research centre.
The US link being the Goddard institute of space sciences(GISS).
All of these institutions are involved in the scandal to at least some extent in the scandal and by some strange coincidence all of them are fanatical believers in and supporters of the AAM narrative, they report the science as settled and beyond doubt, they indulge in attacking sceptical scientists and flout the rules when it suits them, they feel they are above reproach and audit perhaps because they have powerful friends?
Of all the world reach media platforms the BBC has been perhaps the most fanatical in peddling the anti CO2 anti capitalist anti industrial anti free market stories of the ‘eco industry’ it has used its nearly unlimited resources to provide a tsunami of supposed evidence however thin and patchy using state of the art visual manipulation and using the propaganda arts to full effect. The link to all of them is money and/or political affiliation, they are a closed loop of interconnected people with a common agenda showing the arrogance of those who know they have friends in high places, powerful and influential allies.
In attacking the fake cloak of science we are in fact not directly attacking the puppet masters driving the entire narrative, like a bull attacking the matadors cloak? The political forces need this sea change in our civilisation and they feel that they cannot be open and honest with us about the reasons for the required changes so they cover it in lies to hide it. Whatever the real reasons for the massive reordering of our entire civilisation are I suspect that AAM is not one of them.
To get at the truth we must expose the entire chain and the links in that chain, the political classes are moving fast now, faster than perhaps they expected to move because the foundations of their scam ie a warming planet is not happening, without the cloak of fake science to cover it the real movers are exposed for the world to see, if the political classes refuse to alter their policies when the science is exposed as fraudulent we will know who is behind the cloak.
John F. Hultquist (09:21:16) :
‘“a sum-of-squared variable is becoming very, very negative!”
This is an interesting “trick.”…. ‘
I had this happen using a common spreadsheet – which shall remain unnamed here – several years ago. I no longer use any spreadsheet for any serious numbers. Under and overflow conditions can occasionally break things. Commercial programs are particularly poor at documenting such things, and worse at fixing them.
On another note, the QC on the climate history data “Harry” is trying to work with is terrible. Whoever was responsible for supervising the primary data entry should have been running consistency checks, possibly every day, before forwarding it to Hadley or where ever. This is particularly important if the data is keyed from written records. Typos, misreadings etc. can creep in, and often, due to the character of the data recording methods, no useful filters can be employed to catch subtle errors during the entry process (e.g. instead of a proper data entry system the data was keyed into a spreadsheet line by line) – and Harry obviously is trying to deal with such ugly data.
OMG! The sum of squares parameter going negative? This isn’t possible! Even if all computed values from the model deviate negatively from the actual data, the squares of these numbers are positive values and thus the sum of squares is always positive.
I had a similar problem fitting titration data back in graduate school where the fitting statistics weren’t working out. It turned out I had coded in an incorrect equation for a derivative, neglecting to multiply by ln(2) in one line out of a thousand lines of code. It was plainly obvious something was wrong with my model from output plots, but wasn’t obvious in the code.
I feel for the programmer, but it’s his job to straighten out glaring problems, particularly when the output is a mathematical impossibility.
jamespapsdorf(09:40:10)
the probelm is that quoting Limbaugh, Beck et al will get us no-where – far too easy to dismiss them as politically motivated, and they have no credibility in around 50% of the population (in the USA) and of course a much lower number globally (remember – this is a global issue)
No – I think the solution to this is independant review as called for by Lawson in the UK and I understand one of the Senators in the US.
Attack with facts, deconstruct the code issues and eventually the MSM might, just might, start to run with it.
My opinion – for what its worth – I think we are too late.
ralph (09:51:07) : writes:
O.T. but:
Did you know that the BBC held a secret meeting that decided NOT to give balanced time to GW and anti-GW viewpoints. Apparently, the communique said:
“The BBC has held a high-level seminar with some of the best scientific experts, and has come to the view that the weight of evidence no longer justifies equal space being given to the opponents of the consensus.”
This is only partly correct, my friend.
For over 3 years I have been trying to elicit answers from both Mark Thompson (Director General) and Sir Michael Lyons (Trust Chairman). All I had received was sophistry and obfuscation, until I engaged the help of my MP.
Recently it came to light that a report had been commissioned in June 2007 jointly by the Trust and BBC Board of Management entitled “From Seesaw to Wagon Wheel-Safeguarding Impartiality in the 21st Century”. It concluded: ‘There may be now a broad scientific consensus that climate change is definitely happening and that it is at least predominantly man-made… the weight of evidence no longer justifies equal space being given to the opponents of the consensus’.
(SO THEY HAVEN’T EVEN TRIED TO MAKE A SECRET OF THIS…JUST SHOWS THEIR ARROGANCE!)
Despite this damning evidence from their own report, they steadfastly cling to the belief that their impartiality is intact as required by the BBC Charter. Such is their state of denial that Sir Michael Lyons has even tried to deliberately mislead my MP despite evidence I have to the contrary.
In light of this I have posed the question, through my MP: “On whose authority did the BBC cease to be an impartial Public Service Broadcaster, as required by its Charter, and become the judge, jury and sponsor of such dangerously specious political dogma so eloquently described as ‘…the consensus…’?
Answer comes there none! I believe it is time for the BBC to be subjected to an enquiry on this matter.