CRUTEM3 "…code did not adhere to standards one might find in professional software engineering"

This is very important:
it appeared that the organization writing the code did not adhere to standards one might find in professional software engineering
What a big surprise!

…of course.
It’s slowly coming out. CRU, NASA, MET…collusion at it’s worst form.

All this computer code stuff baffles these ‘high level reviewers’. Why would anyone ever use version control when we all know these experts get it right on the first try. A better set of questions is why do they not have some software experts on their review board. No version control poor documentation within the code is criminal, especially with trillions of dollars hanging on every amateur bug in the code. Why has no one asked about version control for the data? Code and data go hand in hand. Amateurs one and all. Yes I have looked at the code and yes I have 45 years as a professional programmer.

So Dr. Graham-Cummings has a BA in Matematics and Computation and a doctorate in Computer Security, has been writing software for 30 years but still only manages, in von Storch and Allen’s opinions, to qualify as an “amateur computer analyst”.
Are they “amature” climate scientists”?

As AJ Strata pointed out, you never let the PhDs write the code if the project is important. People get hurt. If it’s important, you get people who actually have a clue what they are doing to write the code.

This is why climategate is so important.Clever people like this guy are taking an intrest and ripping the warmists to shreds.Wonderful!

Why not bring in professional software auditors to do a thorough review of the CRU software methodology and conformance to that methodology? If the CRU software is not even under version control, that pathetic beyond belief.

Surprise.
I read the code and also saw problems. What shall we do? Jones really doesn’t want this out either.
“It is standard for scientists to manipulate and hide their code”

Professor Acton seems to be quite economical with verifiable and accurate statements of fact! He is quite adept at digging deeper holes, though.

“The computer code is the most important aspect of climate science and the least available.”
It could be worse. What if the code that we have already is all that is available?

Great contribution, but wake me up from this nightmare, please:
All this hype, destroying economies, scaring kids and even killing people by fear, called global warming is based on
– poor weather stations readings, cherry-picked
– enhanced by irreproduceable software, shown here
– compared with unreliable proxies, cherry-picked again
– blaming a trace gas, as only factor, as Schellnhuber’s “linear correlation”
– wrongly mixed with inappropriate statistic methods, as MM have shown
– stonewalled against any checking, as Jones thinks is normal
– arrogantly called this the true and only science by ‘realclimate scientists’
– paid for with a Nobel Peace Price and billions of taxpayers’ money
– brought up as top priority by whatever kind of politicians
You couldn’t invent it.

This must be one of the “sophisticated” climate models I’m always hearing about.

Since Climate Science (one of the youngest branches of science) is all about projecting the future trends for the Earth’s Climate- computer modeling is their primary tool.
It is quite worrying that there is no standardization for the computer code. Nor is there any review or testing process of the code and data sets used for the results.
Computer science (which is also a young science) has extensive requirements for code and data reliability. Lack of following even the basic standards is negligence, bordering on criminal intent to defraud.

The climatescience process really is from normalscience, isn’t it!
Conclusions
Results
Method
Apparatus
Thanks to J C-R and others, hopefully soon we will be able to get to the apparatus.
And, I like the idea of Acton being asked by Parliament to verify his statement that the actual CRUTEM3 production code has been released. When you think about it, IF the answer proves to be negative then the consequence would be enormous. In other words, this could turn out to be ‘match point’
So, can anybody comment on the status of the CRUTEM3 code?

I worked as a programmer for a physics research group as an undergraduate and in industry before going to grad school in the 1980’s. It never occurred to the physicists to publish or make original code or even the data available. If someone wanted to replicate their work they would do the experiment themselves and write their own code. The papers described the mathematical methods used so that anyone could write a program to do the number crunching. In industry I often had to deal with very poorly documented code. The standards in industry have likely improved, but a lot a valid work was done in the bad old days.
Since climatology is such a hot area, I too would like to see greater openness. This will cost more money. Granting agencies will need to provide budget lines for professional programmers (rather than cheap students) and web servers for data archives. But, there is no basis for demonizing researchers who have been following the standards in their field or ignoring past work. If you want to redo someone’s chemistry experiment should you be able to demand the use of their test tubes? Maybe some day chemists will be required to archive all their old test tubes. But, that won’t invalidate all the chemistry that was done before.

I am the ‘computer analyst’ mentioned in 3.2 . . .

Dr. Graham-Cumming diplomatically leaves out the word ‘amateur’ with which Professor Hans von Storch and Dr. Myles R. Allen denigrated his analysis. ‘Nuff said.
/Mr Lynn

John in L du B (08:30:44) :
So Dr. Graham-Cummings has a BA in Matematics and Computation and a doctorate in Computer Security, has been writing software for 30 years but still only manages, in von Storch and Allen’s opinions, to qualify as an “amateur computer analyst”.
Are they “amature” climate scientists”?
****************************************************
You know the drill. If you don’t have the facts on your side, shoot the messenger. The only problem is that as the bodies accumulate, so do those who look for the facts to verify the claims. That’s the stage we’re in now. For years, the McIntyres and Cummings would get lumped with the crackpot oil-bought “denier’ crowd. There are too many McIntyres and Cummings now for that method to work. As far as I can tell, there is no Plan B, and this whole affair called AGW is working its way toward an ugly end.

Should we all weep with laughter or frustration?
I suspect both

“Professor Slingo: … We test the code twice a day every day. We also share our code with the academic sector, so the model that we use for our climate prediction work and our weather forecasts, the unified model, is given out to academic institutions around the UK, and increasingly we licence it to several international met services: Australia, South Africa, South Korea and India. So these codes are being tested day in, day out, by a wide variety of users and I consider that to be an extremely important job that we do because that is how we find errors in our codes, and actually it is how we advance the science that goes into our codes as well. So of course, a code that is hundreds of thousands of lines long undoubtedly has a coding error in it somewhere, and we hope that through this process we will discover it. Most of the major testing is very robust.”
This statement indicates that Slingo knows almost nothing about software. The more salient questions are: how must test automation coverage do you have, how do you maintain your test cases, do you do unit testing upon check-in, what’s your procedure for regression testing, do you perform independent code reviews etc.
“We test the code twice a day every day” — this doesn’t make any sense. It sounds like something she just made up on the spur of the moment.

“But, there is no basis for demonizing researchers who have been following the standards in their field or ignoring past work.”
Mike, thats true but by the same token if the standards (of computer coding out of which are born the models) are rather low in this field, should we take their work as good enough to be the basis of policies which could well cause considerable lifestyle changes not to mention rather more taxes?

Perhaps we should set up a project on SourceForge called “CRUTEMP” and help them along a bit! Although funny, I think we should seriously consider it.
We could start with the 160Mb of code and data that we already have (courtesy of our “insider”). We could even put a wiki together – to help the undergrad’s get up to speed.
Although I think we should only allow amateur admin’s. No Dr’s or post grad’s should be allowed access to the delete button. It’s a trust thing.

I bet if we asked any of these CRU people what “cyclomatic complexity testing” was done – they would reply that “we don’t test for cyclones in our software”.

This isn’t “post-normal” science – it is abnormal science. I never liked that “post-normal” BS anyway.

Seems there was no QA, no source control, no independent auditing; all of which one would expect in any commercial environment of any size. I spent 25 years in IT, in a company which grew from small to huge. When I started, we programmed by the seat of our pants, documentation was scarce, commenting scarce, any testing done, we did, and maybe the customer when they went live!
By the time I left, we had full and comprehensive inter-office (global) source code control, QA controlled documentation, a requirement for ISO9001 accreditation, and regular visits from an independent auditing company.
Frankly, what went on at UAE would be laughable were it not so appalling.

I wonder whether Professor Hans von Storch and Dr. Myles R. Allen bothered to try to find out anything about the person they called an “amateur computer analyst”. And whether they are always so pedantic in their work.

Mike (09:27:15) :
(…)
“If you want to redo someone’s chemistry experiment should you be able to demand the use of their test tubes? Maybe some day chemists will be required to archive all their old test tubes. But, that won’t invalidate all the chemistry that was done before.”
———
Reply:
We don’t need their computers (aka “test tubes”); others are available. What we want is their UNadjusted raw data and their final numbers. Then we’ll see how they tortured the data. If the new results are different than “climate science” results, we’ll certainly show them (and the courts) our algorithms!
But will “climate scientists”, barring some subpoena, ever show us their algorithms?
No worries… subpoenas are on the way.

Graham-Cumming is such an ‘amateur’, with those degrees and real world experience. I agree (having architected, designed and managed large software systems for NASA and the DoD) the code we have seen is garbage. And it is clear in testimony the code used to generate CRUs products HAS NOT been released. When it is, with the data, it MUST produce the same numbers as before.
It won’t of course, because there is no way that buggy code could replicate an error message.

Hotrod asked: “Is there a UK based software developers professional association, or similar group?”
There’s the British Computer Society (BCS) – http://www.bcs.org.

wind turbines and solar cells are costly and impractical for most countries.
the code was ‘..unreliable’, hmm now why does that not surprise me?

Here’s the logical problem with the Dr Jones testimony where he claimed it was not ‘standard practice’ to release data and computer models so other scientists could check and challenge research.
There are three pieces to the work of Dr Jones.
First, we know and he argues that almost all the weather data is posted all over the world on the Internet so that anyone can get at it. Second, he argues that his papers contain all of the methodology that demonstrates that his number crunching code is valid. Sandwiched in the middle is the third fact that the code itself has not been made publicly available. But we all know how easy it is to put that code up and make it available just like the other data and papers are.
So there is an established precedent that exists for two items, to wit, the publishing of data and the papers. And, since we all know that they seem to be able to afford to put that information on the Internet, there is simply NO valid reason to hide the code (by not publishing it) unless Dr Jones is hiding something. Enough with the copyright and proprietary ownership red herrings.
Put the three data sets out there and let the chips fall where they may, Dr Jones.

Dr Graham-Cumming has opened a can of worms here. I’m not referring specifically to the bugs in the code, but the complete lack of even the most basic code quality procedures.
Having worked in commercial applications program development for 30 years, I can only reiterate the disbelief that others have mentioned. In the institutions I have worked, (and they are many), all program code must go through a rigourous process of unit testing and an audit log kept of all bugs and fixes, and the test plans filed away for future reference. The same goes for system testing. This way, if a bug is found, you can check the test plan to see if a particular logic pathway has actually been tested, or tested with all relevant conditions. However, once the code “goes live” the process is tightened up even more.
While pre-live, some corners may be cut in documenting fixes, post-live, every change is logged in stone. The source code is usually kept under some kind of development management system so that it can’t be checked out by more than one programmer at the same time. Any fixes must have an audit trail that ties it to a documented problem. Another test plan is created to test the fix, which is documented in the code and cross referenced to the original problem document. Once the fixed program is re-released live, it becomes the new version. This is the basic version control which Dr Graham-Cumming says was missing. There would normally be an archive for each version of the program code. This is the only way you can deal with unexpected problems caused by adding the fixes themselves and allows you to go back to earlier versions to see what effect these fixes have had on the code.
The situation here seems to be closer to what one would expect from a home coder with no controls or quality management at all.

John Graham-Cumming,
I’ve been a professional programmer in the industrial control sector for almost 30 years. I don’t know if the terminology is different between the US and UK, but over here a piece of squirrely code that nets you a couple billion in government dollars is a feature, not a bug. 😉
Of course, all this code should just boil down to Tglobal = f(CO2), where
f(CO2) is a fairly simple formula or a look-up table. Even a bad programmer should be able to pound it out in an hour, but instead everyone wants to milk the problem for a couple of decades.

I will be very curious where this branch of climate science in the area of software leads. John Graham-Cumming says he has almost 30 years of experience in software, I have written software since 1976. Anyone who has spent that amount of time in the intricacies of proper software generally know what they are speaking, degree or not. However, there are multiple branches in software science and design that immediately bring in certain flavors of expertise; you can no longer know it all.
For many times, the most readable and beautiful code is not necessarily the best software. It usually hinges in the area of efficiency or speed. If you don’t care how slow the software runs, object oriented design will usually produce the most readable and logical code but I have seen it cost a 10, 100, or ever 1000 times slowdown. However, if huge amounts of data are involved, especially when the data’s influence spreads across many sub-areas, structured and logical design tend to no longer apply, you get into the area of spatial integrity and locality. This mainly has to do with the internal layout of memory within the computer and jumps directly into multiple-layer cache layouts and pipeline lengths and stalls.
Inside purely efficient code, usually even the base equations are broken into multiple pieces, especially when integrals are concerned, when performing numeric analysis. A long equation will deal with one mathematical operation across the arrays at a time. This can cause the software to look rather bizarre but it doesn’t necessary mean the software is sub-par quality or blatantly bad. I hope these software specialists as John Graham-Cumming are truly trained in this area of expertise, for those used to software in business and financial are going to get a rude shock when jumping across to scientific software design, they are two different worlds.
Still there are times when bad design is just bad design and bad coding is just bad coding and anytime logic flaws are embedded into the code, it is bad. Error flow is usually one of the first areas to look for purely bad designs.

As a programmer that is also what surprised me the most. Not just the quality, but the techniques, and the faith in the accuracy of them.
I’m still not sure I believe they didn’t use version control. In the back of my mind I think they have to say that in order not to release the code (and more, so that no one would see the versions — it would probably be easy to show manipulation with that). If that is true though, then it shows they know almost nothing about software, and only slightly more about computers.

To: Larry Geiger 10:14:22
I strongly support your position. Indeed, the funding source is very important, and the funding source is tax money. I am surprised that scientists involved in Climategate don’t acknowledge that and behave in nasty and arrogant way. I have never encountered in my practice and neither do I know any of my colleagues who were ever asked for a code or experimental data. Generally, public is either unaware that they could or isn’t interested. This is despite the fact that billions are spent for LHC, modeling of comet impacts with Jupiter satellites and Earth, string theories, and other science that I would call “entertainment” science suitable mostly for Discovery or History channels. And, arguably and only to my taste, relatively little is spent to support research that are of practical value for public. However, in the climategate case when the international policies making was attempted on the base of publically funded science and these policies will obviously cause tax and unemployment increase and standard of living decrease, the science must be scrutinized and it must satisfy the standards of majority of people. In the mean time all some group of climatologists separated us, the people, into “real scientists”, “real science supporters” and “ignorant trolls” and continue belittling the latter group despite the fact that it seems to be growing majority contributing the most to their well funded (by public) research.

I bet Pachauri knows what voodoo science is, since he’s seen it before. Might this be voodoo programmimg???
Ecotretas

Seems like what is lacking according to the code that was released, they lack a decent version tracking system.

The fact of the matter is that if you are developing software for, say, a medical environment, or flight control systems, or engineering design tools, or industrial process control, or any of a thousand other disciplines (even automotive powertrain control) a bug in the code (or any of the code libraries and objects underlying that code) can kill. In thousands of other applications, bugs can result in mechanical failure, excessive operating cost, legal compliance problems, Mars landers missing the planet, etc.
So at an industrial level, there’s a very substantial incentive to get it right.
These scientists are amateurs. For the most part, they plink around trying to produce results, and when they get something that looks like what they want to see they call it finished. There’s no formal test plan, no test data sets to evaluate a full range of possible inputs to the model, no process to ensure that the evaluation is done by properly skilled individuals. Peer review is worthless in this environment because most of the peers are no more structured in their approach to software than the original authors.

I don’t know where these FORTRAN DINASOURS are coming from.
Heck, I was raised with crappy old FORTRAN too.
I’ve now processed datafiles with EXCEL which have 10,000,000 elements in them.
Totally TRANSPARENT on the math formuli.
But, I suppose if you are living in 1975…FORTRAN sounds pretty good.
Max

“Harry” and Dr. Graham-Cumming should get together.

mpaul (09:48:33) :
“This statement indicates that Slingo knows almost nothing about software.”
As someone who was for fifteen years a professional systems analyst and programmer (albeit in the commercial sector), this is one of the few things here I can pass an even half-informed comment on.
Running software however often you like isn’t what tests it per se. You analyse, design, code, test and maybe pilot a system before you let it go live, and subsequently, it gets tested by your users, who will be over you like a rash as soon as they pick up on a problem. You damn well better fix it PDQ, especially if it’s mission critical.
All of us here in the UK are the Met’s end users. We’ve been testing their warmed-over soup for a while now, and found it severely wanting. But have they fixed it? Nah. They have this disconnect between modeller’s fantasy land and reality.
I’m sure if they keep on predicting warm summers and mild winters, they’ll get it at least half right the odd year, but I’m not holding my breath. Their shorter-term predictions aren’t too hot (or maybe they are!) either. No one expects them to be perfect, but if even the BBC is contemplating kicking them into touch, things must be really bad.
Slingo’s brass neck and spin is breathtaking.

re: wayne (11:00:11)
I am going to add one other aspect to my previous comment.
However, it would be good to have the software used by climate science to be re-written in a very common and forgiving language as C-sharp or Java, freely available, in purely OO design and ignoring any efficiencies, so all programmers can understand what exactly is happening, no matter their expertise. Efficiency could be added by parallel implementations when speed plays in. That would add hugely to the insight and clarity into climate science itself, and in fact, seems the best way to move this science along at its highest speed of development. A common, plug-in type design to, albeit slow at run-time, test basic core logic used to compute these temperatures.
Yes, that would be good!

…and in addition to strong version control and regression testing and documentation, may I bring up one more point that IT professionals should be harping on.
Any IT shop should be able to reproduce the exact state of the data on any given day. In this context, “data” means EVERYTHING. Datasets, code bases, e-mails, everything that existed on THAT day. Standard best practices require a weekly full backup and daily incrementals written to tape and stored off site. If an email, or data file, or version of the code existed on a specific day 3 years ago, it should be possible to restore it from backup. The only thing that should escape these best practices is a file etc that was created and destroyed on the same day. Everything else should be there.

Mike — If you want to redo someone’s chemistry experiment should you be able to demand the use of their test tubes?
If this is being used by politicians as the basis to extract an extra 10% of my earnings or make me pay an extra $3 a gallon for fuel, then YES.
Do you not have a concept of what’s at stake here, or why extra diligence might be required for trillion dollar decisions?

Mr Lynn (09:31:39) :
I am the ‘computer analyst’ mentioned in 3.2 . . .
Dr. Graham-Cumming diplomatically leaves out the word ‘amateur’ with which Professor Hans von Storch and Dr. Myles R. Allen denigrated his analysis. ‘Nuff said.

Maybe the word is returning its original meaning, which would rather be a compliment than denigration.
Amateur: through French, from the Latin ‘amator’: lover, devoted friend, devotee, enthusiastic pursuer of an objective.
I seem to recall Anthony was called ‘amateur’ several times, too.

John Graham-Cumming has a BA and DPhil in Maths and Computing from Oxford. Professor Jones read Geography at Lancaster.
Here’s a working hypothesis. Most scientists who advocate AGW didn’t go to the best universities. They’re not all that clever. They got their PhDs by slogging. They are easily swayed by strong personalities. They have second rate minds and do second rate science. It shows.

CodeTech (08:42:51) :
“So… to be a “climatologist” one requires knowledge of:
1. Computer programming
2. Chemistry
3. Physics
4. Statistics
However, failure to have knowledge of or skills in some or any of these disciplines is easily overlooked if you have a “green” attitude and mindset, and are capable of writing suitably alarming predictions. Also the ability to procure grants and scare children is a genuine asset.”
Sounds like they need to have an inter-faith meeting.
I think your point proves quite well that the possibility of of one becoming a Climatologist in any real world sense is slim to none. Climatology should be stricken and fall under the umbrella of theoretical physics. An open and Interdisciplinary approach makes the most sense . With a widened pool of participants it may be possible to achieve greater transparency in the process so long as it is not stacked with eco-ideologues.

The climate scientists just don’t get why they should follow standard software development practices. Does anyone remember the “On Replication” post at RC from about a year ago where Gavin made this telling statement at comment 89:
“My working directories are always a mess – full of dead ends, things that turned out to be irrelevant or that never made it into the paper, or are part of further ongoing projects. Some elements (such a one line unix processing) aren’t written down anywhere. Extracting exactly the part that corresponds to a single paper and documenting it so that it is clear what your conventions are (often unstated) is non-trivial. – gavin]”
http://www.realclimate.org/index.php/archives/2009/02/on-replication/
So how is anyone else (or even Gavin) supposed to replicate the work? We’re supposed to “trust” whatever it is their paper said and the maybe somewhere on their computer is the code and data used to produce the result?
I was already leaning to the skeptic side, but this discussion at RC is what pushed me off the fence.
What’s even more disturbing is that using documentation, source control, etc. would actually make their lives easier in the long run.

George Ellis:
I would write that as:
x = 0x05555;
x = 0x0aaaa;
x = 0;
… bonus points if you know why… 😉

I wrote the software for analyzing data obtained from the BATSE experiment (on the COMPTEL Gamma Ray Observatory) in the late 80’s.
Basically I wrote a program which would allow scientists to run their own models within it (for you techie-types, this was done under VMS using Fortran).
The scientists working on the project had no problem sharing data and models with each other. However, they got miffed at a certain University (which I’m leaving unnamed), because some of their programmers hacked into our computers and stole some of the data – which allowed their researchers to publish first.
The scientists I was working with were unhappy, because they felt, since they had legitimately been involved in the project from the beginning, they should have been given 6 months exclusive use of the data.
They felt that would give them sufficient time to publish their hypotheses, at which time, they would release the data too.
Of course, these were mere astrophysicists and theoretical physicists. What would they know about standard scientific practice?

Re: CO2 Realist (Mar 4 12:39),
So how is anyone else (or even Gavin) supposed to replicate the work? We’re supposed to “trust” whatever it is their paper said and the maybe somewhere on their computer is the code and data used to produce the result?
Think of it as an exam. The professor gives a problem. Each student solves it by his/her method using the information provided with the problem. The results have to agree, not the method of solving the problem.
In this sense, given the data and meta data, another researcher should be able to say if the temperature is increasing or not. It is the data that is important. The method becomes important when there is disagreement in the results. In this case a third ( or fourth etc)independent analysis would clarify the issue as well as finding the error in the method of one of the two original who disagreed, and would be less of a hassle. It is the “independent” that is important. By subverting the peer review process independent analysis was lost.

Thank you John for shining a bit more on the truth behind the cargo-cut cience practised by the CRU. Please keep at it and keep us in the loop here at WUWT as thing progress.

mpaul (09:48:33) :
“Professor Slingo: … We test the code twice a day every day. We also share our code with the academic sector, so the model that we use for our climate prediction work and our weather forecasts, the unified model, is given out to academic institutions around the UK, and increasingly we licence it to several international met services
Nail on head comes to mind. She has absolutely no idea what software testing means. She thinks it means running the program and looking at the result(s) and if it looks alright then it passes the test.

Re: ShrNfr (08:44:02) :
Oh my, such a serious misspelling! Surely you meant to type “clusterfrag” instead.
Moderators! Yoo-hoo! Can you help this nice person out and make that change for them? Thanks in advance!

“amateur computer analyst”
Nobody uses that term anymore. I’m surprised they didn’t say ‘in the data processing centre’ or some other such 1980’s term.
I guess if he was a professional he would be in academia and writing papers instead of, you know, actually delivering real solutions to companies who pay for them.

bill (09:51:52) :
“Mike, thats true but by the same token if the standards (of computer coding out of which are born the models) are rather low in this field, should we take their work as good enough to be the basis of policies which could well cause considerable lifestyle changes not to mention rather more taxes?”
Fair question. (1) These low stabdards did get us to the moon, etc., etc.
(2) Many climate research groups have arrived at similar results.
(3) Most of the code for the data analysis of temps is now availible. Small errors that have been found have not substainly changed the results.
The last two points illustrate the robustness of the climate results.
Small note: the programs at CRU are for data analysis, not climate modeling. The hocky stick is a data set of past temps.
Here is another way to look at it. Suppose we went back to the early medical work showing the link between tobacco and cancer. It probably would not meet these new standards. Did they save all the data? I doubt it. Are all the statistics programs that were used availible? I can’t imagine it. Yet, it would be foolish to run out and start smoking.
Remember that the tobacco companies put up are fight to keep people smoking. Millions died.
If we enact C&T schemes now with high caps for now, at least we will have a system in place. I figure it will take a few yaers to get a the “bugs” out of C&T. If the temps go down, and I’d venture there is a 5% chance of that,
then we keep the caps high. Not much harm done. If we do nothing, and the temps go up and we keep pilling up the CO2, we are going to mess things up big time. No, I don’t think it will be the end of civilization, but major hardships will be impossed.
We do need to weigh the risks of doing nothing. Some people will go to doctor after doctor until they hear what they want. If the first nine doctors tell you to lose weight, eat better and get more exercise but the tenth one says not to worry, it is tempting to go with the tenth doctor, but this is not wise.
If the models are off, it only means the warming will come a few years or decades later. You can’t get around the physics that more CO2 will eventualy cause big problems.

Milwaukee Bob (14:09:09)
THEN, there is that damn data….. ☺
That damn data status can be improved upon. The data sets(raw) out were by no means derived from an exhaustive and through effort.
I would estimate that the completeness of the records could be improved upon greatly by examing closely the archives.
I am getting this very unnerving feeling that few have bothered to look into the differences between what’s in archives and what’s being put out there as “that’s it, nothing more to see”.
How many out there have looked?

E. M. Smith
With modern journalling file systems, you can even capture moment to moment changes. The Network Appliance has a kind of version control built in to it. This is a 10 minute kind of thing to set up>>
Well now we’re talking storage management, not backup 🙂
All storage arrays can do a snapshot of what is stored on them and keep it as a “point in time” copy. Not all snapshot techniques are the same, so some arrays can only support a few snapshots while others (like Netapp) can support hundreds. This is slightly different than version control. Also, it is valid for flat files, but databases require that they be quiesced before snapshot otherwise there is a possibility of the snapshot losing an in flight transaction and compromising the integrity of the snapshot. Many vendors have tools that can do this automatically for common databases like Oracle, SQL, Exchange, etc. As for journaling, yes that can be done at the file system level, and most database and transaction processing systems can do it at the application level. In brief, if you are willing to spend the money, you can capture the state of the whole system down to the second if you want. But as a rule of thumb, the minimum any IT shop would have in place would be weekly fulls and daily incrementals. A typical Netapp shop (or Equallogic, or Sun7000, IBM nSeries which support “re-direct on write” snaphots) would supplement the tape based backup system with hourly snapshots.

Anna:
“No. The raw data are not shared with all and sundry in the accelerator experiments. The groups have rights of publication. Once the data is archived, it is open for sharing, after the experiment is closed, and still there are caveats.
Replication is done by having more than one experiment at a time. In the LHC ATLAS and CMS are competing experiments studying the same physics independently.
One reason is proprietary. It takes ten years of preparation by hundreds of people to set up the experiment and take the data. You would not find people willing to do that if the first theorist who came with a FOI request got his/her hands on the data before publication by the group.
The second is the complexity. Each experiment develops its own computer codes ( not well documented) corrections etc that an outsider would have to spend years to do all over again, given the raw data. That is why at least two experiments are necessary.”
I am a little puzzled by this. Just how do people try to reproduce the results of experiments before those results have been published? Yes, to various extents there is exchange of information prior to publication, but prior to publication the results are subject to revision. This is not about pre-publication embargoes of data so that the experimenters can publish, this is about the post-publication blackout on the data so that the results cannot be tested once they become know to the world at large. Once everyone is done publishing, it is expected that the raw data will become available so that the results can be checked is a perfectly acceptable formulation, but contextually I didn’t (and still don’t) see the need to add a “post publication” qualifier.
a side issue: the problem with climate science is that there are so many possible data sets and means of manipulating them that you can produce a wide variety of results. these are not all equally valuable in determining global trends, in fact many are almost useless. without information about what data sets are used and how they are manipulated it is impossible to determine (among other things) to what extent the results are an artifact of the manipulation, to what extent the data sets are representative of a global trend and what degree of significance to attach to the results.

Simple question for the committee.
ISO 9001. Is CRU accredited?. If not, stop wasting time and money – close the enquiry.
If it is accredited, then when was the latest audit and what were the results. If negative, stop wasting time and money close the enquiry.
Then sanitise the organisation.

davidmhoffer (14:39:10) said:

A typical Netapp shop (or Equallogic, or Sun7000, IBM nSeries which support “re-direct on write” snaphots) would supplement the tape based backup system with hourly snapshots.

Would you be referring to “copy on write?” (The actual implementation is unlikely to copy the old data, rather it would simply allocate a new block for the new data and change some pointers in the metadata [block lookup table/b+tree/whatever].)

Much of the data coming in is such unstructured and incomplete crap that it lends itself to manipulation using QAD code just to try and stitch the data together into something that can even be processed…
It’s like trying to build a car using a collection of bits from different manufacturers – the end result may look like a car but it won’t work like a car…
These whole projects attempting to produce global/regional gridded temperature data need taking right back to square-1 and starting over with consistent methodology using only complete quality assured data-sets. Until then it’s just GIGO…
Maybe they should learn to walk before they can run?

davidmhoffer (15:28:04) :
Don’t worry, there’s another round coming. A lot of the fixes were temporary. they took a range like 0 to 18 or 0 to 34 and wrote a little routine to convert JUST that date range to 2000+ instead of 1900+ on the assumption that a) there were no computer records prior to 1960 or so to conflict with, and b) the sofware would be replaced with new software before the new hard coded fix ran out of runway. Then everyone forgot about the new deadline they created for themselves and went back to day to day emergencies.
Grace Hopper would chuckle

Nah, we went whole hog with the long way ’round. If there were ANY chance I’d have to deal with THAT project I’d be looking for another job!
Of course, what they’re doing to us right now out of DC is just as bad, and may be worse, than Y2K ever dreamed of. If you think CRU is dumb and evil, check out CMS.

Started programming 30 years ago. I can only assume that’s not a current picture.

In partial defense of not using professional programmers, and not using professional software techniques. I am guilty of having done that, as it was standard practice for many years in my engineering profession, and many others as well.
Disclosure: as a chemical engineer beginning in the 1970s, I wrote reams of amateur computer code. It worked – eventually; was tested to my satisfaction for the purpose at hand, and was sometimes left for the next poor fellow to deal with, perhaps years later. This was the norm in many organizations who built and ran the chemical plants, refineries, and many other manufacturing operations.
The reasons we did this were financial and time constraints, as there were deadlines to get things done and very little staff or budget for it. When we published in our technical journals, we did not publish code, rather we published the mathematics that went into the computer code. This was standard practice for many years. (for examples of publications in the U.S., see Hydrocarbon Processing, Oil & Gas Journal, Chemical Engineering Progress, also Chemical Engineering, all available in most university libraries). As was mentioned in a comment above, it was assumed that those reading the publication would have the skills to do the programming – that was considered a trivial task.
Then, after sufficient computer code was created internally, a choice arose when a new task was at hand: use somebody’s old code (probably undocumented and poorly written), make it work for your own purposes, or, start from scratch and write your own code.
Management wanted engineers to go with choice number one, but that created problems for management, as they could not understand why old code did not work the first time and produce valid results for the current problem. Eventually, in some organizations, we resorted to having professional software engineers and managers deal with the legacy computer code. And thus was born the technical group within the IT department. The IT department had the same issues, how to reprogram old code and standardize it. They also brought in version control and some of the other good programming techniques mentioned in comments above.
We also found it more economically attractive to lease or buy professionally written and maintained software, and an entirely new industry arose: software providers for the chemical engineers. A couple of such companies were Simulations Sciences of Brea, California, also Aspen Technologies, but there were usually a half-dozen or so. Some of our legacy code was run by the commercial software as a plug-in subroutine. That in itself created more than a few problems, though.
We did not have the fate of the world riding on our software results, but we did have multi-billion dollar processes that could be harmed (or explode) if our code was wrong, and some smaller processes in the hundred million dollar range.
It would appear that the climate scientists are today somewhere in the state that the chemical engineers were in a couple of decades ago: they could use an IT department and quit doing the programming themselves. This ClimateGate fiasco could also create a competitive commercial software industry, where the climate scientists shove their data in the front, and professionally written and maintained software crunches the data to produce the output.
Lamentably, this probably will not happen. A major drawback to these types of commercial software is the lack of flexibility, and stifling of creativity in writing one’s own code to produce results.
My preference is to at the very least, make sure the software is examined by professionals and brought up to some reassuring standards so that the code is robust and bug-free. This is the minimum for making policies that have the implications as proposed by the climate science community. We were able to make do in the early days with our engineer-written, use-it-once code where the worst outcome was we installed a pump or heat exchanger that did not work. The world’s economies did not suffer much, although the engineer who did this might have been out of a job for sloppy work. The stakes for climate models is, of course, far higher. Those who make policy based on the climate science should demand that the data and the computer codes be as up-to-date and modern as possible. No expense should be spared.

Re: Mike (Mar 4 14:26),
If we enact C&T schemes now with high caps for now, at least we will have a system in place. I figure it will take a few yaers to get a the “bugs” out of C&T. If the temps go down, and I’d venture there is a 5% chance of that,
then we keep the caps high. Not much harm done. If we do nothing, and the temps go up and we keep pilling up the CO2, we are going to mess things up big time. No, I don’t think it will be the end of civilization, but major hardships will be impossed. Bold mine.
This is a very naive statement. You are saying : take civilization back to 19th century levels , and not much harm will be done!!, if the pyramid scheme worked of course in reducing the alleged culprit, CO2.
Already millions starved to death in the Third World with the ethanol fiasco, because the price of corn went artificially up. In Haiti they were eating mud pies before the quake because of this. And you have the hubris to say not much harm will be done. I guess as you and yours survive the pyramid.
There is the law of unexpected consequences that the naive do not know and the sharks of this world know and expect.

I am reminded of that old saw..
“People who live in glass houses should not throw stones”
Seems that Professor Hans von Storch, Dr. Myles R. Allen et al still have a lot to learn. Humility would be a good start.

I run a very small property services company. There are only a few tens of thousands of dollars at stake with each transaction yet there is no way my public liability insurer would stand for me writing code myself to handle management of transactions, nor me doing my own property valuations and not using an independent licensed valuer despite the fact that I have 20 years experience in processing these transactions and assessing valuation documents. I would be sued to bankruptcy. Yet with trillions of public and private money at stake these so called UEA academics and their supporters think it entirely appropriate they should bodge up there own code and statistical methods and that criticism from professional programming engineers and statisticians is out of order. These guys are truly out of there minds.

anna v (21:28:50) Well-said.
Mike: re your conviction that CO2 is causing global warming, there are at least two places you can put your money to make a fortune on Global Warming. These are stock index funds GWO, and PBW. You might be interested to know, however, that the price for each has taken a serious tumble in the past two years – far more than the S&P 500 index. It would appear that the profit-seeking investors of the world have missed a big opportunity, leaving it wide open for you. Good luck with that.
http://sowellslawblog.blogspot.com/2010/02/is-global-warming-good-investment.html

CodeTech (20:30:44) wrote: “You [Mike] figure “not much harm done”? [By C&T] Well, the evidence in Europe and even from this recession say otherwise… that we would literally shut down our first world economies. Not just a slowdown, and not some magical, fairyland of “green jobs and energy”. Not even close. Economic suicide is an understatement. Unlike the 1929 depression, a HUGE percentage of the first world now directly have their savings and investments in the markets, markets that will crash, fail, tumble, tank, and end. It won’t just be stock brokers jumping out of windows.”
Now who is being alarmist? C&T or a carbon tax that started low (or a high cap), which is what I’ve seen advocated, would do little to the economy. If then the majority of climatologists are right, we can rise the tax or lower the cap to limit the damage caused by climate change. It time C&T or a CT would cause some economic drag but nothing like what you are saying. Read the CBO report.
http://www.eenews.net/public/25/11455/features/documents/2009/06/22/document_daily_01.pdf
http://www.cbo.gov/ftpdocs/91xx/doc9134/04-24-Cap_Trade_Testimony.1.1.shtml
While there is no free lunch your alarmism has no basis in known economic principles.
anna v (21:02:35) wrote: “This [My statement regarding C&T] is a very naive statement. You are saying : take civilization back to 19th century levels , and not much harm will be done!!, if the pyramid scheme worked of course in reducing the alleged culprit, CO2. ”
More economic alarmism. I can’t even make sense out of your sentence.
anna v (21:02:35) wrote: “Already millions starved to death in the Third World with the ethanol fiasco, because the price of corn went artificially up. In Haiti they were eating mud pies before the quake because of this. And you have the hubris to say not much harm will be done. I guess as you and yours survive the pyramid.”
Millions did not starve. There were food shortages that some attributed to the shift toward ethanol production. And I agree, pushing ethanol is not backed by the science but by corporate and political interests. Further research may change this. But, we do need to watch where the money is coming from. There is influence peddling all around. But, the big bucks are from the oil and coal companies that have funded a lot of the conservative think tanks that push denial and do nothingism. The “green” corporate interests are far weaker.
Environmental alarmists do exist in the media, but not so much among the actual climate scientists. But be just as weary of the economic alarmists.

@ Mike (14:26:24) :
Others have pointed out most of the many flaws, but I think you demand a proper fisking

Fair question. (1) These low stabdards did get us to the moon, etc., etc.
(2) Many climate research groups have arrived at similar results.
(3) Most of the code for the data analysis of temps is now availible. Small errors that have been found have not substainly changed the results.
The last two points illustrate the robustness of the climate results.

1. Not true. And a bit of a straw man. The entire basis of the AGW demand for world shaping policy changes is statistical analysis of adjusted data using computer software. If the people doing this cannot show their work, it is a major problem. This is not like a reproducible chemistry experiment. And while I would not use actually getting to the moon as the primary test case for NASA software, it does sort of validate. Which is why some have said wait 30 years (or another 15 years or whatever) to have a reasonable period for identifying the validity of the models.
2. Using the same compromised ADJUSTED data, with the same selection bias and most importantly the same adjustments which are what actually introduce the warming.
3. The data analysis is based on adjusting actual historical temperatures, interpolating multiple stations in specific ways to establish grids (which may or may not be defensible depending on the details).
The last two points illustrate nothing about robustness. They illustrate a real problem with basing different analyses on the same adjusted data and a potential problem with

Small note: the programs at CRU are for data analysis, not climate modeling. The hocky stick is a data set of past temps.

The hockey stick is most certainly NOT “a data set of past temps”. It is an amalgamation of multiple different unrelated proxies (Ice cores, tree rings etc) stitched together in sequence, plus some of the same adjusted temperature records referenced above, minus some inconvenient proxy data data (tree rings from 1960 to 1990) that does not agree with their pre-conceptions or their models, and ought to undermine their “calibration” of the proxies (rather than “hide the decline”).

Here is another way to look at it. Suppose we went back to the early medical work showing the link between tobacco and cancer. It probably would not meet these new standards. Did they save all the data? I doubt it. Are all the statistics programs that were used availible? I can’t imagine it. Yet, it would be foolish to run out and start smoking.

This is a flawed analogy. The equivalent would be if those scientists had thrown out the orignal data and not documented their adjustments but demanded that we take it on faith. Or if they re-ran an analysis of some early medical work showing the link between tobacco and cancer and adjusted the data to show that fewer people actually died, or that people who died had died of other causes, or that some of them were not actually dead (just sleeping very soundly), or were not actually people, and then concluded that tobacco had no link to cancer deaths.
And then demanded legislation that everyone had to subsidize smokers, or that all workplaces had to allow chainsmoking at your desk.
That would be foolish indeed

Remember that the tobacco companies put up are fight to keep people smoking. Millions died.

This is would only be irrelevant, except it also carries the implication of this bizarre shibboleth of the AGW crowd that the only reason one would question their pronouncements is being funded by the deep pockets of the big commercial interests. Just stupid.

If we enact C&T schemes now with high caps for now, at least we will have a system in place. I figure it will take a few yaers to get a the “bugs” out of C&T. If the temps go down, and I’d venture there is a 5% chance of that,
then we keep the caps high. Not much harm done. If we do nothing, and the temps go up and we keep pilling up the CO2, we are going to mess things up big time. No, I don’t think it will be the end of civilization, but major hardships will be impossed.

The cap and trade system is what we should be concerned about as what the big business interests actually want. From Enron to Goldman Sachs, these derivative markets have been promoted by the same scum who have done more to subtract value, transfer wealth to the rich and game the entire system

We do need to weigh the risks of doing nothing. Some people will go to doctor after doctor until they hear what they want. If the first nine doctors tell you to lose weight, eat better and get more exercise but the tenth one says not to worry, it is tempting to go with the tenth doctor, but this is not wise.

This could be a valid scenario. Or it could be completely off point. Or if in the early 1990s someone with chronic ulcers went to nine doctors in a row who all trotted out the conventional wisdom that was accepted but not questioned within the profession and so they all prescribed the same dietary and pharmacological prophylaxis or treatment. Which did not work. Then he went to a tenth doctor who said “We don’t really have foundation for these protocols but they clearly have not worked for you. However some Australians think that a plurality, perhaps a majority of chronic ulcers are caused by H pylori infection, so let’s try some antibiotics which while unproven for this use case have well known and limited potential negatives”

If the models are off, it only means the warming will come a few years or decades later. You can’t get around the physics that more CO2 will eventualy cause big problems.

Removing the adjustments from the actual data and/or questioning the validity or robustness of some of the proxies used makes the warming go away. The models could be directly opposite of reality.
And a simplistic reading of the physics or focus on CO2 mitigation as the sole policy amelioration underestimates the potential effect of other greenhouse gases.
It is entirely possible, maybe even likely that human activities have caused/will cause climate changes. Some of these changes might be warming. Some of this might be from CO2. But for the love of God, can we acknowledge that these are not logical dominoes that lead to one conclusion.
Undocumented assertions from liars who appear to be lacking expertise in statistics, physics, geology, computer science, meteorology or any discipline necessary to actually execute on the promise of this pretend discipline of “climate science” is no reason to introduce panicky changes that may not do anything useful, but will cause economic hardship while transferring wealth to the same derivative traders who made such a great accomplishment of credit default swaps, securitized junk mortgage pools and then took trillions in public funds as dessert.
So forgive anyone who asks” Ummmm… can you show us your work?”