Questions on the CRU email backup server.

A typical procedure [for a well-managed server] is the son-father-grandfather system where when a backup of of the son becomes the father, the old backup [father] becomes the grandfather. Earlier backups are not kept.

Backups are controlled by policy and schedules. Part of policy includes retention time and archival method. Following the Enron problem a number of organizations realized that email was discoverable for court purposes and made radical changes to the retention policy. Where once it was common to preserve certain email for 10 – 20 years it is now not uncommon to keep it around for no more than 90 days. There are other considerations as well – if laws regarding Sarbanes-Oxley, personal credit information, or HIPPA (health data) are involved all manner of laws kick in. These are US specific but the UK and the EU have similar regulations and privacy requirements for archived data.
Archiving data means long-term storage and it is still common to use tape systems. With very long-term retention policies it is common to put the tapes on off-site storage as part of a disaster recovery program. More and more, spinning storage is being used as disk storage arrays become more prevalent. This kind of storage often included deduplication so that the same file, email attachments, for example, will be stored once and referenced many times, and will appear to be multiple copies to the operators. Tremendous storage efficiency is achieved via deduplication but oddly, you end up with just a single backup of everything – all eggs in one basket.
My bet is that unless the email were already part of a court matter and has been FOIA request or subpoenaed for a court proceeding, it is not likely to be held long, and will not make it to archive status. However – every server room that handled the mail potentially has a copy as well. That list of servers would be found from the CC: and BCC: lists.

It would be reasonable for a backup/archival strategy to depend on the nature of the owning entity. For a Forbes Top 100 international company there would be an overriding concern with long term liability and therefore very long term electronic info retention that would be well in excess of the legal retention requirements.
For a low tier public UK University whose risks are covered ultimately by the UK Government, it sadly seems reasonable to expect much less rigorous IT strategy on backup/archival. I think that is what we are seeing from CRU’s FOI responses and I think in parallel we are also seeing a resistant attitude by CRU that is less than an open and transparent one.
John

Is this common practice: “the backup server had an automated function that operated so as to remove older backups on a rolling basis”. Wouldn’t it be standard practice to periodically preserve some of the older backups?
>>>>>>>>>>>>>>>>>>>>>
The statement is standard practice. The question seems to be a misunderstanding of the meaning of the statement.
In backup systems, you can make a “full copy” of the data, or you can make an “incremental copy”. The latter is simply a copy of all the changes since the last full copy. The most common rotation is daily incrementals and weekly fulls. So yes, it is standard practice to preserve full copies of the data for some period of time, often 7 years. After that, when the next full copy of the data is made, it overwrites the oldest copy in the system, be it on disk or tape. Or the tapes over a certain age are discarded, or one of several other possibilities. There is nothing “fishy” about removing older backups.

Backups at a larger institution/company are usually governed by two things: legal requirements (i.e. for purposes of taxation or financial auditing) which are obviously not negotiable and a policy for “the rest” which is a compromise of constraints on financial and staffing needs vs. the need to recover old information.
In the academic world one would assume that the researchers would put a big emphasis on having all their work protected with a backup plan, in particular since they usually are notoriously difficult to educate about keeping their files on the network instead of their local computer drives. They also don’t delete stuff as long as they have space, in my experience. The proper backup policy for that would be an incremental backup plan that starts with a snapshot of “what is” and then only records changes from run to run (usually just additions and revisions of single files). If space preservation is an issue, you’d limit the number of total increments and do a new snapshot every year or so and then begin increments again.
As far as email retention is concerned, the only real issue are attachments, not the email body per se, which is just text. That doesn’t take a whole lot of space and additionally can be compressed quite well on the fly. From the perspective of backing up emails sans attachments there is really no reason to be fickle about storage requirements. Considering all the material is centralized on the email server anyway, it can be backed up quite easily and efficiently.

Backups of operating system active drives typically use rolling backups…because why would you need a backup from 3 years prior if your intent is simply to recover the operational state of the machine?
>>>>>>>>>>>>>>>>>
Backups serve more than one purpose. If your only goal is to preserve the last operational state of the machine, then yes. But there are many cases where this is insufficient. Suppose you discover that you have a virus on your machine. How long has it been there? What has it been doing to the data? How do you back those changes out?
If the virus has been making subtle changes to the data, you need to know when that started. If all you have is the last operational state of the machine (say from one week ago), and the virus is on that copy as well….you’re hooped. If the virus has been present for say 9 months, you’ll need a copy of the operational state of the machine from before that.
The above is only an example. I’ve had situations where we needed to recover the operational state of a server from as much as 3 years prior to resolve an issue. For data, I’ve had situations where we had to go back to copies of the data wich were 3 years old to resolve a discrepancy in a financial system. Operational recovery is simply one aspect of backup and archive.

Some resources:
UEA Records Management Policy
http://www.uea.ac.uk/is/strategies/infregs/Records+management+policy
Data Protection Policy
http://www.uea.ac.uk/is/strategies/infregs/dp/Data+Protection+Policy
Data Protection Act 1998
http://www.legislation.gov.uk/ukpga/1998/29/contents

Occasionally an astute IT manager will make an on-going series of back-ups of the Emails without telling anyone else; that way if the C-levels are under supoena, they honestly don’t know it exists, yet he can still say “let me take a look around for what you need and I’ll get back to you” if they need a lost email. I hold little hope that this will be the case in anything doing with climatastrology.
Perhaps it’s time for researchers receiving public grant money to start acting like real-life grownups, quit the embarrassing sophomoric trash-talking, the back channel communications, and the hide the original data BS. We need a data retention law for federal grant funded research data, something like can’t produce the data, give the money back or go to prison, lose the Email, give the money back or go to prison.
Seriously the if the Hockey team is correct, then the fate of the human race rests on the ability that they can convince us to spend U$ trillions to fix the CO2 problem; if that were really true, would they be so lackadaisical about retaining the original data and methods? I honestly feel that these buffoons have failed to demonstrate the maturity to allow independent living without adult supervision.

Don’t confuse “backups” and “archiving”.
A “backup” is a copy of the data to allow recovery in the event of a disaster.
“Archiving” is the process of moving un-used data from primary storage to secondary storage (secondary storage could be tape, or even low-cost slow hard disk).
Retention is set by policy, and it is very uncommon to have long-term retention. Nearly all organisations only set retention policy based on tax laws. If you need to keep something for tax reasons for five years, then retention policies will mostly be in the 5-7 years.
Long-term retention is made difficult due to changing backup software (even upgrades of the same software), hardware changes and even deterioration of the backup media. It can be extremely expensive to cycle your retained data to modern standards. Most quotes I’ve seen come to about one thousand times the cost of originally backing up the data, to cycle it to modern standards.
Steve should however enquire as to what JOURNALING CRU employed. This is the IT secret, and how they’ve been able to recover an email from years ago (when pushed to do it). Basically, nearly all email servers file a copy of every single inbound and outbound email in what’s called the journal. This acts as a log for IT email troubleshooting, as well as compliance with Sarbanes.
However IT generally doesn’t admit it exists, as the journal means they can easily read everyone’s emails. (The journal is also why the IT person is the best person to find out what secret plans an organisation has, and why they seen to know a big decision ahead of time). 😉
I wouldn’t worry too much on the server name. When IT rebuilds a server (for whatever reason) they usually use a number to indicate what iteration they are up to. It’s extremely bad practice to rebuild a server with the same name (can cause all sorts of technical problems on the network). Also, CRUBACK is most likely to be the email backup server, as in there would be an active and backup server in case the active goes out of action for any reason. It’s to ensure uptime, not to hold backups of the email server.

Perhaps it’s time for researchers receiving public grant money to start acting like real-life grownups, quit the embarrassing sophomoric trash-talking, the back channel communications, and the hide the original data BS. We need a data retention law for federal grant funded research data, something like can’t produce the data, give the money back or go to prison, lose the Email, give the money back or go to prison.
I agree completely with regard to the data and methods, but again, that’s a matter for the federal funding agencies to require and enforce. NASA, for example, has the requirement and recently, has been quite stringent about enforcing it. I suspect that within a very few years it will be an across-the-board requirement for federally funded research. As for “trash talking” and “back channel communications”, I fundamentally disagree and indeed, think you are treading heavily on the toes of the first amendment. Both sides in this debate engage in far too much of the former — WUWT is full of trash talk and presumption of guilt and malice aforethought in “all climate researchers”, and there are at least some participants on the CAGW “side” of the discussion that are no better. But this isn’t a legal matter, it is a social one or a political one and grant agencies literally don’t have the right to take trash talk or political agenda into account when considering whether to fund or not fund a proposal.
As for back-channel communications, are you trying to make it illegal for me to sit in my office and have a private discussion with somebody? How about calling them on the phone, am I obligated to make a recording of it if I discuss any sort of research? When I go to meetings, do we need to hire a video crew to make videotapes of not only the presentations but all of the private discussions, including those held while sitting around drinking beer late into the night (which happens remarkably frequently, as you might well imagine)? Where, precisely, is there a privacy boundary that permits my unguarded email conversations with people, where I might well be voicing new ideas, patentable concepts, doubts, concerns, or just blowing off steam with regard to a person that I think is a butt-head for entirely personal or any mix of personal and professional reasons, to be open to all comers armed with the flimsiest of excuses but still protects my right to hold exactly the same discussions offline in any number of venues?
Personally, I don’t think there is one. I think email should be precisely as private as personal mail — literally a federal crime to open or expose without a court order obtained on grounds more substantial than a “FOI request”.
This has nothing to do with whether or not CRU or other agents involved in the climate debate should or should not be required to provide their data and methods not only “upon request” but to make them utterly publicly available without any request. Again, NASA is actually not a bad example of the way it should run — you can get ALMOST any of the data used in NASA funded publications, and MOST of the code (maybe all of it, but I doubt it because thing don’t work perfectly) straight off of the internet in real time. I’m also not commenting on whether or not there was academic dishonesty or unethical behavior exhibited on the part of the hockey team members and revealed by climategate and many other events, or whether or not there were real crimes committed in there somewhere.
While one might wish for a greater level of maturity and tolerance by ALL participants in the discussion, one part of freedom is the freedom to be immature and intolerant in one’s communications. Usually, in science, such behavior in the long run carries the seeds of its own destruction. Nature, like honey badger, just doesn’t give a shit about what people think — it is what it is and does what it does. You can trash-talk all you want and even believe true things for stupid reasons — if you turn out to be right. But heaven help you if you turn out to be wrong, as then you are wrong and stupid, not just wrong. When discussing nature it’s a lot wiser to be open to the possibility that, no matter how passionately you believe something to be true, you could be mistaken, and practice just a teensy bit of humility on that account.
rgb

rgbatduke says:
October 7, 2012 at 1:03 pm

John Whitman says, “A US gov’t grant receiving scientist’s emails discussing the government funded research he is formally and legally contracted to perform are not private emails.”

That is simply not true, and indeed, it is absurd. What a scientist who receives a grant is morally obligated to provide is an end product, not every single step of the process through which that product is obtained, including video recordings of every discussion with a thesis advisor, some sort of reality TV taping of every day’s work in a lab, every single bug introduced or resolved while writing the code, every single concept considered and accepted or rejected along the way. For one thing, there isn’t enough bandwidth even in the modern world to provide that kind of trail. For another, it would utterly squelch the freedom to be wrong, especially in intermediate steps of research, if every half-baked idea one has but might end up rejecting becomes part of a public record that can be trotted out by political enemies of your conclusion seeking to prove that some ideas you have had, some things you have said, some statements you might have made in an utterly unguarded venue as part of the search process that is real science turned out — surprise — to be wrong.
[ . . . ]
rgb
– – – – – –
rgb,
Hey, appreciate your return comment.
The way I understand it, you are saying that the ‘in-process’ part of performing government funded research work is exempt from the rules regarding working for the government, that is the process is exempt from the spirit of openness and transparency in government.’
Are you suggesting that in principle no investigation should be allowed by media, private citizen or enforcement agencies is possible wrt the in-process activities of a government funded scientific researcher? During the time they receive grant money and the time they produce the research product a publically funded research scientist, are you suggesting there should be exemption from investigation by private citizens, media and enforcement agencies? They get a free pass?
My view is that by your argument you appear to be emulating the argument/position of the IPCC AR5 leadership wrt preventing in-process openness and transparency. I do not think we should just trust in their product’s unbiased, open and objective nature. Trust does not work that way. I do not think science works that way.
John

Sounds like the file system that was backed up in 2009 contained email archives dating back some time

rgbatduke says:
October 7, 2012 at 10:53 am:
“There are some cures that are worse than the disease, and the current tendency to wield the FOIA as a political weapon or instrument of a kind of harassment is one of them.”
========================================================
I have never heard of such a “tendency”. You are the first one I hear to claim that such a tendency exists. At the moment I doubt that you or anyone else scientifically studied a sufficient number of FOIA requests and can prove your claim.
As far as understand, FOIA requests concerning climate have a purpose to expose climate liars.

Large disks are now very cheap. It was not always so. Going back a few years, backup servers were generally struggling for space so it was common to delete all but a small number of old backups.
For simple data recovery backups, you frequently kept just two or three backups. The more paranoid (a.k.a. sensible) would use some sort of “grandfathering” algorithm.
For permanent(-ish) archives, they were generally stored on some form of tape, or maybe a DVD or (recently) a Blue Ray disk.
For people without an (implemented) retention policy it would be common to keep email and working copies of files in just a small number of backups. Important files (data, work product, etc.) they might be archived by project at certain milestones (e.g. project release).
Obviously people implementing a retention policy would need a longer-term solution>

Are you suggesting that in principle no investigation should be allowed by media, private citizen or enforcement agencies is possible wrt the in-process activities of a government funded scientific researcher? During the time they receive grant money and the time they produce the research product a publically funded research scientist, are you suggesting there should be exemption from investigation by private citizens, media and enforcement agencies? They get a free pass?
Not at all. Here’s what one can investigate. Is researcher A conducting the contracted research, spending a reasonable fraction of their time on it? If researcher A alleged that they spent salary money hiring postdoc B and grad student C, did they indeed do so? When postdoc B spent valuable grant moneys on a trip that was claimed to be to a workshop relevant to the ongoing research, did they in fact do so or did they instead use the money to purchase cocaine and hookers? Finally, when the contracted work is eventually published, did the author(s) of that work actually do the work alleged to have been done in the publications and obtain the results those publications assert? In the event that the work was done for an agency (such as NASA) that requires data and methods to be made available post-publication, are they in fact both available?
None of this implies carte blanche permission to open their underwear drawer and paw through it at home, or their email spool to paw through it at work. All of the first part of it is satisfied by what we in the business call an “audit” and is a routine part of actually doing work for a granting agency that is administered by e.g. a University. Every year I fill in a form (even though I’m not currently supported by any grants) certifying my “effort” distribution among activities such as teaching, doing research, and non-University work such as consulting or entrepreneurial stuff. This certification must be co-signed by the department administrative staff and correspond within reason with their observations and resolve any complaints of me doing or not doing my job. This goes to sponsored programs, which is responsible to all of the granting agencies supporting work done at Duke. At any time, a granting agency or sponsored program can and cheerfully will conduct a far more detailed audit if there is any reason to suspect abuse.
Nobody gets to sit in judgment of the work itself except my colleagues — which includes everybody in the whole world, but with weight given according to expertise and corresponding effort. Nothing constrains my results, which may or may not be what I expected them to be when writing the proposal, politically popular, or what everybody else gets when they study the same problem. There is no recourse for the granting agency to recover any part of the contracted funds should I or my work prove to be incompetent or for that matter mildly wasteful of their money at a level short of active embezzlement — their only recourse is to not fund me in the future if I waste their money now.
The single exception to this are as follows: If, after my work is published, it proves to be egregiously non-reproducible in a way that suggests that I faked my results and the work itself is broadly challenged as not, in fact, being an objective presentation of work that I actually did, a granting agency and sponsored programs can delve deeper and investigate the possibility that actual fraud was committed. They don’t even need a subpoena — I give them implicit permission to do this when I apply for and accept the grant. At that time I do indeed have to open up my “books” to them, show them the details of my intermediate work, show them the actual data if I’m an experimentalist (I personally am not) and/or work from data, show them my methods or code or algebra if I’m a theorist (which I personally am).
Only in the event that my work proves to be academically dishonest — to put it bluntly, I either faked the data or claimed to have done work I did not actually do — does my employer (the University) and the granting agency have to pursue civil remedies and of course the possibility of criminal prosecution also looms if my actions shade over towards embezzlement rather than mere sloth or academic dishonesty. The most likely outcome in this case would be me getting fired, for cause, end of career in academia on the spot, the University absorbing the loss (they were responsible for my hiring in the first place) and reimbursing the granting agency as a condition for continuing to receive grant money in the future. It is possible, but moderately unlikely, that the University would pursue a civil suit against me to recover some or all of this loss.
Sadly, the University is the home of a poster child for scientific misconduct, Anil Potti, see:
http://en.wikipedia.org/wiki/Scientific_misconduct#United_States
Crime: Faking data extensively, over years, to assert egregious claims in the general area of cancer research IIRC. Revealed by the utter failure of his work to be reproducible. Exposed and verified by interviews with his postdocs and research fellows, by delving into the details of his supposed “patient data”, by interviewing the patients. Punishment (so far): systematic withdrawal of all of his papers. Public humiliation. Termination of employment, with a near guarantee that his future career will be as a used car salesman or working in sanitary engineering. Exposure to lawsuits seeking to recover damages on the part of both the University and any patients who claim to have been hurt by the “therapies” that he claimed were fruitful on the basis of faked data. Possible criminal prosecution, although it will never happen — it doesn’t need to, he’s finished.
Note well that the issue isn’t whether or not his work was mistaken — there is a ton of stuff published in science that turns out to be wrong. It doesn’t have anything to do with whether or not he believed his own claims — I rather expect that he did, so strongly that he didn’t think he needed to actually verify them or to the point where he was willing to bend them by a few percentage points to make them statistically significant instead of insignificant. At issue was scientific misconduct — deliberately faking data, bending it by those percentage points, and publishing something that as a consequence was egregiously false.
Note well that at no time is the implicit right to investigate cases of possible scientific misconduct for cause by the granting agencies and the adminstrators of the grants to be construed as a carte blanche invitation to the general public to embark on a witch hunt at will anytime the results being published by some researcher don’t agree with their own personal and political biases and beliefs. Nor does it extend even to the granting agency and/or administrator complete freedom to open and read all correspondence electronic or otherwise.
Finally, only a tiny handful of even the climategate emails cross the line and provide evidence of “edgy” behavior, behavior shading on scientific misconduct. Incompetence yes. An egregious tendency towards confirmation bias absolutely. An entirely unhealthy superman save-the-world complex sure. But none of these are scientific misconduct; they are scientific incompetence.
To be explicit, Michael Mann’s promotion of bristlecone pine trees from a tiny fraction of the surface of the Earth into the dominant component in a flawed PCA method that produced only hockey stick shapes, and the subsequent promotion of his hockey stick into “the” proxy record of global temperature by political extremists is not scientific misconduct on Mann’s part, it is scientific incompetence on his part and political misconduct on the part of others. If he had faked the data it would have been scientific misconduct. Using handwritten PCA code tuned to produce a result that he expected to find, however, was just incompetent because if he were competent he would have doubted his own work and taken steps to verify that his own results were robust instead of waiting for M&M to do it for him. The one place where his activities edged over into scientific misconduct was his reluctance to release his data and code when M&M requested it, but back then granting agencies were a lot more lax with disclosure rules if they had them at all!
This reluctance was rather understandable from a human point of view. Even if you publish results obtained in the best of faith — and I have little doubt that Mann’s results were good-faith results and do not think to this very day that he doubts that they are correct — the public demonstration that they are crap because you made a number of mistakes and failed to exercise sufficient care checking them is rather bad for a career. To some extent we create this monster by supporting a research system that despises negative results and rewards positive ones. As Feynman points out in the cargo cult lecture, negative results are often more valuable (and honest!) than positive ones but nobody gets tenure and future grant money for negative results. Why should we then be surprised when young researchers with their entire future on the line drag their heels and seek to at least postpone the day of reckoning when they make a mistake that could cost them their entire future career?
The structure of the ivory tower itself created Potti, created Mann, and continues to create piles of poor results every day. Incompetent results. Results tainted with confirmation bias and cherrypicking of data, but still honest results in the sense that the researcher in question honestly believes their conclusions to be true and can rationalize their arguments for inclusion of the data in some way. The whole point of the scientific process is to correct for this nearly inevitable tendency by exposing it to the process of verification and falsification. It is, or should be, all right to make a mistake, as long as you don’t lie and create false data to support it.
I am far from immune to this myself. I spent a decade or so investigating the critical behavior of the classical Heisenberg ferromagnet via Monte Carlo computations. This is a model that looks like it “should” be analytically solvable, but in fact it has so far proven intractable (only one major magnetic model — the Ising ferromagnet — has proven solvable). In the course of my simulations I discovered that the dimensionless critical temperature/coupling of the model was very, very close to , and I came to believe that it was this value (it fits to between four and five significant figures). This was naturally very exciting — it suggested that a simple/tractable solution might exist where this result could be proven!
If one had opened up my email records to scrutiny, I’m quite certain that they would have contained all sorts of discussions of this that I had with colleagues I was working with. I was quite convinced of it — too many digits to be a coincidence, surely, I thought. However other researchers got numbers that were very slightly different from mine — still within my error bars, but they claimed smaller error bars. I doubted their error bars were as small as they claimed they were, they agreed that my results were correct as far as they went but that I needed to make my error smaller.
They, as it turns out, were quite correct. Eventually I accumulated enough data to positively reject my own hypothesis of a “nice” analytic form for — in the fifth and sixth significant digit — within my own error bars. The seductive pretty story sadly gave way to actual computations that eventually falsified it.
At no time did I participate in scientific misconduct, mind you. I didn’t fake my results. I didn’t misreport them. I presented my hypothesis as a hypothesis, not a proven fact (one would have to have the derivation itself to report it as a proven fact). But I damn skippy sure believed, throughout a couple of years of work, that the result was analytic, and presented results that were very slightly erroneous by assuming (as a stated assumption) that this was .
This is only one of several pretty stories I’ve been misled by over the years. The formation of a scientific hypothesis is the formation of a pretty story. It is by its nature seductive, the observation of a pattern and a proposed explanation for that pattern. It isn’t always easy to do work to accept or reject it, either — one’s results tend to reflect the bias of assumptions made in obtaining them, the bete noire of scientific research and the reason many eyes and hands looking at things and challenging one another are good.
In the end, there are checks and balances in the scientific process a plenty. Treating the email spools of every scientist in the world as open to public scrutiny is not, and should not, be one of them, not without far more cause than what is behind a typical FOI request. If someone wants my code — they are welcome to it. If someone wants my data (well, I personally don’t have any data, theorist) sure, why not. If somebody wants to read through my private emails, many of them discussing students with special needs and other things that would violate privacy laws if exposed, just so they can see whether or not there was a time I believed something that turned out to be incorrect, such as the value of for the critical Heisenberg ferromagnet, well, all I can say is get a life. Go away. Leave me and my email spool alone.
That’s a line that can and should only be crossed armed with the equivalent of a warrant issued on the basis of actual hard evidence of a committed crime, precisely like my real underwear drawer in my own house, precisely like my real, federally protected paper mail. Email and paper mail should absolutely enjoy the same protections under law, and those protections should completely block and actively punish its exposure by third parties armed with anything less than a court order, or even my employer (who technically “own” the spool file itself) acting as an agent of granting agencies that support me without just cause and due process.
I repeat: Without a reasonable expectation of privacy, frank discussions and speculations essential to the scientific process cannot occur. Freedom of scientific discourse includes the freedom to be wrong, the freedom to be misled, the freedom to pursue the wrong path to a fault, to stubbornly stay on it even in the teeth of contrary evidence. The system has checks and balances galore built into it, and is loose enough to tolerate the iconoclast in the short and intermediate run while still ending up quite rigorous in the long run. FOI is often being used as a weapon and tool for harassment in a political war, not a scientific war, and is just as repugnant to me as the gatekeeping and other over-the-edge activities revealed in climategate. It is one thing to ask for data and methods, quite another to ask for open access to email spools so one can see if there is a “conspiracy” of some sort that can be discovered there.
Whatever happened to ordinary civility in scientific discourse?
rgb

rgbatduke says:
October 8, 2012 at 7:56 am
[ . . . ]
I repeat: Without a reasonable expectation of privacy, frank discussions and speculations essential to the scientific process cannot occur. Freedom of scientific discourse includes the freedom to be wrong, the freedom to be misled, the freedom to pursue the wrong path to a fault, to stubbornly stay on it even in the teeth of contrary evidence. The system has checks and balances galore built into it, and is loose enough to tolerate the iconoclast in the short and intermediate run while still ending up quite rigorous in the long run. FOI is often being used as a weapon and tool for harassment in a political war, not a scientific war, and is just as repugnant to me as the gatekeeping and other over-the-edge activities revealed in climategate. It is one thing to ask for data and methods, quite another to ask for open access to email spools so one can see if there is a “conspiracy” of some sort that can be discovered there.
Whatever happened to ordinary civility in scientific discourse?
rgb

———-
rgb
Again, appreciate your continued dialog in this important subject of the role of open transparent professionalism by climate scientists in the performance of the entire process of executing a US government contract for climate research.
We are from different philosophical foundations, apparently.
Private is private, but conduct of public research is inherently not private in any common sense way.
I see that a climate scientist receiving US government funding is not prevented from fully and creatively conducting activities via the scientific method especially if there is a clear public expectation of ‘in-process’ openness and transparency in conduct of government funded climate research. I really see no problem for said scientist. I see no dilemma, whereas you have expressed, at length, a critically severe concern over ‘privacy’ and ‘suppressing’ free thought’.
What scientist fears being seen, during the entire process of grant through final research product, as a highly professional and open scientist in public research? I see no suppression of free thought and no intimidation. I take that as civility in the highest possibly mode. Where has civility gone you ask? That is where civility should be.
In my long professional experience in corporate America, there was never the expectation of any ‘in-process’ privacy from the owning corporation’s eyes; it did not reduce creativity nor dampen open discussion of even our failed attempts at achieving original goals . . . on the contrary the openness stimulated new ideas and solutions. Likewise, that same expectation of no privacy from owning government (and public) eyes during the conduct of government funded research is not inherently inimical to creativity, false starts and failure to meet original goals; in fact it can and should enhance the creative ideation process and scientific objectives.
As to fundamental philosophical differences; I live in a benignly indifferent universe.
John

rgbatduke says:
October 8, 2012 at 7:56 am:
“That’s a line that can and should only be crossed armed with the equivalent of a warrant issued on the basis of actual hard evidence of a committed crime, precisely like my real underwear drawer in my own house, precisely like my real, federally protected paper mail.”
=======================================================
Look, talking about your real underwear, I’ll give you one example: going through Customs. They can touch your real underwear without a warrant. They can confiscate or arrest it under circumstances. You can not reasonably say they should not be allowed to go through your luggage, because you might have some real underwear there.
Back to the taxpayers funded climate liars, it looks like there are quite a few around and they do not just perpetrate their “research” alone, but in teams. In this context their emails are relevant and no way are their taxpayers funded working emails private. And then there is a law allowing taxpayers to look into that stuff. This is the way the liars can be caught and I hope they will.