New Climategate 1 and 2 combined search engine

Jo Nova writes (and I’m repeating here to get he word out)

Ecowho FOIA Grepper

Behind the scenes, I’ve been playing with a new neat tool for hunting hypocrisy, corruption, bias and unprofessional behaviour and I’m pleased to announce its ready to share with the world. The kudos for this all belongs to, as usual, a skilled volunteer. Thanks to EcoGuy for turning his rapid-fire coding ability onto this.

On the  EcoWho site he has helpfully placed all of Climategate I and II together into a combined searchable database. It’s fast, easy to scan, it copes with tricky search requests and provides a link to the full email from the results page of the search.

Welcome to the ClimateGate FOIA Grepper !!!

 

 

0 0 votes
Article Rating
22 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
John A
November 25, 2011 2:14 pm

Now if someone will come up with the key to the all.7z file we can find out what was being said entirely out-of-context…

Harriet Harridan
November 25, 2011 2:30 pm

Thanks for the link. This one is also great: http://di2.nu/foia/foia.pl
You can search CG I & II, you get three summary lines, which pop open on clicking.

November 25, 2011 2:31 pm

Thank you, Jo and EcoGuy.

jaypan
November 25, 2011 2:50 pm

Dangerous tool … for some people involved, but very helpful for all of us.
Thank you.

tallbloke
November 25, 2011 2:56 pm

Excellent!

Baa Humbug
November 25, 2011 3:55 pm

tallbloke says:
November 25, 2011 at 2:56 pm

Excellent!

hehehe were you rubbing your hands together while saying that Mr Burns?
On a grateful note, where would we be without all those volunteers? Thank you EcoGuy and Jo.

John
November 25, 2011 7:59 pm

Unbelievably nice – I was able to get search data after only a few seconds. This community is awesome!

dp
November 25, 2011 11:19 pm

It would be nice if it could handle this:
egrep -i -B3 -A3 “delet|foi” *.txt
It means find delet (which includes delete, deleted, deleting…) OR foi, foia, foible, etc. Earlier the site was struggling with “foia” and returned pointers to the file space.
If you want a fat dose of deceit, try entering “lawyer” as a search pattern. They clearly have no clue that FOIA is not negotiable. Some time in jail away from their broken thermometers will afford them some time to study the act.
I built a table of From: -> To: addresses to get an idea of who was talking to whom. Very interesting stuff. I’m now unconvinced all this mail came from one source.
Keep up the good work

FrancisT
November 26, 2011 3:47 am

@dp
I have been thinking about creating a more advanced grepper with support for regexes etc. on my one (http://di2.nu/foia/foia.pl ) currently it’s just doing “grep -i ..”
I’ll probably add a separate page with this more advanced version, if you have any other suggestions please leave them here because I cna easily change it again

Bloke down the pub
November 26, 2011 4:01 am

I hope that while everyone is combing through climategate 2.0 we’re not missing something else going on in the world. The Team seem almost glad of the distraction for the time being.

November 26, 2011 4:30 am

Early criticisms from “Team” supporters have dismissed email quotes from FOIA 2011 as being taken “out of context”.
But stringing together pearls of evidence from searches using “Kendall” is highly revealing. http://di2.nu/foia/foia.pl allowed me to view many fragments simultaneously. I’ve extracted key elements simply to allow the story to flow, to reveal evidence that I think is even more damning when taken “in context”. Comparing these 2011 results with JoNova’s grepper’s results shows that all the Kendall-related emails were released only just now – in other words, the 2.0 Climategate release is, as we already know, putting everything more and more “in context”.
Dr Alan Kendall taught about climate at the University of East Anglia until he retired in 2010, apparently. He is clearly of a different mettle to his “Team” colleagues. The emails tell a whole story of his parting company with his colleagues over time.
Also, tellingly, the references in emails 3114 and 4341 to Douglas Keenan, Steve McIntyre, Willis Eschenbach and Climate Audit show how legally inappropriate it was to omit all these key witnesses in the Oxburgh and Muir Russell / Boulton “enquiries”. The full references have been omitted below, but they are highly relevant.
EMAIL 1443 – In August 1999, Tim Osborn invites Alan Kendall to join a “discussion seminar on detection of anthropogenic climate change”. The atmosphere is congenial, friendly, relaxed.
EMAIL 2380 – By September 2006, the atmosphere has changed. Dr David Viner writes, apropos a letter Kendall sent to The Guardian, “Whilst Alan may have his own personal views on climate change (that are not supported by the science) I do not believe he should be using our School’s affiliation on these letters”.
EMAIL 3143 – December 2006 is the time when battle lines are drawn, judging by the tone and quantity of comments.
Alan Kendall emails “Dear all, I trust you enjoyed the seminars yesterday”:

…3.Next week’s seminars are upon 1) clouds and landuse changes influencing climate change, 2. the deficiencies of climate models, and 3.other causes of climate change, in particular solar changes…
4. …Dave Viner and Kieth Briffa have “threatened” to come as well. Is this because they were stimulated by the idea of these seminars or because you need to be “put right” after being subjected to undue influence by your’s truely? Regardless, they are very welcome.
5. Finally can I emphasize that you are being asked to present the evidence for the proposition that evidence exists that is contrary to the commonly accepted “consensus” and to answer questions from this particular viewpoint. You are not being asked yourselves to assume any particular stance…

David Viner emails Alan Kendall in reply to this:

There is a nice article in this weeks New Scientist on the “Hockey Stick” graph, page 9. If anyone can send me any published literature (in the peer-reviewed journals) that does support the view that the current rapid rise in CO2 (and other GHG concentrations) and associated temperature changes are not down in part to human activity please do send them to me. Also if anyone can find the literature that states that 1990 were not the warmest decade of the last millennia please send me the paper.

Alan Kendall replies:

David, pity the author of that New Scientist piece doesn’t know all of the literature. As you are aware, there are now dozens of peer-reviewed papers identifying the MWP and LIA outside of the North Atlantic… Objections to the “hockey-stick” do not entirely reside with the reality (or not) of the MWP or LIA. As I read the criticism, it mostly concerns the invalidity of the statistical manipulations used.
I also thought that the Osborn & Briffa diagram you used last week did identify the MWP and LIA, so why are you seemingly supporting the hockey-stick” by recommending a “nice article in this weeks New Scientist” ?
As for supplying references, why bother? Any, like the Peruvian glacial study, are immediately ignored or explained away as being local.

EMAIL 4341 – In March 2007, Dave Palmer, UEA FOIA expert, emails Kitty Inglis

Got a call from Alan Kendall, ENV, expressing concern regarding UEA’s ‘visibility’ on a Climate Audit website. He was quite concerned regarding the comments surrounding the publication on this website of our letters of response to requests… I’m not sure if this website has a limited readership or is something we need to be concerned with but the extent and nature of comments is multiplying rapidly. Not sure I wanted my ’15 minutes of fame’ this way lol

Palmer adds a message to Phil Jones

Phil, you are quoted in a comment to the 9 March 2007 posting…

Jones replies to Alan Kendall

I appreciate your concern about UEA and ENV’s image, but I don’t appreciate you calling our press office about what is happening on the Climate Audit website. The website is run by a self appointed group, who ignore most climate facts. They are not interested in getting at the truth.If you want to learn more about the subject I would suggest the Real Climate website. http://www.realclimate.org/ I do occasionally look at Climate Audit. The group are clearly very right of centre, and just make things up to suit their arguments… We have signed agreements with some national Met Services not to pass on the raw station data.

to which Kendall replies:

Phil, I’m sorry you take exception to my contacting David Palmer about this matter. I felt that, since the University was being “slagged off” on a much read website and people were advancing possible legal ways in which the information they request could be obtained from the university, it was important to warn David Palmer about this development. I assumed that you would be aware of what might be being said on that website, whereas David Palmer would not be expected to be so informed. In fact he was not, and thanked me for the information. I’m not exactly sure what you are complaining about and if the same situation arose again I think I would act in exactly the same way again.
What you do with your data is not my concern, nor was it ever. I would not presume to interfere in this area, nor have I done so. I am, however, concerned that UEA will be beset by possible legal challanges and I consider it only prudent to warn those involved. Here ends any further envolvement.
Also please don’t you presume to lecture me about what particular website I should or should not be consulting. I’ll make my own mind up. In actual fact , I read both ClimateAudit and RealClimate.

To this note, Jones replies:

I have told these people on several occasions that some of the data are restricted. They refuse to believe this. We make all the gridded data available on the CRU web site. 99.9% of climate scientists are happy with this, and out data are widely used. I have met Dave Palmer and told him of the web site and that his first response letter appeared there, so he did know about it… Also, these people are self-appointed auditors. They have no interest in doing their own work. I suggested once that they produce their own gridded temperature database, but they said they weren’t interested in that. This to me tells me where they are coming from – and it isn’t the science. I have tried engaging with them in 2003/4 but realised after a few months it was pointless. Anyway, we can differ on this. Glad to hear you look at Real Climate.

Lastly, Kendall replies:

Phil, I can assure you that Dave Palmer did not know about the particular threadline that I drew his attention to because, during my telephone conversation with him, his voice expressed surprise (and perhaps resignation?) as he opened up the website and because he thanked me for drawing the information to his attention.
I repeat, I am not primarily interested in the dispute between yourself and ClimateAudit and will keep my opinions to myself. However when I read that people are suggested methods of legal redress against the University for not supplying research data, I felt that I needed to act. I could not contact you by telephone, so resorted to informing Dave Palmer.
I would do exactly the same if similar circumstances arose again.
AlanK

EMAIL 3114 in March-April 2007 is mainly a sequence between Dave Palmer and Douglas Keenan, starting at around the same time as the above. Dave Palmer is slowly waking up and we see his fresh reactions which seem reasonable in the circumstances. His emails widen to include references to requests from Steve McIntyre and Willis Eschenbach, and draw in Phil Jones for comments. Jones includes this titbit:

I had a brief exchange with Alan Kendall of ENV, which I didn’t find very useful.

EMAIL 3281 – In October 2008, Alan Kendall emails:

Dear all, I don’t intend to get into a debate about this (my suggestion was made primarily on the basis that climate changes are different in varied parts of the Earth) but Phil’s disparaging comment about “a few Russian mavericks” denies the existence of almost an entire group of specialists worldwide – the solar physicists – who seem to have had a particularly rough time in recent years – being demonized by AGW advocates for their views about climate that fly counter to the “IPCC consensus”…
PS Phil, I’d be willing to make a small bet with you that over the next 5 years we will have increasing evidence of a cooling – with the PDO entered into a cool phase; cycle 24 of the Sun still not officially arriving yet and increasing “Earthshine” I think I would be onto a winner. But then I’m probably a maverick! …

and Jones replies by pulling rank:

It would be unfair to get involved in a bit with you, as I know a couple of things you don’t.
1. The Arctic issue. We’re getting SST data in from ships travelling around in regions where we haven’t had any data from for the base period. We’re still figuring out how to use these. However we do it, it will only raise temperatures.
2. SST is being measured differently now than it was in the 1980s…

EMAIL 5182 – In September 2009, Professor Neil Adger emails his staff on the schedule for the incoming semester on ‘Adapting to Climate Change’: “Welcome back to 1A01 – where we inspire the next generation of environmental scientists. Attached you will find the updated course manual for the semester. It is all but unchanged from last year…”
Alan Kendall replies to all recipients, including Tim Osborn:

Neil, last year I gave two lectures and a practical-seminar in ENV 1A01. The lectures were in week 7 and 12 and were upon 1) climate change and 2) peak oil and its effects respectively. I had an agreement with Graham that my week 12 lecture would be transfered to week 7 for 2009 so consolidating my teaching in this module… Now I find that, without any consultation, my second lecture (and probably the most important one) has been eliminated… Can you confirm this and give me the rationale behind your decision?

EMAIL 4333 – A fortnight after email 5182, Prof Adger writes to Tim Osborn:

Tim, I have had to go through 1A01 lectures to extract reading lists etc and I read the slides of Alan Kendalls lecture from last year which he is planning to give this year. I attach the slides. As you will note there are many assertions that will be very confusing to students. And the lecture gives no sources for where the students can follow up on these and verify or falsify what Alan is saying. I would like a quick chat about this. Not sure what to do perhaps just let is slide given that this is the last year of Alans teaching etc.

Tim Osborn passes on this note to Phil Jones:

don’t pass on further, but let me know what you think!

and emails Mick Kelly (EMAIL 3252):

… On another note, do you know whether, when you were teaching ENV-1A01, Alan Kendall was giving a counter climate change lecture later in the module full of climate skeptic nonsense? It turns out that he has done so for the last year or two, and possibly before that, and is planning to do so again this year. His lecture slides are full of the usual stuff (CO2 absorption is already saturated, CO2 lags temperature, instrumental temperature is unreliable so it isn’t warming and anyway it has cooled since 2001, etc.)…

Mick Kelly answers:

…That’s amazing re Alan Kendall (always thought he was rather a loose cannon)… Kind of ironic that I used to invoke academic freedom when criticised by ENV for ‘promoting’ concern about global warming and I suspect Alan might respond in a similar fashion….

EMAIL 2639 – A week after email 4333, Tom Wigley apparently emailed Ben Santer and Phil Jones: “See the item by Pat Michaels [from Benny Peiser to “cambridge-conference”] – “THE SUN COULD BE HEADING INTO A PERIOD OF EXTENDED CALM…”
Ben Santer replied:

Dear Tom, This is a vicious and unjustified attack – not only on Phil, but also on you and on CRU. Please let me know if there’s any way I can help in responding to Michaels. I’ll do anything I can.

Phil Jones replied:

Seen this one… So much else wrong with this piece…
I have stopped sending data out to anybody after the stupid comment on Climate Audit by Peter Webster. We’ve had over 60 FOI requests for data. They are varied – many can be answered by telling people to read the literature. We’re refusing those for the data… It is just a pain having to respond to them – someone else at UEA does this though. I did send one of the requests to Myles as it was from one of his fellow profs in Physics at Oxford! [Prof Jonathan Jones] Myles knows him well and … can’t understand why he’s getting his climate education from Climate Audit and not from colleagues in his own dept! This annoys me too. I’d read up and talk to people if I were to ever attempt moving to another field! It is just common sense.
Neil Adger has taken over the running of First Year course here in ENV. He asked Alan Kendall for the ppt for 2 lectures he gives. He sent them and 40 slides are taken from Climate Audit! A student asked Neil why Alan was saying things opposite to what Neil and Tim Osborn were saying!!!
Alan is retiring at the end of this year….thankfully.

Editor
November 26, 2011 5:43 am

dp says:
November 25, 2011 at 11:19 pm

It would be nice if it could handle this:
egrep -i -B3 -A3 “delet|foi” *.txt

I tried that. Then resorted to adding -l and wc because -l was still too long to post:
(-l lists file names, one per line, wc lists # lines, words, and characters):
tux:mail> egrep -l -i -B3 -A3 ‘delet|foi’ *.txt | wc
572 572 5148
Even uppercase only FOI has a lot of refs, few are off target:
tux:mail> egrep FOI *.txt | wc
wc: standard input:542: Invalid or incomplete multibyte or wide character
913 9763 72065
(haven’t checked out that complaint.)
I did notice some BASE64 encoded Emails, I haven’t checked to see if
those are expanded at web archives.

Crispin in Waterloo
November 26, 2011 11:29 am

@Lucy
That was an amazing set of communications and it is obvious you are not removing context – the gist is very clear: there are ‘enemies’ in science and everything possible should be done to ‘shape the message’ the public hears.
What is inevitable is the conclusion that the BBC is deeply involved with a person or persons at UEA and an organisation or organisations and further that a high level decision has been taken to cover for appalling scientific misconduct and blatant misrepresentation of the climate facts. Experienced as I am at hearing and reading this CAGW nonsense, even your early cull of a few Climategate 2 files shows that the dishonesty and petty bitterness of the Team renders their future involvement in IPCC matters untenable.
Is it possible that Phil was removed as part of a behind-the-scenes agreement with the whitewashing groups that they (persons/organisation) will bury the matter as well as they can but he has to go? What reasons were given for his removal? These days UEA means Unbelievable from Every Angle. There is not a single example of honourable academic conduct from this vituperative lot. They have rejoiced at the slow death of balance, truth and the individual investigation of truth by UEA students. Of what worth is a degree granted by UEA? Are they not just trained fools with Names to be Wise?
The BBC has supported this lot, hook, line and stinker. Something connects them and clearly it is not science or morals. The argument that is their common (and heavy) investment in carbon trading is wearing thin. Science and ideology only find common cause in back rooms.

November 26, 2011 12:07 pm

Hi,
Just added a checkbox so you can search with case sensitivity on if you wish, helps with targeting names and places.

November 26, 2011 4:48 pm

Thanks EcoGuy!
Thanks Jo!
Duly linked.

dp
November 26, 2011 4:51 pm

Ric:
Not to turn this into a techie thread but it would probably be a better effect to use good old ht:dig or Estraier which are true indexers that provide context. Takes a lot of real-time load off the server, indexes once, serves many. But my point with that use of egrep was to allow OR as well as AND. Even more is possible using the PHP intrinsic prcegrep functions. Taken to extremes it could probably be wedged into Splunk.

dlb
November 26, 2011 5:25 pm

Thanks Lucy for some very intresting reading. It’s like being a fly on the wall and in their ointment all at once! Also thanks of course to ecoGuy and that most troublesome fly, FOIA.
Crispin, I wouldn’t go as far as saying “There is not a single example of honourable academic conduct from this vituperative lot ” Alan Kendall comes accross as a shining light of honour in an otherwise disreputable bunch.

November 27, 2011 3:48 pm

Thanks Lucy, very informative! Maybe Dr. Kendall may speak out even more forcefully, now that he is retired. Or not, if he is the whistleblower…

crosspatch
November 27, 2011 3:53 pm

Thanks Lucy, too bad we only get to see what went on a UEA. I’d love to be a fly on the wall at Columbia or Carnegie Mellon.

Dave
November 30, 2011 8:12 pm

Jo Nova has a finger on the pulse of the 7z files and a possible explanation that is more the more than interesting.
THE ONE THINGS THAT SCARES THE POLITICIANS THAT HAVE SUPPORTED THE CAGW POLICY’S, IS EXPOSURE THEY WERE IN ON THE SCAM ALL ALONG?
http://joannenova.com.au/2011/11/pointman-a-dead-mans-hand-detonator-on-hidden-emails-may-protect-climategate-whistleblower/
Climategate 2
Pointman — A dead man’s hand detonator on hidden emails may protect ClimateGate 1 & 2 whistleblower FOIA
He points out there are no emails released yet between key scientists and people in power
We do not have a single one of those high-level political emails but they must of course exist.
I strongly suspect we now have them in our possession.
In the high-powered risky game of whistle blowing there are ways to make the Climategate 2 leaker a less attractive target.
Pointman analyzes the ClimateGate whistleblower’s tactics and explains why he, she or they probably released those other 200,000 emails but kept them hidden behind the 4000-8000 character almost unbreakable password.
He points out there are no emails released yet between key scientists and people in power, hence the worst, most damaging emails may be kept under a ” dead man’s hand detonator”. If politicians are afraid of what might be in those released-but-hidden emails, they may not want to expose or attack the whistleblower for fear of unleashing the other emails. The hidden emails buy the whistleblower protection.
Jo

Brian H
December 7, 2011 5:19 am

Baa Humbug says:
November 25, 2011 at 3:55 pm
tallbloke says:
November 25, 2011 at 2:56 pm
Excellent!
hehehe were you rubbing your hands together while saying that Mr Burns?

I doubt it, as he was typing at the time! Simultaneous rubbing would have to make for lotsa typoz…
😉

December 7, 2011 5:24 am