![gleickpic[1]](https://wattsupwiththat.files.wordpress.com/2011/10/gleickpic1.jpg?resize=184%2C184&quality=83)
Readers may recall that on February 22nd, I offered up some open source stylometry/textometry software called JGAAP (Java Graphical Authorship Attribution Program), with a suggestion that readers make use of it to determine the authorship of the faked Heartland strategy memo disseminated to the media by Peter Gleick.
A link to that article is here:
The reason I did that was that many had speculated that Dr. Peter Gleick was the author. Gleick, who admitted to obtaining the Heartland board meeting documents under false pretenses, and likely illegally, denies he wrote it. Except for a few holdouts and those who won’t give an opinion, like Andy Revkin, other prominent voices of the online community such as Megan McArdle of The Atlantic think otherwise, and she doesn’t even see it as a professionally written memo:
“…their Top Secret Here’s All the Bad Stuff We’re Gonna Do This Year memo…reads like it was written from the secret villain lair in a Batman comic. By an intern.”
In posting about JGAAP software crowdsourcing, I had hoped that the wide professional base of readers could make use of this software and would be able to come to conclusions using it, but there were complications that made the task more difficult than it would normally be. These complications included the fact that there were cut and pasted elements of other stolen Heartland documents in the “Climate Strategy Memo,” making it difficult for the software to delineate the separate writing styles without knowledgeable fine tuning.
These complications became especially evident when writer Shawn Otto at the Huffington Post used the JGAAP software to do his own analysis, coming to the conclusion that Joe Bast, president of the Heartland Institute, had authored the fake memo. The problem was that Mr. Otto did not perform the due diligence required in his selection of documents and the JGAAP software controls, and this led to an erroneous result.
In the end I realized that only professionals familiar with the science of stylometry/textometry would be able to make a credible determination as to the authorship. So, I asked for help.
On February 23, 2012 I sent the Evaluating Variations in Language Laboratory (the group responsible for the JGAAP software) a request for assistance. Mainly what I was looking for initially was tips on how to best operate their software, but given the high profile nature of this issue, and the unique situation, they referred me to Juola & Associates and its president, Patrick Brennan, who responded with an even better offer. They would use their larger collection of tools and techniques reserved for their forensics consulting work and apply it to the task, pro bono. Normally such professional analysis for courtroom quality work nets them fees comparable to what a metropolitan lawyer might charge, so not only was I extremely grateful, but realized it was an offer I couldn’t refuse.
In my email to Brennan on Fri, Feb 24, 2012 at 5:07 PM I wrote:
For the record, I do not know what the outcome might be, but it is always best to consult experts externally who have no financial interest in the outcome of the case.
Here’s the background on the group:
Juola & Associates (www.juolaassoc.com) is the premier provider of expert analysis and testimony in the field of text and authorship. Our scientists are leading, world-recognized experts in the fields of stylometry, authorship attribution, authorship verification, and author analysis. Every written document is a snapshot of the person who wrote it; through our analysis, we can determine everything from sociological information to biographical information, even the identity of the author. We provide sound, tested, and legally-recognized analysis as well as expert testimony by Dr. Patrick Juola, arguably one of the world’s leaders in the field of Forensic Stylometry.
We have worked with groups as wide-ranging as multinational companies, Federal courts, research groups, and individuals seeking political asylum. We have literally written the book (ISBN 978-1-60198-118-9) on computational methods for authorship analysis and profiling.
The lead analysis was conducted by Patrick Juola, Ph.D., Director of Research, and director of the Evaluating Variations in Language Laboratory at Duquesne University in Pittsburgh. Juola & Associates, headed by President Patrick Brennan is a separate commercial entity that provides analysis and consultation on stylometry.
Dr. Juola has published his analysis of the “Climate Strategy Memo,” which I present first and in entirety here at WUWT.
First, the short read:
Stylometric Report – Heartland Institute Memo
Patrick Juola, Ph.D.
Summary
As an expert in computational and forensic linguistics, I have reviewed the alleged Heartland memo to determine who the primary author of the report is, and more specifically whether the primary author was Peter Gleick or Joseph Bast. I conclude, based on a computational analysis, that the author is more likely to be Gleick than Bast.
And the larger excerpt of the document, bolds mine:
Analysis
24 This task is challenging for several reasons, some technical and some linguistic.
25 First, the Heartland memo as published contains a great many quotations taken from other sources. As originally published, the memo contains approximately 717 words, but at least 266 of those words have been identified as belonging to phrases (or paraphrases of phrases) found elsewhere in the stolen documents). [N.b. this identification was done by the Heartland Institute, who admit that these 266 words are “paraphrases [of] text appearing in one of the stolen documuments.”
As paraphrases, they may nor may not reflect the style of the original authors, and they also may or may not reflect the style of the alleged forger. For this reason, we analyzed both the full document as well as the 451-word redacted document with the controversial passages removed.
26 Second, even the full-length document is rather short for an accurate analysis. Most authorship attribution experts recommend larger samples if possible. (E.g., Eder recommends 3500 words per sample, noting that results obtained from fewer than 3000 words “are simply disastrous.”)
27 Thirdly, perhaps as a result of the previous factors, we have observed that Bast and Gleick appear to have extremely similar writing styles.
Results
28 Despite this difficulty, we were able to identify and calibrate an appropriate analysis method. Using this method, we analyzed both the complete Heartland memo and the selections from the Heartland memo that had been identified as not copied from other stolen documents. In both analyses, the JGAAP system identified the author as Peter Gleick.
29 In particular, the JGAAP system identified the author of the complete (unredacted) memo as Peter Gleick, despite the large amount of text that even Bast admits is largely taken from genuine writings of the Heartland Institute. We justify this result by observing, first, that much of the quotation is actual paraphrase, and the amount of undisputed writing is still nearly 2/3 of the full memo.
Conclusions
30 In response to the question of who wrote the disputed Heartland strategy memo, it is difficult to deliver an answer with complete certainty. The writing styles are similar and the sample is extremely small, both of which act to reduce the accuracy of our analysis. Our procedure by assumption excluded every possible author but Bast and Gleick. Nevertheless, the analytic method that correctly and reliably identified twelve of twelve authors in calibration testing also selected Gleick as the author of the disputed document. Having examined these documents and their results, I therefore consider it more likely than not that Gleick is in fact the author/compiler of the document entitled ”Confidential Memo: 2012 Heartland Climate Strategy,” and further that the document does not represent a genuine strategy memo from the Heartland Institute.
It seems very likely then, given the result of this analysis, plus the circumstances, proximity, motive, and opportunity, that Dr. Peter Gleick forged the document known as ”Confidential Memo: 2012 Heartland Climate Strategy.” The preponderance of the evidence points squarely to Gleick. According to Wikipedia’s entry on the “legal burden of proof”:
Preponderance of the evidence, also known as balance of probabilities is the standard required in most civil cases. This is also the standard of proof used in Grand Jury indictment proceedings (which, unlike civil proceedings, are procedurally unrebuttable).
Further, it is abundantly clear that this document was not authored by Heartland’s Joe Bast, nor was it included as part of the board package of documents Dr. Gleick (by his own admission) phished under false pretenses from Heartland.
The complete analysis by Dr. Juola is available here: MemoReport (PDF 101k)
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Steve S says:
March 14, 2012 at 7:12 am
Based on all this, I wonder if P. Gleick would voluntarily submit to a polygraph. I’m guessing…’No’.
Steve, I’d say “no” as well, and I’d recommend the same for everyone. The most favourable results I’ve seen were by the National Acadamy of Sciences which, “after examining 57 polygraph studies the NAS (national Academy of Sciences) concluded: ‘In populations of examinees such as those represented in the polygraph research literature, untrained in countermeasures, specific-incident polygraph tests can discriminate lying from truth telling at rates well above chance, though well below perfection.’ Their analysis of the 30 most recent polygraph data sets showed an overall accuracy of 85 percent, and an analysis of seven field studies involving specific incidents showed a median accuracy of 89 percent.” An 11-15% failure rate and the fact that a sufficiently trained or knowledgable person can confuse the machines and the operator make it a poor source of evidence.
Other scientists and non-scientist curmudgeons like me, consider the polygraph to be pseudoscientific trickery similar to the witchdoctor’s opinion or to torture. Both will “work”…to use the term loosely… by being perhaps useful in investigations and can show “accuracy,” by playing with profusion of questions and subjective definitions and interpretations. Fortunately, your Supreme Court’s United States v. Scheffer (1998) and your 11th Circuit Court of Appeals in 2005 were not impressed either.
Unless the victim complains, law enforcement won’t get involved. If they’re not getting involved, HI hasn’t complained. Here’s the reason, I think: In one of these threads, someone commented that it would be bad to involve the Feds, because their criminal suit would automatically stall the civil suit for years while the prosecutor dragged his feet and then made a “mistake” that led to the case being dismissed and Gleick looking vindicated. Also, the higher threshold of proof in criminal trials means that Gleick could be found “not guilty” and wind up looking exonerated.
“Nevertheless, the analytic method that correctly and reliably identified twelve of twelve authors in calibration testing also selected Gleick as the author of the disputed document. ”
If I’m reading this right does this mean that they tested Gleick against 12 random authors also.
Anthony Watts says:
March 14, 2012 at 11:24 am
Can anyone post a link to this essay in comments over on Shawn Otto’s HuffPo story?
http://www.huffingtonpost.com/shawn-lawrence-otto/joe-bast-fake-document_b_1297042.html
I have been denied access.
BWA-HAHHAHAH ! That’s priceless. May we, er, politely inquire perchance as to the reasons for this, hmmm, unfortunate “misunderstanding”? Or perhaps you’d rather not talk about it.
Anyway, Anthony, a colleague with a proxy ISP account just tried and got blown away within a minute. They are after the content and must have mods with their eyes on the screens and fingers on the nuke button. Weird; it’s an essay, not bawdy limericks about their moms.
I think “the 12” refer to 12 samples of Gleick’s own work, which Juola downloaded from the web. He did the same with several samples of Bast’s writing. These establish a basic stylistic “fingerprint” for each man. Someone with computer savvy might be better able to explain what he did with this info (from the PDF):
In a world of infinite possibility it will always be “possible” that Gleick did not author the forgery, but in the end and in the larger context it doesn’t seem to be all that relevant to know whether he did or not, Given what he has already stipulated too and the fact that only the most extreme loons are still willing to challenge the notion that the “alleged forgery” is an actual forgery, the nefarious nature of his actions in this incident has been fairly well established. The negative judgement he has coming would be worse if it could actually be proved that he wrote the forgery but the difference would be only incremental.
As to those questioning why Gleick has yet to be charged for his actions, you might begin your inquiries by asking why it is that AG Holder still remains in office. His tenure has been notable for a growing litany of transgressions against the rule of law, equity, and the Constitution, any one of which would have been sufficient to send any Republican occupant of his office down the memory hole.
I’m surprised that the company would even take this on, as it exposes them to litigation risk. Sure, it’s a publicity gambit, but it might backfire. Even though they included the quote about the accuracy anything less than 3000 words is simply disastrous. A not-properly qualified result is a libel. The problem is, of course, advocates will trumpet the results and conveniently forget to *always* include the disastrous disclaimer. As the author of this post did, using words like “very likely”, “preponderance”. “Disastrous” means no better than random.
Such a result is only perhaps useful in the initial phases of a police investigation, to help the investigators prioritize their resources. It’s a lead, a tip; not a fact.
Dave Wendt says: March 14, 2012 at 3:21 pm
In a world of infinite possibility it will always be “possible” that Gleick did not author the forger
Wouldn’t Trenberth say: “the null hypothesis must be reversed … it is now up to Gleick to prove his innocence”.
Or perhaps we should apply the precautionary principle says … whilst we cannot be certain beyond all doubt that Gleick did it, the consequences of the whole of climate science being so corrupt is so utterly damaging, that we must act as if it were proven beyond all doubt that they are a complete bunch of charlatans.
Or as Patterson said: “Suppose you see a bird walking around in a farm yard. This bird has no label that says ‘duck’. But the bird certainly looks like a duck. Also, he goes to the pond and you notice that he swims like a duck. Then he opens his beak and quacks like a duck. Well, by this time you have probably reached the conclusion that the bird is a duck, whether he’s wearing a label or not.”
John A. Fleming: Once Gleick admitted to fraud and mumbled nonsense about getting the document in the mail, something an eight year-old wouln’t believe, he opened the door for speculations, guesses and accusations. He’s little credibility or reputition now to ligitate for, anyway. Otherwise, I agree twith you that such techniques, like polygraphs, handwriting analysis and psychometric wonders are all, at best, leads and tips which may be marginally better than tossing a coin. In a civil court, though, they may tip he scales.
Ric Werme says: “There’s no rush. It’s more important to get the charges right and the evidence lined up than it is to arraign him with an incomplete investigation…Patience.”
If he acted alone, then there’s probably no urgency. If, however, he acted in concert with someone else, well, what happens when that someone connects the dots…?
Anyone attempting to muddy the waters in the manner of Otto and, by association, the Huffington Post, is a few sandwiches short of a picnic. Attempting to malign the HI using inverted logic flies in the face of Gleick’s own admission of doing really naughty stuff.
that’s pretty cool – another star for mr Watts.
i hope it’s all going to add up to at least one prosecution in 30 years after a few hundred billions have been looted. just one. once there’s a first one, the rest will be easy.
call me when he’s called to testify at trial.
The headline says “internal documents”. The story says UN documents.
> The headline says “internal documents”. The story says UN documents.
No inconsistency there: What is internal to Greenpeace, is internal to the UN, and vice-versa 🙂
(my 5.14pm comment is redundant – and wordpress is giving me a hard time logging in to post)
I wish to second the opinion that Anthony Watts runs one of the fairest sites on the net.
As far as I know censorship is limited to behaviour that would lead to expulsion from any civilised gathering and abusive, insulting behaviour is certainly justifiable reason for dismissal of comments.
Kudos to you.
“….those who won’t give an opinion, like Andy Revkin…”
What?!? But Andy Revkin was so quick to ‘authenticate’ the documents why wouldn’t he want to clear the tarnished reputation of his good friend Gleik now by authenticating the identity of the forger?
So what if Andy’s authentification of the documents was wrong. His employer, The NY Times, didn’t seem to be bothered by that lie, why should anyone else care….there are no rules in the free-for-all, unethical world of climate alarmism. So c’mon Andy, give it your best shot. Authenticate the forger and help out your good friend Gleik.
Falsus in uno, falsus in omnibus. An admitted liar.
‘Falsus in Uno, Falsus in Omnibus is a legal maxim which means false in one thing, false in everything. A Roman legal principle indicating that a witness who willfully falsifies one matter is not credible on any matter. The underlying motive for attorneys to impeach opposing witnesses in court: the principle discredits the rest of their testimony if it is without corroboration.’
http://definitions.uslegal.com/f/falsus-in-uno-falsus-in-omnibus/
> Falsus in uno, falsus in omnibus. An admitted liar.
I’d have trouble believing the story about the anonymous mail, and it just happening to quote (and only quote) a smalll selection of documents that were later published – even if we knew nothing else about Gleick
BUT
Gleick has already confessed to forgery, fraud, and lying.
He’s confessed to fraudulently obtaining documents from Heartland by forging the identity (which we know to include the name & signature block) on a series of emails, and a series of lies – both a course of conduct over several days, rather than a single incident.
He’s confessed to sending the email to the “15” bloggers – ostensibly his allies, which includes yet more lies:
1. It lies about the sender being a Heartland insider
2. It lies by omission – not mentioning that the strategy memo and form 990 were obtained separately from the other documents.
3. It lies by commission in that it says all the documents, including strategy memo, were from Heartland, but even according to Gleick’s “confession” he doesn’t know where the strategy memo came from.
In short, Gleick is a self-confessed liar, a self-confessed fraudster, and a provable forger. The only question is whether he lied about and forged the strategy memo AS WELL AS the other lies, frauds, and forgeries that he has committed.
RichieP, good one to remember, the falsus in uno, falsus in omnibus. Pardon the excursus into pedantry, but as this is a Roman Law principle, is it possible that it may not be directly applicable in Common Law, where presumption of innocence and credibility predominate?
Peter, I think it’s that the credibility of the witness has been impeached, and I think that principle still applies. Innocence and credibility don’t necessarily intermix.
I had to post this on my face book because Huffington allow posts that conflict with their views…
“I’m enjoying the censorship exercised by the thought police at the Huffington Post…come read their defense of the indefensible as they make excuse after excuse for why Peter Gleick was forced to lie and forge a fake memo to prove his point…”
And this was the post (in response to someone asking why people were upset at Gleick and not Heartland) they wouldn’t allow:
“I’m more disappointed at the censorship on the site…obviously the goal is to shape opinion rather than provided a forum for pen discussion…
But to your point, you wonder why people are outraged that Peter Gleick obtained documents under false pretense, then, not finding the smoking gun he expected, crafted a hybrid memo stating all those deeply help beliefs he had expected to find, and then passed the whole package off as original data from Heartland…Yea, why would anyone be upset with that?
Seems Gleick believes that the only way truth will win in the debate is to lie…”
Of all the blather above, eg Smokey, I find Copner’s 7:31 comments most cogent and compelling. Personally, I don’t know, but my point, as I think Anthony has induced from comments I’ve made elsewhere, is that this is somewhat of a fool’s errand. It also seems that some experts feel the same: the sample is too small to distinguish from noise, which as someone who understands a very little bit about statistical analysis I would tend to agree with. For example, I made a similar argument when the WHO came out with their nonsense last year claiming that cell phones “might” cause brain cancer. Statistically, the blip they observed was indistinguishable from chance. I got lots of similar blather from lefties for criticizing them. http://www.huffingtonpost.com/shawn-lawrence-otto/cell-phone-use-cancer-link_b_914005.html
While antiscience types on the left think they are using science when they grind their axes over hidden health dangers of cell phones and vaccines, you guys on the right blind yourselves in a similar way, sure you are being scientific when you dismiss anything that doesn’t conform to your preexisting conclusion. Juola, to preserve his credibility, admits that what he’s about to say is, like the WHO paper, indistinguishable from chance, and yet you people lap it up, you are so anxious to sate your confirmation bias. Now, it is possible Peter Gleick wrote that. We cannot say, using science, with certainty at this point. But we can say that some JGAAP tests show Bast is the most likely author, and some show Gleick. Some might argue that it was irresponsible of Anthony to suggest readers adopt the JGAAP approach on his blog for the same reason, and it highlights the flaws in this new “study” he has published purporting to reveal Gleick as the author. In my view this study has limited credibility because A) JGAAP showed different answers using different, reproducible tests (and mine were, arguably, simpler and more recommended under he circumstances, and also correctly identified a known author as a control); B) unlike my “study” the Juola report does not actually supply any data or methodology and so is less scientifically reliable; C) the study was commissioned by an interested party (big surprise he got the results he had hoped for) and D) as noted by other experts, the sample both studies considered was small to the point of being indistinguishable from noise, ie chance. From Maciej Eder: “It becomes quite obvious that samples shorter than 5000 words provide a poor “guessing”, because they can be immensely affected by random noise. Below the size of 3000 words, the obtained results are simply disastrous.”
See http://dh2010.cch.kcl.ac.uk/academic-programme/abstracts/papers/html/ab-744.html
Anyone experienced in statistical analysis like Anthony Watts claims to be should surely recognize the merits of that observation, and should, to maintain any credibility, apply equal standards of reason, whether goose or gander. And Juola should be at least as suspect as I am by any real skeptics worth their salt for coming to to a public conclusion that is not supportable, according to the literature he himself cites, beyond chance. If we were talking about climate change you guys would be all over that, ripping it to shreds, and yet now you are lapping it up like happy puppies. And yet somehow I’m not surprised. 😉
REPLY: Oh, please. You were fine with it all when you thought you had snagged Joe Bast, and to use your words “your readers lapped it up”. But now that somebody who actually understands the science of stylometry has correctly configured JGAAP and run the calibration and produced results that don’t support you it’s now “oh wait, the method is flawed”. Right, sure, anything to defend Gleick and keep yourself from looking foolish for running the program in an evening and not knowing enough about the science to say your results weren’t understandable. Instead you simply took the numbers you threw out on that blog post (without understanding what they mean) and created a smear piece with a conclusion unsupported by your work. Typical HuffPo drive by.
Then tonight, you throw out some distraction drivel about cell phones and vaccines, then created a straw man lie to say “Juola, to preserve his credibility, admits that what he’s about to say is, like the WHO paper, indistinguishable from chance…” I’m sorry Otto, that’s bull. He never said that, you did. How pathetic.
Similarly, Mr. Otto, you planted another lie of suggestion, that of the study was “commissiened” by me. Uh no, apparently you can’t read. I asked for help on how to run the program and explained the situation, THEY made the offer to help, pro bono. I didn’t ask for it. Something can’t be “commissioned” when it happened that way. I resent your implications, sir.
I’m even less impressed with this comment than your original blog article. Readers should note that Otto is not an unbiased observer here, he’s written a book about “anti-science” that is endorsed by Bill McKibben, who’s about as far from science and as close to a political hack as anyone could ever be, and curiously also by Bill Nye the Science guy, who couldn’t be bothered to actually check the science in Al Gores “Climate 101” video before it was distributed worldwide with his own name attached to it. Have a look at the results of that Nye fiasco here: Replicating Al Gore’s Climate 101 video experiment shows that his “high school physics” could never work as advertised
From his comment, it is clear that Otto is just another drive by Joe Romm clone that when presented with real science, trashes it and labels anyone who disagrees with him as “anti-science”.
And at Otto’s HuffPo site, I can’t comment, and others in the thread above can’t get their comments retained, they disappear. So much for any pretense of fairness or discussion of science there.
Mr. Otto, be as upset as you wish. – Anthony Watts
To summarize: The memo was paraphrased by Peter Gleick, because the memo was paraphrased, except for what wasn’t paraphrased. Therefore, Peter Gleick paraphrased it. Maybe, but more than certainly.
Sorry, but this isn’t all that convincing.
REPLY: You are correct, your argument isn’t convincing at all, it sounds like paraphrased nonsense- Anthony
shawnotto,
So you presume that you are privy to the straight skinny, and for your personal opinion that court appointed experts are all wrong. Got it: you’re deluded. Thanx for the heads-up.