Guest Post by Willis Eschenbach
I’ve been involved in climate science for a while now, this is not my first rodeo. And I’ve read so many pseudo-scientific studies that I’m starting to develop a list of signs that indicate when all is not well with a particular piece of work.
One sign is whether, how, and when they cite the IPCC “Bible”, their “IPCC Fourth Assessment Report”. The previous report was called the “T. A. R.” for “Third Assessment Report”, but the most recent one is called “AR4” rather than the “F. A. R. “, presumably to avoid using the “F-word”. This report is thousands upon thousands of pages of … of … of a complex mix of poorly documented “facts”, carefully selected computer model runs, good science, blatantly political screeds from Greenpeace and the World Wildlife fund, excellent science, laughable errors, heavily redacted observations, poor science, “data” which turns out to be computer model output, claims based on unarchived data, things that are indeed known and correctly described, shabby science, alarmist fantasies, things they claim are known that aren’t known or are incorrectly described, post-normal science, overstated precision, and understated uncertainty. That covers most of the AR4, at least.
Since many of the opinions expressed therein are vague waffle-mouthed mush, loaded with “could” and “may” and “the chance of” and “we might see by 2050”, you can find either support or falsification within its pages for almost any position you might take.
I have an “IPCC fail-scale” that runs from 1 to 30. The higher the number, the more likely it is that the paper will be quoted in the next IPCC report, and thus the less likely it is that the paper contains any actual science.
I’d seen some high-scoring papers, but a team of unknowns has carried off the prize, and very decisively, with a perfect score of 30 out of 30. So how does my “IPCC Fail-Scale” work, and how did the newcomers walk off with the gold?
First, there are three categories, “how”, “whether”, and “when”. They are each rated from zero to ten. The most important of these is how they cite the IPCC report in the text. If they cite it as something like “IPCC Fourth Assessment Report: Climate Change 2007 (AR4), Volume I, pages 37-39 and p. 40, Footnote [3]”, they get no points at all. That’s far too scientific and too specific. You could quickly use that citation to see if it supports their claims, without blindly searching and guessing at what they are citing. No points at all for that.
If they cite it as “IPCC Fourth Assessment Report: Climate Change 2007 (AR4), Volume I” I award them five points for leaving out the page and paragraph numbers. They get only two points if they just omit the paragraph. And they get eight points if they leave out the volume. Leaving out a URL so their version can’t be found gets a bonus point. But to get the full ten points, they have to disguise the report in the document. They can’t seem to be building their castles on air. So how did the winning paper list the IPCC Fourth Assessment Report in their study?
They list it in the text as “Solomon 2007”. That’s absolutely brilliant. I had to award the full ten points just for style. Plus they stuck the landing, because Susan Solomon is indeed listed as the chief culprit in the IPCC documents, and dang, I do like the way they got around advertising that they haven’t done their homework. 10 full points.
Next, where do they cite it? Newcomers to the field sometimes cite it way at the end of their study (0 to 5 points) or in the middle somewhere (six to nine points). But if you have real nerve, you throw it in as your very first reference. That’s what got them the so-called “brownie point”, the extra score named after the color of their nose, the final point that improves their chances of being in the Fifth Assessment Report. Once again, 10 out of 10 points to the winner, “Solomon 2007” is the first reference out of the box.
Finally, do they cite the IPCC at all? Of course, the authors not citing the IPCC Report greatly improves the odds that the author has actually read, understood, and classified the IPCC document as a secondary source, so no points if they don’t cite it, 10 points if they cite it. One points per occurrence for citing it indirectly through one of their citations, to a maximum of 8. And of course, the winner has ten points in this category as well.
And what is this paragon of scientific studies, this ninja reference-master of analyses, this brazen grab by the newcomers for the crown?
Quite appropriately, it is a study which shows that when the Arctic is warmer, we should expect Northern winters to be colder.
Lately there have been a string of bitterly cold winters … who would have guessed? Well, as the authors of the study point out, none of the climate models guessed it, that’s for sure.
The study is “Arctic warming, increasing snow cover and widespread boreal winter cooling“, by Judah L Cohen, Jason C Furtado, Mathew A Barlow, Vladimir A Alexeev and Jessica E Cherry. This study proves once again that in the topsy-turvy world of climate science, all things are explainable by the AGW hypothesis … but only in hindsight.
It’s also a curious study in that the authors, who are clearly AGW supporters, are baldly stating that the climate models are wrong, and trying to explain why they are wrong … man, if I say the models are wrong, I get my hand slapped by the AGW folks, but these authors can say it no problem. It does put them into a difficult position, though, explaining why their vaunted models got it wrong.
Finally, if they are correct that a warmer Arctic has cooler winters, then for the average Arctic temperature to be rising, it would have to be much, much warmer in the summers. I haven’t seen any data supporting that, but I could have missed it. In fact, thinking about cooling winters, one of the longest underlying claims was that CO2 warming was going to lead to warming winters in the extra-tropics and polar regions … what happened to that claim?
CONCLUSIONS in no particular order
• I have no idea if what they are claiming, about snow and cold being the result of warming, is correct or not. They say:
Understanding this counterintuitive response to radiative warming of the climate system has the potential for improving climate predictions at seasonal and longer timescales.
And they may be right in their explanation. My point was not whether they are correct. I just do love how every time the models are shown to be wrong, it has the “possibility of improving climate predictions”. It’s never “hmmm … maybe there’s a fundamental problem with the models.” It’s always the Panglossian “all is for the best in the best of all possible worlds.” From their perspective, this never ever means that the models were wrong up until now. Instead, it just makes them righter in the future. They’ve been making them righter and even righterer for so long that any day now we should reach righterest, and in all that time, the models have never been wrong. In fact, we are advised to trust them because they are claimed to do so well …
• Mrs. Henninger, my high school science teacher, had very clear rules about references. The essence of it was the logical scientific requirement that the reader be able to unambiguously identify exactly what you were referencing. For example, I couldn’t list “The Encyclopedia Britannica, Volume ‘Nox to Pat'” as a reference in a paper I submitted to her. I’d have gotten the paper back with a huge red slash through that reference, and deservedly so.
Now imagine if I’d cited my source as just “The Encyclopedia Britannica”? A citation to “The Encyclopedia Britannica” is worse than no citation, because it is misleading. It lends a scientifically deceptive mask of actual scholarship to a totally unsupported claim. And as a result …
• Citing the IPCC TAR in its entirety, without complete volume, page, and if necessary paragraph numbers, is an infallible mark of advocacy disguised as science. It means that the authors have drunk the koolaid, and that the reviewers are asleep at the switch.
• Mrs. Henninger also would not let us cite secondary sources as being authoritative. If we wanted a rock to build on, it had to, must be, was required to refer to the original source. Secondary sources like citing Wikipedia were anathema to her. The Encyclopedia Britannica was OK, but barely, because the articles in the Britannica are signed by the expert who wrote each article. She would not accept Jones’s comments on Smith’s work except in the context of discussing Smith’s work itself.
But the IPCC is very upfront about not doing a single scrap of science themselves. They are just giving us their gloss on the science, a gloss from a single highly-slanted point of view that assumes what they are supposed to be setting out to establish.
As a result, the IPCC Reports are a secondary source. In other words, if there is something in the IPCC report that you are relying on, you need to specify the underlying original source. The IPCC’s comments on the original source are worthless, they are not the science you are looking for.
• If the global climate models were as good as their proprietors claim, if the models were based on physical principles as the programmers insist … how come they all missed it? How come every one of them, without exception, got the wrong answer about cold wintertimes?
• And finally, given that the models are unanimously wrong on the decadal scale, why would anyone place credence in the unanimity of their predictions of the upcoming Thermageddon™ a century from now? Seriously, folks, I’ve written dozens of computer models, from the simple to the very complex. They are all just solid, fast-calculating embodiments of my beliefs, ideas, assumptions, errors, and prejudices. Any claim that my models make is nothing more than my beliefs and errors made solid and tangible. And my belief gains no extra credibility simply because I have encoded it plus the typical number of errors into a computer program.
If my beliefs are right, then my model will be accurate. But all too often, my models, just like everyones’ models, end up being dominated by my errors and my prejudices. Computer climate models are no different. The programmers didn’t believe that arctic warming would cause cooler winters, so guess what? The models agree, they say that arctic warming will cause warmer winters. Fancy that. Now that the modelers think it will happen, guess what future models will do.
Now think about their century-long predictions, and how they can only reflect the programmers beliefs, prejudices, and errors … here is the part that many people don’t seem to understand about models:
The climate models cannot show whether our beliefs are correct or not, because they are just the embodiment of our beliefs. So the fact that their output agrees with our beliefs means nothing. People keep conflating computer model output and evidence. The only thing it is evidence of is the knowledge, assumptions, and theoretical mistakes of the programmers. It is not evidence about the world, it is only evidence of the programmers’ state of mind. And if the programmers don’t believe in cooling winters accompanying Arctic warming, the models will show warmer winters. As a result, the computer models all agreeing that the winters will be warmer is not evidence about the real world. No matter how many of the models agree, no matter how much the modelers congratulate each other on the agreement between their models, it’s still not evidence.
My best to all,
w.
Well said: “models cannot show whether our beliefs are correct or not, because they are just the embodiment of our beliefs”
A good example is how Dr Hansen’s treatment of the Earth’s surface as being a blackbody (even though heat transfers by several means other than radiation) is incorporated into the models via the infamous 33 degrees of warming figure. They calculate radiation on the basis of it transferring all thermal energy as in a blackbody, and then double count the “extra” energy in conduction, convection, evaporation etc. Then they assume backradiation adds back some of the thermal energy that was radiated (or was it conducted?) because they assume it will be converted to thermal energy (perhaps by those mass-less photons colliding with it and causing friction or something) – but it isn’t, and so it doesn’t.
See top of my Home page (updated today) http://climate-change-theory.com for more detail on this.
I despair. Trying to teach a bunch of 9th graders the importance of proper citations is obviously futile when I see supposed scholars getting Willis’ full Monty of 30 points. I should require them to read this thread.
Isn’t it grand to know that each error we find only serves to improve the models?
What an accomplishment.
So productive!
You are wrongly critical of all models in general. The climate AGW “models” are indeed calcified prejudice, protected by fudging the data.
And you say ” I’ve written dozens of computer models, from the simple to the very complex. They are all just solid, fast-calculating embodiments of my beliefs, ideas, assumptions, errors, and prejudices. Any claim that my models make is nothing more than my beliefs and errors made solid and tangible” ???
WTF! You don’t calibrate your models, run them against an independent test set, align your initial guesses with objective reality? I’ve written dozens of models in the past 6 months, all data driven, self learning and objectively testable and verifiable. Many years ago I wrote several discrete simulation models. They are also testable and verifiable. What’s wrong with your models? If they contain only your preconceptions, why bother to write them at all.
Take Stanford’s free Machine Learning class – advanced track. You’ll feel better when you can write objective models.
Conrad
maybe co-author Jessica E Cherry was cherry-picking the citations?
I agree with all.
Let us at least make science cleaner and sharper, following rigorous rules, if nothing else. The some of us skeptics might be persuaded a bit more.
So, in summary:
1) Hearsay is not evidence
2) The Map is not the Territory
Mrs. Henninger reaches from the grave through the hand of Willis Eschenbach. Her teachings are through her red pencil. We seem to learn and remember most by the mistakes we make, pointed out to us, and we being held accountable. Science seems to work best under this scenario. It does take a certain attitude and maybe even personality to grow from one’s errors. One’s goals need to be aligned with the philosophy of science, an awe and wonderment on how this whole thing works. I kind of understand how one becomes attached to one’s creation, the love of Pygmalion. Unfortunately for climate scientists, there is no Aphrodite to give models life and tell us the future. Instead of Gods and Goddesses intervening in our work, we, and in reality scientists are left with rules of engagement; rules on how we do things, how we proceed; what can be said from the machinations of our models. When we violate such rules, stretch the true a bit, don’t back up our claims to the satisfaction of others, then Mrs. Henninger needs to revisit us, reach out, and draw a red line through our fanciful presentation.
conrad clark says:
February 1, 2012 at 8:04 pm
Jeez, conrad, come down off of your high horse. Of course I do that. I told you up front, this is not my first rodeo. You should pay attention when a man says that, and adjust your foolish expectations accordingly.
The last six months? I don’t know how old you are, conrad, but I’ve been writing computer programs for 49 years now, nearly half a century. Your assumed air of superiority from your “dozens of models in the past 6 months” is unwarranted.
My models are like everyone else’s models including yours. If your claim is that you’ve never written models with errors, you are lying. If you have written models with errors, you’re no better than me. Your choice.
I said above:
So yes, I do test my models. And guess what? Some of them are 100% right. And you know what else? Even though I test them, some of them still turn out wrong.
I would say that’s the nature of models, that they can fool you. However, clearly in the past I failed to realize that there are perfect programmers like yourself, who never have errors in their models. Thanks for straightening me out on that.
Seems like what you should do, conrad, is write climate models. I mean, since your models are all perfect, you could write a climate model with no flaws, and we could all go home.
w.
Mr. Eschenbach, “…the most recent one is called “AR4″ rather than the ”F. A. R. “, presumably to avoid using the “F-word”.”
Or perhaps because there was already a FAR, First Assessment Report.
Seconding everything Willis said in response to conrad clark.
conrad clark wrote:
WTF! You don’t calibrate your models, run them against an independent test set, align your initial guesses with objective reality?
Even with the best models and independent test, there are still questions regarding interpolation versus extrapolation.
They’re still just embodiments of beliefs, even if they fit well within the interpolated regions.
w.
Strawman much? I never claimed or implied that my models/programs are without error. I never claimed to be a perfect programmer. Trying to write a climate model with no flaws??? Where did I imply that I would attempt that? My models are all perfect? Where is “perfect” in my comment?
My 1st paying programming job was in 1964. I’ve done very well with that, slowly and with much effort improving and keeping up. You should try that. (that’s a snark vs your “get off your high horse” comment).
It seems that the word “model” doesn’t mean the same between the two parties in this conversation. You should try to keep opinion and prejudice out of your programs (I won’t call them models anymore, since we disagree on some definitions, apparently). Look up “objective(philosophy)” and “science wars” in Wikipedia.
Hmmm: do these thoughts apply to the GHG theory?
PS—conrad, you say:
I just took a look. It’s a class on machine learning and neural networks … what use is that to climate science? Iterative models are the order of the day in climate science, my friend, get with the picture.
In any case, I was doing work in neural networks in 1986, and was involved in studying evolutionary machine learning systems as well at that time. So I fear you are about, oh, maybe a quarter century behind the times with your advice as far as I’m concerned.
w.
Thank you Willis.
Reminds me of my time working on my Master’s degree. I used a statistical analysis program to analyze my data. Low and behold, the output was marvelous and supported my theory. Unfortunately, that is why there are students and there are professors. I learned that statistics tell you pretty much what you want to. I had used the wrong variables and even the wrong statistical test.
sceptical says:
February 1, 2012 at 9:05 pm
As Foghorn Leghorn used to say, “Son, I say son, it’s just a joke, y’know, a bit of humor.”
w.
[SNIP: -REP]
w.
Re 1986 neural networks and machine learning. Were you in the same IBM SRI class as me? Believe me, you need to be dragged into the 21st century.
Iterative models don’t seem to work with climate science (or am I misreading the lack of actual predictions)?
Conrad
Pandora looks super cool! I’d like to go there someday…..oh wait its fiction, but it looks so real!
conrad clark says:
February 1, 2012 at 9:21 pm
Oh, please. You came in criticizing my claim, getting on my case because I admit that I have written models with errors. You speculated that I “don’t calibrate [my] models, run them against an independent test set, align [my] initial guesses with objective reality?,” as though I were some newbie.
Now you are whining about a strawman, and butter wouldn’t melt in your mouth, oh, no, you never said anything like that at all.
Look, conrad. Either lead, follow, or get out of the way. Coming in to school me on models isn’t working. I know models. Here is what I said, and I hold to it:
I see nothing in there that I would change. And yes, I do test my models, I do calibrate them, I do run them against independent data, and I have done so for years.
w.
I agree, all models , once bug free (lol) should be validated against reality.
This is where the GCM’s seem to be having very major issues.
Maybe it time for applying the circular file. 😉
I just saw a chart today that I had saved knowing that it would be perfect for one of these threads. Little did I suspect Willis would have the perfect post to respond to with it. Enjoy!
What they say vs what it actually means:
“It has long been known” — I didn’t look up the original reference.
“A definite trend is evident” — The data are practically meaningless.
“While it has not been possible to provide definite answers to the questions” — An unsuccessful experiment, but I still hope to get it published.
“Three of the samples were chosen for detailed study” — The other results didn’t make any sense.
“Typical results are shown.” — This is the prettiest graph.
“These results will be in a subsequent report.” — I might get around to this sometime, if published/funded.
“A careful analysis of obtained data.” — Three pages of notes were obliterated when I knocked over a glass of beer.
“After additional study by my colleagues.” — They didn’t understand it, either.
“Thanks are due to Joe Blotz for assistance with the experiment and to Cindy Adams for valuable discussions.” — Mr. Blotz did the work and Ms. Adams explained to me what it meant.
“A highly significant area for exploratory study.” — A totally useless topic selected by my committee.
“In my experience.” — Once.
“In case after case.” — Twice.
“In a series of cases.” — Three times.
“It is believed that.” — I think.
“Correct within an order of magnitude.” — Wrong.
“According to statistical analysis.” — Rumor has it.
“It is clear that much additional work will be required before a complete understanding of this phenomenon occurs.” — I don’t understand.
“A statistically-oriented projection of the significance of these findings.” — A wild guess.
“It is hoped that this study will simulate further investigations in this field.” — I quit.
All credit to: Scientists Research Paper Chart
JM
You read my mind. or I read yours. Only yesterday I’d started to wonder how the IPCC reports were being cited. For example many are happy to cite it when referencing the future 2-4oC temp increase predicted this century. Yet as you say this is only a review document.
w.
OK, go quote yourself some more. That’s an iterative model that seems to work.
Conrad
One can see it this way:
If we take the assumption that warming makes winters colder, we can speculate that past glacial eras were actually triggered by global warming.
Now we really have to worry about the coming ice age…
The notion that a warmer Arctic means colder NH winters is directly falsifiable via a simple thought experiment. Just make the Arctic warmer and warmer… until it is physically impossible to have a cold NH winter. Now at what point did the original notion fail? There is no point of failure — because it’s all a failure. Similarly, the authors must therefore hold that if the Arctic gets colder, the NH winters will be warmer — similarly absurd. The paper is not science.
“As a result, the computer models all agreeing that the winters will be warmer is not evidence about the real world. No matter how many of the models agree, no matter how much the modelers congratulate each other on the agreement between their models, it’s still not evidence”
If you with evidence mean that only something that has been measured and agree with the assumption made is an evidence then I agree on above statement. But for me a model output that agree with measured data and predicts future data with the same accuracy as it has modeled past known data is “evidence” that I may be on the right track understanding the process. Maybe evidence is not the right “scientific” word in my case.
I can only agree on what you write about what a model represent, your own prejudice, assumptions, etc. I am doing an own study on simple models for modeling the cause for the CO2 increase and one thing that struck me is that a model using the exact same formula but with completly different input values in 2 cases can give an output that matches real world data. When you look at the details of how the model came to the same conclusion in both cases there will be some factors that are completly different but you have no real world data that can tell you which one that is most accurate on that specific factor, so which model do you put most trust on?
The one that best fits your assumption ofcourse…
I wonder how many AGW model runs that has ended with a result that matches real world data but they do not fit the modelers prejudice of what should be the main factors and the result is thus thrown away as faulty and never gets documented.
conrad clark says:
February 1, 2012 at 10:12 pm
Quit trying to hijack the thread, Conrad. Go to school or to a nunnery and learn some manners. Better yet, take your nastiness and just go away!
Here is the IARC web page at uaf.edu.
http://ine.uaf.edu/werc/projects/seward/team.html
Here is a video telling us we must live within our means. And of course, the children…
I actually think AR4 is quite moderate. You do not find the doomsday predictions of several meter increase in sea level or any certain predictions extreme weather events in the IPCC assessment reports. Those worst predictions, which are typically cited in the news, are usually not cited in the IPCC reports.
The estimate of sea level rise in AR4 is 39 cm rise from 1990 to 2100, down from estimated 48 cm for the same period in the TAR. The rise is thought to be caused by both thermal expansion of sea water and glacier melting.
AR4 do say that we know for certain that the global temperature has increased substantially in the last century, and that the level of GHG has also increased, but they do not claim that we know for certain that human activity is a cause of the temperature increase. AR4 says that it is very likely that human activity is a cause for the temperature increase, but this is not certainty. I think this comes as a surprise to many.
Willis and Conrad, what’s with the pissing contest? None of you brought anything good to that part of the discussion, and both of you came out of it looking bad. Willis, it really is not worth getting dragged into the mud like that.
From my point of view, Conrad had some interesting thoughts in that first comment. Had he put them a bit differently, or had Willis been a bit more overbearing, we might have learned something. Now, Conrad will leave this site thinking that WUWT is a place full of brats and whacky skeptics. 30 points for both of you, and that is leaving out the bonuses for self-referencing.
when you’re good, you’re awesome.
great article willis.
I notice that computer models didn’t prevent design flaws from creeping into the F-35 Joint Strike Fighter.
http://www.wired.com/dangerroom/2011/12/joint-strike-fighter-13-flaws/
And the climate system is vastly more complex than this little object.
Why isn’t AR4 called PHOARRR?
@ Gary Mount .. from that link…
“Trillion-Dollar Jet Has Thirteen Expensive New Flaws”
add… “that have been found so far”
trouble with the AGW hypothesis, is that it is based on one MAJOR FLAW.. being that CO2 causes warming…. everything else, models included, is just corollaries from a flawed hypothesis.
Willis
I am currently reviewing the draft AR5 like, no doubt, many others on this site. Not having been involved in the process before I was quickly astonished by the amount of assertion, speculation and conjecture that was immediately apparent within the areas I am competent to comment on.
I had a very circular argument with the IPCC in Geneva when I queried a comment that ‘unpublished research has shown’ regarding a particularly contentious aspect of ocean warming. I queried as to who authored the unpublished research and what it actually said, which brought forth a string of emails between us as I asked for a copy (which we are permitted to see)
This culminated in the surprising (to me) response that only cited (unpublished or otherwise) material i.e that with a number and a correspomnding reference to the authors, could be provided. A (seemingly) wild assertion with no citation could not be provided as it…er…had no citation.
I know Mosh has written elsewhere as to how much better the draft of Ar5 is than Ar4. That may be so, but it therfore beggars the imagination as to what the draft AR4 was like.
It may well be of course that the final AR5 may be very different to its draft-but if it isn’t I can only comment that the eventual summary for policymakers-which previously seemed to bear little relationship to the original final assessment- is going to be well worth scrutinising. It is likely to contain numerous examples of puffery and hype.
tonyb
Sometimes models are programmed because the programmer has little or no idea what the outcome will be. Monte carlo methods are often used in such models. The results are sometimes the opposite of what the modeler was expecting.
I agree with anders valland.
http://wattsupwiththat.com/2012/02/01/cites-and-signs-of-the-times/#comment-882392
No way to conduct a dialogue. A very interesting article none-the-less, and I take it’s point.
I just love the way the programmers “adjust” their models to correlate with the past – especially when they totally failed before the “fixes” or should I say fudges and new routines to model the past – and then they proudly say – “see, we were right and it’s worse than we thought”.
I still remember GIGO and managed to program in that vein several times.
To help me slim down I built a predictive computer model of my weight given diet and exercise as inputs. So far the model output has been within 99.2% of my measured weight. By running this model 23 weeks and 1 day forward I have evidence that I will be lighter than air before the end of the year. Alert the media: I am going to fly like Shazam!
Anders Valland says:
February 1, 2012 at 11:24 pm
Anders, if you think Conrad had some interesting thoughts, why the heck haven’t you discussed them with Conrad? I’m certainly not stopping you. Speak up, man, and ask him questions or comment on his ideas. Because I assure you that blaming me because you and Conrad haven’t been discussing interesting things goes nowhere.
Me, when a man starts off the discussion by attacking me, and tops it by saying I should go back to school, I bite back. If you don’t like it, skip my replies. Some of us defend ourselves. If that offends you, look elsewhere. If you want someone who will blow in Conrad’s ear and rub his tummy when he comes into the discussion and acts like a pompous prat, you are reading the wrong man’s work. I’m not that guy.
Anders, you are about the 413th person who has come through the door whose first words are to tell me I ‘m doing it all wrong, that my comments are too boring, that my topics are not topical, or what I say is too overbearing, or I’m too dismissive of serial stupidity, that I should write shorter pieces, or longer pieces, that I shouldn’t puncture pomposity, or a hundred other reasons why I’m wronger than wrong. You are merely today’s version of, in Spiro T. Agnew’s memorable words, the “nattering nabobs of negativity”. By tomorrow there will be another frustrated would-be “Miss Manners” popping up to tell me that my morals or my methods or my mannerisms are terribly wrong.
So here’s the deal for me, here’s how I resolve it. It’s like what the song says:
“But it’s all right now,
I learned my lesson well.
You can’t please everybody, so
You got to please yourself.”
What will Conrad think when leaving this site? I haven’t a clue. I have written over 200 posts on WUWT, and I had a million page views last year of my work. I’m positive that not all of the million were happy with what I said. If 1% of them were dissatisfied, that’s hundreds and hundreds of folks out there who, like you, feel I should go back to school, or feel I should have been nicer to poor Conrad, or have some other complaint about my actions.
So, I can’t please everybody, and although the percentage is small, that’s still a lot of unpleased people. Some of them, like Conrad, want to start out by lecturing me, telling me I’m a jerk who doesn’t understand computers and should go back to school. Others clutch their pearls and gasp in horror when I call someone out for coming here and starting by lecturing me and going back to school.
You may be accustomed to people who trim their sails based on polls, and votes from the readers, and expressions of concern, and the like. Me, I’m a seaman who follows the stars and the winds. Focus groups mean nothing to me. So please, save your expressions of concern for someone who might appreciate them.
The thing that all you guys seem to be missing is this. The Alexa statistics show WUWT is far and away the most influential climate science blog on the web. The blog statistics show I am far and away the most popular guest author on the site. I have been pillorying serial stupidity and not putting up with abuse since I started writing here. Why should I change now?
Hard to argue with success, WUWT has been very successful, and generally my motto is, if it ain’t broke don’t fix it. So no, I fear I won’t take your well-meant and perhaps reasonable advice, Anders. I am who I am, I don’t suffer either abuse or fools gladly, it’s no good for me to pretend otherwise, and some folks actually appreciate my honesty. I welcome people who come with honest questions or scientific objections. People who just come to abuse me? Yes, they might leave dissatisfied, as you suggest Conrad might be going.
Or they might stick around and contribute to the conversation. Up to them.
All the best,
w.
“Seconding everything Willis said in response to conrad clark.
conrad clark wrote:
WTF! You don’t calibrate your models, run them against an independent test set, align your initial guesses with objective reality?”
Adding my tuppence worth… This time last year the Formula one racing teams were putting the finishing touches to their 2011 cars ready for testing. McLaren, who employ some of the best computer modellers and simulators in the world, put their new car on the track and expected it to be a competitive car. However, it was a case that the “models” were wrong. Do McLaren do all the calibration and testing required? Hell yes to the nth degree. They even sent their 2010 car to the first 2011 test to get accurate measurements to use for calibration and comparison for the parameters of their 2011 car, but a computer model cannot possibly integrate everything that happens in reality AND it does indeed integrate the assumptions (prejudices and beliefs) of the people who design the system and code the models.
McLaren had gone from the front of the grid at the end of 2010 to the back of the grid (in terms of pace) at the second test of 2011, which is a massive loss of pace. What the computer models told them about the 2011 car was wrong.
Unlike climate change “experts”, McLaren engineers HAVE to deal in reality as the results on the track cannot be fudged. They recognised that there was an error in their simulations and put in place measures to correct them. They constantly take the data gathered from the race weekends and enter that into the simulation software to calibrate their software against reality and that is with the relatively small set of parameters effecting a race car in any given weather on any given track. Even then, the simulator is wrong to a greater or lesser degree from the reality when the team gets to an actual race weekend.
Climate models have to incorporate an almost infinitely more complex system, with far less predictability, in a chaotic, non-linear integration of competing and synchronous climatic systems and sub-systems. The bias of the people designing the models, (or at least those who give them the data and operational parameters) are going to have a massive effect on what the models tell them.
I know enough about computer modelling and computer simulations to know that they are a valuable tool in running multiple variants to gather general data quickly, and only when filled with predictable logic, but they are less than useless as a predictive tool when dealing with chaotic systems with multiple complex, unpredictable and many unknown feedbacks. Tossing a coin is as effective a predictive tool, as a computer model of the climate.
Models are useful in that they can be used to encapsulate a hypothesis that in some sense is mathematically consistent. This is better than a hypothesis that is not mathematically consistent. But I would agree that they should not be cited as if their output were empirical observations.
Willis do us all a favour and don’t change.
Why say in a couple of sentences what can be said in six pages?
d.
Agnostic says:
February 2, 2012 at 12:15 am
Great, Agnostic. The vote is now 9,412 to 3 that I’m a jerk. Thanks for voting.
I notice that you and Anders haven’t had a dialog with Conrad at all, but you don’t like my dialog with him. If you can do better, why aren’t you doing so?
w.
As a non-scientist following these discussions for some time, it seems to me there is a philosophy of science question here that is more general than those raised by climate science: When is computer code “science”? Is it ever science? Can rules be laid down that would make it science? When Einstein writes down E=MC**2, any scientist on the planet (or elsewhere for that matter) can understand it, formulate experiments to test it, draw conclusions from it, etc. Where are we when someone writes a big, complex computer program claiming to be a representation of reality? At best it seems like the doodles that perhaps Einstein used as motivation before he actually formulated the science. Writing code to fit past data and then make predictions seems little more than a sophisticated form of curve fitting. If it doesn’t work or new data doesn’t fit, just tweak it and run it again. After it is tweaked, is it the same model or something different? In essence, running the code is not sufficient to say that the code is “science”. Somehow, the construction of the code and the assumptions of the code have to be accessible and atomically adjustable to start to say that some model is “science”. And most importantly, models must be capable of being compared to experimental/observed data; if they do not agree, then they are not right and must be rejected as bogus, failed attempts to grok reality.
Doug Cotton there is a problem with your paper. Page 2 is corrupted with overprinting. So I got all but that part.
What I could read I liked and it is nice to see that someone else agrees that heat cannot flow spontaneously from cold to hot. This is a major failing of the GHG theory.
Climatereason says:
“I am currently reviewing the draft AR5 like, no doubt, many others on this site. Not having been involved in the process before I was quickly astonished by the amount of assertion, speculation and conjecture that was immediately apparent within the areas I am competent to comment on. ”
The combination in AR5 of assertion, speculation, conjecture embedded in an array of good science, mediocre science, bad science mixed with politics and financially dependent ego’s is a recipe for disaster.
Willis,
I agree with a comment given earlier: please don’t ever change!
Your articles are all gems.
Doug Cotton, Sorry it is my computer. Asking it to print your article, through several routes, produces the overprinting on page 2. Your web page is OK. I thought Windows 7 sorted these problems.
Do you have a PDF version?
Anders, I agree with you. But don’t you mean “a bit less overbearing” rather than “a bit more…”?
We have been treated to a head-butting contest between two antagonists, each of whom has much good to offer. But their obvious self-regard is beyond any level to which mere mortals such as you and i dare aspire.
Ken Hall. The one thing you forgot to mention was THE DRIVER. He is the one guy who can actually come back in and say, this is wrong. All the computer programmers in the world cannot come up with that final response. This is why drivers and test drivers are so highly valued. Some are faster than others and they get the champagne and the kudos, but every team has to have a driver who can come back in and say WHY the car isn’t going as fast as the models and simulations. Schumacher was an example of the driver that could do both, drive fast and identify design flaws on the track. Prost was another one. What is needed in climatology is less modelers and a few more guys out on the track doing the work. It would help if even one of them occasionally looked up from his keyboard and glanced out of the window at the weather.
Conrad Clark,
Write us a program showing/explaining the relevance of the number 9 to infinity
Willis, what a wonderful world we live in, and you Sir, make it more worth while.
The authors of that study ought to look out the window. This year the polar jet is strong, there is little blocking and cold air is bottled up in Canada and Alaska (where Jan was 20F below normal in many places). The models “predicted” this back in the 90’s and early 2000’s. I put quotes around that word because models don’t predict anything. Also the polar jet was strengthening at that time, so the models matched up with reality. Those model results directly contradict the claims of the last few winters that Arctic warming or low sea ice was responsible for continental cooling, here in the US and in Europe.
The original models seem more applicable since the strong polar jet theory has a physical basis in a cooling stratosphere. One of the problems with verifying the theory is that the stratosphere is cooling from lowered solar ultraviolet and that seems like a more probable cause of the recent colder Northern winters.
Willis, get a grip. A pissing contest on who has done the most advanced programming for the longest time gets you nowhere – and even if you say you don’t care, for the CAGW believers you give them food for telling everybody else what a jerk you are. I care, because I really think you are not the jerk you act like sometimes. So – you were both wrong, doing the “best defense is attack”-variety has no effect with me.
Conrad, from what you write I guess you haven’t been here long. Willis position on models is quite clear in my view, he has been very vocal on the issue of models vs. reality (AKA observations), where observations trump models anytime. Although that is not very apparent if this post is all you have read from Willis. I would like to know what you think when you say that the Machine Learning class has any relevance to this – do you feel neural networks and self-learning models should be used for modelling climate? Do you think that is feasible, given the complexity of the issue?
John Marshall, February 2, 2012 at 2:11 am
“Do you have a PDF version?”
John, if it’s any help, and if you have a Kindle, Chrome has a gizmo that allows you to send any web article straight to your Kindle (you don’t get the comments, though).
Thanks for a great post Willis. I once had a go at what could be called modelling, but which we then called robotics. I had to code a PC to interact with and control six systems. The initial code was about 2k lines. And it worked. But then, we sat back and tossed around all (ALL?? Hah!!) the ‘what ifs’. That added another 20k lines to the code. It was pretty small beer in those days, but fun, and very educational. It worked, but it was NEVER perfect.
Straight on to sunrise Willis, your bearings are fine. Quite frankly, I don’t know where you find the time nor fortitude to sieve these brain damaging theses. I certainly appreciate your ability to do so. Thank you sir.
The fact that it only takes them 5 words into the abstract to use the word “consensus” tells you absolutely everything you need to know. “Consensus” is a word that has application in politics – it has no place at all in science. In fact the application of the word “consensus” in science actually prevents the evolution of scientific theories towards scientific truth. The use of the word consensus in science stifles alternative views and the evolution towards truth because the required skeptisicm that goes with the formulation of alternative views immediately puts those who articulate it on the outside of society. In the context of scientific progress it is a blocker. As an example,. we do not have a “consensus” that the world is spherical. That is something which is quite simply recognised as a fact, made clear by countless and repeatable observations that match scientific theory and with evidence from multiple sources and angles. The ground-truthing of that particular fact did not go through some magical phase called “consensus”.
It only took 5 words.
I agree with Willis that models cannot prove that your theory is correct, but they can prove that it is wrong.
Model output says “if my theory is correct, I expect the following real-life behaviour”. If said behaviour is observed, the theory may be correct or you may just be lucky. However, if the behaviour is not observed then clearly the theory is wrong.
If numerous attempts fail to prove the theory wrong, then you may have some hope that the theory is correct, but never certainty.
Note: the above fails if you “tune” the model with known results. As soon as you depart from pure theory and add in fudge-factors, the results no longer say anything useful about the theory.
Thanks for the interesting article.
Willis Eschenbach is always a good read, in spite of (or perhaps because of) being a stroppy blighter, who knows his own mind only too well. Read him on his terms, and you will be better informed and possibly a little wiser.
conrad clark says:
February 1, 2012 at 10:12 pm
—
You criticize Willis for what he says. Willis quotes himself to show that you misunderstood what he said.
Then you criticize Willis for quoting himself.
You claim to have been writting for years, yet your behavior is nothing more than that of a poorly schooled grad student.
“The study is “Arctic warming, increasing snow cover and widespread boreal winter cooling“, by Judah L Cohen, Jason C Furtado, Mathew A Barlow, Vladimir A Alexeev and Jessica E Cherry. This study proves once again that in the topsy-turvy world of climate science, all things are explainable by the AGW hypothesis … but only in hindsight.”
================================================================
How many points for “More handwaving than a Rose Bowl Parade?”
conrad clark says:
February 1, 2012 at 9:35 pm
w.
Re 1986 neural networks and machine learning. Were you in the same IBM SRI class as me? Believe me, you need to be dragged into the 21st century.
Iterative models dont seem to work with climate science (or am I misreading the lack of actual predictions)?
Conrad
—
Conrad, before you start saying stuff like “iterative models don’t seem to work” please educate yourself on how numerical methods work for solving systems of partial differential equations, which is what a climate model is at its core. Essentially, you are numerically solving a set of discretized equations which estimate the time rate of change of key physical variables such are air temperature, velocity, pressure, moisture content, etc. (if you have a coupled ocean model, then the time rate of change of ocean current velocities, temperature, etc. will be determined). There are many submodels associated with radiation, cloud formation, aerosol transport, etc. which also are solved in support of the basic equations. You then “march” the numerical solution iteratively over “short” time step (1 time step = several hours for a climate model) to get the solution at the next (future) time level. Do this thousands and thousands of times to cover days, months, or years of prediction time.
The problem is that the equations being solved are non-linear and coupled, with many different characteristic time scales. Even with the much simpler compressible Navier-Stokes equations, it can be very difficult to get solutions to basic problems like separated flow over a wing at high angle of attack. Climate simulations are at least one order of magnitude more complicated. In addition, with these kinds of problems, then solutions can be highly sensitive to initial conditions and boundary conditions. And depending on your assumptions, you may or may not get a valid solution – nothing can be guaranteed for non-linear systems!
I’m not against using models for climate predictions, except that many research groups (particularly NASA-GISS and Model E) are sloppy in how they document what they’re solving. In Model E’s case, there is NO one place that you can find all the equations being solved adequately documented! And the FORTRAN code is not well written at all. They have some word descriptions which are a bunch of unverified fluff and say almost nothing about their numerics – and this is the code being used by Hansen and his cronies to assert that the “missing heat” is 0.58 W/m^2 and that humans are responsible!!
On the other hand, groups like the GFDL at Princeton and NCAR do a great job with their models, both in coding and documentation. Which leads me to the inevitable question about redundancy – why do we need to be funding dozens of these climate models??? Get a competent group to develop a single code and go with that. It would save a LOT of money and remove the uncertainties associated with presenting climate model results from disparate groups (and then averaging them as ensembles – yikes!).
“And finally, given that the models are unanimously wrong on the decadal scale, why would anyone place credence in the unanimity of their predictions of the upcoming Thermageddon™ a century from now?”
I don’t know (though we can all guess) but they still do.
http://www.telegraph.co.uk/earth/earthnews/9038988/Climate-change-will-make-UK-new-holiday-destination.html
At least they are no longer suggesting that warmer means unmitigated disaster – a big step for some.
Mrs. Henninger must have been a real gem. Not only did she have you writing research papers, but she actually schooled you in proper citation! We never got further than elementary textbooks. My ‘science teacher’, Mr. Cooper (who also coached girls’ softball), believed that rockets would not work in space, because “they had nothing to push against.” Citing Willy Ley (The Conquest of Space) and Newton was of no use.
/Mr Lynn
In many areas such incorrect referencing can lead to undergrads work being failed, perhaps only in climate ‘science’ would it be regarded as acceptable professional standard for published research.
sceptical says:
February 1, 2012 at 9:05 pm
“Or perhaps because there was already a FAR, First Assessment Report.”
Maybe they should have called AR4, “2 FAR”
“Isn’t it grand to know that each error we find only serves to improve the models? What an accomplishment. So productive!”
Yes, and any day now we’ll have the geometry of those epicycles completely worked out. Every error we find in predicting planetary motion improves them just that little bit more. Won’t the future be grand!
@Anders
“Willis position on models is quite clear in my view, he has been very vocal on the issue of models vs. reality (AKA observations), where observations trump models anytime.”
One of the ultimate crimes against science is to get a set of observations and compare them to what spits out of a model in which one has invested a lot of time. Then, correct the observations to align with the modelled output and publish the model as a better and more complete record of what is real. This has to stand as the essence of lunacy. Utter madness. Even alchemists had more common sense and logic than that. It passes beyond wilful blindness into the realm of madness for only a madman thinks altering reality will make his fantasy come true.
Climate ‘science’ is the only branch of anything where such lunacy is given a plinth from which to dictate social, economic and political action. Normally, in the land of pre-post-normal science, exposure of craziness or fraud or incompetence is rewarded with oblivion.
It would be interesting, in a morbidly fascinating way, to see the CAGW climate science community itself modelled to predict their collective behaviour. There would be a problem finding suitable analogous models in the animal world because even weasels are not that perfidious. It will have to be done using publications, websites and personal observations. Given the level of consistence seen thus far, and the trends to replace reality with models, Willis’ 0-30 scale may be a useful metric for making chart predictions about the content, warmings, predictions and facts one might find in the coming AR5.
Willis, I agree again with anders:
http://wattsupwiththat.com/2012/02/01/cites-and-signs-of-the-times/#comment-882480
The problem is, if you conduct a debate in such combative terms, it detracts from the very laudable and interesting points you make. You should relish the opportunity to take on someone like Conrad to drive home your point.
Also, we normally associate such combativeness with defensiveness. It’s been noted that those on a losing side of an argument start attacking their detractors rather than their arguments. As far as i am able to tell, you are no where near losing this specific point as an argument, so no need for the defensiveness. And I am not sure justifying it by pointing to the blogs popularity or your proliffacy as a guest poster is wise either, given that we generally object to argument by consensus or argument by authority.
As far as engagement with Conrad is concerned, I am nowhere near qualified. But you are….so engage! Don’t beat him up straight away at least. And while you are at ‘im the rest of us can learn something by way of the exchange. We don’t like the way RC deride anyone who questions the orthodoxy, so let’s not start here.
Thinking back to several years spent at the Cavendish Laboratory, and several decades spent in high-tech research, I never, ever, heard the term “THE SCIENCE” used by a scientist. On the few occasions when this terminology surfaced, it usually came from lawyers or public-relations flacks. Maybe Willis doesn’t need his ingenious “scoring” system for spotting the fakers and poseurs — any time “THE SCIENCE” pops up in a paper or discussion, we can discount the source!
Can I cite this as “Eschenbach W in Watts A, 2012”?
Mr Lynn: reminds me of a science teacher I had in high school who said that you could put a fan on the back of a sailboat and it would drive the boat.
Willis:
Could I put my hand up in a Tee, and ask for a TIME OUT between you and Conrad?
One of my specialties is FINITE ELMENT ANALYSIS. When you write up an FEA program, and you make the “stiffness matrix” for the elements, it’s based on the well established “mechanics of materials”.
If I just use the PURE MATH of the mechanics of materials, and write up a proper set of stiffness matrices, and do a model…say of a pressure vessel, when I run it if I do the simplest evaluation (say the hoop stress, at the center of a vessel) I should match to about the numerical accuracy of the materials properties. (Commensurately, in the literature, we can actualy find historic TESTING on various PV’s and match their measured strains (displacements) with our FEA and know it’s a fairly robust system.)
There are a multitude of “closed, limited variable” systems for which this work extremely well.
HOWEVER the Atmosphere is neither a completely closed, nor a limited variable(s) system.
It is hugely complex. I really liken some of the Atm models to attempts to model the economic system and therefore, say the stock market.
As Willis says, this WILL be dominated by prejudices, “beliefs” and the results will depend as much on the programmer as any data or manipulation thereof.
Max
Heh… I have to do it.
“Willis Eschenbach says:
February 2, 2012 at 1:40 am
Response to Agnostic:
“…I notice that you and Anders haven’t had a dialog with Conrad at all, but you don’t like my dialog with him. If you can do better, why aren’t you doing so?”
Dunno about A & A, but my take on Conrad is that it’s ‘Mind over Matter.’ I don’t mind and he doesn’t matter. You can find a twit in any parking lot, all you have to do is look. He’ll be the one wrestling with the shopping cart.
BTW Willis, great article!
Most of climate science has things all wrong. They really need to mathematically model their hypothesis of all the underlying physics (about a hundred different specialties). Then use the computer to numerically solve those equations so they can figure out what and how to measure in the real world to determine whether or not the hypothesis makes any sense. Any other use of computers is just playing video games.
Phil: It will, but not if you point it at the sail 😉 . Love those swamp boats that work on that principle.
I have seen science teachers in training around here and some are excellent (pay attention in their science classes and do well) and others are pitiful (very social but not learning the science). I’m afraid our future k-12 students are in for a mixed bag.
Model Problems:-
http://www.bbc.co.uk/blogs/paulhudson/
My apologies to Willis because I looked at the Solomon reference, “Solomon S et al (ed) 2007 Climate Change 2007: The Physical Science Basis (Cambridge: Cambridge University Press)”, and my first thought is that there is no way that could be the IPCC document but a short search later showed it was. Therefore, I think that it rates a score over 30!
Climate models should be simulators based on first principles. A simulator that can model 3 bouncy balls in a revolving barrel should be able to model 3000. From that any macro behaviours should then arise naturally.
(You don’t model the big things that arise from the little things: you just model the little things accurately and let sh*t happen.)
The way they discuss the model’s lack of alignment with reality and the approach to amending it seems to indicate that the first principles just aren’t right i.e. the scientific assumptions are wrong, and they’re going to whack it with a hammer until it looks right (fudge).
sceptical says:
February 1, 2012 at 9:05 pm
Mr. Eschenbach, “…the most recent one is called “AR4″ rather than the ”F. u. b. A. R. “, presumably to avoid using the “F-word”.”
There sceptical and Willis I fixed the F.A.R. thingy for you.
Frank K. has correctly summarised many of the problems associated with climate models. “In addition, with these kinds of problems, then solutions can be highly sensitive to initial conditions and boundary conditions. And depending on your assumptions, you may or may not get a valid solution – nothing can be guaranteed for non-linear systems!!”
Like him I am not against models being used for climate predictions. I am against those who pretend that such predictions can have any serious degree of accuracy – except, perhaps, as to the boundaries which have been imposed upon them in the initial assumptions.
Some thirty years ago, using using a variety of the most sophisticated econometric models then available, three hundred and sixty-five leading economists in the UK forecast that Prime Minister Thatcher’s financial, industrial and economic policies would fail and lead to all sorts of disasters with a few years. None of which came about. Instead the UK thrived.
Econometric models are not nearly as complex as climate models nor do they pretend to forecast events so far into the future.
George Edward Pelham Box, professor Emeritus of Statistics at the University of Wiconsin, famously said “Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.” Empirical Model-Building by Box and Draper, p. 74
Anders Valland says:
February 2, 2012 at 2:49 am
Anders, since your first appearance in this thread your whining posts have been one unending litany of complaints. Are you like this in real life? Do you walk into a room, seek out the host whom you’ve never met, and start the conversation by telling him he’s being a jerk?
Get it straight, my friend. At this point your own credibility has sunk below that of Conrad, and that takes some doing. Conrad’s boasting about how he knows all about models and I should go back to school makes no difference to me. But at least he’s not mewling about it, he’s a man.
So why do you think that your wingeing and whimpering about what a bad boy I am will affect me in the slightest? I don’t care even one little teeny bit what you think, Anders. As I said above, you are just the latest jerk to appear and tell me I’m doing it wrong … and as if to prove my point, just after I posted that to you above, somebody came along to tell me what I had posted was too long …
I ignored him, just like I ignore your complaints. You don’t get it, Anders, I’m not one of the pet toy poodle people you hang around with that are all sensitive new age guys. I don’t give a damn if the CAGW folks are upset when I tear their science apart. I don’t care if you think I’m a big meanie to Conrad. I am one of the real 1% who don’t live their lives and set their courses based on what other people might think, but who follow their own compass. You might try it sometimes, it is exhilarating to be responsible for your own successes and failures.
You have a choice, Anders, both here and in the world. It is the choice that Buddha put before his followers. In its simplest form, it can be expressed as “dig it or bitch about it”. For everything that enters your ken, for everything that happens, you can either enjoy it or hate it, the choice is up to you.
Now, ever since you entered this thread, you’ve been bitching about it, and about me, and about my actions, and about what a bad person I am.
That’s your choice, my friend … but other than a serious empathy for what living with such a nasty mental condition must be like for you, I fear that your unending litany of complaints is meaningless to me. Sure, if you insist that the world is totally unacceptable to Anders and complain about it all day long, that’s how the world will really be … but it’s your choice.
w.
Turns out Mark Twain didn’t say “there are lies, damn lies and statistics”. But he should have. Willis, did I read that post right? Were you instructed to look up something in Wikipedia?
conrad clark says:
February 1, 2012 at 9:35 pm
Thanks, conrad. No, I was teaching myself. And because I was self-taught and living in a remote location, as far as I knew at the time I invented the idea of computer-based evolutionary learning. Of course I didn’t actually invent it, far from it. But I worked on it off and on for some years in my isolation, which was most entertaining and educational, I learned a lot.
Sadly, you are quite correct about iterative models. As I’m sure you know, iterative models are tricky, finicky beasts that love to not only go off the rails, but to fall off the table entirely. They are also black boxes in that you may not know a) where it went wrong or b) how to fix it.
So when three of them come up with widely differing results, it’s hard or even impossible to sort out why … and getting the right answer for the wrong reasons is more dangerous than just getting the wrong answer.
All the best to you,
w.
Agnostic says:
February 2, 2012 at 7:47 am
And your problem is, you think I care in the slightest whether you agree with Anders or what Anders thinks of my evil ways. Get it straight. I don’t care about you or your anonymous opinion.
I’m a guy who doesn’t suffer fools gladly and who defends himself when attacked. I don’t run my life based on votes from my friends about my behavior. I pay no attention to focus groups, and even less attention to earnest advice from random internet talking heads who won’t even sign their own names to their opinions.
Sometimes most of my readers agree with what I say or do, sometimes most of my readers think I’ve gone off the rails. I pay absolutely no attention to either reaction, I “treat those two impostors just the same”. I do what I think is right, not what you or the vox pop think is right.
You are correct, I’m a throwback, a dinosaur, an atavism living in the wrong era, a fool who truly doesn’t care in the slightest what you think about my actions.
You clearly don’t like that.
I’m sorry to hear that, but I have no plans to change based on the likes or dislikes of a man who won’t even sign his own words.
w.
Ron Manley says:
February 2, 2012 at 7:51 am
Yeah, but don’t forget the page numbers …
w.
Willis, when you say “The blog statistics show I am far and away the most popular guest author on the site. ” the phrase ‘car crash TV’ comes to mind.
I had to laugh. I’d just written to Agnostic and Anders above about why I didn’t put any weight on their opinions of my actions.
After that, I went off to cruise the web and came across an article by a nurse about the 5 biggest regrets people had when they were dying. Topping the list, number one with a bullet, was the following:
As you might assume, that regret will not even make it on my top five, much less be number one …
w.
What happened to the kinder, gentler Willis?
Wow Willis, you’ve really stirred something up. I can’t help but laugh at the ludicrous nature of the conversations going on with those three posters. The real joke is the other two complaining about you being mean to poor ol’ Conrad, while completely ignoring the baseless, illogical attacks by Conrad at your credibility over completely true things you write–let alone the debasing nature of his tone. But, even then, at least even Conrad was still discussing the article, even if challenging a relatively unimportant fragment of it!
This is the first time a WUWT thread as made me smile with amusement.
I guess you rankle feathers when you point out the incapacity of computers of model “reality” (or their version of it) with any relevance. To me, models are only useful in getting a handle on what variables deserve extra attention for -empirical observations and experiments-. If you put in all the relationships you do know, and see it go off the rails, it can help in a reductionist way to dissect out what relationships need further study to get a true view of reality built on science not computer programming. To me, a scientist, models may be a tool in the box, but they are not an experiment, or observation, or science; and it’s a travesty we have this field of climate science trying to change our ways of living based on what the computer programs they are writing are saying, rather than reality! Or what about geological history? It pains me that we sit here relaying on computer simulations rather than the observable geological record, or the normal swings going on outside our windows.
Thank you for sticking it to them, and upholding the credibility of science.
This paper makes me laugh, and is contrary to every known property of climatology. Melting ice doesn’t increase cold, or add cold, it just holds temperatures steady in its immediate area until it finishes melting. Have climatologists forgotten thermal dynamics at the same time they jettisoned the geologists out the airlock?
Mr Eschenbach,
After reading your eye-opening argument, I was reminded of a Wiki article, and managed to remember the term and the title for the faulty scientific processes in climate science you describe: pathological science. It’s a curious term apparently coined by one Irving Langmuir in the ’50s. It can be found at http://en.wikipedia.org/wiki/Pathological_science and may be worth your time, if you aren’t already familiar with it.
What I find interesting is that in rejecting input by “non-specialists,” i.e. scientists from other disciplines, “mainstream” climate science is digging a hole for itself by relying exclusively on government and “green” industry funding which gives it a false impression of legitimacy, stability and longevity. The glory of Soviet science, like Lysenkoism and “poly-water” come to mind. The problem for climate science, of course, is that by severely limiting the pool of scientific contributors and reducing the number of papers that are “acceptable,” climate science will drift off the map to become another oddity categorized as “pathological science.” Add to this the fact that climate scientists are (mis)behaving very much like pampered pseudoscientists with an attitude, we may expect that climate science will sooner, rather than later, find its proper place among the “traditional” pseudosciences like phrenology, homeopathy, chiropraxy, cold fusion, chem trails, astrology and perpetuum mobile “research.” It’s worse than we thought!
@Alan Watt,
How is Willis being mean or unreasonable? He isn’t being a jerk, nor unkind! Just saying he isn’t going to dance to their tunes, or bow to their preconceived notions of… whatever it is they are trying to make him do, I have no idea what they are hoping to accomplish, do you?
Nothing Willis has said is a personal attack, unlike the personal attacks or derogatory statements the other three posters have made (particularly Conrad with his “go back to school” snark; quite the insult! Why haven’t you called out Conrad for a direct insult like that?).
You should be asking Conrad to be kinder, gentler, and more reasonable… and get back on topic of the thread.
Phil, your science teacher was right. IF the fan is big enough and blows towards the rear end of the boat. You don’t even need a sail. These things are called air-boats. 🙂
Alan Watt says:
February 2, 2012 at 11:23 am
Bad news, Alan, this is me being kinder and gentler, you didn’t see all the stuff I left on the cutting room floor before posting my responses to the commenters … hey, it’s a work in progress, what can I say?
w.
Alan Watt,
What Willis was that. I have never noticed him around here.
Willis, please don’t change, I like your attitude just like it is.
PhilH says (February 2, 2012 at 8:03 am): “Mr Lynn: reminds me of a science teacher I had in high school who said that you could put a fan on the back of a sailboat and it would drive the boat.”
Mythbusters blew their own sail with limited but definite success:
“Solomon Green says:
February 2, 2012 at 9:25 am
Frank K. has correctly summarised many of the problems associated with climate models. ”
Not to hammer a dead horse guys, but I learned in stat 101 that curve fitting is not modellling(that was physics 101). Anytime you use equations with adjustable parameters you are curve fitting, and a curve fit “model” is only usefull within the data range that was used to generate it. Primarily because you have no way of knowing or testing whether or not the equation actually fits the physics behind the observations and can give correct results when out of its fitted range. Any physical model (without fitted parameters) will at the very least model the correct sign at any point. I like to use the photo-voltaic effect as an example. Once Einstein visualized the correct model, photons as particles instead of waves, he had a physical model that worked. Further work in particle physics allowed them to calculate the photo-electric voltage and match measured results. and in some cases spot poor meaurements.
No climate model is based solely on first principles. so none of them are useful in making predictions, or projections, or wahtever they call them.
You have to look at how the terms are derived and applied. In IPCC-speak, “very likely” means either expert consensus (their own) or >95% statistical reliability Neither one is worth spit. Yet having so pronounced, both “likely” (90%) and “very likely” are then taken as givens and the sand-castle-in-the-air building proceeds apace.
Ged says:
February 2, 2012 at 11:52 am
I think it is a bit of a stretch to go from my rather mild query:
to the conclusion that I believe Willis is being mean, unreasonable, a jerk or unkind.
What prompted my comment was the recollection (quite possibly in error), that on WUWT within the past several months (I realize with Willis this encompasses a lot of posts) Willis stated he was turning over a new leaf and trying to be less abrasive (probably not the exact wording). In that context, the exchanges between Willis and Conrad seemed to suggest a certain weakening of said resolve.
If I am mis-remembering and Willis never expressed any intent tone down a bit, then my apologies to all — carry on as it were. Although I find it hard to believe I have Willis confused with anyone else who posts here regularly.
I now return you to your regularly scheduled free-for-all …
A lot of this meta-discussion discussion reminds me of a great term/phrase invented by Gestalt Therapist George Bach in “The Intimate Enemy”, when discussing argumentation styles. It labels telling someone what THEY think, feel, or believe “Mind-fucking”. It’s intended to, and very effectively does, stimulate irrational rage reactions. It necessarily implies contempt, the most corrosive of interpersonal attitudes.
If you MUST do it, prefacing it with “I think/suspect/surmise/guess” can somewhat mitigate the loudly implicit and very arrogant insult. But better to resist the temptation altogether.
Willis,
I don’t think you are a jerk. But then I also know that it is not the best idea to push the wrong button with you, particularly when your cranky side is acting up. I’m fine with others doing so, as the results are pretty entertaining.
BTW – as usual, I agree with you on both the failure to accurately use reference material and on the idea that models are little more than an output of what the modeler put into it, which is not the same as an accurate reflection of anything real.
The question arises, could any amount of aerodynamic modeling have predicted that a fan can in fact blow a sail forward? Once the experiment has been done it’s easy to explain why it happens: the backward component of the wind force bouncing off the sail exceeds the difference between the force on the fan and the force on the sail. So much depends on how close the fan is to the sail. Any aerodynamic engineers out there? See if you can model at what distance between the fan and the sail the boat will not move forward. –AGF
I wonder if Ms. Laframboise has regarded references to FAR, SAR and TAR as “peer-reviewed” ones or “grey” ones during her team auditing of the AR4 citation.
interesting to view the frequency of comments on this post, i.e. the time comments were published – maybe something to be gained by that analysis – ‘hot’ topics? are those warmists at it again ?
it seems to me that this discussion was highjacked by the Conrad Clark criticism, (a debating ploy?) which takes from discussion of more important matters. i.e. the rebuttal of warmist propaganda. possibly some censorship protocols might be appropriate here.
re ‘academic integrity’ – ‘oxymoronic’ jumps to mind. most of my experience tells me that yes/no pass/fail views are formulated in the first second of sighting. there might be a 10% chance of revision, and a 10% bias towards disposition. yes, ‘blind’ assessments help to avoid this, and multiple choice questions. it’s obviously harder at the more complex end of the intellectual spectrum.
slightly off topic, but I’m interested in the ‘groupthink’ here – maybe via a key word count. does anyone here have a snappy little program that can do that ? even better, put the findings into a summary ?
Alan Watt says:
February 2, 2012 at 11:23 am
Where’ve you been?
Willis. You are so right and it isn’t just computer models, it goes right across our life experience.
Everything is dictated by our beliefs, ideas, assumptions, errors, and prejudices.
The model may be validated most of the time or even ALL of the times we’ve tried it.
Still doesn’t mean it’s right.
On the other hand, once it’s wrong, we’ve made a mistake somewhere”
DaveE.
Here’s a sample of what Oz’s once proud Commonwealth Scientific and Industrial Research Organisation has been reduced to ‘exploring’-
http://dancass.com/static/files/assets/08eed16e/Report__Exploring_community_acceptance_of_rural_wind_farms_in_Australia_a_snapshot_CSIRO2012.pdf
Well if the dumb ignorant slobs out there have gone off the Global Warming, Climate Change, Climate Disruption mantra and carbon taxing then it’s much safer to stick to exploring whether they have any violent objections to windmills. Nope! Whew that’s a relief! Quick, print it and get out the press release before they change their minds.
Welcome to our new Climatology Spruikers and Inventive Research Organisation folks!
Willis don’t ever change. Does that mean I agree with you all the time HELL NO. 😉 but I have learned a great deal from you about many things and I for one will continue to read every one of your posts when I see then on my daily travels on the web. Thank you sir and keep it up.
Brian says ” In IPCC-speak, “very likely” means either expert consensus (their own) or >95% statistical reliability Neither one is worth spit.”.
Well, the global warming debate is very polarized. It seems like almost everyone who have taken a standpoint think that their opponents are either idiots or having a hidden agenda.
I think it is very interesting that IPCC say that it’s not certain, but a 95% chance that human activity plays a role in the observed temperature increases.
This makes room for a serious debate on whether the 5% chance is actually happening.
Willis, you spend a lot of time on us telling us you don’t care… You don’t know me, I don’t know you. I will not stoop to the level of trying to tell you who you are.
I never said I thought you were too harsh on Conrad. I never said I believe you should shy away from people who start their argument with “WTF!”. What I tried to tell you is that your behaviour detracts from your points. When you stoop to the level of that commenter you make a mistake. Stooping lower makes it worse. You know that. Yeah, yeah, I know you don’t care. I have a teenager, I know how that works.
Now, I care about what is on the pages of WUWT because I tend to use it to show people some very good arguments and posts. There is even more gold in the comment section. But not this time. I’ll step out of this thread and leave you to it.
Reblogged this on riversmoon416.
Anders Valland says:
February 3, 2012 at 2:55 am
So you’ve come back to accuse me of acting like a teenager, and then you are taking your ball and going home in a snit … and you think I’m the one acting strange here?
WUWT is the most popular climate blog on the web, Anders. It has become that way in good measure because people here tell the truth. Me, I go over the edge sometimes in that regard, I’m a really bad liar, so I often tell the truth when others might be able to successfully and artfully skirt the truth. Skirts are not my style, I wear pants.
And despite my telling the truth, or perhaps because of it, I am far and away the most popular guest blogger on the site, with about a million page views last year. That’s about twenty times the page views of any other guest author here. And now, you show up to tell me how I’m doing it all wrong, and to school me in how to straighten out my wicked ways …
I can understand you wanting to leave now, Anders. Conrad came in to patronize me and to tell me I should go back to school, and I answered him straight up. It could have just stopped there. Conrad was over the line, and I suspect he realized it.
But then you went all Miss Manners on me, and when I didn’t kneel and kiss your ring, now you’re taking your ball and going home, leaving your mess behind. I wouldn’t want to face it either.
And yes, Anders, I can understand why you wouldn’t want to show this thread to your friends as you say. Me, I’m happy to show it to my friends, but then I haven’t been the idiot trying to tell a very successful man that he’s doing it all wrong and claiming that the idiot knows just how to fix it. I wouldn’t want my friends to see that either.
Let me be clear. You are welcome to stay. You are even welcome to continue your run of 100% science-free posts, since all you seem to want to do is tell people that they are not doing something right. However, I’d suggest that you actually touch lightly on some scientific subject, say every dozen posts or so, to fool the rubes into thinking you’re not just here to be that dickhead kid in class who is always going on about “Ooooh, Willis said something mean, you shouldn’t do that, that’s not nice, I’m gonna tell the teacher”.
Conrad’s a grown man, Anders. He doesn’t need you punching for him, particularly since you punch so far below your weight. Here’s a novel idea—how’s about you stay out of other people’s fights and disagreements? There’s already enough puritanical junior G-men on the planet out trying to stop people from having fun, how about you be a force for science instead of just wingeing about how I’m doing things all wrong?
All the best,
w.
Good grief Willis, I’ll say it again:
Willis, when you say “The blog statistics show I am far and away the most popular guest author on the site. ” the phrase ‘car crash TV’ comes to mind.
[SNIP: Louise, put all your zingers in one post and be done with it. You’re trying to provoke a fight. -REP]
Louise says:
February 3, 2012 at 10:56 am
Good grief Louise, thanks for saying it again. It sounded stupid the first time you said it, and sounds even dumber the second. You do realize you are making a fool of yourself, I hope.
I am popular for two reasons. First, because I write clear, interesting, detailed posts about climate science that explore hidden corners of the climate and our responses to them.
Second, because when jerks like you come to give me grief over imaginary slights, I don’t pat their tummies and blow in their ears and pretend that they are here to help.
The crazy thing is, I have people like you who read my posts, complain, read some more, complain again, read some more, tell me I’m doing it all wrong, and then read some more.
So if anyone is addicted to what you are calling “car-crash TV”, Louise, that would be you. If you didn’t like it, you wouldn’t be still reading … and yet, here you are. You remind me of the little old lady who calls the hotel Security up to her room on the 53rd floor of the hotel.
“There are two people making love right out in the open where everyone can see them,” she says. “I want you to stop it, it’s worse than car-crash TV, it’s disgusting.”
The security man goes to the window and looks out. He looks all around, and says “I can’t see anyone at all, lady.”
The old lady draws herself up to her full height and says, “Well, young man, if you don’t believe me, just jump up and stand on the dresser, and take a look with these binoculars!”
w.
I tried to explain exactly why Willis’ posts are so popular and REP decided to SNIP me.
Willis – your posts aren’t popular “because I write clear, interesting, detailed posts about climate science that explore hidden corners of the climate and our responses to them.” but because it is human nature to stop and watch a street fight (or car crash TV).
Most of your fans (of which I am one as I contribute to the numbers who read your posts more than any other guest poster) are tuning in for the vitriol, not the science. Everyone enjoys a bun fight from the safety of their own laptop.
Louise says:
February 3, 2012 at 11:34 am
Louise, you are welcome to believe that if it helps you to sleep. But heck, if they want to come for the fireworks and stay for the science that’s OK too.
My scientific posts have been cited and reposted all over the web. I have been quoted inter alia in the New York Times, the UK Telegraph, and the Australian Herald Sun. I have written several pieces that have appeared in the peer-reviewed scientific journals. I was the first person to send a Freedom of Information request to the CRU, unknowingly starting a train of events that would lead to Climategate. I’m mentioned in a number of those Climategate emails.
So let’s take a moment, Louise, and look at the balance. I’m a well-known and active participant in both the scientific and political debates about the climate, quoted around the globe and read by thousands of climate scientists and other interested folks. People on both sides of the climate aisle read my work, just as I read theirs.
You, on the other hand, are a random person of unknown sex posting anonymously on the internet. Unlike me, you are an observer rather than a participant in the climate science wars.
Yet you want to claim that my science, both in and out of the peer-reviewed literature, is not the main reason for my being among the most-read authors regarding climate science today? You’re not even in the game, “Louise”, or whatever your name is. How would you even know?
As far as I recall, all you have ever done is show up on my threads and make snide comments about my writings. I can’t recall you ever once touching on the science, although I’m glad to be reminded if you have.
Given that, your judgement about why I am well-known likely traces to one thing and one thing only …
Your obvious fascination with car-crash TV. Hurry along now, you might be missing some …
w.
w. says:
“Sadly, you are quite correct about iterative models.”
Wow – something that I can reply constructively to. Your statements about models being nothing more that codified preconception and bias may be true, but, in my opinion, only at the margin.
In the early 1970s while working for a 3-letter security agency, I was asked to justify a major computerized-learning (as it was called at the time) project, using the Army Signal School’s very large FORTRAN iterative model, with lots of inputs and computations, no comments, and obfuscated code (partially the by-product of FORTRAN itself). After days of reverse engineering I found the essential algorithm to be the statement EFFTV=EFFTV*1.08 !!! That is, that the effectiveness of computerized-learning was 8% greater than the alternative! Tracing back EFFTV showed that the only input used in its calculation was the computed effectiveness of the measured non-computerized class.
So that particular “model” went beyond codifying preconception and bias to actually serving only to hide the explicit preconception. Bad as that was, the Signal School at least provided test data (real class measurements) and source code, which none of the AGW crooks are willing to do.
I don’t think that understanding the power and limitations of models is peripheral to any discussion of AGW, climate change, or whatever alarmists call it today. Models are the alarmist’s only argument. After we get a look at their real data and code, I believe they will have no basis for avoiding prosecution for fraud.
Conrad
conrad clark says:
February 3, 2012 at 1:08 pm
Thanks, Conrad. Given the depth and breadth of your computer experience, I suspect we actually agree on many things.
For example, you seem to think that I said that models are “nothing more that codified preconception and bias”. This misrepresents my position. I actually said that it also represented the programmer’s beliefs, and that insofar as those beliefs were true (and the coding good) the model would also be true. However, I also noted too often it is what you describe accurately as “preconception and bias”.
Finally, you say that in your opinion the idea that models are nothing more that codified preconception and bias holds “at the margin”. Unfortunately, what you call “the margin” we call “climate models”, which claim to be “physics based” and “first-principle” models, but are nothing of the sort.
My best to you,
w.
w.
Your comment to Anders missed his attempt to engage me (which I also missed in the haze of heated discussion (heated discussion is always better with beer and a dart board, not in discussion threads)).
Anders Valland says:
February 2, 2012 at 2:49 am:
“… I would like to know what you think when you say that the Machine Learning class has any relevance to this – do you feel neural networks and self-learning models should be used for modelling climate? Do you think that is feasible, given the complexity of the issue?”
I really don’t have any idea what could be used to model climate with predictions and falsifiability. However, the newer statistical, self-learning and even neural networks could be very useful (using back modeling) in determining which observations seem to be useful over climate time scales, which are echos, and which don’t even have back predictive capability. That’s different from:
1. Using the model to adjust the data (CRU et. al.).
2. Using the models to predict the immediate future as a basis for expensive policy (CRU and various statist criminals).
3. Adjusting the model as more data comes in. Not always evil, but all too often results in overfitting (in the technical model sense) and introduction of experimenter bias.
The data rules. It should be relatively easy to conduct data experiments as above, with little experimenter bias (see http://www.sciencenews.org/view/generic/id/337088, in which the results startled the drug researchers, and even me (note there may in fact be truth or untruth in their model, but it really doesn’t seem to be correctly described as a network analysis)). When I say “relatively easy”, that means we have the computation and storage technology, that doesn’t mean easy or inexpensive. The climate data is a mess, even where it hasn’t been intentionally corrupted.
Conrad
Conrad,
thank you for your reply. My own programming skills are very basic, but we do use models such as you describe in our research. I work with people who makes it possible for us to do analysis using such techniques, I don’t need to know how to make them in order to use them. But I need to know how to set up the algorithms and how to sort out all of the traps that follows what Willis describes. I agree with you that climate data is a mess, and that there is quite a bit of work to do before you can get down to working with whatever part of it that is good.
conrad clark says:
February 3, 2012 at 4:54 pm
That’s an interesting idea, and one that I touched briefly on in a prior post. There is a recently invented general purpose algorithm which finds relationships between data regardless of the form of the relationship (sinusoidal, linear, exponential, etc. I had the same thought you describe, of seeing which observations seem to be related and which don’t.
T-I-M-E is my usual problem, too many roads, not enough time to follow them all.
w.
Louise says (February 3, 2012 at 11:34 am): “Willis – your posts aren’t popular ‘because I write clear, interesting, detailed posts about climate science that explore hidden corners of the climate and our responses to them.’ but because it is human nature to stop and watch a street fight (or car crash TV).”
Personally, Willis had me at “Steel Greenhouse”:
http://wattsupwiththat.com/2009/11/17/the-steel-greenhouse/
I read all his articles and most of the comment threads for the information therein, and sigh in amused frustration at the distracting bickering.
Let’s do an experiment. From here on, let’s all stick to the science in WE’s comment threads, ignore any perceived impoliteness in his replies, curb our inner snark, and see if his popularity at WUWT plummets. If it works, we can extend the passion-free zone to the rest of WUWT and drive its popularity as low as a snake’s belly. Or even lower, down to Realclimate’s level.
Oops.
OK, we’ll start after this comment!
Anders says:
“My own programming skills are very basic, but we do use models such as you describe in our research.”
Conrad says – If that is the case, you can relatively easily improve your skills and critical abilities re experiment design and a deeper understanding of (statistically and data-wise) what is right, wrong or stupid. I will again promote the Stanford Machine Learning (ML) class. It’s free, it’s HARD (the advanced track, which also requires that you rapidly learn the one of the goofy but powerful languages: octave(free) or MatLab(not free)). At the end you will have a good grasp of perhaps 15-30% (a very fuzzy number) of what current models do in the real world. As you describe your job, however, ML may be much more than 30% relevant.
Conrad
Don’t feel specially favoured, W.; she/it/he does the same thing over at Climate, Etc. Sneering ad-hom denigrations of Judith. Not even worth the brief preliminary skim needed to verify that it’s just more of the same-old.
It’s a calling, I guess.
@contrad;
“3. Adjusting the model as more data comes in.”
As I once observed to a pro-tweakist, as soon as you adjust a model (or hypothesis), especially by tuning to new data, you have done a Reset, and must start your validation testing all over, back to square one. So for GCMs, that means a new 10 (or 15, or 17, or 30, etc.) year period starts — during which your predictions are “frozen” and stand as-is, unfiddled, live or die!
Not a popular POV in Climastrology.
typo:
@contrad@conradWillis;
Don’t know if it’s related to the Rashef study you linked, but have you heard of Eureqa? Finds standard and novel “rules” that are implicit in any raw data. E.g., derived the laws of motion from observations of a jointed dual pendulum. Free download from Cornell.
typo:
RashefReshefclimatereason says: February 2, 2012 at 12:06 am
Unbelievable, Tony! Perhaps they are trying to “hide” the fact that this uncited “unpublished research” is a blogpost (which their new, improved “rules” have decreed is not acceptable as a reference source!) And speaking of the IPCC and “peer-reviewed” / non-peer-reviewed …
tokyoboy says: February 2, 2012 at 4:44 pm
As one who participated in this Citizen Audit project, I can confirm that the only references we considered as “peer-reviewed” were those that bore all the hallmarks of a Journal; references to any of the IPCC reports did not meet the criteria for “peer-reviewed”.
[Pls See: http://www.noconsensus.org/ipcc-audit/quality-assurance.php for details]