Guest essay by John Davies *
Being able to evaluate the evidence behind any scientific claim is important. Being able to recognize bad science reporting or faults in scientific studies is equally important. These following points may help you separate the real science from the pseudo science.
Speculative Language
Speculations from any research are just that – speculations.
Look out for ambiguous, obfuscatary or weasel words & phrases such as …
– can, clearly, could, conjectured, considered, expected, may, might, perhaps, possibly, projected, robust, unprecedented
– “Experts suggest…” “It has been said that …” “Research has shown…” “Science indicates …”
“It can be argued…” “Scientists believe….” “A high level of certainty” “Models predict….” etc,
…as any real evidence, for the conclusions being claimed is doubtful.
Sensational Language & Headlines
The media will ‘Never let facts spoil a good story’
Words like – Unprecedented, unparalleled, unmatched, extraordinary, groundbreaking, phenomenal, apocalyptic, bizarre, cataclysmic, catastrophic, devastating, extreme.
Phrases like – ‘Since records began’, ‘The majority of scientists concur’ ‘Never on such a scale’: are used to convey a message, not necessarily the truth or facts, they rely on the reader having a short memory or being too lazy to check. ‘Unprecedented’; now often means…not within the last 9 months !!
Headlines of articles are regularly designed (with no regard to accuracy) to entice readers into reading the article.
At best they oversimplify the findings, at worst they sensationalise and misrepresent them.
E.g. – ‘Margarine makes mayhem in Maine !!’
Correlation & causation
Be wary of confusion by assuming that correlation equals causation.
See some entertaining examples – http://tinyurl.com/oqhw24g – 6 min.
Correlation between 2 variables doesn’t automatically mean one causes the other; there could be many other causes.
E.g. – Divorce rate in Maine has a 99% correlation with the consumption of margarine. http://tinyurl.com/qb4n9mf
(So is eating margarine, the cause or result of divorce ?…or are there other reasons ???)
Misinterpreted results
News articles often distort or misinterpret the findings for the sake of a good story, intentionally or otherwise.
If possible try to read the original research paper; rather than relying on ‘quotes’ from a news article (by a pressurised hack journalist on a deadline, who is trying to build a story to fit the catchy headline), roughly based on a poor press release.
‘Cherry-picked’ results
This involves selecting bits of data which support the conclusion, whilst ignoring those that do not.
Trend lines plucked from the middle of a graph may not show the real picture, you need to see the full graph to compare. If a paper draws conclusions from just a selection of its results, it may be cherry-picking.
Data Presentation
Check the start & finish points in every data set to pick up any cherry-picking. Look at the X Y scales on graphs, is one truncated to show a distorted result ? A neat often used trick is to just show the anomaly, so a small amount looks enormous.
Beware of graphs that suddenly go exponential, Are the results out of normal range ?? Look for the error bars, If there are no error bars, ask why ??
Graphs & statistics can help summarize data; but are also often used to lead people to make incorrect conclusions. This video shows a few of the many ways people can be misled with statistics and graphs.
-13 mins –
Journals and citations
Research published to major journals should have undergone a review process, but can still be flawed, so should be evaluated with this in mind. Similarly large numbers of citations do not necessarily indicate that research is good quality or highly regarded.
Un-replicable results
Results should always be replicated by independent research and tested over a wide range of conditions. Extraordinary claims require extraordinary evidence. You always need more than one independent study.
If it can’t be reproduced or the full data & methodology is not made available, then it’s probably another example of junk pseudo science.
Peer-review
The peer-review process** is supposed to be one of the cornerstones of quality, integrity and reproducibility in science & research. Peer Review does not mean the conclusion is correct.
It only means that it was reviewed by similar people for obvious errors.
Judging by the number of peer-reviewed papers that have had to be withdrawn in the last few years, the system clearly isn’t working any more: http://tinyurl.com/lahsgrl http://tinyurl.com/pwbsvzx
A scientist / journalist shares his story of 2 sting operations on the scientific publishing process with frightening results. http://www.sciencemag.org/content/342/6154/60.full & http://tinyurl.com/pweth63
“Of the 255 papers that underwent the entire editing process to acceptance or rejection, about 60% of the final decisions occurred with no sign of peer review. Of the 106 journals that discernibly performed any review, 70% ultimately accepted the paper. Most reviews focused exclusively on the paper’s layout, formatting, and language. Only 36 of the 304 submissions generated review comments recognizing any of the paper’s scientific problems and 16 of those papers were still accepted by the editors despite the damning reviews.”
It can be argued that the peer-review process has actually worked against reproducibility in research.
Desk top ‘peer-review’ has replaced reproducibility as the standard of good research.
Having your research pass a peer review is what gives researchers the moral license to say things like this.-
“Even if WMO [the World Meteorological Organisation] agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.” – Prof Phil Jones UEA 2005.
This cuts to the heart of the matter. Science must be falsifiable: otherwise it’s not science. Those who seek to find something wrong with your data are the first people who should have access to it, not the last.
Challenging, refining and improving other people’s work is the means by which science proceeds.
It’s not science until it has been reproduced several times over.
No matter HOW good the figures look, or HOW smart everyone thinks you are, or HOW pretty your graphs are – if you can’t reproduce the results on demand, it isn’t scientific – it’s just hinting in that direction. If a model is unable to predict direct observations, then the parameters, variables, or basic theoretical concept must be wrong. You should change the model ….NOT the observed data. The motto of the Royal Society of Great Britain is: nullius in verba – take nobody’s word for it. Never take any thing at face value.
Science is based on provable facts not blind belief;
Question everything.
But remember, the worlds most threatening words are – How, What, Who, Why.
John Davies is a retired engineer with interests in engineering, physics, history, power supply & transmission, steam engines & over the last few years the climate.
*Inspired by an original idea of Andy Brunning at Compound Interest http://www.compoundchem.com/wp-content/uploads/2014/04/Spotting-Bad-Science.png
** Peer-review alternatives http://tinyurl.com/pbdykgj & more importantly http://tinyurl.com/6souaom
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
I’ve just assumed that if it is called Climate Science then it is bad Climate Science.
I’m with you on that.
I often call it climate Siants, ’cause it sounds like science ; )
How about climate séance?
If the “science” doesn’t mention air density, it is garbage.
You really don’t have to look for more.
The other day I spoke to an old student of mine from way back in the early 90’s when the enginnering college I was teaching at had say 10 final year students. My former student went on tho do a Masters and got a job there running a research facility and is still there. After a couple of decades he udpgraded to a PhD. The solege now has 450 to 100 graduates per year now but get this, they have about 50 PhD studcents. PhD’s have been commoditosed. Its not good, its not healthy. All of these people will be considered as ‘experts’ by the media, politicians and bureaucrats yet they will have very narrowbanded educational experience and even more limited life experience. It is abit like replacing natural forest with a plantation of GM trees. Just what is the plantation optimised for? Intellectual wood chip? Media fodder to produce free content?
Fortunately, it appears that some of the dubious terms now have a quantitative definition. Terms like ever, on record and in history can now be interpreted as meaning “since 1850”. Kinda like time immemorial means from 6 July 1189. 🙂
I eat margarine, but it hasn’t worked yet.
Ah, summertime and our engineering interns are in the manufacturing plant; and one of them has fallen into my hands. I have him working in property and parts management. Count and organize everything – inventory is not perfectly deterministic I tell him; we are looking at a statistical population and I want to know the errors…Now issue the parts and check for fit up…What!? We have a fit problem?! Check everything. Doubt everything. Trust nothing, and trust no one; but distrust in a kindly, respectful way; but let it be known that error is everywhere, and this is indeed a general rule.
I asked him where he is in school and he’s between second and third year electrical engineering. I mentioned to him that at that point in his education, he is likely being jammed with information and I compared his education experience to my admonishments to be skeptical and look for error. I advised him to pay particular attention to proofs and if they are glossed over in class to take time to learn them anyway. I told him, “if you practice doing proofs, you get better at them.”
And one does get better at proving things rather than simply passively accepting them. It is a mindset and dedicated engineers are all about the proof. They have to be; because when the phone rings ask not for whom the bell tolls for it tolls for thee; and you’d better know what is going on top to bottom – you will need to prove it.
Gregole
You are helping him to be educated instead of merely schooled.
Congrats.
Another science paper cliche, signaling it’s time to stop reading:
‘A growing body of evidence . . . .’
Another example from a current post:
‘UNSW PhD candidate Laurence Delina’
A graduate student, then. Use of the title PhD is used to inflate the credibility of the subject. Another flag indicating you should stop reading.
“You’re wrong because it is obvious you have never taken a science class. Scientific theory trumps all laws and scientific facts which are used to develop the theories. It is true science is the best explanation but not necessarily the truth. The truth will never be known. AGW is scientific theory with 100% consensus among all current climate science researchers who publish their findings.”
How do you combat such statements? (Taken from the comment section on a newsarticle.)
Reality Check,
You don’t HAVE to combat such statements. It’s often useless to try.
But if you DO try, use logic and reason. Question their statements. Question their questions. Make them back UP what they are saying. If they cannot, or will not, it exposes their arguments as weak and flawed.
“You’re wrong because it is obvious you have never taken a science class”. =illogical. Someone who has never taken a science class can be completely right about something scientific. And reading what someone writes on the internet is not enough to make anything about that person’s education “obvious”.
“Scientific theory trumps all laws and scientific facts which are used to develop the theories.” =that doesn’t even make sense in the first place. Scientific theories rise and fall all the time as knowledge is collected. An explanation for something can never “trump” the laws and facts upon which it is based.
“A scientific theory is a well-substantiated explanation of some aspect of the natural world that is acquired through the scientific method and repeatedly tested and confirmed through observation and experimentation.” wiki
“A scientific law is a statement based on repeated experimental observations that describes some aspects of the universe. A scientific law always applies under the same conditions, and implies that there is a causal relationship involving its elements.”wiki
“In the most basic sense, a scientific fact is an objective and verifiable observation, in contrast with a hypothesis or theory, which is intended to explain or interpret facts.” wiki
“The truth will never be known”.= huh? Talk about ironic! That statement contradicts itself. If the “truth will never be known”, then the person speaking cannot KNOW that “the truth will never be known”. Right?
“AGW is scientific theory with 100% consensus among all current climate science researchers who publish their findings.”= Ask “Please show me irrefutable evidence that “all current climate science researchers who publish their findings agree with AGW 100%.” (or whatever number they give you). They cannot show you because it’s never, ever, been factually and concretely established. Period.
If they cannot back up what they claim with facts, and empirical evidence, then their claims amount to no more than opinion and should not carry any more weight than one would normally give an opinion. You cannot fix stupid. And fighting with it is a waste of time, unless you have the time to waste and you enjoy it.
Aphan: The person actually insulted a couple more times, calling the other commenter a liar about knowing anything about science. Then the commenter got bored and left. I like your ideas and will remember them if this comes up again. Some of these people will argue for days, insulting but never providing proof. It’s quite disheartening at times—there’s just so much stupid out there.
choose your battles – comments like your example disprove themselves by their sheer incompetence – so why bother – if you are commenting elsewhere in the thread – you may be able to address particularly annoying points of this commenter there
You missed the most important “tell”:
There is a prediction of the future climate.
Since no one knows what causes climate change with any accuracy, it is impossible to predict the future climate, except with a lucky wild guess.
Since the climate is always cooling or warming, I suppose you could pick one and be “right” 50% of the time.
Even if the causes of climate change were known precisely, the future climate still might not be predictable (unless there are regular repeating cycles, or an easy to measure variable causes climate change with a consistent lag)
So when you hear, or read, a prediction of the future climate, or future anything else, just plug your ears with your fingers and hum loudly until the person who wants attention goes away.
Source: 60+ years of learning common sense,
and forgetting what I learned in 18 years of school.
‘Since no one knows what causes climate change with any accuracy, it is impossible to predict the future climate, except with a lucky wild guess.’
This is critically important. Should someone produce a model today that apparently works, be assured, it doesn’t. We don’t know enough to produce a useful model.
You assume we do not know enough. Any disciples of Hansen for instance will never produce the right answer because they assume water will change its physical processes and instead of moderating temperatures water suddenly enhances temperature changes. What fools!
No, those of us who have studied it without those errors realize these simple facts:
Natural climate change trumps what humans do for now by a very healthy margin
Water moderates forcings through the hydrological cycle
And knowing that we can make some predictions that are testable.
We can probably expect a temperature drop of about two tenths of a degree over the coming 20 years because of the sixty year ocean cycle.
Co2 will have a small warming impact but it will continue to be undectable.
The longer term trend will predominate in that we will slowly cool off until the next glacial episode. Will we have another warm period like the 20th century? I would say yes and it will probably wait 200 years to come about. Until than we will probably gradually cool off.
And my last testable prediction:
Climate scientists will still insist that the earth has a fever while glaciers crash through their homes in Chicago and new york
‘We can probably expect a temperature drop of about two tenths of a degree over the coming 20 years’
Your making a prediction is not proof you know. Even if it comes true, it’s not proof. That is my point.
Anyway, a change in Global Mean Temperature is NOT climate change. I live in a Köppen Cfa climate. A change in GMT of .2 degrees will have no effect on my climate whatsoever, though it might change some climates at the boundaries of climate regions.
My personal favs from MSM: it’s worse than we thought; time is quickly running out; we may be approaching the point of no return; world must act now…
Gem after gem repeated tirelessly to the masses mainly preoccupied with Black Friday discounts and the like…
Least we forget “tipping point”.
How to spot bad science? Look for the description “climate science” and you know it’s a fraud. That my friends is about all you need to know.
If climate scientists did not want to be known as peddlers of fraud they would start being truthful and since we know that is impossible for blinkered fools like Hansen and the rest we can rest easy knowing that in the future climate science will be synonymous with fraud and cons.
Here is a few:
-It’s worse than we thought
-its unprecedented
-ever accelerating
-ever decelerating
-things staying the same is not an option
-the poor will suffer the worst
-it’s the rich countries fault
-it’s already happening (then why is it secretly and mysteriously hidden?)
-things were much better in the golden past
-things will be much better if only people would make extreme sacrifices for the greater good
-there is a real danger….
-the danger level has increased
-it’s never happened before, but its likely to happen more often
–
etc
here is a few more:
-we need to have stopped doing it yesterday
-we don’t know how bad its going to get, and we don’t want to know
-something needs to reverse that hasn’t started yet
-its not a matter of if, but when
-its overdue
-its not worth the risk
-we need more money….
The absence of this is a better indicator of bad science or bad scientific reporting than its presence.
“The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt.” – Russell
Those are all good methods for analyzing the actual science done, but when it comes to science news REPORTING online, there’s a one simple fact that usually sets off my BS meter:
Did they provide a link to the original scientific report?
If not, it’s either because they didn’t read it themselves or it doesn’t actually say what they are reporting it does. In either case, they aren’t being journalists but just spreading rumors. And as is often the case, once you go back to the source you often find the rumors are false. Once I find the original source, then I can use the methods mentioned above to see if the science rings true even if I don’t understand every aspect of the published paper.