Guest post by David Middleton
ADAM ROGERS
SCIENCE
07.30.17 07:00 AM
DARPA WANTS TO BUILD A BS DETECTOR FOR SCIENCE
ADAM RUSSELL, AN anthropologist and program manager at the Department of Defense’s mad-science division Darpa, laughs at the suggestion that he is trying to build a real, live, bullshit detector. But he doesn’t really seem to think it’s funny. The quite serious call for proposalsRussell just sent out on Darpa stationery asks people—anyone! Even you!—for ways to determine what findings from the social and behavioral sciences are actually, you know, true. Or in his construction: “credible.”
Even for Darpa, that’s a big ask. The DoD has plenty of good reasons to want to know what social science to believe. But plenty more is at stake here. Darpa’s asking for a system that can solve one of the most urgent philosophical problems of our time: How do you know what’s true when science, the news, and social media all struggle with errors, advertising, propaganda, and lies?
Take a scientific claim. Do some kind of operation on it. Determine whether the claim is right enough to act on. So … a bullshit detector?
“I wouldn’t characterize it that way, and I think it’s important not to,” Russell says. He doesn’t want to contribute to cynicism that lets people think if scientists admit uncertainty, that means they can’t be trusted. “I have a deep faith that there is real science. It’s not that we know nothing about the world.” Science is still the best way of knowing stuff. Darpa just wants to know what stuff science is really sure about, and how it knows it. And how it knows it knows it.
You can imagine why Darpa and the DoD might want to shore up the social sciences.
[…]
No, I cannot “imagine why DARPA and the DoD might want to shore up the social sciences.” I can’t even fathom why the Federal government would spend any taxpayer money on social sciences… So, I certainly can’t imagine why the Defense Advanced Research Projects Agency and the Department of Defense would be interested in shoring up the social sciences, much less, employing anthropologists.
This almost reads as if Douglas Adams wrote it:
“Science is still the best way of knowing stuff. DARPA just wants to know what stuff science is really sure about, and how it knows it. And how it knows it knows it.”
Why not just build a supercomputer to find the Answer to the Ultimate Question of Life, The Universe, and Everything… /SARC
Besides, bullschist detectors are readily available on the Internet…
The benchmark BS metering in science is the phrase, and any permutation of the phrase.: “The science is settled”. Even more than BS, it is 97% likely an outright lie.
“… one of the most urgent philosophical problems of our time: How do you know what’s true when science, the news, and social media all struggle with errors, advertising, propaganda, and lies?”
I can’t express how frustrated and angry I am every time I see a statement like this. Where did we get the idea that today’s supply of lies and distortion is somehow vastly larger and significantly more dangerous than at any other time in history? The ability to fact check has been virtually nil throughout most of humanity’s existence, and only in the past century or so has it been at all available to the general public, with electronic communications opening up the field exponentially. Sure, there is greater access to untruth, but damn it, there’s also more access to truth. It balances out.
So why this rather sudden and slightly hysterical call to action to “do something” about bullshit?
I suspect it’s because one particular form of bullshit is starting to be seen for what it is, and this call for a “return to science,” “evidence-based reasoning” and the like is a huge smoke screen meant to bring the public back on board to the official bullshit.
‘Where did we get the idea that today’s supply of lies and distortion is somehow vastly larger and significantly more dangerous than at any other time in history?’
It’s the reach, the scope, and the consolidation of the messaging that is unprecedented. The modern media is Goebbels’ wet-dream.
And while there is good information out there, you have to know what to look for, and (more importantly), be aware enough to do so.
As opposed to past eras in which authorities could spread whatever lie they wanted and the people had virtually no ability to check the truth. For the first time in history the average person now has the ability to dig into the facts and unearth information that would have been almost impossible to obtain 50 years ago — or even 30. And yes, a lot of people doing just that, which is why the Church of Global Warming is so widely questioned. Think of how much hold such a message — apparently being voiced by all the world’s scientists and backed by our very own governments — would have had in the 1940s.
We love to doomsay how terrible it is that all this false information is spread around, but totally ignore the fact (yes, fact) that good solid information is also widely spread and that a great many people want to access it and know how to do so.
We’re certainly far, far better off in this regard than we’ve been through most of our history when entire populations had to simi;y accept whatever their kings and priests were telling them.
‘No, I cannot “imagine why DARPA and the DoD might want to shore up the social sciences.’
The social sciences are a very good primer for enacting propaganda.
I think you’ve hit the nail on the head (as have a couple of other commenters): It’s all about the propaganda.
If you understand how the best BS is created, you can build a better BSer. On the flip-side, you can also tell when you’re being fed a disinformation campaign by your enemies/opponents. And if a machine can flag that, the machine can also back-trace to the “truth”, if there happens to be one.
Probably the best BS detector I have ever seen belonged to a second line manager I had many years ago. He might know absolutely nothing about the subject you were presenting, but within about five or 10 minutes he knew how much you knew about your subject. I doubt he is still around, but he would have been the perfect subject for developing an expert system BS detector.
Guess I should add that the best BS detector probably has more to do with reading the character and personality of the presenter than it has to do with understanding the subject of the presentation.
Over my career, I have been rather critical of government contracting and management. However, DARPA is one of the more successful government efforts. DARPA has had some significant successes over the years (Internet, stealth technology, other classified efforts, etc.). They have carried out their mission with a relatively small staff. DARPA typically awards contracts to Industry, Academics, and/or government labs.
DARPA’s mission is to look for game changing technologies / systems rather than the typical government program that has to show success. A lot of DARPA’s efforts don’t work out but the ones that do have major impact on the US industrial base. I have never worked at DARPA but I have some experience with DARPA programs.
While this DARPA Request For Proposal (RFP) has drawn snickers, there is no guarantee that any program will receive funding. DARPA will review proposal and determine if any are worth the investment for further study or proof of concept.
For those of you who haven’t bothered to read the original Wired article, there are some interesting and useful links included, such as the “reproducibility crisis” and “P-hacking.”
It seems to me that one of the most useful tests for hypotheses or claims made in the ‘silly sciences’ is, “Does it demonstrate any predictive or explanatory value?” As Carl Sagan was fond of saying, “Extraordinary claims require extraordinary evidence.” If it doesn’t pass the personal smell test acquired by years of living, then it deserves additional scrutiny and there should be demands for more evidence. Explanations that seem to conveniently support some politically correct position should be placed in the queue for prompt re-examination by a Red Team before it gets a chance to be cited. If it doesn’t contribute any obvious value to the use to which the ‘knowledge’ might be applied, then it should just be ignored.
For any who might be interested in additional reading on the “P-Hacking” issue:
https://www.vox.com/science-and-health/2017/7/31/16021654/p-values-statistical-significance-redefine-0005
If it can’t be demonstrated with a robust experiment, then it’s BS.
where “robust” = repeatable, by anyone.
The so-called Social Sciences can’t call themselves a Science without such validation.
Svensmark took his proposal of GCRs modulating cloud cover to experiment with his SKY 1 experiment.
More details about cloud formation specks and their rates of growth were determined with SKY 2.
Both experiments blew so-called “cloud experts” … umm…. hypotheses of cloud formation out of the sky.
One small matter they settled was that cloud droplets require a seed particle or speck before water vapour will condense around it to form a cloud droplet.
Other sources of cloud modulation and the details of cloud formation chemistry have been determined by CERN’s CLOUD experiment. Experimental exploration of the matter continues.
The AGW crowd haven’t done that. They’ve projected plenty of BS from non-validated models—another clue for the BS meter. Observation has shown up the poor performance of the models yet we are continually told “they aren’t bad.”
B. S. They’re terrible. They should have been tossed out over a decade ago, even two decades ago.
I thought the answer was 47 …
As I recall, the real meaning of life was found by a supercomputer to be…27? Or per KD, 47? Poor memory.
42
He was a computer geek. 42 in ASCII is * which could be anything or everything…
As I recall, the real meaning of life was found by a supercomputer to be…27?
At one time, the meaning of life was calculated:
It seems that Douglas Adams was right after all: the answer to Life, the Universe and everything, is 42.
Cambridge astronomers have found that 42 is the value of an essential scientific constant – one which determines the age of the universe.
http://www.independent.co.uk/news/yes-the-answer-to-the-universe-really-is-42-1351201.html
Google already has an algorithm that knows everything about everything! Artificial intelligence certainly beats the intelligence of governments.
(Douglas Adams’ “Hitchhiker….”
How are they going to zero it ?
Feynman Test.
That will be $109,099 please.
Done
Simplest BS detector in the world..
Spend your own money on it.
It being the “brilliant social science break through”.
It is amazing how few of our self appointed superiors will spend their own money to cure the “problems” they claim to see.
Yet they see no limit to how much of our money they must seize to “attempt” a cure.
I have seen it done using an ordinary polygraph and don’t knock the idea. Once it was seen in operation no one in the management ever agreed to any further trials.
It will be as epic a failure as lie detectors
What happened to repeating the work and reproducing the results?
I was reading this in Australia during a break in the movie Idiocy. They seemed to blend together well.
” If you have one bucket that contains 2 gallons and another bucket that contains 7 gallons, how many buckets do you have?”
“Science is still the best way of knowing stuff. DARPA just wants to know what stuff science is really sure about, and how it knows it. And how it knows it knows it.”
Seamless.
To DARPA, read my lips
Somehow I imagined you not to be quite so good-looking.
Better-looking?
Interesting parallel to the “positivist” movement in philosophy at the beginning of the 20th century (see “Vienna Circle”). Then, the shiny new toy that people imagined should yield a prescriptive algorithm for “verifying” proposed laws of empirical science was formal logic. Karl Popper developed his falsification principle as a response to this strangely naive idea. Today’s shiny new toys are computers and computer programs — and again, credulous people credit them with magical powers of divining the “truth.”
It appears that credulity/scepticism and intelligence are orthogonal — many highly intelligent and accomplished people are also very credulous, whereas people with otherwise unremarkable intellect can have very good built-in bullshit detectors.
It’s actually quite simple – the validity of a scientific assessment is inversely proportional to the amount of press it gets
I detect BS. This is satire, Shirley?
The OP is rubbish. This has nothing to do with BS detection and everything to do with assigning a weighting factor when using the results of Social and Behavioral Science experiments, studies, meta-analyses, and other products — to draw conclusions related to devising solutions to defense related concerns issues problems whatever.
Actually read the RFI linked in the OP and you can easily ascertain it is not a BS detector at all.
What the RFI asks for is proposals on how to automate or semi-automate the assignment of confidence levels to the conclusions of studies, reports and other product data created in the Social and Behavioral Sciences tradespace.
You just described one of these:
For this…
https://www.fbo.gov/index?s=opportunity&mode=form&id=3f2885886c97d05491ff3ac8f7968912&tab=core&_cview=0
Reliability, reproducibility, and robustness are on the left side of the gauge.
No, you perceive that’s what a described.
Low confidence does not = the study results were incorrect or flawed (bullshit)
High Confidence does not = the study results are correct.
The whole piece is written with a bias toward Social and Behavioral Sciences.
There is your bullshit
And to falsify your RRR argument:
Faulty instrumentation can reliably reproduce robust bullshit —
There are a host of other conditions that can result in reliably reproduced and robust — faulty findings. Poor statistical sampling methods for one.
I thing that has been proved out on this board with regard to hard sciences.
David,
It looks like a prototype already exists: http://tsedl.com/media/uploads/fraudoscope_general_web_eng.pdf
Now, if we could only get Mann et al. to sit down and answer a few questions. Or, maybe, give it a trial run at the Red/Blue team debate.
“what findings from the social and behavioral sciences are actually, you know, true.”
I’m tempted to answer with a four-letter word. The first letter is “n”.
Tell that to pavlov’s dogs
But do they work for BS that is not set in stone?