It depends on how you ask: Three wording decisions can significantly alter estimates
Annenberg Public Policy Center of the University of Pennsylvania
![Question wording & response option effects across experimental conditions. Hollow circles correspond to mean levels of agreement with anthropogenic climate change, observed across experimental conditions. 95% confidence intervals extend out from each one. These analyses use weighted survey data and do not include Independents who 'lean' toward one party over the other. See the study and Supplementary Materials for additional information. Conditions: (1) Discrete, Hard DK [Don't Know], No Explainer; (2) Discrete, Soft DK, No Explainer; (3) Discrete, Hard DK, Explainer; (4) Discrete, Soft DK, Explainer; (5) Likert, Hard DK, No Explainer; (6) Likert, Soft DK, No Explainer; (7) Likert, Hard DK, Explainer; (8) Likert, Soft DK, Explainer Credit Motta et al. as published in Climatic Change.](https://i0.wp.com/wattsupwiththat.com/wp-content/uploads/2019/05/200575_web.jpg?resize=543%2C349&quality=83&ssl=1 "200575_web")
What percentage of Americans believe in human-caused climate change?
The answer depends on what is asked – and how. In a new study, researchers at the Annenberg Public Policy Center (APPC) of the University of Pennsylvania found that “seemingly trivial decisions made when constructing questions can, in some cases, significantly alter the proportion of the American public who appear to believe in human-caused climate change.”
Surveying more than 7,000 people, the researchers found that the proportion of Americans who believe that climate change is human-caused ranged from 50 percent to 71 percent, depending on the question format. And the number of self-identified Republicans who say they accept climate change as human-caused varied even more dramatically, from 29 percent to 61 percent.
“People’s beliefs about climate change play an important role in how they think about solutions to it,” said the lead author, Matthew Motta, one of four APPC postdoctoral fellows who conducted the study. “If we can’t accurately measure those beliefs, we may be under- or overestimating their support for different solutions. If we want to understand why the public supports or opposes different policy solutions to climate change, we need to understand what their views are on the science.”
The study, published in Climatic Change, is based on an online survey of 7,019 U.S. adults conducted from September 11-18, 2018, who were presented with questions in one of eight formats.
Three ways of asking
The study tested three variations in question format in different combinations, in which respondents were:
- Given the option to respond with a choice of “don’t know” or allowed to just skip the question (a “hard” don’t know vs. a “soft” don’t know);
- Provided with explanatory text saying that climate change is caused by greenhouse gas emissions, or given no additional text apart from the question;
- Presented with discrete, multiple-choice responses and asked to pick the one that comes closest to their views – or shown a statement and asked how strongly they agreed or disagreed with it, using a seven-point agree-disagree scale.
The two extremes
Two formats produced the most extreme results:
- The “Pew Style” approach, which uses a clear “don’t know” option, no explanatory text, and a discrete choice among statements as to which best represents your views, produced the lowest acceptance of human-caused climate change: 50 percent of U.S. adults and just 29 percent of Republicans.
- The van Boven et al. approach cited by Leaf van Boven and David Sherman in a 2018 New York Times op-ed, “Actually, Republicans Do Believe in Climate Change.” This approach uses an agree-disagree scale and explanatory text and does not offer a “don’t know” option. In the present study, this format combination found that 71 percent of U.S. adults believe in human-caused climate change and 61 percent of Republicans – the highest level of acceptance among the eight question formats studied.
The researchers said that the differences show how question construction can produce widely varying reports about what the public purportedly thinks. But they cautioned that because the respondents in this study were not a representative sample of the U.S. adult population, the raw estimates can’t be read as definitively reflecting the acceptance of anthropogenic climate change (ACC).
How format choices matter
While other differences in wording and question structure have been studied, the researchers said these three choices in format have not been examined closely. Questions that lack a hard “don’t know” response may nudge participants to pick a response that doesn’t reflect their lack of an opinion – and thereby inflate acceptance of human-caused climate change. Likewise, they said questions that use explanatory text may push respondents toward a greater acceptance.
However, they found that the most substantively and statistically significant increases in belief in climate change came from the use of an agree-disagree scale (so-called Likert-style response options) instead of distinct choices in response. In other fields, the researchers wrote, the agree-disagree format has been shown to introduce acquiescence bias, which occurs when respondents “agree” with a statement in order to “avoid thinking deeply about the matter at hand” or “avoid appearing disagreeable to the interviewer…”
“We find evidence that questions featuring Likert-style response options tend to produce higher levels of belief in ACC than those offering discrete choices,” the researchers said. In the case of self-identified Republicans, the researchers suggested that the agree-disagree scale and absence of an alternate series of positions to choose from may have presented them with “more difficulty identifying and selecting the party’s stance on the issue.”
The researchers said that they found no differences in the way that these question format changes affected Republicans and Democrats. “We hope that our research can help to broadly raise awareness of measurement issues in the study of climate change opinion and alert scholars to which specific design elements are most likely to impact opinion estimates,” the researchers said.
They added that additional research is needed to understand the psychological mechanisms underlying the effects seen here.
In addition to Motta, the study was written by Annenberg Public Policy Center postdoctoral fellows Daniel Chapman, Dominik Stecula, and Kathryn Haglin. “An experimental examination of measurement disparities in public climate change beliefs” is published in Climatic Change.
###
The Annenberg Public Policy Center was established in 1993 to educate the public and policy makers about the media’s role in advancing public understanding of political, health, and science issues at the local, state and federal levels. Follow APPC on Twitter and Facebook.
Obviously, the real question is ‘do most Americans believe human-caused climate change is destroying the planet’.
Whatever that number is, the reason is because they’ve been lied to.
Polling is another “soft science”, about as useful as that old joke about the three accountants.
The true measurement of the public concern for an issue is how much of their own money they will put toward it.
Maybe the Pollies should try crowd sourcing their stupid ideas before committing political suicide.
Not half as entertaining for the political junkies, but far less damaging to the national economy.
Canada is running the experiment right now.
Forced contribution to a “Cost of pollution”.
A carbon tax, but not a carbon tax when it comes to court?
Deceit being the bureaucratic way.
Now our falsely accused Vice Admiral crowdsources $400 000 dollars for his legal defence and will raise more.
Compare these two.
One “issue” the government has to extort the taxpayer with a new sliding fee(A tax) and a whole bunch of innumeric lies.
on the other issue,people gave of their own free will,to fight an criminal injustice.
So which one is the more accurate poll?
I wonder what the response if the question was asked in the form “Given the near zero accuracy of climate science beaten significantly by astrologer’s predictions do you believe their theory that climate change is man made is beyond question.
You guys do realize you are going to the FEMA camps first, right?
I have never in any scientific class or endeavor considered what I “believed” to matter. It was always what I could prove or disprove, or what my results were and how accurate they were. Not only are the current models not accurate, they don’t know the sensitivity of the climate to various factors. In other words, not only is the science not settled, it’s not even close to being fully developed.
And, IMO, any claims of belief at this point are more akin to theology than science.
Many years ago there was a brilliant BBC TV show, “Yes Prime Minister.
Bernard was bailed up by a bunch of reporters and blutted out a answer
which the reporters liked.
Later Sur Humphrey was explaining to Bernard about how to answered
difficult questions.
He quoted a typical question at that time. “Do you believe in conscription
of youths into the armed forces ?
He then said that there were two answers.
Good answer, make them better citizens by a bit of discipline,
Bad answer “Create a well trained bunch of thugs, which could cause
massive civil disruption.
I am not saying that all polls are rigged, the political pollsters if they want
to be asked again and again to conduct them, have to be as accurate as
possible, but if say a Green group wants to make politicians scared, easily
done, then they want a result which favours their point of view.
MJE VK5ELL
Mixing of statements: “human-caused climate change”
What to answer if you accept that climate change happens but not that it is caused by humans?
And climate change in itself is not defined at all. Some concider it to be weather and some think it is global warming and anything in between.
“Look out your window and you can see climate change”.
The polls used to be about AGW, but now I see that the polls are about ‘climate change’. At least we understood what the metric was for global warming. What is the metric for climate change? Is it rate of change of the climate? I’ve never seen a chart with ‘rate of climate change’. What exactly is the unit of measure? If anyone thinks they know what is meant by climate change, I don’t have much confidence that they’ve thought it through. Can we please go back to debating and polling global warming?
Three ways of asking:
They forgot to include:
1. (Money Where Mouth is): Despite colder winters featuring superabundant snowfalls which have left reservoirs brimming across the West, the (…cough) “threat” posed by global warming is certainly growing and I would personally like to pay utility companies more money to help fix it to the tune of … a) $500 b) $5,000 c) $50,000 d) $500,000 e) all the above
2. (Energy): To help fight the war against the EVIL, DIRTY UTILITY companies my family would agree to suffer periodic mid-summer and mid-winter brown and black outs for periods of up to: a) two hours b) two days c) two weeks d) indefinitely e) just until we have saved the planet
3. (Transportation): I have already a) garaged my car and started taking the bus b) sold my car and begun legging it c) publicly demolished my car with a sledge hammer to demonstrate my “green elan” d) willed my car to the homeless and quit my job to teach the Man that I won’t be made a pawn of
4. (Science): One of the following is NOT A CAUSE OF global warming. Which one? a) drought b) floods c) toenail discoloration d) polar vortex e) None – they ALL cause global warming
5. The Annenberg Public Policy Foundation can be trusted to a) not run with sharp scissors b) present tendentious public policy surveys on global warming with a deliberate bias c) not touch your retirement 401 Ks and Roth Account numbers even if you told them, “Take it, damn it! It’s yours!” d) send your mother flowers today for Mother’s Day