A frank admission about the state of modeling by Dr. Gavin Schmidt

This is something I never expected to see in print. Climate modeler Dr. Gavin Schmidt of NASA GISS comments on the failure of models to match real world observations.

Gavin_models_BSCapture

Source:

[ http://twitter.com/ClimateOfGavin/status/340605947883962368 ]

While the discussion was about social models, it is also germane to climate modeling since they too don’t match real world observations. Below is an example of climate models -vs- the real world; something’s clearly not right.

IPCC_AR5_draft_fig1-4_without

Graph source: IPCC AR5 draft

Is it maths or assumptions (or both) that cause the divergence?

UPDATE: In comments, I had a discussion with reader “jfk” which I think is worth sharing. He made some good points, and it helped hone my own thinking on the issue:

jfk says: Submitted on 2013/06/01 at 8:40 am

Well, I still think it’s a bit unfair to Gavin (and I am no fan of his). But hey, it’s Anthony’s site.

For a good review of the many failures of statistical modeling in social sciences (and one or two successes) see the book “Statistical Models: Theory and Practice” by David Freedman. Whether or not climate modeling has devolved to the point where it is social science rather than physics, well, I hope it’s not quite that bad…

REPLY: And I think it is more than a bit unfair to us, that if he believes what he tweets, he should re-examine his own assumptions about climate modeling. We have economies, taxes, livelihood, etc. hinging (or perhaps failing) on the success of these models to predict the climate in the future. The models aren’t working, and Dr. Schmidt knows this. Unfortunately his job is tied to the idea that they do in fact work. I feel no regrets at making this comparison front and center. – Anthony

UPDATE2: RussR in comments, provides this graph below showing Hansen’s modeled scenarios against real world observations. He writes:

Here’s an excel spreadsheet comparing observed temperatures vs. model projection from: Hansen (1988), IPCC FAR (1990), IPCC SAR (1995) and IPCC TAR (2001), in pretty charts.

It can be updated as more observations are added.

https://dl.dropboxusercontent.com/u/78507292/Climate%20Models.xlsx

giss-vs-observations

UPDATE3: Dr. Roger Pielke Sr. adds this in comments.

Climate models are engineering code with quite a few tunable parameters, and fitting functions in their parameterization of clouds, precipitation, land-atmospheric interfacial fluxes, long- and short-wave radiative flux divergences, etc. Only a part of these models are basic physics representations – the pressure gradient force, advection, the Coriolis effect.

The tunable parameters and fitting functions are developed by adjustment from real world data and a higher resolution models (which themselves are engineering code), but only for a quite small subset of real world conditions.

I discuss this issue in depth in my book

Pielke Sr, R.A., 2013: Mesoscale meteorological modeling. 3rd Edition, Academic Press, in press. http://www.amazon.com/Mesoscale-Meteorological-Modeling-International-Geophysics/dp/0123852374/ref=sr_1_2?ie=UTF8&qid=1370191013&sr=8-2&keywords=mesoscale+meteorological+modeling

The multi-decadal global climate model projections, when run in a hindcast mode for the last several decades are showing very substantial errors, as I summarize in the article

http://pielkeclimatesci.files.wordpress.com/2013/05/b-18preface.pdf

Advertisements

  Subscribe  
newest oldest most voted
Notify of
jim2

Dr. Schmidt is stating the obvious and frankly I am aghast that Taleb would make such a obviously idiotic statement. Math is exact, but as Gavin points out, essentially, garbage in –> garbage out.

Kurt in Switzerland

There is a crack in the armor. It will grow.
The metal is not of the self-mending type.
Kurt in Switzerland

jfk

Gavin is right in this case, in the social sciences and maybe also in climate science, most of the calculations are perfect; the assumptions behind the models just don’t remotely apply to the situation being studied. I have always thought Taleb only half knows what he’s talking about.
The Mann hockey stick which used principal components without centering is a notable exception; it is mathematically wrong.

Corey S.

Gavin:”Perfect maths plus bad assumptions still equals BS”
Really…you don’t say. Does that go for Climate Science as well? That is a rhetorical question….

Schmidt showing a shadow of a doubt in the Church of Climatology’s Delphic models? Out, out damn spot! It must be the harbinger of end of days for them …
Pointman

Jon

They have lost control over “reality” and can no longer change it?

jfk

Wait a minute, is Gavin talking about climate models or social science models? Let’s try not to put words in his mouth that he didn’t say…
REPLY: That is a point, and I dropped the word climate from the title to be fair. But the same issues apply to climate models, perhaps even more so than social models. You have to make assumptions, you have to set starting points. Maths generally can be debugged quickly, assumptions, not so much. – Anthony

Steven Devijver

Gavin Smith disappoints. — Gavin Smidt

Les Johnson

@climateofgavin @nntaleb Sounds a lot like “Method Wrong + Answer Correct = Bad Science” (Wegman)— Les Johnson (@LesJohnsonHrvat) June 1, 2013

Mike H

“failure of models to match real world result far more likely a result of erroneous assumptions”
Like, CO2 is the significant factor of global warming?!?!?!!?

This is really irritating how things keep tying together thus necessitating comment
– An excellent book that seems to me to explain the mind of Warmist “True Believers” it’s called On Being Certain : Even If You Are Wrong by Robert Burton
..he explains how our unconsciousnes sis programmed to give us a feeling of certainty (cos that blur over the hill might be a tiger) . So when we OFTEN get the same feeling of absolute certainty when we don’t have full evidence that we get when we
do have certainty.
..AND he then references Fooled by Randomness” by Nassim Taleb
..I guess that would refer to the human habit of seeing patterns in the wallpaper “seeing Elvis’s face etc. when it’s not actually there”
– Now I wonder what kind of people would let that process spillover into their work ?

Jeff Norman

Does anyone else find it ironic that Gavin Schmidt would not debate John Christy on camera because he does not consider himself an entertainer and yet he tweets all the time to entertain his followers? Maybe ironic is the wrong word… hypocritical?

Jeff Norman

jfk,
Statistically wrong but not mathematically wrong, after all all the numbers added up the way Mann wanted them, I presume.

Ryan

Perhaps adding the word “social” to the title? Wouldn’t want the flock to be confused. It appears a couple already are convinced that he was talking about some failure of climate models.

Kurt in Switzerland

I agree that the title of the post ought to be changed.
Gavin wasn’t [consciously] talking about climate models in their exchange.
But he HAD to realize that the statement could be equally applicable in another discipline!
Here’s hoping that a non-dogmatic individual takes the Hansen-vacated spot at GISS.
Kurt in Switzerland

Jimmy Haigh.

Steven Devijver says:
June 1, 2013 at 7:47 am
Absolutely! He’s dissing his own ‘discipline’.
I thought our Gav would be one of the last rats to leave. He’s actually jumping quite early. I suppose now that Grandad Hansen has gone … There’ll be a whole flock of lemmings jumping off now.

jfk

Jeff Norman – OK, you mean the arithmetic was correct, and maybe it was. I was using the word mathematically in a broader sense. It’s a mathematical error not to center the vectors before orthogonlization when calculating PCs. If a college freshman did this on a test he would probably not get any points.

Blarney

I think this post should be removed or very clearly put in context. The twitter conversation between Schmidt and Taleb is about social sciences models. Schmidt statement, that perfectly working math models can be totally wrong if the assumptions are wrong, is unquestionable and generally valid, therefore it can’t be framed as a statement about “the state” of modeling, as it doesn’t refer to any current state of any particular modeling but to a general rule. You may think that Schmidt may be hinting at climate models but he doesn’t say it explicitly, so it should be made very clear in the post title and in the reported twitter conversation that he’s not.
REPLY: The same issues applies to climate models. They don’t match real world data, as seen in the IPCC AR5 graph above. Mind you, this isn’t long term modeling failure out a century, this is failure to model the present. Is it bad maths, bad assumptions or both? Dr. Schmidt may not think the issues apply to climate models, or he may be making a mental slip, either way his comment is germane to the current state of climate models and it is a fair question to pursue. – Anthony

jfk

Blarney is right. I suggest that this post should just be removed, Schmidt is not saying what you claim he said. Aren’t there enough things to complain about with Gavin without making stuff up?
REPLY: The issue is fair to discuss, since climate models also don’t match real world observations (see AR5 graph), and nothing was “made up”. Though, I did update the post to make it clearer for people who have issues such as yours. – Anthony

Master_Of_Puppets

I suspect Gavin is campaigning (mostly within NASA) for the GISS Directorship and needs to at least appear Non-Hansenian. If NASA caves to him, Schmidt will be Hansen at Warp 10.

FAH

I saw a man pursuing the horizon; 

Round and round they sped. 

I was disturbed at this;
I accosted the man. 

“It is futile,” I said, 

“You can never — ” 


“You lie,” he cried, 

And ran on.
Stephen Crane

michael hart

Yes, that’s quite refreshing from Dr. Gavin.
The bigger problem is that intuitive criticisms such as this, even when coming from experienced modellers and scientists, amount to little more than “Based on my experience, I think it’s wrong”, which doesn’t carry much currency either way.
A greater attention to resolving lack of predictive skill should have been applied much earlier in the process, ideally before exaggerated claims were made, or scientific cul-de-sacs entered.

Peter Stroud

Of course Gavin Schmidt is stating the obvious. Has he, at last, realised that the assumption of positive feedback, leading to enhanced water vapour content might be false? Pity he has not listened to those pesky deniers!

Gerry Parker

No matter how biased toward a particular conclusion one begins, if the data diverges far enough you have to recognize there is a problem. This could really happen to either side of the climate debate going forward. This appears to be a critical inflection point, and the next 20 years will be fascinating to see which way it breaks.

jfk

Well, I still think it’s a bit unfair to Gavin (and I am no fan of his). But hey, it’s Anthony’s site.
For a good review of the many failures of statistical modeling in social sciences (and one or two successes) see the book “Statistical Models: Theory and Practice” by David Freedman. Whether or not climate modeling has devolved to the point where it is social science rather than physics, well, I hope it’s not quite that bad…
REPLY: And I think it is more than a bit unfair to us, that if he believes what he tweets, he should re-examine his own assumptions about climate modeling. We have economies, taxes, livelihood, etc. hinging (or perhaps failing) on the success of these models to predict the climate in the future. The models aren’t working, and Dr. Schmidt knows this. Unfortunately his job is tied to the idea that they do in fact work. I feel no regrets at making this comparison front and center. – Anthony

Laurie Bowen

It’s probably good to read a great rendition of the “History of Physical Science” . . . . as it demonstrates that for every answer comes many more questions. Wiki is brief, incomplete; but, a good start. Maybe others can recommend some good reading. I can not site the source of my first reading on “History of Physical Science” as it was long long ago and far far away. Gavin gives me pause to hope!
http://en.wikipedia.org/wiki/History_of_physics#The_physical_sciences

milodonharlani

Both assumptions & math are wrong in climate forecast modeling. Gavin won’t admit it, but deep down he must know the facts of the case.

Tilo Reber

Interesting that those two should be talking. Taleb is a financial guy with a big interest in probability statistics. He has written a couple of books about how probablility applies to investing. I believe that he is a hedge fund manager and his teqnique is to bet on very low probability events (black swan events). The events rarely pay off, but when they do, they pay off big in comparison to the small investment that was made to place the bet. I didn’t realize that he had an interest in climate change.
I think that Gavin has it right in this case. He made bad assumptions about CO2 forcing feedback. He made bad assumptions about Svensmark being wrong. He made bad assumptions about the importance of ENSO. He made bad assumptions about cloud behavior. He made bad assumptions when he decided that he understood enough climate factors to model the climate. And there is also the GIGO factor. When fitting your models to instrument data that has been maximized by adjustments, the chances of the models having a good future are slim.

klem

Interesting comment. I wonder what Nasim Taleb’s views are regarding climate models and AGW. Just curious really.

Dodgy Geezer

If Gavin is willing to admit that Climate Models have problems and do not match the real world, then I think that someone should mention this to the UK Met. Office.
Because they have just said that their stats may be questionable, but physics Climate Models back them up. Effectively they are running a circular argument – the physics must be right because the Climate Models built from it work, and the Climate Models must be right because the statistically modified temperatures match these (with assumed ocean warming!), and the temperatures must be right because they follow the physics….
What’s not to like…! 🙂

Gavin is exactly right. His climate models and the entire literature on impacts and policy based on them and all the futile CO2 emision cap nonsense – in short the whole CAGW house of cards are buillt on three illogical and irrational assumptions.
1, That CO2 drives temperature- when obviously it is the other way around- they clearly have the cart before the horse.
2. Their climate sensitivity for CO2 includes the effect of water vapour as a feed back to CO2 when again clearly it too is driven by temperature quite independently .
4.They assume that when they iterate their programs they can specify the initial parameters with such precision that the output will bear some relation to reality.
Naturally the ecoleft democratic politicians and MSM pundits lack the ordinary common sense to see that this quixotic emperor never had any clothes to begin with.

Wondering Aloud

This is exactly right Gavin. Why have you been denying it for 20 years?

I just finished reading the UN’s Post-2015 plans for all of us so I wouldn’t get too excited that anyone is seeing the light. They are using climate change literally as the basis for remaking societies, economies, and new mindsets and values.
With David Cameron as the UK’s representative on the “High-Level Panel of Eminent Persons” and John Podesta of the Center for American Progress representing the US, we may not be paying attention to the UN’s plans for all of us. But the UN certainly is paying attention to us.
I especially love the part about how all this restructuring is necessary. Otherwise “we will be on a path to at least a 4-degree Celsius warming over pre-industrial levels by this century’s end.”
It is BS but it is coming in through education and especially videogaming designed to get students to visualize false scenarios. It will be difficult later to dissuade the adults who have spent their school years modelling all these false, Limits to Growth, Club of Rome scenarios that reality is different.

John F. Hultquist

So, what were they tweeting about? Is there a ‘social model’ gone wrong that either of these folks has an interest in? Was this exchange over anything in particular? Just 2 guys commenting on QE2? To buy bonds or not buy bonds, that is the social model of the era!
The multi-color chart out to be linked to the original WUWT post where it was introduced. As I recall the chart needs clarification.

jfk

Anthony, re your response “And I think it is more than a bit unfair to us..”, I agree, and that was well said.

milodonharlani
DirkH

Kurt in Switzerland says:
June 1, 2013 at 8:36 am
“I agree that the title of the post ought to be changed.
Gavin wasn’t [consciously] talking about climate models in their exchange.
But he HAD to realize that the statement could be equally applicable in another discipline!”
Probably a Freudian slip.
As for assumptions in GCM’s I found something that I overlooked at the time and that is absolutely stunning. It touches on the Loschmidt / Miskolczy / Virial theorem complex but goes further:
The principal and terminal fault with climate science and climate models seems to be that they assume the atmosphere to be hydrostatic. It is known that it isn’t yet this simplification allows them to use the barometric equation.
In other words the models don’t contain vertical mass exchange and assume that the atmosphere as a whole does not expand when warmed and contract when cooled.
Quite amazing isn’t it?
https://tallbloke.wordpress.com/2012/01/04/the-loschmidt-gravito-thermal-effect-old-controversy-new-relevance/comment-page-1/#comment-13004
And this is probably related; the QBO is an oscillation in the stratosphere. Density waves. They don’t simulate it, citing too high computational effort.
WG1 of the IPCC says GCM’s cannot simulate the QBO.
http://www.ipcc.ch/ipccreports/tar/wg1/279.htm
Now maybe I’m wrong, and the GCM’s don’t assume hydrostasis. If a climate modeler reads this and can show that I’m wrong I would be interested in source links. Thanks.

Kurt in Switzerland

Dirk H:
Good comment — probably a Freudian Slip.
I find it hard to fathom that Climate Models fail to handle vertical mass exchange, however.
That would be a major fail.
We’ll just have to see if the climate continues to fail to cooperate in the coming years.
These guys must be praying for warming by now! (Bad form for atheists).
Kurt in Switzerland

Mike Haseler

Every engineer learns the saying “it works in theory”. This is the real difference between science and engineering. Scientists develop theories about what should happen. Engineers try to use those theories to predict in real world situations what actually does happen.
Perhaps Climate “Science” is finally learning after many decades what any scientist in an engineering company learns in the first few weeks of their job?

GIGO all over again.

William Abbott

klem,
I read most of Talib’s book, (Anti-fragile?) He takes AGW seriously.

michaelspj

Coupla things:
Note to Gavin: That erroneous assumption would be the sensitivity. Multiply it by 0.40 and GISS comes much closer to reality. How hard is that?

RockyRoad

But is anyone on that side, even Gavin, willing to concede that reality trumps models?
That’s the big question.

Scott Basinger

I work in engineering. I’ve been bombarded recently by people who throw theory at me and ask that it be applied in a system with a slightly different set of circumstances to the actual testing that was done. My response is that it works in theory, but we have to test it in our specific system before we implement it – especially if it has safety implications.
The scariest thing is the thought of sitting in court saying ‘we hadn’t considered that…’.

Jeff Norman

If you want Nassim Taleb’s opinion about anyone who claims they can predict anything about the future based on statistical modelling, read his book The Black Swan. His thesis is that you cannot predict a future calamity. The only way you can prepare for it is to have a surplus of wealth that allows you to recover from whatever happens. This is very similar to Lomborg’s thesis, which may not be too surprising given that they are both economists.

Jimbo

What did Gavin and Mann mean when they stated the following in a paper a while back?

“Modellers have an inbuilt bias towards forced climate change because the causes and effect are clear.”
Michael Mann, Gavin Schmidt, et. al. – 2004
http://www.meteo.psu.edu/~mann/shared/articles/Schmidtetal-QSR04.pdf

Then there’s this from WUWT in April.

Michael Mann says climate models cannot explain the Medieval Warming Period – I say they can’t even explain the present
http://wattsupwiththat.com/2013/04/11/michael-mann-says-climate-models-cannot-explain-the-medieval-warming-period-i-say-they-cant-even-explain-the-present/

Yet they are baffled by the temperature standstill. They have yet to learn what GIGO is.

Jimmy Haigh.

If Our Gav thinks he is being misrepresented here he is more than welcome to come along and defend himself. He’s a big boy now.
Having said that he doesn’t like debating – see his puerile performance against Professor sSpencer recently…..
He used to think he was good at debating – before he got wiped across the floor by Professors Stott and Lindzen and Michael Crichton.

Greg Goodman

“Is it maths or assumptions (or both) that cause the divergence?”
Assumptions. Developing from Willis’ volcano stacking, it is pretty clear that the assumption that climate takes a heavy energy deficit after major eruptions is totally erroneous.
http://climategrog.wordpress.com/?attachment_id=278
However the synchronised pattern that emerges is quite significant.
It basically shows that dip is surface temperature centred two years after an eruption is mainly part of a natural cycle.
what also comes as a surprise is that this relationship implies there is a relationship between these variations in climate and the timing of major volcanic events. Now that merits further study.
The erroneous assumption of a strong negative forcing (which exists) leaves and energy deficit at the surface leads to need to exaggerate the real CO2 forcing with hypothesised feedbacks. These feedbacks are not based of “basic science” nor measured nor proven. They are simply there to make up the energy budget on the assumption that volcanoes leave a large energy deficit.
Willis’ tropical “governor” is simplistic but seems to be a more accurate description.
However, it will be a long night before those so heavily invested in AGW will be able to admit such a fundamental short-coming. My guess is that they will be hedging around for another ten years with tweaks and adjustments trying to back-pedal so slowly that no one notices and that they can do so without loss of face.
There is a grudging acknowledgement that climate sensitivity is too high, but this kind of non linear, self-correcting behaviour means abandoning the CS concept altogether.
There is kudos to be had for the first to come up with a new paradigm that captures this kind of behaviour, but I don’t see an avalanche of researchers working in that direction.

Lance Wallace

The headline as of now (10:02) still refers to “the state of climate modeling”. This is unfortunate.

Chris4692

Lance: mine does not. Try reloading the page.

Russ R.

Here’s an excel spreadsheet comparing observed temperatures vs. model projection from: Hansen (1988), IPCC FAR (1990), IPCC SAR (1995) and IPCC TAR (2001), in pretty charts.
It can be updated as more observations are added.
https://dl.dropboxusercontent.com/u/78507292/Climate%20Models.xlsx

Lance Wallace

Sorry, I had refreshed and still saw “Climate” for some reason; now, I don’t see it. I apologize to Anthony.