From the INSTITUTE OF ATMOSPHERIC PHYSICS, CHINESE ACADEMY OF SCIENCES and the “pyramid schemes” department:
A new method to evaluate overall performance of a climate model
Many climate-related studies, such as detection and attribution of historical climate change, projections of future climate and environments, and adaptation to future climate change, heavily rely on the performance of climate models. Concisely summarizing and evaluating model performance becomes increasingly important for climate model intercomparison and application, especially when more and more climate models participate in international model intercomparison projects.
Most of current model evaluation metrics, e.g., root mean square error (RMSE), correlation coefficient, standard deviation, measure the model performance in simulating individual variable. However, one often needs to evaluate a model’s overall performance in simulating multiple variables. To fill this gap, an article published in Geosci. Model Dev., presents a new multivariable integrated evaluation (MVIE) method.
“The MVIE includes three levels of statistical metrics, which can provide a comprehensive and quantitative evaluation on model performance.”
Says XU, the first author of the study from the Institute of Atmospheric Physics, Chinese Academy of Sciences. The first level of metrics, including the commonly used correlation coefficient, RMS value, and RMSE, measures model performance in terms of individual variables. The second level of metrics, including four newly developed statistical quantities, provides an integrated evaluation of model performance in terms of simulating multiple fields. The third level of metrics, multivariable integrated evaluation index (MIEI), further summarizes the three statistical quantities of second level of metrics into a single index and can be used to rank the performances of various climate models. Different from the commonly used RMSE-based metrics, the MIEI satisfies the criterion that a model performance index should vary monotonically as the model performance improves.
According to the study, higher level of metrics is derived from and concisely summarizes the lower level of metrics. “Inevitably, the higher level of metrics loses detailed statistical information in contrast to the lower level of metrics.” XU therefore suggests, “To provide a more comprehensive and detailed evaluation of model performance, one can use all three levels of metrics.”
###
The paper: https://www.geosci-model-dev.net/10/3805/2017/
Abstract:
This paper develops a multivariable integrated evaluation (MVIE) method to measure the overall performance of climate model in simulating multiple fields. The general idea of MVIE is to group various scalar fields into a vector field and compare the constructed vector field against the observed one using the vector field evaluation (VFE) diagram. The VFE diagram was devised based on the cosine relationship between three statistical quantities: root mean square length (RMSL) of a vector field, vector field similarity coefficient, and root mean square vector deviation (RMSVD). The three statistical quantities can reasonably represent the corresponding statistics between two multidimensional vector fields. Therefore, one can summarize the three statistics of multiple scalar fields using the VFE diagram and facilitate the intercomparison of model performance. The VFE diagram can illustrate how much the overall root mean square deviation of various fields is attributable to the differences in the root mean square value and how much is due to the poor pattern similarity. The MVIE method can be flexibly applied to full fields (including both the mean and anomaly) or anomaly fields depending on the application. We also propose a multivariable integrated evaluation index (MIEI) which takes the amplitude and pattern similarity of multiple scalar fields into account. The MIEI is expected to provide a more accurate evaluation of model performance in simulating multiple fields. The MIEI, VFE diagram, and commonly used statistical metrics for individual variables constitute a hierarchical evaluation methodology, which can provide a more comprehensive evaluation of model performance.
We already have methods to evaluate any model, sounds like these guys are trying to re-write the rules. Of course for many so-called climate scientists the real criteria is if the model helps them to get more funding.
Lame….models will never be right when they are constantly changing/adjusting temp history
History that they back cast to today….will not even be the same by the time they run the model
…and all the other crap they do to temp history
…and a few hundred other things
When applied to current climate models do the results correlate with real world performance?
Lipstick on a pig.
The paper has lots of gobbledygook and horseshit, so good on them for that. Could definitely use more cowbell though.
Funny how ALL models could be evaluated by their actual predictions. I guess that would be too easy.
Trying to reduce something as complex as climate to a single number reminds me of Swift and his parody of science.
Start with a thorough understanding of what causes climate change.
Even a one page summary is acceptable.
Without that understanding, there are no real climate models.
Unfortunately that understanding does not exist today.
Therefore, we have only wild guess computer games, falsely called “climate models”,
that will make wrong predictions … until the temperature actuals are eventually “adjusted”
enough so the predictions look better!
The average temperature of our planet ‘s surface has remained in a one degree C. range since 1880,
even with haphazard measurements, lots of surface area not measured at all, and “adjustments” that may have doubled the warming in the raw data, and the data is owned by people who WANT to see a lot of global warming … yet we’ve been in a one degree C. range for 137 years!
Why would anyone with a functioning brain think such a tight average temperature range over 137 years is a ‘coming climate change catastrophe”?
Climate blog for non-scientists:
http://www.elOnionBloggle.Blogspot.com
over 12,000 page views so far
No ads – no money for me – a public service
Poorly-worded, but as it notes, most models seem to be scored on an individual variable (global temperature). This at least would seemingly call BS on models that fair miserably to reproduce other parameters with accuracy.
My understanding is that the Chinese categorically reject CAGW and Climate Change altogether.
So they might be up to something else, like developing a new suite of models that actually produce useful long range forecasts. And maybe taking a poke at Western climate science in the process.
This climate model effort is analogous to making further adjustments to the epicycles and deferents in a Ptolemaic planetary system. With each refinement, new complications arise elsewhere in the model.
Their underlying fundamental assumptions are wrong in both cases.
The first rule of climate ‘science’ , if the models and realty differ in value , it is reality which always in error, somewhat underminds the need for this research. In addition the authors have made an error in their maths. For it is clear the ‘value’ of any model is not a function of its validity, rather it is in a direct relationship with the degree of support the model offers to further the AGW faith. Science has f all to do with it.
I’ll wait for McIntyre or Briggs to work this over! First it’s already compromised science when you have to use statistics at all (that ought raise a din from all sides). It’s essential for the social sciences and we know the wiggle room it makes for these irredeemably ideologically corrupted disciplines (of which climate science is the best example).
To me, this analysis constructs a phony “index” that will give a totally wrong model a pass. If you have a hard-wired falsified theory based on CO2 as the basis and you adjust this with a concocted uber-aerosols effect to protect the theory and an additional wrong-signed cloud parameter, you could end up with an excellent forecast with a little prestidigitation and get an index of 0.99.
I’ve been fearing this possibility as a pretext for jailing deplorables and putting the world under elitist governance. Thankfully their hubris and post no-idjit-left-behind enrollment policies in institutions of higher learning seems to have blocked their vision. Adding a coefficient “c” equal to 1/3 to multiply their formula by would have given them a heck of a scary fit.
It appears to me that the authors have presented a rigorous, quantitative method for evaluating multiple model results to determine the best compromise model. However, one often is more concerned about one of the variables than the others. They then recommend assigning subjective weighting to the variable(s) of primary interest. They have then degraded the quantitative approach with subjective assessments of the weighting to be assigned.
The authors acknowledge that is is generally recognized that some models do a better job of predicting future temperatures than they do future precipitation, and vice versa. That is an interesting state of affairs because there is strong interaction between temperature and precipitation. That is, the surface commonly cools down during and immediately after a Summer rain, and high surface temperatures may result in virga. So, at first blush, it would appear that there are serious problems with the assumptions or constructs within the models when these interacting variables have different inaccuracies.
It should be obvious that the models aren’t fit for the purpose for which models are usually built, i.e. to predict future states with the perturbation of one or more input variables. Being able to identify the best compromise model is indeed like putting “lipstick on a pig.” What is needed is a paradigm shift in modeling where the numerous output variables, such as temperature and precipitation, are consistent with each other and track historical records much better than the 3X overestimate of future temperatures currently seen.
The right way to evaluate a climate model is to watch and wait for a few hundred centuries. This just isn’t compatible with human lifespan.
The same methodology might be useful in detecting chronic bias not just of the model but the operator and the users.
How can the predictions for a chaotic system compared to reality be anymore than chance?
Off topic a bit, but I’m curious to get your takes on the flurry of propaganda articles coming out just before the UN climate talks starting on 11/6 and the findings of the “major federal climate report” that was release today.
A computer big enough to run a programme complex enough to realistically represent the evolution of the weather through the ages thus creating a picture of the climate, would run very slowly. In fact it would run at about the same speed as the actual climate.
This coincidence would give the modelers something to gauge their success by, as the actual performance could be compared with the computer-calculated performance in real time, side by side, for generations. After some time, tweaking and all, they would be able to demonstrate they can back-cast the whole climate accurately. I think this would be a major step forward.
Mathematical onanism with lubricants.
A comment from a management school where the pyramid of management was being explained
“Oh is that how it works? I thought it was like a vegetarian’s outhouse where the turds float to the top”
Just saying.
The utility and skillfulness of computer models depends on:
1. how well the processes which they model are understood,
2. how faithfully those processes are simulated in the computer code, and
3. whether the results can be repeatedly tested so that the models can be validated and refined.
Specialized models, which try to model reasonably well-understood processes like PGR and radiation transport, are useful, because the processes they model are manageably simple and well-understood.
Weather forecasting models are also useful, even though the processes they model are very complex, because the models’ short-term predictions can be repeatedly tested, allowing the models to be validated and refined.
But more ambitious models, like GCMs, which attempt to simulate the combined effects of many poorly-understood processes, over time periods too long to allow repeated testing and refinement, are of dubious utility.
E.g., NASA’s ModelE2 consists of about a half-million lines of moldy Fortran code, which it is safe to assume nobody actually understands. They’ve got so many fudge factors, “knobs” and pseudo-random number generator calls in there that they can make it do just about anything at all, but It doesn’t in any sense represent an understanding of the Earth’s climate system. What’s more, unlike weather models, which are comparably complex but get tested every week, the predictions of those GCMs are untestable. Ask any computer scientist whether he would trust an untestable 500,000 line Fortran program as the basis for multi-million dollar decisions!
Worst of all are so-called “semi-empirical models,” which aren’t actually models at all. So-called “semi-empirical modeling” is an oxymoron: “modeling” that doesn’t actually model anything. It is similar to modeling, but without reference to any physical basis. It is really just curve-matching. It can be made to produce just about any desired result.
GCMs are subject to criticisms that they don’t accurately model the real world, because of inconsistency with observations of things like clouds and the predicted tropical mid-tropospheric hot spot. Semi-empirical modelers neatly avoid such criticism, by not even trying to model the real world. It’s the worst sort of junk science.
“Inevitably, the higher level of metrics loses detailed statistical information in contrast to the lower level of metrics.”
Does this mean destroying accuracy with averaging?
*accuracy of ‘first level’ metrics values
“According to the study, higher level of metrics is derived from and concisely summarizes the lower level of metrics ”
lower resolution?