From the INSTITUTE OF ATMOSPHERIC PHYSICS, CHINESE ACADEMY OF SCIENCES and the “pyramid schemes” department:
A new method to evaluate overall performance of a climate model
Many climate-related studies, such as detection and attribution of historical climate change, projections of future climate and environments, and adaptation to future climate change, heavily rely on the performance of climate models. Concisely summarizing and evaluating model performance becomes increasingly important for climate model intercomparison and application, especially when more and more climate models participate in international model intercomparison projects.
Most of current model evaluation metrics, e.g., root mean square error (RMSE), correlation coefficient, standard deviation, measure the model performance in simulating individual variable. However, one often needs to evaluate a model’s overall performance in simulating multiple variables. To fill this gap, an article published in Geosci. Model Dev., presents a new multivariable integrated evaluation (MVIE) method.
“The MVIE includes three levels of statistical metrics, which can provide a comprehensive and quantitative evaluation on model performance.”
Says XU, the first author of the study from the Institute of Atmospheric Physics, Chinese Academy of Sciences. The first level of metrics, including the commonly used correlation coefficient, RMS value, and RMSE, measures model performance in terms of individual variables. The second level of metrics, including four newly developed statistical quantities, provides an integrated evaluation of model performance in terms of simulating multiple fields. The third level of metrics, multivariable integrated evaluation index (MIEI), further summarizes the three statistical quantities of second level of metrics into a single index and can be used to rank the performances of various climate models. Different from the commonly used RMSE-based metrics, the MIEI satisfies the criterion that a model performance index should vary monotonically as the model performance improves.
According to the study, higher level of metrics is derived from and concisely summarizes the lower level of metrics. “Inevitably, the higher level of metrics loses detailed statistical information in contrast to the lower level of metrics.” XU therefore suggests, “To provide a more comprehensive and detailed evaluation of model performance, one can use all three levels of metrics.”
###
The paper: https://www.geosci-model-dev.net/10/3805/2017/
Abstract:
This paper develops a multivariable integrated evaluation (MVIE) method to measure the overall performance of climate model in simulating multiple fields. The general idea of MVIE is to group various scalar fields into a vector field and compare the constructed vector field against the observed one using the vector field evaluation (VFE) diagram. The VFE diagram was devised based on the cosine relationship between three statistical quantities: root mean square length (RMSL) of a vector field, vector field similarity coefficient, and root mean square vector deviation (RMSVD). The three statistical quantities can reasonably represent the corresponding statistics between two multidimensional vector fields. Therefore, one can summarize the three statistics of multiple scalar fields using the VFE diagram and facilitate the intercomparison of model performance. The VFE diagram can illustrate how much the overall root mean square deviation of various fields is attributable to the differences in the root mean square value and how much is due to the poor pattern similarity. The MVIE method can be flexibly applied to full fields (including both the mean and anomaly) or anomaly fields depending on the application. We also propose a multivariable integrated evaluation index (MIEI) which takes the amplitude and pattern similarity of multiple scalar fields into account. The MIEI is expected to provide a more accurate evaluation of model performance in simulating multiple fields. The MIEI, VFE diagram, and commonly used statistical metrics for individual variables constitute a hierarchical evaluation methodology, which can provide a more comprehensive evaluation of model performance.
The obfuscation is strong in this one 😀 I smell a rat
quantifying how bad they stink
Meanwhile:
“Directly contradicting much of the Trump administration’s position on climate change, 13 federal agencies unveiled an exhaustive scientific report on Friday that says humans are the dominant cause of the global temperature rise that has created the warmest period in the history of civilization.”
Oh, those rats.
The swamp requires draining and we need to get rid of that rat infested hell hole.
These are truly “if” and “then” models. If the input conditions actually come to pass, then the results will be accurate. The problem seems to be that they never get the “if” anywhere near reality.
“The first level of metrics, including the commonly used correlation coefficient, RMS value, and RMSE, measures model performance in terms of individual variables.”
No. Performance is measured by the results being compared to actual measured conditions. These metrics are structured to measure the assumptions of one model compared to the assumptions of other models, and has no relevance to empirical data. Here is one more example of modelers trying to justify their existence.
The cast of the TV show Stargate had/has a word to describe papers of this kind: technobabble – it means all the things ‘babble’ means and it sounds suitably technical (meaning obtuse)
In the settled science scenario, on one side CO2 is increasing with the time and on the other the climate sensitivity factor coming down with the progression of time as presented by IPCC from their reports. This means the resulting temperature presents a zero trend. What will be the result of model tests???
Dr. S. Jeevananda Reddy
very few climate models are
intended to “predict” climate, and
that’s not how scientists use them.
they use them as experiments — change this
part over here, and see if it matches reality.
might be a particular parametrization, or a
different way of handling sea ice, or clouds,
or aerosol
pollution.
warming to 2100 can’t be predicted anyway.
models are run to 2100 with some assumed
scenarios, none of which will
actually take place.
models are calculations, very very complex
calculations, & the
interesting
questions are what if you change
this term A here to be a different term A’.
So if the computer models are not actually trying to describe the natural world climate, and be judged by their conformity to that real world, perhaps the writers of such exercises could be moved to the philosophy or theology departments, and no longer pretend to be doing science.
crackers345,
If the models can’t predict the future, then there is no evidence that they are simulating reality. Without being able to trust the outputs to be realistic, then how can one trust that anything that comes from them tells you anything about reality. The only thing that one can say with confidence is that if you change “term A,” then you will probably get results that are bounded by an uncertainty range of an ensemble. Actually, it is worse than that because of the tuning that goes into the models. It is not unlike picking a number between 1 and 1,000 to characterize the average age of men on Earth.
How about taking CO2 out of the mix and using RAW temp data from stations wholly not impacted upon by UHI, and hey presto.
In all likelihood the temperature today is no warmer than it was in around the late 1930s/1940 notwithstanding that approximately 95% of all manmade CO2 emissions have taken place since that date.
The funny thing is that that is what Biffas/Mann’s tree ring data was saying, and that is why they truncated it and spliced on the adjusted thermometer record.
I trust most of you recognize the following as the Stefan Boltzmann radiation equation that quantifies the amount of energy emitted from a surface.
Q = σ * ε * A * T^4
Two points: this radiation is a surface property and NOT a bulk property and its direction is from hot to cold. There are those that suggest since all surfaces emit based on their surface temperature energy can flow from colder to hotter with a resulting “net” flow. This supposedly explains how “back” radiation can flow from the cold troposphere to the warmer “surface” (1.5 m above the ground). This phenomenon is not present in the radiative flow calculation from the sun to the earth (1,368 W/m^2) and earth’s ToA “back’ radiation (240 W/m^2) to the sun.
As often seen in text books this “net” phenomenon is reflected in a modified equation where ΔT, (T1 – T2), is simply substituted for T, i.e.:
Q12 = σ * ε * A * (T1^4 – T2^4)
So, if 1 is hotter than 2 “net” energy flows from hot to cold. If 2 is hotter than 1 the result is negative and “net” energy still flows from hotter 2 to colder 1.
This substitution is mathematically illegal. Here is how it actually works.
********
Two S-B surfaces a & b, any temperature. (BB if ε = 1.0, GB if ε < 1.0)
Qa = σ * εa * Aa * Ta^4 Qb = σ * εb * Ab * Tb^4
Which surface is hot or cold is irrelevant. Energy radiates from the hot to the cold and according to RGHE “theory” heat also radiates from cold to hot leaving a “net” radiative LWIR heat flow.
So, let’s do that math.
Subtract
(Qa – Qb) = (σ – σ) * (εa – εb) * (Aa – Ab) * (Ta^4 – Tb^4)
(σ – σ) = 0, i.e. ZERO!!!
Right side of equation goes to zero! Also goes to zero if ε, A or T are equal.
What does this illustrate/prove?
Conservation of energy: Qa = Qb
ZERO algebraic evidence of “back” cold to hot or “net” radiation.
Good thing, since that would grossly violate the laws of thermodynamics.
Hey, this is MY marked up graphic!! R&C thoughts?
so you believe that, unlike
all other objects/substances in the universe,
the atmosphere doesn’t
radiate??
Crackers,
No, it radiates – from 32 km where the molecules end not primarily from the ground/surface.
nickreality65 commented >>No, it radiates – from 32 km where the molecules end not primarily from the ground/surface. <<
so atmospheric gases radiate at 32 km altitude,
but these gases don't radiate
near the surface?
and you have evidence of this?
if so it would completely
rewrite physics.
can't wait to read it
Crackers,
At the surface they radiate 63 W/m^2, NOT 396 W/m^2. BTW “surface” is 1.5 m above the ground.
nickreality65 November 5, 2017 at 9:51 am
Per Stefan-Boltzmann, a black body radiating at 63 W/m^2 has a temperature of -90°C … I believe you are not referring to “radiation” (how much the body is radiating). Instead, you seem to be referring to “net radiation” (how much the body is radiating MINUS how much the body is absorbing).
While both are valid ways to look at a situation, for mathematical calculations you need to consider the individual energy fluxes.
Next, you say that the “333 W/m2 comes from nowhere does nothing” … but in fact it comes from the atmosphere, and it leaves the surface warmer than it would be without the radiation from the atmosphere.
Finally, you say:
While this is true for what is called the “surface air temperature”, it is NOT true for the K/T diagram you are discussing. In that diagram, the surface is the actual surface.
w.
nickreality65,
I’m afraid that you have made a mistake in your algebra. The two sigmas do NOT equate to zero!The S-B constant of proportionality is the same for both expressions. Assume for the sake of illustration that both emissivities are equal. And, let’s assume that the areas are equal. (Actually, the area of the atmospheric emissions is slightly larger than the surface of Earth, but I want to keep it simple for illustrations.) Therefore, the three parameters are common to both difference expressions and can be extracted. Thus, Qnet simplifies to the product of the 3 parameters multiplied times the difference of the absolute temperatures to the fourth power. That is, the net energy is proportional to the difference between the temps raised to the 4th power.
Tink about it for a moment. If the temperatures were equal, there would be no net energy difference. If one temperature were absolute zero, there would be only one term surviving, the one with the positive temperature. All values in between these two extremes are possible. I think that the atmospheric energy component should be divided by two to account for the fact that half is radiating into space, and half is radiating back towards the surface. That can be taken care of with the area term.
“…half is radiating into space, and half is radiating back…”
BUNK!
Sounds like W.E. and ACS’s infamous, opaque, dull, multi-shell models. Bogus. See my papers:
http://writerbeat.com/articles/14306-Greenhouse—We-don-t-need-no-stinkin-greenhouse-Warning-science-ahead-
http://writerbeat.com/articles/15582-To-be-33C-or-not-to-be-33C
http://writerbeat.com/articles/16255-Atmospheric-Layers-and-Thermodynamic-Ping-Pong
nickreality65,
You didn’t respond to my major criticism that your algebra is wrong. Why should I bother reading more of the same?
Clyde is right. Algebraicly, what you wrote is abcd-efgh = (a-e)(b-f)(c-g)(d-h). This is a pretty big mangling of distributivity (if I have my terms correct).
Q = σ * ε * A * T^4
“Surface” σ ε A T Result
A 5.670E-08 0.9 10000 288 3.511E+06
B 5.670E-08 0.5 12000 213 7.002E+05
A-B 2.810E+06
Q = σ * ε * A *(TA^4 – TB^4)
A 5.670E-08 0.9 10000 288^4 – 213^4 2.460E+06
NOT THE SAME!!!!!
Q = σ * ε * A * T^4
“Surface” σ ε A T
A 5.670E-08 0.5 12000 288 2.340E+06
B 5.670E-08 0.5 12000 255 1.438E+06
A-B 9.020E+05
Q = σ * ε * A *(TA^4 – TB^4)
A 5.670E-08 0.5 12000 273-255 4.512E+05
STILL NOT THE SAME!!!!!
My point is that you can’t just replace T with dT.
Yeah, I screwed up pretty good. I ASSUMED that if T^4 could be replaced with dT^4 than so could the other terms, A w/ dA, ε w/ dε, σ w/ dσ. But that’s not what happened. There is an ASSUMPTION that σ, ε and A are constant so they can be pulled out of the parens leaving behind dT^4.
Surface a and Surface b
Watob = (σ * ε * A * T^4)a – (σ * ε * A * T^4)b or w/ ASSUMPTION σ * ε * A *(Ta^4 – Tb^4)
As you mentioned, if Ta = Tb than the line goes to zero and Wa = Wb.
However, what happens of Ta = Tb and ε and A are NOT equal?
Than if Aa is larger than Ab, energy will flow from a to b EVEN THOUGH Ta = Tb.
Than if εa is larger than εb, energy will flow from a to b EVEN THOUGH Ta = Tb.
So, if Aa is larger than Ab, heat will flow from a to b even though Ta = Tb.
Sun Aa is huge compared to earth Ab.
And if εa is larger than εb, heat will flow from a to b even though Ta = Tb.
If surface a is opaque and dull and surface b is shiny and translucent, heat will flow from a to b even though Ta = Tb.
Should be easy enough to demonstrate in the lab, to Feynman’s satisfaction.
So, the area of the earth’s “surface” (How is that defined? The ground? or 1.5 m above the ground?) is enormous compared to the surface area of the GHGs (How is that even defined period?) as is the net flow.
The atmosphere is 99.96% transparent and with albedo reflective? How does that εa compare to GHGs εb?
I think what it all boils down to is the notion of “net” and “back” radiation is incorrect.
Which brings me to another point. Had a discussion w/ Scott Denning about this.
Denning’s hypothesis is: at 396 W/m^2 upwelling radiation the surface/ground will lose so much heat/energy so fast that it will get really^3 cold, even frozen. All that prevents this from happening is the 333 W/m^2 “back” radiation that compensates by warming/slowing the loss (396-333=63) the surface/ground, i.e. RGHE theory.
However, based on type K T/Cs I placed in the ground and the surface (1.5 m above the ground), the air heats and cools rapidly and a lot compared to the ground which heats and cools slowly.
During the day the sun heats both the air and ground and the air can be hotter than the mulch and grass covered ground. (Notice that the car, asphalt, hunks of iron, etc. can get much hotter than the air.)
At night the air cools quickly, becoming cooler than the ground and the ground cools slowly staying warmer then the air all night long.
As Feynman observed, if your theory doesn’t pass experiment, it’s wrong and RGHE, air warming the ground, fails the experiment.
Here’s how to evaluate a climate model.
At the top of the pyramid is a single index. Would that, properly, be: 42?
My plan is to invent a system, kinda like the Chinese system above, to judge scientists instead of climate models.Here’s the basic form, just as above.
It will have the same structure, where it starts by evaluating scientists on several different metrics. Those metric results move upwards to the midlevel in the graphic above, where they form the inputs to the multivariable integrated statistics system (MISS) regarding the scientist in question.
Then in the final step, shown at the top of the pyramid above, these MISS statistics feed into the high-level integrated topmost system (HITS). This final step “summarizes the three statistical quantities of second level of metrics into a single index and can be used to rank the performances” of various climate scientists.
At the end, this will give us a single number, the Scientific Value Index that perfectly expresses that scientist’s value to society. I predict that this system, which I have dubbed the “HIT and MISS System”, will allow us to … what was it … oh, yeah, “concisely summarize and evaluate” each scientist’s performance.
This is great, because it will settle all scientific debates immediately, definitively, and painlessly. If your Scientific Value Index is greater than that of your scientific opponent, you will be judged to have won the debate.
==========================
And if you can see what is wrong with my proposal … well, that’s exactly what is wrong with these scientists’ proposal for judging climate models.
Regards to everyone on what is a lovely rainy night here,
w.
How about we get Google to lend us one of their A.I. bots?
You know, the ones with fiendishly clever algorithms they use to “tease out” meaning from billions of unrelated pages about cats. The algorithms that can identify fake news, after an initial training period under the loving keystrokes of a Google/YouTube Hero, who will teach the long-suffering computer how to recognize and correct wrong-think in climate science and beyond.
Imagine such a vista…
Two alternate methods:
1: The Fyenmann method, if the data/results don’t match the model, the model is wrong.
2: The BOM method, if the data/results don’t match the model, change the data.
except the data/results are themselves
sometimes wrong or (esp) incomplete, so evaluation is far
more
nuanced and complicated.
Realistically, the “average global temperature” parameter, however defined, should be unimportant in molding climate changes. The models need to be able to predict not only how temperature patterns change seasonally, regionally, and over the course of an average day, they should also be predicting humidity and precipitation on those levels as well. If they get those all wrong, then there is no point in even checking is their global average somehow tracks reality.
On that note, the most ridiculous thing I have seen on this whole topic is the way that measured increases in temperatures in specific conditions (winter, night, high latitude) are used to elivate the global average, which is then used to predict uniform warming everywhere and at all times.
co2isnotevil November 4, 2017 at 11:57 am
co2, it seems you’ve missed my point, likely my lack of clarity. Let me give it another shot.
First, the important correlation of temperature is not with the polar ice albedo.
The important correlation of temperature is with the tropical albedo, which is ruled by cloud cover and which responds quickly and dynamically to local temperatures. This in turn imposes strong controls on local temperatures.
Second, it is exactly that relationship between temperature and clouds that is missing in your analysis. When it gets warm in the tropics, clouds form. This changes the amount of solar energy available after albedo reflections … but you do not have any equation in your analysis for that most important connection.
Or in your terms,
a = f(T)
which means that the albedo (a) is some unknown function (f) of the temperature (T).
Where is that in your analysis?
Regards,
w.
Not trying to criticize this analysis, but:
Unless it manages to properly carry uncertainties all the way up FROM the data TO the results (and I hope it does), it has potential to be just as wrong as all the others.
I have only skimmed through this article – lack of time – and have been unable to find anything that relates to step changes in climate. Have I missed it, or is it not there?
Presuming that it is not there, no climate model is going to fit the real data adequately. Climate frequently changes abruptly. Where should I go to read something about this, either a refutation or a support for my ideas?
Robin, that’s an interesting question that points to a more general problem with climate modeling. This is that many times, nature doesn’t do “gradual”. Nature does “edges”.
For example, there is no gradual transition from cloud to clear air. You are either in the cloud or you are not. Another example. Fifty miles out off the coast where I live, you often come across a clear line with green water on one side of the line and blue water on the other side of the line. It doesn’t shade gradually from one to the other. It undergoes, as you point out, a “step change”.
Computers, on the other hand, are the reverse of nature. They do “gradual” quite well … but step changes, not so much. I’m not saying that computers can’t do them … I’m saying that step changes are much harder to model accurately than are gradual changes.
Unfortunately, most of the interesting climate processes (tropical cumulus fields, dust devils, thunderstorms, the PDO, squall lines, the El Nino/La Nina pump, tornadoes, williwaws, cyclones, etc) are temperature-threshold based. When the temperature (or more accurately the temperature difference between surface and altitude) exceeds some local threshold, the phenomenon appears.
So ALL of them represent step changes. Makes for a very challenging system to model … and it’s the reason that the current class of climate models don’t work. All of those phenomena listed above act to regulate the temperature … but they are far too small to be included in the climate models.
As a result, they are trying to model the future temperature evolution of the planet, while not modeling the very climate phenomena that regulate the temperature … which is a fool’s errand.
w.