guest post by Nick Stokes
There has been a lot of discussion lately of error propagation in climate models, eg here and here. I have spent much of my professional life in computational fluid dynamics, dealing with exactly that problem. GCM’s are a special kind of CFD, and both are applications of the numerical solution of differential equations (DEs). Propagation of error in DE’s is a central concern. It is usually described under the heading of instability, which is what happens when errors grow rapidly, usually due to a design fault in the program.
So first I should say what error means here. It is just a discrepancy between a number that arises in the calculation, and what you believe is the true number. It doesn’t matter for DE solution why you think it is wrong; all that matters is what the iterative calculation then does with the difference. That is the propagation of error.
A general linear equation in time can be formulated as
y’ = A(t)*y+f(t) ……….(1)
y(t) could be just one variable or a large vector (as in GCMs); A(t) will be a corresponding matrix, and f(t) could be some external driver, or a set of perturbations (error). The y’ means time derivative. With a non-linear system such as Navier-Stokes, A could be a function of y, but this dependence is small locally (in space and time) for a region; the basics of error propagation follow from the linearised version.
I’ll start with some bits of DE theory that you can skip (I’ll get more specific soon). If you have another solution z which is the solution following an error, then the difference satisfies
(y-z)’=A*(y-z)
The dependence on f(t) has gone. Error propagation is determined by the homogeneous part y’=A*y.
You can write down the solutions of this equation explicitly:
y(t) = W(t)*a, W(t) = exp(∫ A(u) du )
where the exp() is in general a matrix exponential, and the integral is from starting time 0 to t. Then a is a vector representing the initial state, where the error will appear, and the exponential determines how it is propagated.
You can get a long way by just analysing a single error, because the system is linear and instances can be added (superposed). But what if there is a string of sequential errors? That corresponds to the original inhomogeneous equation, where f(t) is some kind of random variable. So then we would like a solution of the inhomogeneous equation. This is
y(t) = W(t) ∫ W-1(u) f(u) du, where W(t)=exp(∫ A(v) dv ), and integrals are from 0 to t
To get the general solution, you can add any solution of the homogeneous equation.
For the particular case where A=0, W is the identity, and the solution is a random walk. But only in that particular case. Generally, it is something very different. I’ll describe some special cases, in one or few variable. In each case I show a plot with a solution in black, a perturbed solution in red, and a few random solutions in pale grey for context.
Special case 1: y’=0
This is the simplest differential equation you can have. It says no change; everything stays constant. Every error you make continues in the solution, but doesn’t grow or shrink. It is of interest, though, in that if you keep making errors, the result is a random walk.
Special case 2: y”=0
The case of no acceleration. Now if there is an error in the velocity, the error in location will keep growing. Already different, and already the simple random walk solution for successive errors doesn’t work. The steps of the walk would expand with time.
Special case 3: y’=c*y
where c is a constant. If c>0, the solutions are growing exponentials. The errors are also solutions, so they grow exponentially. This is a case very important to DE practice, because it is the mode of instability. For truly linear equations the errors increase in proportion to the solution, and so maybe don’t matter much. But for CFD it is usually a blow-up.
But there are simplifications, too. For the case of continuous errors, the earlier ones have grown a lot by the time the later ones get started, and really are the only ones that count. So it loses the character of random walk, because of the skewed weighting.
If c<0, the situation is reversed (in fact, it corresponds to above with time reversed). Both the solutions and the errors diminish. For continuously created errors, this has a kind of reverse simplifying effect. Only the most recent errors count. But if they do not reduce in magnitude while the solutions do, then they will overwhelm the solutions, not because of growing, but just starting big. That is why you couldn’t calculate a diminishing solution in fixed point arithmetic, for example.
This special case is important, because it corresponds to the behaviour of eigenvalues in the general solution matrix W. A single positive eigenvalue of A can produce growing solutions which, started from any error, will grow and become dominant. Conversely the many solutions that correspond to negative eigenvalues will diminish and have no continuing effect.
Special case 4: Non-linear y’=1-y2
Just looking at linear equations gives an oversimplified view where errors and solutions change in proportion. The solutions of this equation are the functions tanh(t+a) and coth(t+a), for arbitrary a. They tend to 1 as t→∞ and to -1 as t→-∞. Convergence is exponential. So an error made near t=-1 will grow rapidly for a while, then plateau, then diminish, eventually rapidly and to zero.
Special case 5: the Lorenz butterfly
This is the poster child for vigorous error propagation. It leads to chaos, which I’ll say more about. But there is a lot to be learnt from analysis. I have written about the Lorenz attractor here and in posts linked there. At that link you can see a gadget that will allow you to generate trajectories from arbitrary start points and finish times, and to see the results in 3D using webGL. A typical view is like this
Lorenz derived his equations to represent a very simple climate model. They are:
The parameters are conventionally σ=10, β=8/3, ρ=28. My view above is in the x-z plane and emphasises symmetry. There are three stationary points of the equations, 1 at (0,0,0),(a, a, 27)and,(-a, -a, 27) where a = sqrt(72). The last two are centres of the wings. Near the centres, the equations linearise to give a solution which is a logarithmic spiral. You can think of it as a version of y’=a*y, where a is complex with small positive real part. So trajectories spiral outward, and at this stage errors will propagate with exponential increase. I have shown the trajectories on the plot with rainbow colors, so you can see where the bands repeat, and how the colors gradually separate from each other. Paths near the wing but not on it are drawn rapidly toward the wing.
As the paths move away from the centres, the linear relation erodes, but really fails approaching z=0. Then the paths pass around that axis, also dipping towards z=0. This brings them into the region of attraction of the other wing, and they drop onto it. This is where much mixing occurs, because paths that were only moderately far apart fall onto very different bands of the log spiral of that wing. If one falls closer to the centre than the other, it will be several laps behind, and worse, velocities drop to zero toward the centre. Once on the other wing, paths gradually spiral outward toward z=0, and repeat.
Is chaos bad?
Is the Pope Catholic? you might ask. But chaos is not bad, and we live with it all the time. There is a lot of structure to the Lorenz attractor, and if you saw a whole lot of random points and paths sorting themselves out into this shape, I think you would marvel not at the chaos but the order.
In fact we deal with information in the absence of solution paths all the time. A shop functions perfectly well even though it can’t trace which coins came from which customer. More scientifically, think of a cylinder of gas molecules. Computationally, it is impossible to follow their paths. But we know a lot about gas behaviour, and can design efficient ICE’s, for example, without tracking molecules. In fact, we can infer almost everything we want to know from statistical mechanics that started with Maxwell/Boltzmann.
CFD embodies chaos, and it is part of the way it works. People normally think of turbulence there, but it would be chaotic even without it. CFD solutions quickly lose detailed memory of initial conditions, but that is a positive, because in practical flow we never knew them anyway. Real flow has the same feature as its computational analogue, as one would wish. If it did depend on initial conditions that we could never know, that would be a problem.
So you might do wind tunnel tests to determine lift and drag of a wing design. You never know initial conditions in tunnel or in flight but it doesn’t matter. In CFD you’d start with initial conditions, but they soon get forgotten. Just as well.
GCMs and chaos
GCMs are CFD and also cannot track paths. The same loss of initial information occurs on another scale. GCMs, operating as weather forecasts, can track the scale of things we call weather for a few days, but not further, for essentially the same reasons. But, like CFD, they can generate longer term solutions that represent the response to the balance of mass, momentum and energy over the same longer term. These are the climate solutions. Just as we can have a gas law which gives bulk properties of molecules that move in ways we can’t predict, so GCMs give information about climate with weather we can’t predict.
What is done in practice? Ensembles!
Analysis of error in CFD and GCMs is normally done to design for stability. It gets too complicated for quantitative tracing of error, and so a more rigorous and comprehensive solution is used, which is … just do it. If you want to know how a system responds to error, make one and see. In CFD, where a major source of error is the spatial discretisation, a common technique is to search for grid invariance. That is, solve with finer grids until refinement makes no difference.
With weather forecasting, a standard method is use of ensembles. If you are unsure of input values, try a range and see what range of output you get. And this is done with GCMs. Of course there the runs are costlier, and so they can’t do a full range of variations with each run. On the other hand, GCM’s are generally surveying the same climate future with just different scenarios. So any moderate degree of ensemble use will accumulate the necessary information.
Another thing to remember about ensemble use in GCM’s is this. You don’t have to worry about testing a million different possible errors. The reason is related to the loss of initial information. Very quickly one error starts to look pretty much like another. This is the filtering that results from the vary large eigenspace of modes that are damped by viscosity and other diffusion. It is only the effect of error on a quite small space of possible solutions that matters.
If you look at the KNMI CMIP 5 table of GCM results, you’ll see a whole lot of models, scenarios and result types. But if you look at the small number beside each radio button, it is the ensemble range. Sometimes it is only one – you don’t have to do an ensemble in every case. But very often it is 5,6 or even 10, just for 1 program. CMIP has a special notation for recording whether the ensembles are varying just initial conditions or some parameter.
Conclusion
Error propagation is very important in differential equations, and is very much a property of the equation. You can’t analyse without taking that into account. Fast growing errors are the main cause of instability, and must be attended to. The best way to test error propagation, if computing resources are adequate, is by an ensemble method, where a range of perturbations are made. This is done with earth models, both forecasting and climate.
Appendix – emulating GCMs
One criticised feature of Pat Frank’s paper was the use of a simplified equation (1) which was subjected to error analysis in place of the more complex GCMs. The justification given was that it emulated GCM solutions (actually an average). Is this OK?
Given a solution f(t) of a GCM, you can actually emulate it perfectly with a huge variety of DEs. For any coefficient matrix A(t), the equation
y’ = A*y + f’ – A*f
has y=f as a solution. A perfect emulator. But as I showed above, the error propagation is given by the homogeneous part y’ = A*y. And that could be anything at all, depending on choice of A. Sharing a common solution does not mean that two equations share error propagation. So it’s not OK.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
If I understand this correctly, Nick’s informative post is making a very simple logical point in criticism of the original paper.
He is basically arguing that the simple emulation of GCM models that Mr Frank has used in his paper does not behave in the same way, with regard to error propagation, as the originals. He gives reasons and a detailed analysis of why this is so, which I am not competent to evaluate.
But this is the logic of his argument, and its quite straightforward, and if he is correct (and it seems plausible) then its terminal to Mr Frank’s argument.
It is restricted in scope. It does not show that the models are valid or useful for policy purposes or accurately reflect climate futures. It just shows that one particular criticism of them is incorrect.
The thing that has always puzzled me about the models and the spaghetti graphs one sees all the time is a different and equally simple logical point. We have numerous different models. Some of them track evolving temperatures well, others badly.
Why does anyone think its legitimate to average them all to get a prediction going forwards? Why are we not simply rejecting the non-performing ones and using only those with a track record of reasonable accuracy?
Surely in no other field, imagine safety testing or vaccine effectiveness, would we construct multiple models, and then average the results to get a policy prediction, when more than half of them have been shown by observation not to be fit for purpose.
Well, michel, if I have any inkling of the gist of the reality, the original models cannot ever account for information to such an extent that they have any predictive value. So, how does a person show this using the original models, when the original equations upon which they are based are unsolvable? It seems that you model the models, which, yes, might not be the original models, but, remember, the models are not the original climate either — they are simulations based on limited input.
Tools that have inherent reality limitations might be subject to a set of limitations themselves that analyze them, in this respect.
The paper demonstrates that the models are linear air temperature projection machines, michel.
Nick’s post is a complicated diversion, is all. A smoke screen.
I doubt stability analysis is the same as propagation of uncertainty. A stable numerical solution to a differential equation still propagates uncertainty. They are related because stability is required, otherwise any further analysis is impossible.
I’ve used (and written) many simulation programs for technical applications (flight simulation) that involve solving differential equations. I’m familiar with propagation of uncertainty in this kind of programs. It has a distinctive mathematical form. I’ll try to illustrate below.
Wikipedia shows propagation of uncertainty involves the Jacobian matrix (and its transpose) of the function under analysis, see:
https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Non-linear_combinations
This shows the distinctive pattern: J*cov(x)*trn(J)
were J is the Jacobian, cov(x) is the covariance of x, trn() means transpose()
You can see how uncertainty propagation works in the prediction and update steps of a Kalman filter (linear case btw), see:
https://en.wikipedia.org/wiki/Kalman_filter#Predict
Sure enough we see the pattern J*var(x)*trn(J).
Since we are dealing with differential equations y’=f(x) I expected to see the Jacobian of the derivative function f(x) and its transpose to emerge in this article. But I don’t see the Jacobian anywhere. What’s up? 😉
(Sorry if I messed up the links. I’m unfamiliar with this forum system.)
“I doubt stability analysis is the same as propagation of uncertainty.”
No, it’s in effect a subset. If there is a component that propagates with exponential increase, corresponding to a positive eigenvalue of A, then it is unstable. As you say, that is the first thing you have to establish about propagation.
“involves the Jacobian matrix (and its transpose) of the function under analysis”
That is for a mapping with a prescribed function. Here we have a function indirectly prescribed by a differential equation. The equivalent of Jacobian is the matrix W(t) = exp(∫ A(u) du )that I defined.
“But I don’t see the Jacobian anywhere.”
If the de is non-linear, y’=g(y,t), then A is the Jacobian of g.
I left the equation mostly at the deterministic stage, but gave the mapping of an added term f as W(t) ∫ W⁻¹(u) f(u) du. If f(u) is a random variable, then the integral is stochastic, and should be evaluated as sqrt( ∫ w*u*cov(f)*u*w du) with appropriate transposes, covariance including autocovariance, w standing for W⁻¹. Same pattern but you have to build in the integration. That is the generalisation of random-walk style integration in quadrature for general DE’s. W(t) is also the fundamental solution matrix, which you can take to be the set of solutions with initial conditions the identity I.
“Since we are dealing with differential equations y’=f(x)”
Did you mean y’=f(x)*y? That is what I had, with A for f.
I’m just a simple engineer. I’m used to the recurrence equations used to find the state evolution of systems described by X’ = f(t, X) with X as system state vector:
X_n+1 = X_n + h*A(t, X_n)
Where h is the time step and A(t, X_n) is an approximation of the slope between X-n and X_n+1. Usually A(t, X_n) is a Runge-Kutta scheme that evaluates f(t, X) at intermediate points. If one uses the Euler integration scheme then A(t, X_n) is equal to f(t. X_n).
Analysing the propagation of uncertainty by this recurrence equation produces another recurrence equation that describes the evolution of uncertainty in the system state (same as Kalman filter does):
cov(X_n+1) = J*cov(X_n)*trn(J)
With Jacobian J = d(X_n+1)/d(X_n) = I + h*d(A(t, X_n))/d(X_n)
For complex systems finding the Jacobian of function f(t, X) can be difficult. But, in principle de propagation of uncertainty is straight forward and can be combined with the evolution of the system state itself. If the uncertainty exceeds acceptable limits: stop the simulation.
“Did you mean y’=f(x)*y? That is what I had, with A for f.”
Yeah, I messed up. I meant to write y’=f(y, t).
“For complex systems finding the Jacobian of function f(t, X) can be difficult. But, in principle de propagation of uncertainty is straight forward and can be combined with the evolution of the system state itself. If the uncertainty exceeds acceptable limits: stop the simulation.”
Yes, I agree with all that. And your Jacobian approach will tell you whether in one step solutions are diverging or converging. Again it comes down to whether your dA/dX has a positive eigenvalue.
I’m tending to look at multi-steps where you say you have a basis set of independent solutions W, and say that any solution is a linear combination of that basis. You could get that from your Runge-Kutta recurrence. Another way of seeing it is as if the multistep is the product matrix of your single steps I+h*dA/dX.
Thanks Nick, an interesting overview of just one flawed feature of climate models.
In my earlier days, I was involved in a project to computerise the fluid dynamics of molten aluminium as it solidified in a rapid chill casting process. This was an attempt to predict where trapped air would congregate and create areas of potential failure under extreme stress.
The variables are not so great in this set up, as they are in global climate modelling. Finite element analysis was deployed and some of the best mathematical minds were engaged to help write the code and verify the model’s potential.
I won’t go into detail, but it’s safe to say my confidence in academia and computer modelling was crystallised during that exercise, if only the castings had experienced such predictable crystallisation….
The difficulties with trying to capture all the variables that impact a chaotic system are where the challenge actually is. The known flaws in the computer algorithms and even the maths deployed in the code is not where the challenge is. Just missing any variable that impacts the model, renders the model useless.
The ability of climate models to predict the future is zero.
The evidence of this is there for all to see. The models are all running hot when compared to real observation. That is telling us something.
It is telling us the models are missing a feedback or are based on a flawed hypothesis, possibly both!
When weather predictions can only be confident/meaningfully accurate to three days out, and as weather patterns most definitely play a part in our experience of climate, who out there, is going to bet on the same weather/climate people getting it right 100 years out?
“When observation disagrees with the hypothesis, the hypothesis is wrong” Feynman
https://www.presentationzen.com/presentationzen/2014/04/richard-feynman-on-the-scientific-method-in-1-minute.html
“In my earlier days, I was involved in a project to computerise the fluid dynamics of molten aluminium as it solidified in a rapid chill casting process. This was an attempt to predict where trapped air would congregate and create areas of potential failure under extreme stress.”
Well, well. My group did high pressure die casting with aluminium, but using smoothed particle hydrodynamics. It worked pretty well. The problem with FEM is the fast moving boundary; hard to do with mesh. GCMs don’t have anything like that.
“The models are all running hot when compared to real observation.”
But they are all running, and they produce a huge amount more of information than just surface temperature. And they aren’t describing weather, which covers much of the comparison time. It is quite possible that the Earth has been running cool, and will catch up.
Stokes
You said, “It is quite possible that the Earth has been running cool, and will catch up.” Almost anything is possible! What is the probability? On what would you base the estimation of such a probability? Your remark is not unlike all the scare stories based on words such as “may, could, conceivably, etc.”
+1
So first I should say what error means here. It is just a discrepancy between a number that arises in the calculation, and what you believe is the true number. It doesn’t matter for DE solution why you think it is wrong; all that matters is what the iterative calculation then does with the difference. That is the propagation of error.
So what if, instead of knowing an error, you know only the range or confidence interval of an important parameter? How do you propagate the range or confidence interval of model values reasonably concordant with the range or CI of the parameter? That is the problem addressed by Pat Frank that you have never addressed yet.
You wrote of the “scale” problem of using a meter stick that was 0.96m in length. What if all you know is that the stick is between 0.94 and 0.98 m? The distance measured to be equal to 1 stick length is between 0.94 and 0.98m; two lengths would be between 1.88 and 1.96; …; N lengths would be between N*94 and N*98, and the uncertainty would be N*0.04. That’s for absolute limits. With confidence intervals instead, the propagation of the uncertainty is more complex.
Given the CI of the cloud feedback parameter addressed by Pat Frank, what is your best calculation of the uncertainty of the GCM forecasts? Less than his calculated value? More than his calculated value?
As I wrote, you have not yet come to grips with the difference between propagating an error and propagating an interval or range of uncertainty.
It would be good of you, in the spirit of scientific disputation, to submit your article for publication.
“How do you propagate the range or confidence interval of model values reasonably concordant with the range or CI of the parameter?”
In the same way as for point pairs or groups. A DE determines the stretching of the solution space; the range or CI’s stretch in accordance with the separation of two points, or however many are needed to provide an effective basis to the dimension of the space.
“That’s for absolute limits.”
No, it’s just for scaling. The ruler doesn’t change between measurings. You may not know what the number is, but no stats can help you here. If you think it is wrong relative to the standard metre, you have to consult the standard metre. Calibrate.
“Given the CI of the cloud feedback parameter addressed by Pat Frank, what is your best calculation of the uncertainty of the GCM forecasts?”
Can’t say – we have just one number 4 W/m2. There is no basis for attaching a scale for how if at all it might change in time. There is also the issue raised by Roy, which I would put a little differently. If it is just a bias, an uncertainty about a fixed offset, then in terms of temperature that would be taken out by the anomaly calculation. It is already well known that GCM’s have considerable uncertainty about the absolute temperature, but make good predictions about anomaly, which is what we really want to know. If it does have a fairly high frequency variation, that will just appear as weather, which for climate purposes is filtered out. The only way it might impact on climate is if it has secular variation on a climate time scale.
“It would be good of you, in the spirit of scientific disputation, to submit your article for publication.”
I doubt if it would qualify for originality. A lot of it is what I learnt during my PhD, and sad to say, is therefore not new.
Nick Stokes: You may not know what the number is, but no stats can help you here. If you think it is wrong relative to the standard metre, you have to consult the standard metre. Calibrate.
Boy are you ever dedicated to missing the point. When you recalibrate you are given the range (or CI) of likely true lengths, not the true length. Do you believe that in actual practice you know either the true length or any specific error? The error does not have to change randomly from placement to placement of the “meter” stick in order for the uncertainty induced by the calibration uncertainty to accumulate.
Can’t say – we have just one number 4 W/m2.
That is not true. Besides that best estimate, you have a CI of values that probably contains the true value within it (but might not). From that, what is reasonably knowable is the range of GCM outputs reasonably compatible with the CI of reasonably likely parameter values.
The “point” you and Roy emphasize is exactly wrong. What we have here is that the parameter value is not known, only a reasonable estimate and a reasonable estimate of its range of likely values; therefore what the output of the model would be with the correct parameter value is not known. What is a reasonable range of expectations of model output given the reasonable estimates of the parameter value and its likely range? A GCM run gives you one output that follows from the best estimate; what is a reasonable expectation of the possible range of model outputs compatible with the range of possible parameter values? That is what Pat Frank focused on calculating, and what you and Roy are systematically avoiding.
I doubt if it would qualify for originality.
I meant as a point-by-point critique of Pat Frank’s article, not an introduction to what you have learned. It really isn’t such a point-by-point refutation.
Mr. Stokes –> You said ” “Given the CI of the cloud feedback parameter addressed by Pat Frank, what is your best calculation of the uncertainty of the GCM forecasts?” Can’t say”.
Therein lies the problem. You can’t say what it is, but you insist that Dr. Frank can’t either.
You first need to answer for yourself and others, whether there is uncertainty or there is not. If you agree there is, then you need to tell the world what your values are and provide numbers and calculations and how you derived them. If you continue to claim there is no uncertainty, then we have an immediate answer.
Your math is above my pay grade in resolving. Been too long since I delved into this. However, as an old electrical engineer, I can tell you that all the fancy equations and models of even simple circuits never survived the real world. Kind of like they say a battle plan never survives contact with the enemy. What is the moral of this? There is always uncertainty. Those of us who have done things, built things, answered the boss’s question about how certain you are about your design know this first hand.
KISS, keep it simple stupid! Dr. Frank has done this. All your protestation aside, you have not directly disproved either his hypothesis nor his conclusions. You’re basically trying to say that Dr. Frank is wrong because your mathematics is correct, which is basically saying that you believe there is no uncertainty.
Until you man up and can derive your estimation of the amount of uncertainty in GCM projections you are not in the game. Dr. Frank has put his projections out for the whole world to see, let us see what you come up with.
“You can’t say what it is, but you insist that Dr. Frank can’t either.”
That’s a common situation. It in no way follows that if I can’t get an estimate, any other guess must be right.
In fact, what I say is that the way to find out is to do an ensemble and look at the spread of outcomes. I don’t have the facilities to do that, but others do.
Nick Stokes: In fact, what I say is that the way to find out is to do an ensemble and look at the spread of outcomes.
I advocated that, as have many others, with the proviso that the “ensemble” runs include (random) samples from the CI of the parameter values (in a response to Roy I mentioned “bootstrappping”.)
Until then, an improvement on Pat Frank’s analysis is unlikely to appear any time soon. By addressing only one of many uncertain parameters, he has likely produced an underestimate of the uncertainty of GCM model output.
“That’s a common situation. It in no way follows that if I can’t get an estimate, any other guess must be right.”
ROFL! No, the issue is that you can’t say that other estimates are wrong! You are trying to say that since you don’t know then no one else can know either!
“In fact, what I say is that the way to find out is to do an ensemble and look at the spread of outcomes. I don’t have the facilities to do that, but others do.”
Ensembles simply can’t tell you about uncertainty. All you wind up with is a collection of different output values. You still don’t know what the uncertainty is for each of those different output values.
Nick deserves this post as head post for a few days in keeping with the status afforded to Pat.
Irrespective of the time Pat Franks post was pegged, I would say this is one of the most informative posts in a while here and deserves visibility above the typical low level opinion posts and “guest rants” of which we see several per day. Posts of this quality do not happen.
As the Author said just above , the lead has no originality.
So it actulally brings nothing new to the mad climate tea party, except more cups.
He said it was nothing ground breaking that would merit a paper. That does not mean it is not a valuable contribution to the ongoing discussion of error propagation started by the prominent coverage given to Pat Franks “discovery”.
If you want any more plastic throw away cups , why do you still read WUWT?
To quote above : “I doubt if it would qualify for originality. A lot of it is what I learnt during my PhD, and sad to say, is therefore not new.”
Now Pat Frank’s thorough and devastating uncertainty analysis is only new for climate actors.
Say for the sake of argument that the parameter estimate was 0.10, and the 95% CI was (-1.9, 2.1). One way to propagate the CI would be to calculate the GCM model output by running it repeatedly with this parameter sequentially assigned values -1.9, -1.8, …, -0.1, …, 2.1, keeping all other parameter estimates at their best estimated values. If model output were monotonic in this parameter, the CI on outputs would be the set of lower and higher outputs from this run; without the monotonocity, the calculation of a CI would be more complicated.
Once again, the point is that propagation of uncertainty requires propagation of an interval of uncertainty. Nick Stokes has illustrated the propagation of an error.
“Besides that best estimate, you have a CI of values that probably contains the true value within it (but might not). “
I think it would be useful if someone would actually write down, with references, what we do know about this datum. I don’t know where you get that CI from, or what the number might be; AFAIK, Lauer just gives the value 4 W/m2 as the 20 year annual average rmse. No further CIs, no spectral information. Pat insists that it can be compounded in quadrature – ie variance/year, but the factual basis is very weak, and I think wrong. It all hangs on Lauer including the word annual, but it seems to me that this is just saying that it is a non-seasonal average, just as you might say annual average temperature. Anyway, people are trying to build so much on it that the bare facts should be clearly stated.
Nick Stokes: I think it would be useful if someone would actually write down, with references, what we do know about this datum.
That’s potentially a good idea, but would you care? Your argument has been that propagating the uncertainty in the parameter value is something that you have never studied (implication being that it is intrinsically worthless); besides that, you have never responded when your other questions were answered, such as What use is made of the correlation of the errors at successive time intervals.
Where-ever any GCM modeler got any parameter estimates, those estimates were reported with “probable errors”, and the implications of the probable errors have been ignored up til now. You advocate continuing to ignore them.
Incidentally, I am reading a book titled “Statistical Orbit Determination”. Surprisingly, some of the parameter estimates (called “constants”) are reported to 12+ significant figures of accuracy. Nothing in climate science can claim to be that well known. N.B. “accuracy” is the intended adjective, since accuracy of the orbit of a satellite is of great importance. My reading of the GCM modelers is that they treat their parameter estimates as though such accuracy has been attained for them.
So, perhaps someone has already tabulated the parameter estimates, their sources, and their confidence intervals. Surely the GCM modelers have done this?
Firstly, many thanks to Nick Stokes for a very informed and enlightening introduction to the subject in a clear and accessible way.
There is an unjustified jump here. We know that gas laws work because of extensive observational verification. We have zero verification for GCM output, thus the claimed “so GCMs give information about” does not follow. It is not sufficient that both are working is similar ways to infer that GCMs “give information” which is as sure and tested as the gas laws.
This should read more like ” GCMs have the potential , after observational validation, of give information about climate “. At the moment they give us what they have been tuned to give us.
It should be clarified that this means ensembles of individual runs of the same model. Not the IPCC’s “ensembles” of random garbage from a pot-pourri of unvalidated models from all commers. Just taking the average of an unqualified mess does not get us nearer to a scientific result.
“It would be good of you, in the spirit of scientific disputation, to submit your article for publication.”
I doubt if it would qualify for originality. A lot of it is what I learnt during my PhD, and sad to say, is therefore not new.
“We know that gas laws work because of extensive observational verification. “
Well, yes, Boyle and Charles got in first. But Maxwell and Boltzmann could have told us if B&C hadn’t.
“At the moment they give us what they have been tuned to give us.”
That’s actually not true, and one piece of evidence is Hansen’s predictions. There was virtually no tuning then; Lebedev’s log of the runs is around somewhere, and there were very few extensive runs before publication. 1980’s computers wouldn’t do it.
“It should be clarified that this means ensembles of individual runs of the same model”
That is certainly the simplest case, and they are the ones I was referring to. However I wouldn’t discount the Grand ensembles that CMIP put together quite carefully.
Nick,
Your comment on Hansen’s predictions is a logical fallacy. Back in 1976, Lambeck and Cazenave looked at the strong quasi-60 year cycle in about 15 separate climate indices, which they related to (lagged) changes in LOD. This was at the time when the consensus fear was still of continued global cooling. They wrote:-
“…but if the hypothesis is accepted then the continuing deceleration of m for the
last 10 yr suggests that the present period of decreasing average global temperature
will continue for at least another 5-10 yr. Perhaps a slight comfort in this gloomy
trend is that in 1972 the LOD showed a sharp positive acceleration that has persisted
until the present, although it is impossible to say if this trend will continue as it did
at the turn of the century or whether it is only a small perturbation in the more general
decelerating trend.”
If Hansen’s predictions are “evidence”, then by the same token we can conclude from Lambeck’s prediction that the post-1979 upturn in temperature is natural variation following a change in LOD.
“Your comment on Hansen’s predictions is a logical fallacy”
No, I can’t see the logic of your version. I said that Hansen’s successful predictions were evidence of the lack of need for tuning in a GCM, because he didn’t do any. Lambeck and Cazenave didn’t have a GCM at all. They made an unsuccessful prediction based on over-reliance on a supposed periodicity.
Nick,
I think you meant to say that they made a successful prediction. They did successfully predict the upturn in temperature. The “gloomy trend” was the continuing cooling of average temperature.
Hi Nick,
you have avoided my main point that you try to infer GCMs tell us something by comparing to gas laws, yet this is a non sequitur fallacy. We can rely on gas laws because they have been validated by observation. This NOT true of GCMs.
ALL climate models are tuned and always have been because a lot of processes are not modelled at all and are represented by ill-constrained parameters. Hindcasts do not work first time because we have such a thorough and accurate model of the climate that they can model it from “basic physics” as some pretend to con the unwashed, but because parameters are juggled until the hindcast of the post 1960 record fits as well as possible.
It is true that at least Hansens group have thrown out physics based modelling of volcanic forcing for arbitrarily adjusted scaling more recently, so this is a situation which is getting worse not better.
further more, Hansen more recently introduced the concept of “effective forcing” where 1W/m^2 of one forcing is not necessarily the same as 1W/m^2 of another : each gains an “effective” scaling.
Whether this is legitimate physically or not, it has introduced a whole raft of arbitrary unconstrained parameters which add more degrees of freedom to the tuning process. Von Newman’s elephant is no longer just wiggle its tail, it is now able to dance around the room.
The problem with an ensemble of dancing elephants is that stage they are pounding just happens to be our physical economy, and fissures already appear!
One very specific case in point is the scaling of AOD to W/m^2.
Lacis et al did this by basic physics. The result under-estimated the effect on TLS ( and one may suggest this indicates it under-estimated the effect on lower tropo climate ).
Ten years later they dropped any attempt at calculating the forcing directly and just introduced a fiddle factor, making an arbitrary scaling to “reconcile” their model’s output with the late 20th c. surface record. This resulted in an effect twice a large as observed on TLS ( and presumably twice as strong at the surface ).
If you make the model change twice as much as observed, you have doubled the climate sensitivity to this forcing. In order to get you model to match lower climate record you will need to double a counter-acting forcing ( eg CO2 ) twice as strong to balance it. You then hope no one spends much time worrying about your failure to match TLS and pretend your model has “skill”.
Since there have been no major eruptions since Mt. P they are left with a naked CO2 forcing which is twice what it should be with no counter balance. Hence the warming is about twice what is observerd.
This is not the only problem, the whole tuning process is rigged. But I suspect this is one of the major causes of models’ exaggerated warming since 1995.
Greg,
“We can rely on gas laws because they have been validated by observation. This NOT true of GCMs.”
Two aspects to that. One is that a great deal of GCM behaviour has been validated by observation. That is the part used in numerical weather forecasting. Some still scoff at that, forgetting how vague forecasts used to be (when did you ever get a rainfall forecast in mm?). But even without that, the fact is that they do render a reasonable, tested, emulation of the whole globe weather system over that time period.
The other is that they provide an immense amount of detail which corresponds with observation. I often post this u-tube
https://www.youtube.com/watch?v=KoiChXtYxOY
Remember, GCMs are just solving flow equations, in this case ocean, subject to energy inputs and bottom topography. And they get not only the main ocean current systems right, but also the mechanics of ENSO.
I think that GCM are based on observations, not validated by them.
“Some still scoff at that, forgetting how vague forecasts used to be “
Yes, 10% chance of rain at 5.00 o’clock is so informative.
Remind me again how many millimeters that is Nick?
And will it actually rain at 5.00?
Incredible accuracy.
No “error” in that figure.
+1 Greg Goodman.
What Nick Stokes is arguing is frequently not what he appears to want you to think he is arguing.
Climate models tell you nothing about what the climate will be like in 2100
They can’t tell you anything, because the uncertainty by 2100 is enormous.
So whatever actions are taking to mitigate alleged CO2 driven climate, cannot be measured for effectiveness.
What do we know? We know the models have not been reliable to date, and the general discussion around this topic largely does not address the huge number of problems with the models, let alone the tuning.
There is little value at all in these models, as observations have shown.
It amounts to crystal ball gazing dressed up in some sort of “scientific exercise”, the uncertainty in the real world is so huge, that any actions based on this nonsense that harms people now, for a completely unknown future (and unknown results of mitigation) is ludicrous.
Climate model output is directly responsible for developing nations and 3rd world country getting investment and world bank loans for energy sources that would dramatically improve their lives.
Maybe send climate modelers to live in the same conditions so they get some perspective on how damaging this crystal ball gazing is to real people.
Lastly, Mann’s latest twitter meltdown is hilarious, Heller merely pointed out Mann was starting his doom graph from the coolest point in the data to claim doom, and Mann had a complete meltdown, talking about Russians!! 😀
I guess Mann’s problem with the Russians is that they abandoned the communism he and his colleges are trying to impose on the world by rigging climate science and hijacking the peer review process.
Hansen 2002 states that climate models can be tune to produce whatever climate sensitivity is desired. There are so many poorly constrained variables that you can easily ( intentionally or otherwise ) get the ‘right answer for the wrong reasons’ by tweaking parameters to approximately reproduce your calibration period. This does not mean you are correctly calibrated or that even the shortest extrapolation will be informative. Anyone clamouring to redesign the world economy based on non validated models which are already significantly wrong, is basing his ideas on climate models or science. That is simply a pretense.
oops: is NOT basing his ideas on climate models or science. That is simply a pretense.
“Anyone clamouring to redesign the world economy based on non validated models which are already significantly wrong, is NOT basing his ideas on climate models or science. That is simply a pretense.”
Still makes no sense – the ensemble of models validates the other dancers, right? Mann’s well known hockey stick is in fact the conductors batton.
From this self-validating ensemple, or Troupe, dances out the one true real climate?
Shades of von Hayek’s spontaneous unknowable economics, the Fable of the Bees.
“vary large eigenspace of modes”
Simple typo or something deep and profound that is beyond my pay grade?
As for the rest. It requires days/months/years of thinking. My gut feeling. Ensembles are, as Pat Frank says, great for testing. And I can believe they are good for handling slightly divergent modelling. However, I’m inclined to doubt they have the magic property of extracting truth from a menage of faulty analyses.
“Sharing a common solution does not mean that two equations share error propagation.” Correct. But neither does it mean that the equations don’t have similar/identical error propagation. If the two equations always have the same solution (as Pat would maintain?), isn’t it quite possible that they DO have similar error propagation?
“isn’t it quite possible that they DO have similar error propagation?”
Not really, because the homogeneous part is just that very simplest of equations y’=0. That has only one error propagation mechanism, the random walk cited. GCM’s have millions, and they are subject to conservation requirements. That’s one reason why random walk should always be regarded with great suspicion in physical processes. They are free to breach conservation laws, and do.
A randomly varying radiative ‘forcing’ would simplistically lead to a random walk in temperature but the larger excursions would be constrained by neg. feedbacks. It would still look and behave pretty much like a random walk.
Part of the neg f/b may be loosing more energy to space, so be careful with the boundaries of your conservation laws, Earth is not a closed system.
Thread message, IMO, no matter how much Stokes twists and turns? Models are rubbish, not fit for purpose (To change the world as we know it that is) no matter how hard you “polish” them. Even a turd can be polished, but it is still a turd!
I’ve always heard that you can’t polish a turd. Maybe you have more experience that I do. I guess there’s some Teflon polymer spray-on finish available in aerosol format from Walmarts now.
Myth Busters tried it: Busted- you can polish excrement. Why anyone would?
“So first I should say what error means here. It is just a discrepancy between a number that arises in the calculation, and what you believe is the true number.”
So it’s the difference between the cargo cultist pseudo scientific model result and the cargo cultist religious blind belief.
LMFAO – That’s IT!!
How is the postulated Soden-Held water vapor feedback mechanism implemented in the GCM’s? And is the implementation handled differently among different GCM’s?
While interesting, does it really matter whether the math is right or not since the GCM’s are not modelling this planet’s climate? Don’t they admit to not including all know forcings because they don’t know how much effect each has while they include CO2 as if they know its effect?
The GCMs results are not useful because they aren’t “of this world”.
Stokes ==> The REAL problem with GCMs is Lorentz as clearly demonstrated in the paper:
“Forced and Internal Components of Winter Air Temperature Trends over North America during the past 50 Years: Mechanisms and Implications*”
““[T]he scientists modified the model’s starting conditions ever so slightly by adjusting the global atmospheric temperature by less than one-trillionth of one degree”.
Run for 50 years, they got 30 entirely different projections for North American winter.
see the image here:
Read my essay about it about it here: Lorentz Validated at Judith Curry’s.
GCMs are extremely sensitive to “initial conditions” as demonstrated by NCAR’s Large Ensemble experiments. They are also sensitive to “processing order”.
GCMs can not and do not offer reliable projections of future climate states.
Kip,
“Lorentz Validated”
Another title might be “Ensembles validated”. Your case illustrates first the disconnect between initial conditions and later path. And it shows the variability that can result. In the case of N American winter temperatures, that is a lot. It is well acknowledged that GCMs do not reliably predict regional temperatures. The reason is that there are more degrees of freedom to vary and still comply with overall conservation laws.
Exactly , there are too many degrees of freedom to solve the set of equations. It is ill-conditioned.
This means that there are any number of solutions which will roughly fit the hindcast but we have no idea whether this is because we have the correct weighting ( parameterisation ) of various forcings or just a convenient tweaking to get the result.
As I pointed out above , if you double AOD ( volcanic ) sensitivity and double CO2 sensitivity you will get about the right hindcast while both are present. When one is no longer there ( post 1995 ) your model will run hot.
This is exactly what we see.
The problem is far more messy than just two uncertain parameters so there may be another similar reason models run hot. The current state of the GCM art is useless for even modelling known, recent climate variation.
“This means that there are any number of solutions which will roughly fit the hindcast”
You’d think this would be obvious after the 100th or so model that could hindcast yet had different results than all the others.
Greg,
“This means that there are any number of solutions which will roughly fit the hindcast”
It means it has a nullspace. That is where, for example, the indifference to initial conditions fits in. It means that you can’t pin down all the variables.
But often with ill-conditioned equations, you can determine a subset. They are rank-deficient, but not rank zero. And that is what CFD and GCM programs do. They look at variables that are based on conservation, and are properly constrained by the equations. Energy balance does not tightly constrain winters in N America. But it does constrain global temperature.
Stokes ==> Well, we disagree about that — you correctly state their CLAIM but that is not what the study actually shows. It shows that if one makes the tiniest, itty-bitty chnges to initial conditions (starting point) the resulting projections are entirely different — This illustrates that Lozentz discovered in his very first toy models — that because ot=f the inherent non-linearity in the mathematics of these models, they will be extremely sensitive to initial conditions and therefore long-term prediction/projectionof climate states IS NOT POSSIBLE.
The NCAR experiment shows this categorically.
It is nonsensical to state the the models mathematical chaos — sensitivity to initial conditions — illustrates “climate variability”.
…bad fingers!
….because of the inherent non-linearity….
….long-term prediction/projection of climate states IS NOT POSSIBLE. …
Hey Nick,
Nice general intro! But – what are you actually trying to say with respect to work of Mr Frank? What I can only see is that for some cases of differential equations error propagation/accumulation is not linear. But how that translates to earlier discussions? Are you trying to say that linear growth of uncertainty documented in the article of Dr Frank is not how the error in climate modelling behaves?
Secondly, there was mentioned here couple of times that error propagation is not the same as uncertainty propagation (or more precisely error calibration uncertainty). Do you buy that and if not why not?
“CFD solutions quickly lose detailed memory of initial conditions, but that is a positive, because in practical flow we never knew them anyway.”
Really? So there is no such thing as an initial, boundary value problem in computational fluid dynamics? And what do we mean by “quickly”?
Isn’t the 4 W/m2 uncertainty actually an indicator of our lack of understanding of how clouds behave? So it doesn’t matter what models are doing, because this uncertainty surrounds any calculations. The uncertainty increases as time passes, because time is a factor in the physical processes and the future state is dependent upon what actually happened previously. The models might happen to seem reasonable, but we don’t know exactly how the clouds can be expected to behave so the models can’t be proven to be accurately correct.
“So there is no such thing as an initial, boundary value problem”
There is, of course, and CFD programs are structured thus. But as I said about wind tunnels, the data for correct initial conditions is usually just not there. So you start with a best guess and as with GCMs allow a spin-up period at the start to let the flow settle down. There are some invariants you have to get right, like mass flux. The fact that it usually does settle down gives an indicator of the status of initial conditions. Most features don’t influence the eventual state that you want to get results from, and that is true for error in them too.
Nick, maybe you’d like to take another stab at addressing the non sequitur fallacy .
https://wattsupwiththat.com/2019/09/16/how-error-propagation-works-with-differential-equations-and-gcms/#comment-2797025
OK. Imagine we have a 1 m cube of hot steel at 200 F (the initial condition). We place this in a room at ambient conditions (70F another initial condition). The steel cube cools via natural convection and we are interested in the time history of the steel and air temperatures as well as the flow field. If we calculate this using conjugate heat transfer CFD, how long will it take before the fluid flow “forgets” it’s initial condition? Will the initial condition matter?
In a more general context, are there fluid dynamics problems for which the transient behavior depends entirely on the initial condition? Can there be multiple solutions to fluid dynamics problems for which the particular solution you obtain depends on the initial condition? What if the system has multiple phases (e.g. liquid water, air, dust particles)?
what you are describing is one cell of a GCM with totally constant surrounding cells and the rest of the system in total equilibrium. In a word irrelevant to the current discussion.
The properties of initial boundary value problems in computational physics is irrelevant to the current discussion? OK….
“are there fluid dynamics problems for which the transient behavior depends entirely on the initial condition”
I once spent far too long studying a problem of solute dispersion in Poiseuille flow. It made some sense because there was an experimental technique of instantaneously heating a layer with some radiation, and then tracking. That is for an existing steady flow, though. Generally I think that dependence is very rare. There are some quite difficult problems like just creating a splash, like this:
https://www.youtube.com/watch?v=Xl0RGPa57rI
The steel cube you describe is dominated by cooling, which has the continuity of heat conservation. Conserved quantities are of course remembered; the rest fades.
Random walks are based on the possibility of variation that occur with physical processes.
There are conservation of energy laws of particles and an entity called entropy that is inherently anti conservation of state.
There are no conservation laws as you state it in climate but I believe you can put it in as part of a computer programme ie TOA has to be conserved as a stated value.
The fact that this is totally unrealistic due to among other things cloud cover is why the computer programmes cannot predict accurately a future climate state.
Apparently everyone wants to talk about CGMs and not about whether we know enough about clouds to program the CGMs.
We don’t that is well known and one of the main reasons by GCM output is whatever it has been tweaked to produce, and not something with any objective validity.
The “basic physics ” meme is a lie.
Some time ago I downloaded a free training module from UCAR/NCAR entitled “Introduction to Climate Models”. Here is a link to a few screenshots concerning how they work. Some modeled behavior is “resolved”, i.e. each time-step results from solving a set of equations for motion, temperature, etc. for each grid-box. The rest is parameterized. Take a look. The parameterized outputs dominate those aspects of GCM’s which relate to clouds and longwave radiation, which of course is the supposed purpose of using GCM’s to simulate the climate impact of increases in greenhouse gases.
The GCM’s make sausage. Would you like it spicy hot, or milder? Take your pick, tweak the parameterizations.
So Nick Stokes may have some great points in this post, but I don’t see how it matters much, nor does it invalidate Pat Frank’s approach to determining the reliability of air temperature projections.
https://www.dropbox.com/sh/9trnmu9vepf1e2b/AAA7EZKmSmAnGVT9unkHeleYa?dl=0
Wow, it’s worse that we thought:
“when we simulate the climate system, we want no intrinsic climate drift in the model. In an process akin to calibrating laboratory instruments, modellers tune the model to achieve a steady state ”
This implies two assumptions.
1. there was some point in the measurable record when climate was in an equilibrium state, where we can initiate the “control run”.
2. Without human emissions and land use changes , climate will remain in a “steady state” and show zero trend in all major metrics over a test period.
Who are the climate change deniers in this story ?!
I think he just means they tune it against actual temperature and other measurements.
In fact it used to be argued that climate model predictions must be true because models can hindcast. The fact that lots of models with different results can also hindcast is more obvious now.
whoever argued that new nothing about modelling or fitting and degrees of freedom, ie they know nothing. Maybe it was Nobel lauriet Mickey Mann.
TallDave
Tuning the models to hindcast is essentially curve fitting. Anyone familiar with fitting high-order polynomials should be accustomed to the fact that such curve fitting does a good job on the data used for fitting, but produces unreliable results for extrapolations beyond the domain of the data used for fitting.
Some time ago I downloaded a free training module from UCAR/NCAR entitled “Introduction to Climate Models”. Here is a link to a few screenshots concerning how they work. Some modeled behavior is “resolved”, i.e. each time-step results from solving a set of equations for motion, temperature, etc. for each grid-box. The rest is parameterized. Take a look.
Interesting. I’ve had a brief look into SimMod – simplified climate model developed at the Berkeley. Authors claim that even such simplified model follows closely more advanced ones and is suitable for researches. In this model forcings are front-loaded from different RCP models. There is no specific forcing due to clouds so I suppose this forcing is lumped into non-greenhouse gases forcing. Default time for simulation run spans for over 300 years (1765-2100). Non-ghg forcing is a simple difference between total forcing frontloaded from RCP model and total forcing due to greenhouse gases. Default climate sensitivity is 1.25. Till 1850 non-ghg forcing is set up to zero. For the next 50 years there is a steady decrease of non-ghg forcing from ~0.3 W/m^2 to 0, then the downward trend continues from 1900 till 2000 when the non-ghg forcing drops to -0.4 W/m^2 and then from 2000 there is a steady increase from -0.4 to 0.2 W/m^2. Have a look at the chart.
According the the model forcing due to CO2 alone dwarfs non-ghg forcing, especially towards later years – again the graph may be helpful.
y’ = A(t)*y+f(t) ……….(1)
y(t) could be just one variable or a large vector (as in GCMs); A(t) will be a corresponding matrix, and f(t) could be some external driver, or a set of perturbations (error).
I am not sure how many ways to say this, but the case analysed by Pat Frank is the case where A is not known exactly, but is known approximately; and where the goal is to analyze the effects on knowledge of output given the approximate knowledge of A. He is not focusing, as you are, on the divergence of solutions resulting from known A and different starting values.
I am sensing that the idea that elements of A are only known to be in intervals, but not known exactly, is an idea that you have never addressed. You write as though A is either known or not known; not that its values are uncertain but likely within limits.
Lots of practical information is available with limits on accuracy: resistances and capacitances of electrical components; concentrations of active ingredients of cough medicine; milliamp hours of output from rechargeable batteries; calories in a cup of cooked rice; driving speed displayed on a speedometer. Imprecise knowledge of component characteristics leads to imprecise knowledge of the effects of using them.
Which brings us back to a question addressed by Pat Frank: what is the “probable error” of GCM forecasts? Much greater than you expect if you ignore the probable error of the parameter inputs.
Mathew RM wrote:
Untested assumptions of component characteristics leads to even greater imprecise knowledge.
For example, my understanding is that models operate on the assumption that the atmosphere is only in local thermodynamic equilibrium. I have been led to believe that real-world data shows this to be wrong.
Then according to your CGM’s Anthony’s CO2 jar experiment should have shown an increase in temperature with an increase in CO2, but the temperature did not increase. It should be easy with only one variable the CO2 ppm.
“So first I should say what error means here. It is just a discrepancy between a number that arises in the calculation, and what you believe is the true number. ”
Well, there’s the problem. What about the discrepancy between what you believe is the true number, and the physically true number? That would be a function of the physical uncertainty in your measurements underlying the parameters.
Consider the difference between Step 0 (measurement) and Step 1 (simulated). If you could measure all the parameters again in Step 1, their measurement error wouldn’t change. But since you can’t, it has to increase. It doesn’t make sense simulated values would have the same physical error at every step, just like actual measurements.
You’re proven the models are fit for making guesses about the numbers you believe are correct, but proved nothing about whether they are fit to make reliable predictions about the future states of the physical properties you’re actually measuring.
This is why there are so many abandoned models, and so many current models with different ECS — all of them can’t be right, but all of them can be wrong.
I’m comfortable with Nick Stokes explanation that cumulative error won’t send the GCMs to extremes. Now with cumulative error out of the way the only remaining reason for GCMs to be so very wrong is because the models themselves are wrong and not repairable via refinement to reduce error.
Nick doesn’t get that uncertainty is not error, and that uncertainty doesn’t affect model outputs at all.
His entire analysis is one long non-sequitur.
This was an excellent post, but it might have failed to address another source of error.
Nick mentions that a major source of error is grid discretization, and suggests using finer grids until no further changes in error are observed. This can possibly be done with weather forecasting models, where the time span being forecast is on the order of five or 10 days. If the model can be run on the computer within a few hours, a weather forecast can be generated before the forecast weather actually happens. If later observations are different than those predicted by the model “one day out”, those observations can be incorporated as initial conditions into a new model run, and the model can be corrected.
But short-term weather forecasts do tend to diverge from reality (later observed weather) within 5 to 10 days, even using relatively small grids. If a model is to be used to predict the climate 50 or 100 years from now, the number of time steps simulated needs to be drastically increased. In order to keep calculation times reasonable, that may mean increasing the grid size, since calculation time is proportional to (number of grids) * (number of time steps).
If the grid size is increased, the spatial errors per time step are likely to increase (if the temperature, pressure, humidity, wind speed, etc. at the center of a grid cell do not correspond to the calculated linearized “averages” from the edges of the grid). Since the time steps for a climate model will probably be longer than those for a weather-forecasting model, there would be greater temporal errors per time step, also propagated out over many more time steps.
This qualitative analysis has not been subjected to any differential equations, but it would be expected that random errors due to imperfect knowledge of conditions within a grid cell and those between time steps would tend to increase much faster for a Global Climate Model than for a short-term weather-forecasting model. It does not seem that Nick Stokes’ analysis has addressed this problem.
Steve Z
“In order to keep calculation times reasonable, that may mean increasing the grid size, since calculation time is proportional to (number of grids) * (number of time steps).”
I fully agree with you.
But are not we living in a perverted world where the American military will soon get a computer to simulate ultramodern nuclear power heads that climate research in fact should benefit from as well?
https://insidehpc.com/2019/08/cray-to-build-el-capitan-exascale-supercomputer-at-llnl/
El Capitan is projected to run national nuclear security applications at more than 50 times the speed of LLNL’s Sequoia system.
I’m living since very long time in Germany, but can tell you that in France, the computing power dedicated to climate research is, to say the least, incredibly low. The people there are proud to obtain, in some near future, a ‘supercomputer’ doing no more than 11 teraflops!
El Capitan will come around in a few years with 1.5 petaflops, i.e. over 100 times more.
Rgds
J.-P. D.
“It does not seem that Nick Stokes’ analysis has addressed this problem.”
Well, I did mention grid invariance in CFD. It’s a big issue in all PDE solution, including GCMs.
You’re right about the trade-off. GCMs have to run many time steps, so have to use lower spatial resolution. Grid sizes are large, usually 100 km or so horizontally. They can’t really do hurricanes, for example. But they are not trying to emulate weather.
Here are two questions for Nick Stokes and Steven Mosher concerning the scientific credibility of GCM model runs which produce exceptionally high predictions of future warming; for example, 6C of warming, a figure which Steven Mosher regards as credible.
Question #1: What kinds of evaluation criteria should be applied when assessing the scientific credibility of a climate model run which produces 6C of future warming?
Question #2: Will the basic list of evaluation criteria be different for a model run which produces 2C of future warming, as opposed to a run which produces 6C of warming?
I’ve always been concerned about how GCMs that approximate climate can be trusted very far into the future. I think that everybody would agree that the GCMs only approximate how climate will change?
If we agree on that much, then for each step of a GCM run we should be able to agree that the result of that step is also an approximation? I’m not asking about how big a bound of eror on all the paramters, just agreement that the result won’t match reality in some amount.
If we agree on that, then consider how that approximation is affected by the next step of the run?
A trivial example that came to mind based on an experience of laying a large circular wall where I was using a spirit level for each wall block. I was making sure that each wall block was level and level to each of the three blocks before it. Spirit levels are approximations on true level and is off true level by +/- .1%. Using the eye for each step of laying the wall blocks is also an approximation. My eyesight isn’t that great and I use progressive lenses so my accuracy on assessing true level by the spirit level is +/- 1%. With 100 wall blocks in the circle and using the method described, how close did my last wall block come to being level with the first block laid? Was it above or below the first one?
I think that is the sort of uncertainty that Pat Franks is trying to get at. GCMs are a tool used to measure future climate in the same way that the spirit level and my eye was a tool I used to level my circular wall blocks. To answer my question about the wall blocks you’d have to know how far off each assessment of level was and in which direction. Without that detail, you can’t really say how far off I would be. You can put a bound on how far I could be, though based on the uncertainty numbers I provided. The same could be said for GCMs.
That is what I think Pat Franks is trying to characterize in his paper. From what could see in Nick Stokes post, this aspect of the problem with GCMs is not addressed. In the comments Nick did appear to acknowledge that this type of thing is very hard to calculate.
From Nick’s respone to David:
“David,
“Now how can we quantify model uncertainty?”
Not easily. Apart from anything else, there are a huge number of output variables, with varying uncertainty. You have mentioned here tropical tropospheric temperature. I have shown above a couple of non-linear equations, where solution paths can stretch out to wide limits. That happens on a grand scale with CFD and GCMs. The practical way ahead is by use of ensembles. Ideally you’d have thousands of input/output combinations, which would clearly enable a Type A analysis in your terms. But it would be expensive, and doesn’t really tell you what you want. It is better to use ensembles to explore for weak spots (like T3), and hopefully, help with remedying.”
I believe that Pat is willing to stipulate that GCMs are all internally consistent, have a balanced energy budget, and converge on solutions. But for all that, they are still only approximations of reality. Pat is saying that the results the GCMs end up with are only approximations to what the climate will actually be and is trying to put an upper bound on how far off those final states will be. We can argue on whether the bound he is using is too large or not and can discuss how to improve the assessment of uncertainty bounding the final results.