Another uncertainty for climate models – different results on different computers using the same code

Great report and read Anthony.
Tree Rings in trees also do the same thing in a way. The roots are grounded but soil conditions under each tree can vary from tree to tree giving different results even though the trees are same species and grow right next to each other or in same area.
Like how computer models run on different software.

I don’t know that I thought much about computing errors, but it should be known by everyone who wants to be taken seriously in the climate debate, that even the approximations to the actual equations of science being modeled in the Climate models cannot be kept from diverging for very long. This is why I cringe whenever i see the modelers trying to clam that their models are using the actually scientific equations underlying earth’s climate. They aren’t and they can’t.

Now did I lambast people for years here now with the mathematical definition of chaos as used by chaos theory, which states that a system is chaotic IFF its simulation on a finite resolution iterative model develops an error that grows beyond any constant bound over time? And was this argument ignored ever since by all warmists (to be fair; our resident warmists surely had their eyes glaze over after the word mathematical)?
Yes and yes.

honest questions, I’m no longer sure that I fully understand these things:
1. the beginning of each model run is really hindcasting to tune the models??…if so, they missed that too
2. each model run is really an average of dozens/hundreds/thousands of runs, depending on how involved each model is?
3. if they hindcast/tuned the models to past temps, then they tuned them to past temps that have been jiggled and even if the models work, they will never be right?
4. the models have never been right…not one prediction has come true?
..I’m having an old age moment…and doubting what I thought I knew…mainly because it looks to me like people are still debating garbage

[snip – I nipped his initial comment, because it was wrong and gave him a chance to correct it, you can comment again too – Anthony]

Nick Stokes says:
July 27, 2013 at 11:13 am
[snip – I nipped his initial comment, because it was wrong and gave him a chance to correct it, you can comment again too – Anthony]

For a non-tech savy person like me and others too, this is undoubtedly an eye opener and vital piece of information.

Let me repeat a comment I made over a years ago:

I’ve often thought it would be interesting to run one of these models with 128 bit floating point numbers and then repeat the exercise with 64 bit and 32 bit numbers and then compare the outputs. Any differences would highlight the futility in attempting to use models to predict a chaotic system.

Terry, It would cost a lot of money to run at 128bit – one would need millions of dollars of coding, then the program would run 20 – 100x slower at 128 bit. At 32 bit you get muddy sludge in minutes with this kind of code.
The authors of the paper have done the right thing – its much cheaper to change compilers and optimization settings while staying at 64 bit.
Essentially this is the butterfly effect in numerical computing.

So it’s really important to split hairs about the differnce between ‘global forecast models’ and ‘climate models’? So, until a ‘scientist’ produces a peer-reviewed paper showing that this divergence also occurs in ‘climate models’, we can safety asssume they are unaffected? Ha!

I have been a programmer since 1968 and I am still working. I have been programming in many different areas including forecasting. If I have undestood this correctly this type of forecasting is architected so that forecastin day N is built on results obtained for day N – 1. If that is the case I would say that its meaningless. Its hard enough to predict from a set of external inputs. If you include results from yesterday it will go wrong. Period!

Ingvar Engelbrecht says:
July 27, 2013 at 11:59 am
“I have been a programmer since 1968 and I am still working. I have been programming in many different areas including forecasting. If I have undestood this correctly this type of forecasting is architected so that forecastin day N is built on results obtained for day N – 1. ”
Yes, Ingvar, weather forecasting models as well as climate models are iterative models (basically very large finite state machines; where the program describes the transition table from one step to the next).

Nick Stokes says:
July 27, 2013 at 11:30 am
“They no longer attempt to predict from the initial conditions. They follow the patterns of synthetically generated weather, and the amplification of deviation from an initial state is no longer an issue.”
That’s great to hear. As every single model run will develop an individual ever-growing discrepancy in its state from the state of the real system ( in the vector space, this can be imagined as every single model run accelerating into a different direction; picture an explosion of particles here), how is the mean of a thousand or a million model runs meaningful?
TIA

mpaul says:
July 27, 2013 at 11:26 am
Excellent! When are you selling shares? 1 penny from everyone’s bank account will make me a millionaire in no time, 0.001 of a degree each month will prove the warmists are right!

Excellent article about an excellent article. If you “looked under the hood” of a high level model and talked to the very high level programmers who manage it, you would learn that they use various heuristics in coding the model and that the effects of those heuristics cannot be separated from the model results. Run a model on different computers, especially supercomputers, and you are undoubtedly using different programmers, heuristics, and code optimization techniques, none of which can be isolated for effects on the final outcome. Problems with rounding errors are peanuts compared to problems with heuristics and code optimization techniques.
But the bottom line is that no VP of Finance, whose minions run models and do time series analyses all the time, believes that his minions are practicing science. He/she knows that these are tools of analysis only.

Frank K. says:
July 27, 2013 at 12:16 pm
“Unfortunately, Nick, climate (as formulated in most GCMs) is an initial value problem. You need initial conditions and the solution will depend greatly on them (particularly given the higlhy coupled, non-linear system of differential equations being solved).
“They follow patterns of synthetic weather”??
REALLY? Could you expand on that?? I have NEVER heard that one before…”
Synthetic weather is what we have in Virginia. Nature could not produce the rain that we have “enjoyed” this year. The synthetic “pattern” is endless dreariness.

Tom says:
July 27, 2013 at 11:53 am
Well said. And many a decision is made on the basis of overall cost.

Well, whenever I read about these computer “glitches”, a whole flood of thoughts come over me. Dirk H’s point on chaotic systems, being just one of those concerns. I think about these problems often, when driving.
It occurs to me (all the time) that all those traffic lights are programmed by the same sort of people who gave us Micro$oft Windows; the largest of all computer viruses. Well the keep sending out new viruses every few days too.
So traffic lights are programmed to answer the question: “Which car(s) should I let go (if any) ?”
This results in most traffic lights being mostly red, most of the time, and most cars standing still burning gas very inefficiently.
If they changed the algorithm, to answer the question: “Which car(s) should I stop (if any) ?”
Then most traffic lights, would be mostly green most of the time, and most of the cars would be moving (safely), and conserving gasoline.
So a lot depends not on the programmer, but on the designer of the algorithms.
Let me give you an example from Optical Ray Tracing, to demonstrate the problem.
In lens design, you are dealing with optical surfaces, that most commonly (but not always) are portions of spheres of radius (R). Now sometimes the value of R can be large, even very large.
In a typical camera lens, the difference between a surface with a radius of 100 cm and one of radius 1,000 cm is not tat much in practice. You even have surfaces of infinite radius, sometimes called “flats” or planes.
Well now we have a problem, because my keyboard doesn’t have an infinity key on it. Well not to worry, we can write a separate routine, to deal with plane surfaces. That doesn’t deal with the 100-1,000 problem.
Well instead, we recognize that it is the CURVATURE of the surface that determines the optical power; not the radius. Moreover, most keyboards, DO have a zero key, to designate the curvature of a plane.
Does it ever occur to anyone, that no matter how small a segment of a circle (or sphere), you take, whether a micron of arc length, or a light year of arc length, the curvature is still the same. it NEVER becomes zero.
Well the issue is still not solved. Suppose, I have a portion of a spherical surface of radius of curvature (R), and that zonal portion has an aperture radius of (r).
We can calculate the sag of the surface (from flat) with a simple Pythagorean calculation, which will give us:-
s = R – sqrt (R^2 – r^2) How easy is that ? ………….(1)
Well now we have a real computer problem, because if R is 1,000 mm, and r is 10 mm , then we find that sqrt is 999.95 with an error of about 1.25 E-6.
So our sag is the small difference in two large numbers; a dangerous rounding error opportunity.
So we never use equation (1) aside from the infinity problem..
I can multiply (and then divide) by (R + sqrt (R^2 – r^2)) to get :
(R^2 – (R^2 – r^2)) / (R + sqrt (R^2 – r^2)) = r^2 / (R + sqrt (R^2 – r^2))
So now I divide through by (R); well why nor multiply by (C) (= 1/R)
This gives me s = Cr^2 / (1 + sqrt (1- C^2.r^2)) ……………..(2)
So the small difference problem has vanished, replaced by the sum of two nearly equal numbers, and suddenly the plane surface is no different from any other sphere.
Most geometers might not immediately recognize equation (2) as the equation of a sphere; but to a lens designer; well we live and breathe it.
This is just one example of writing computer algorithms, that are mathematically smart, rather than the red light traffic light codes.

Nick Stokes says:
July 27, 2013 at 12:40 pm
“If you run that on another computer, the time features won’t match. You’ll see eddies, but not synchronous. That’s because of the accumulation of error. There is no prediction of exactly what the temperature will be at any point in time. But the main patterns will be the same. The real physics being shown is unaffected.”
We get whirly patterns that look like real whirly patterns, ergo we’re right? Really? That’s it? Now in that case I have a surprise for you – see first graph in the headpost – the momdel temperatures don’t look anything like real ones. Ergo your “looks about right” argument can be used by skeptics to say – hmm, yeah, ok, according to the arguments of the warmists, the GCM’s are junk because it doesn’t look right. Well thank you.

It’s disturbing to hear them blame this on rounding errors. As Ric noted all of the hardware should be conforming to the same IEEE standard which means it will result in the same precision regardless. Now the libraries and compilers are a different beast altogether and I could see them causing divergent results. Different numerical integration methods could also be at play here. Whatever the ultimate cause this is a creative definition of the word, “robust.”

Back in my 286/287 ASM days, I was playing working with the then-novel Mandelbrot set, which has incredible detail down to a gazillion decimal places. Double precision with the 287 ran only about 17 places, though. As long as I was working in the zero point something region of the set, I could get down to the sub-atomic details, but once I got above one point zero, all that fine stuff went away and I was limited to larger patterns. That’s a characteristic of floating point math, but with the models, I don’t think it makes a nickel’s worth of difference.
The models are not real climates, the initial values are not real weather values, only approximations, so computing to 40 or 50 decimal places just wastes irreplaceable electrons. Sort of a reverse of the ‘measure with a micrometer, cut with a chainsaw’ idea.
Dan Margulis, the Photoshop guru, did some work trying to see if there was an advantage by image processing in 12 bits per primary as opposed to the usual 8 bits, that’s 1024 shades of red, green, or blue, as opposed to 256 shades apiece. There’s a similarity to climate in that you have a great many little pieces adding to an overall picture. His conclusion was that examining at the pixel level, you could see subtle differences, but the overall picture looked the same. Waste of time.
The unreality of the climate models is not due to any imprecision in the calculations or to any failure to translate any particular model into verified code. It’s due to the assumptions behind the models being wrong.

A few years ago (2008), I tried to git GISS GCM Model-E, one of the climate models used by James Hansen and Gavin Schmidt at NASA, to run under Windows. Since it is written in a combination of Fortran, Perl, and Unix shell scripts, I needed to make a few, fairly simple, modifications. After simulating about 28 days of weather, it would always crash with because the radiation from some source was less than zero. Since inspection revealed that the code was full of tests for other parameters being less than zero, I assumed that these were added as necessary.
Besides the fact that it wouldn’t run, there were a number of other issues.
* Years were exactly 365 days long – no leap years
* Some of the physical constants were different than their current values
* The orbital computation to determine the distance between the Earth and the Sun was wrong
It was when I discovered a specific design error in using Kepler’s equation to compute the orbital position that I quit playing with the code. I wrote a short paper explaining the error, but it was rejected for publication because “no one would be interested”.
Ric Werme says:
July 27, 2013 at 11:35 am

I would think that IEEE floating point should lead to near identical results, but I bet the issues lie outside of that.

Yes .. except that single precision floating point numbers may use a different number of bits, depending on the compiler and the compile flags.

Jonathan Abbott says:
July 27, 2013 at 1:11 pm
“Very interesting. I manage a team of software engineers but am new enough to coding to not fully understand some of the discussion here. It appears to be suggested that unless the coding team really know their stuff, differences between models when run on different machines are to be expected. Could someone expand on this a bit or tell me where to start reading?”
As to the precision question, George E Smith put it best with his example. Yes, you must know what you’re doing when working with number formats. The first Ariane 5 rocket was lost because they copied an algorithm for the stabilization from a smaller Ariane and didn’t have the budget to re-test it. Turned out that the bigger mass of the new rocket lead to an overflow. The algorithm was fine for the small rocket but not for the big one. All that would have been necessary was going from, I think, a 16 bit integer to a 32 bit integer or something like that; I think it wasn’t even a floatingpoint number.
Didn’t test; lost the rocket.

Dennis Ray Wingo: That these people are just now studying this and figuring it out is the height of incompetence!
It does strike me as rather late in the game for this. Imagine, for the sake of argument, that a patent application or medical device application depended on the validation of this code.

Climate is a stochastic process, driven by the laws of random probability. To emulate this, a GCM will need a random number generator. There must be thousands of random number generator algorithms in use in computer systems, but none of them are anywhere near perfect. To get around this, you set constraints for the upper and lower limits of the number returned by the random number generator, and if the return value doesn’t pass the test, you ask the random number generator to try again. So to debug an unstable GCM, the first thing I’d look at is the constraints.
Some variation between identical programs running on different hardware and software platforms is to be expected. The usual way to seed a random number generator is from the system clock. Small differences in system speed can yield large departures.
Then there’s CROE, or cumulative round off error. This was a big problem back in the 1950’s but it has since been solved, so that CROE will only occur once in maybe 10 to the 20th calculations. However. A typical GCM will run on a superfast computer for 20, 30, 40 or more days and perform gazillions of calculations. Inevitably CROE is going to happen several times.So the second place I’d look at in an unstable GCM is the waypoints where numbers are examined for reasonableness.
Normally as I said the odds on a computational error are so small it’s not worth wasting time checking the results, but here the law of large numbers comes into play.

Paul Linsay says:
July 27, 2013 at 12:50 pm
[…] The climate is a very high dimensional system with severe uncertainty about the actual dynamical mechanisms, and mediocre data that covers the atmosphere and oceans very sparsely.
———————————————————————————————————
Very well stated but, also, very irrelevant.
Whether or not the models have, or are even capable of developing, skill in forecasting in no way impacts their utility as long as 9.7 out of 10 Climate Scientists (who’s owners expressed a preference) accept there output.

Matthew R Marler says:
July 27, 2013 at 1:30 pm
“Dennis Ray Wingo: That these people are just now studying this and figuring it out is the height of incompetence!
It does strike me as rather late in the game for this. Imagine, for the sake of argument, that a patent application or medical device application depended on the validation of this code.”
When you write code for medical or transportation or power plant applications you know from the start about the safety integrity level (SIL 0..4) the critical core of the application must fullfill, and the level of validations necessary to get it certified. Including code reviews, test specifications, documentation that proves that the tests have been fullfilled by the software – including signatures of the responsible persons etc etc etc. (In the case of the Ariane 5 lost I mentioned above – no human life was endangered so no biggy.)
As climate science is not directly affecting human lifes it has no such burden and doesn’t need code validation, it is all fun and games. The life-affecting decisions only come about through the ensuing policies; and what would happen if we had to validate political decisions for their impacts on human life…. probably the state would not be able to get ANY decision through the validation process.

My recollection from first classes in chaotic systems is that the starting point matters in the extreme and there are no insignificant digits. The underlying precision of the computing hardware, the floating point/integer math libraries, and math co-processors all contribute error in different ways at very small and very large numbers (regardless of sign).

I am absolutely flabbergasted !!! This is a novice programming error. Not only that, but they did not even test their software for this very well known problem.
Software Engineers avoid floating point numbers like the plague. Where ever possible we prefer to use scaled integers of the appropiate size. IEEE floating point implementations can differ within limits. Not all numbers can be exactly represented in a floating point implementation so it is not just rounding errors that can accumulate but representational errors as well.
When you do floating point ops, you always need to scale the precision of the calculation and then round to a lower precesion that meets your error bar requirements.
Thye problem is not the libraries they have used, or the differing compillers, or the hardware. It is the stupid code that makes the system non-portable.
If they can not even manage these basic fundementals. What the hell are the doing using multi-threading and multi-processing! These are some of the most difficiult Comp Sci techniques to master correctly. So I would be less than surprised if the also had race conditions throwing out their results.
Incredible!
/ikh

Just the tip of a rather large iceberg.
I know how much time and effort goes into identifying and fixing errors in commercial software of similar size to the climate models. With the crucial difference in commercial software that you can always say categorically whether a result is correct or not, which, of course, you can’t with the climate model outputs. Even so, commercial software will get released with hundreds of errors, which come to light over time.
Then the iterative nature of climate models will compound any error, no matter how minor.
As I’ve said before, baseing climate models on iterative weather models, was a fundamentally wrong decision from the beginning.

“Yes .. except that single precision floating point numbers may use a different number of bits, depending on the compiler and the compile flags.”
I believe it’s even more complex than that?
It’s a long time since I did any detailed floating point work on x86 CPUs, but from what I remember, the x87 FPU would actually perform the math using 80-bit registers, but when you copied the values out of the FPU to RAM, they were shrunk to 64-bit for storage in eight bytes. So, depending on the code, you could end up performing the entire calculation in 80-bit floating point, then getting a 64-bit end result in RAM, or you could be repeatedly reading the value back to RAM and pushing it back into the FPU, with multiple conversions between 64-bit and 80-bit along the way. That could obviously produce very different results depending on the code.
I would presume that a modern compiler would be using SSE instead of the old FPU, but I don’t know for sure.

The problems of floating point calculation (mainly rounding and losing significance by mixing very large an very small numbers as others have commented) are well-known to any competent computer scientist. For models of this complexity it is essential to have someone who specialises in numerical analysis write/check the code to avoid problems. When I worked in a University computer science department we frequently despaired about the “results” published by physicists and other scientists who were good at their subject but thought any fool could write a Fortran program full of floating point operations and get exact results.
After following many of the articles (at Climate Audit and elsewhere) about the lack of statistical knowledge displayed by many climate scientists, I am not surprised that they are apparently displaying an equal lack of knowledge about the pitfalls of numerical calculations..

Pointman says:
July 27, 2013 at 2:19 pm
“Non-linear complex systems such as climate are by their very nature chaotic,”
No. Only when they amplify low order state bits. Complexity alone is not necessary and not sufficient. The Mandelbrot equation is not very complex yet chaotic.

MarkG says:
July 27, 2013 at 2:15 pm
“It’s a long time since I did any detailed floating point work on x86 CPUs, but from what I remember, the x87 FPU would actually perform the math using 80-bit registers, but when you copied the values out of the FPU to RAM, they were shrunk to 64-bit for storage in eight bytes. So, depending on the code, you could end up performing the entire calculation in 80-bit floating point, then getting a 64-bit end result in RAM, or you could be repeatedly reading the value back to RAM and pushing it back into the FPU, with multiple conversions between 64-bit and 80-bit along the way. That could obviously produce very different results depending on the code.”
Yes. OTOH you are allowed to store the full 80 bit in RAM; the “extended” data type supported by various compilers.
“I would presume that a modern compiler would be using SSE instead of the old FPU, but I don’t know for sure.”
SSE is a newer instruction set and requires either handcoding or an according library (maybe BOOST numerical does that, I’m not sure) or a very smart compiler who recognizes opportunities for vectorizing computations; when you’re lazy and don’t have top performance needs your throwaway computations like
double a=…;
double b=…;
b *= a;
will still create instructions for the old FPU core.

There is a general theory of non-equilibrium stationary states in reproducible systems (a system is reproducible if for any pair of macrostates (A;B) A either always evolves to B or never)
Journal of Physics A: Mathematical and General Volume 36 Number 3
Roderick Dewar 2003 J. Phys. A: Math. Gen. 36 631 doi:10.1088/0305-4470/36/3/303
Information theory explanation of the fluctuation theorem, maximum entropy production and self-organized criticality in non-equilibrium stationary states
Now, the climate system is obviously not reproducible. Earth is not black as seen from outer space, although in systems radiatively coupled to their environment maximum entropy production occurs when all incoming short wave radiation is thermalized, none reflected.
Unfortunately we do not have any theory at the moment, rooted in statistical mechanics, about non-reproducible (chaotic) systems.
Therefore the guys are trying to do computational modelling based on no adequate physical theory whatsoever. That’s a sure sign of coming disaster, according to my dream book.

Berényi Péter says:
July 27, 2013 at 2:43 pm
“Unfortunately we do not have any theory at the moment, rooted in statistical mechanics, about non-reproducible (chaotic) systems.”
Again: Reproducibility (or Determinism) and Chaos do describe different aspects.
A real life chaotic system of course has a for all practical matters infinite resolution of the state word; so the exact same starting condition cannot be maintained between two trials, giving the impression of “randomness”. Randomness might be present; but is not necessary for chaos.

Pointman says:
July 27, 2013 at 2:59 pm
“DirkH,
“Non-linear complex systems such as climate are by their very nature chaotic,”
“No. Only when they amplify low order state bits. Complexity alone is not necessary and not sufficient. The Mandelbrot equation is not very complex yet chaotic.”
*************************
When you can show me the math that handles turbulence (or even the physics of clouds), I’ll retract. Beyond the math, the behaviour of climate exhibits all the characteristics of chaotic behaviour. The more iterations, the quicker is spirals out to a completely different result. Simply compare the accuracy of a 1 day forecast to a 30 day one.”
You are of course completely right for climate; but you said
“Non-linear complex systems such as climate are by their very nature chaotic”
– where “such as climate” mentions an example, so let’s reduce it to the statement
“Non-linear complex systems are by their very nature chaotic”
which is not always correct. I’m picking nits, but definitions are all about the nits. 😉

While the differences might appear as small to some, bear in mind that these differences in standard deviation are only for 10 days

So we only have about 31,755 days to go until the year 2100. So this means garbage in, garbage out into computer software handling differences = accurate IPCC projections for the year 2100. Oh, our children and grandchildren will have a field day in 2100. Historians won’t be able to type their books because of the huge belly laughs.

What a fascinating discussion and what a privilege it is to be privy to the massive crowdsourcing that is made possible by this site.
The range of expertise available and the freedom to express ones ideas (within the bounds of decency) made possible by the world’s most viewed science blog is quite breath-taking.
Thank you Anthony and all those who contribute (for better or for worse) to demonstrate the future of learning and enquiry.

Correction: Depending on the nature of a CPU erratum, it could be present in the deterministic or in the nondeterministic camp. Many CPU errata are internal race conditions inside the CPU.

“Since the software effects they observed in this study are cumulative”
Not necessarily. Models sometimes diverge and they may diverge for any number of reasons, floating point precision being the most common. However the models *should* be stable, that is return to a state (or trend) no matter any errors in the initial conditions or calculations. If the models are *not* stable then they are glorified curve fitted, unstable, extrapolations. Which they are.

davidmhoffer says:
If the programmer didn’t take errata into account, the most likely results is that they are ALL wrong.
No. Unless you are programming in Assembler language. It is generally the job of the compiller to deal with CPU bugs. A rare exception to this was the Pentium FDIV bug.
It is the job of the GCM programmer to understand the programming language guarntees and to know how to correctly perform numerical calculations accurately to the desired precission. They also need to have a good understanding of how errors can propagate.
Unless they wanted to use SIMD hardware ( Single Instruction Multiple Data ) such as SSE instructions or an FPU, then you should be using integral data types. I.e a temp of 15.11C could be stored in an interger data type as e.g. 1511. or as 151100 depending on the precession you need.
They would also need to avoid division as an intermediate operation. And you always need to be careful with division, in order to maintain precession.
Heather Brown’s comment above are spot on. We have Climate Scientists and Physicsits codimg climate models without sufficient trainning in Computer Science. No wonder thery produce GIGO results.
I tried to look at the open source climate models including GISS E. All of the models I looked at were coded in Fortran. Most used mixed versions of the language making it very difficiult to understand or reason about the computing model that they are fiollowing. Not one of them was even minimally documented and they were almost completely lacking in comments. This makes them almost undeciferable to an outsider. Fortran is an archiac language that is almost never used in the commercial world because we can get the same performance from more modern languages such as C or C++ with much better readability and easier reasoning for correctness.
We can not even check these models ourselves, because we do not have access to the hardware necessary to run them. Typically a cluster or supercomputer..
Irrespective of what we may think of the mathematical modelling in GCM’s We have a seperate and independant criticism. That they are not reproduceable across hardware platforms.
/ikh

For those of sufficient curiosity, get the old Mandelbrot set code, and set it up to use the maximum resolution of your machine. Now take a Julia set and drill down, keep going until you get to the pixels. This is the limit of resolution for your machine, if you’re lucky, your version of the Mandelbrot algorithm lets you select double precision floating point numbers which are subsequently truncated to integers for display, but still give you some billions of possible colors. The point is, every algorithm numerically approximated by a computer has errors. A computer can only compute using the comb of floating point numbers, not the infinite precision that the real world enjoys. Between every floating point number, there are a very large (how large? for the sleepless among us) number of numbers with infinite decimal places.
Solving PDEs using approximate solutions, gets you errors, period. Parameters can make the pictures look pretty, but they can’t make the solutions better. The precision isn’t there. That’s one of the reasons that “Cigar box physics” still rules. If you can’t fit the problem description and solution in a cigar box, you probably don’t understand the physics yet.
To regular readers here, note that Willis’ contributions are generally a perfect example of CBP.

DirkH says:
July 27, 2013 at 3:37 pm
Jimmy Haigh says:
July 27, 2013 at 3:25 pm
“Would the models even produce the same result when run on the same computer on different runs?”
I am sorry DirkH but you are wrong. You are assuming far too simlistic a computing model. W£hat you need to remember is that each core can do “Out Of Order” excution, as long as the operations are non-dependant. This means that numercial calculations can be re-ordered.
And that is just on a single cored cpu. Then add multi-threading on multi -cores and multi-processing across a cluster that makes up a super computer and you have a completely non-determanistic piece of hardware. it is uto the programmer to impose order. And the Climate Modelers do not have that skill.
/ikh

When I took a numerical computation class back in the 1970’s (using Fortran 4 and WATFIV) we spent weeks going over error terms and how to try to minimize them. They are generally represented by epsilon. Anyone with even a minimal background in computation would be aware of this. I’m sure there are advanced methods to try to compensate for these errors but these clowns don’t seem to find it necessary to get advice from experts in this field.
And this is even more remarkable when you consider the fact that NASA computer people must be well aware of these problems since their space craft seem to get where they are intended to go, for the most part anyway. You know NASA that where Mr Hansen used to work.

Unsurprising. For a chaotic system this behavior is completely expected. There is a mathematical theorem that errors for each run of a numerical approximation (with roundoff) there is a set of real world conditions close to the original ones that would yield exactly the same real world result. So having suchsensitivity to roundoff doesn’t add a new type of error to the models. It can be dealt with by looking at the sensitivity to initial conditions.
The real world is itself chaotic. Even if we had an absolutely perfect model of the climate on a godlike compuler with no roudoff errors, runs with slightly different initial conditions would still give you a pile of spaghetti. The pile of spaghetti is not a sign that the models are defective. The only way to get rid of the spaghetti is to find that damned butterfly and kill it.

ikh says:
July 27, 2013 at 4:51 pm

“DirkH says:
July 27, 2013 at 3:37 pm
Jimmy Haigh says:
July 27, 2013 at 3:25 pm
“Would the models even produce the same result when run on the same computer on different runs?”
I am sorry DirkH but you are wrong. You are assuming far too simlistic a computing model. W£hat you need to remember is that each core can do “Out Of Order” excution, as long as the operations are non-dependant. This means that numercial calculations can be re-ordered.”

As you say, re-ordering means non-dependancy. Assuming no CPU erratum this does not affect the outcome. Determinism is maintained.

“And that is just on a single cored cpu. Then add multi-threading on multi -cores and multi-processing across a cluster that makes up a super computer and you have a completely non-determanistic piece of hardware. it is uto the programmer to impose order. And the Climate Modelers do not have that skill.”

That’s why I mentioned race conditions as a possible source of non-deterministic behaviour.

Ah. My mind drift back to 1978 and the “State Of Computing.”
A graduate-level class, I was an undergraduate at that time, in numerical analysis where we learn about “heat”, i.e. heat buildup in the CPU and Memory and the growth of round-off error, I/O misreads and miswrites and byte misrepresentation in memory from one cycle to the next cycle.
And I thought this was going to be a class on ‘Mathematics.’
And it WAS. 🙂
Then came “Endianness”.
From Wikipedia:
“Endianness is important as a low-level attribute of a particular data format. Failure to account for varying endianness across architectures when writing code for mixed platforms can lead to failures and bugs. The term big-endian originally comes from Jonathan Swift’s satirical novel Gulliver’s Travels by way of Danny Cohen in 1980.[1]
[1] ^ a b c Danny Cohen (1980-04-01). On Holy Wars and a Plea for Peace. IEN 137. “…which bit should travel first, the bit from the little end of the word, or the bit from the big end of the word? The followers of the former approach are called the Little-Endians, and the followers of the latter are called the Big-Endians.” Also published at IEEE Computer, October 1981 issue.
🙂
Thank you Anthony for reminding me!

“As you say, re-ordering means non-dependancy. Assuming no CPU erratum this does not affect the outcome. Determinism is maintained.”
Indeed. For out-of-order execution to work effectively, particularly when running old code built for in-order CPUs, it has to produce the same results as though the program ran in-order. We realised twenty years ago that relying on the programmer or compiler to deal with the consequences of internal CPU design was a disaster; e.g. some of the early RISC chips where the results of an instruction weren’t available until a couple of instructions later, but there were no interlocks preventing you from trying to read the result of that instruction early and getting garbage instead. One compiler or programmer screwup, and suddenly your program starts doing ‘impossible’ things.

As a mathematician by training, in the 60s all were taught Numerical Analysis, the art of avoiding rounding error by hand and with those new fangled computer things! But MUCH more serious is
CHAOS Theory 1996 which showed that the integration of differentiable manifolds was unstable with respect to initial conditions. This is a proven Mathematical Theory, which means that:
Chaos: When the present determines the future, but the approximate present does not approximately determine the future.
Models RIP, MFG, omb

Someone mentioned compilers and hardware platforms as a dangerous beast for FP calculations. About 1000 years ago I was coding numerical algorithms and – when moving code from one computer to another – I got nasty surprises on FP results due to the ‘endianness’ problem (little endian vs. big endian, etc). Then I left the field but I guess some of the code embedded into the models is pretty old and is never been reviewed or adapted. I remember a routine on splines I wrote which produced absurd results depending on the hardware platform used. However astronomers in the team where using it to smooth data – despite my alert not to trust 100%- like if my routine were the word of the Lord. Sigh! So long time ago!

We are trying to model an open system. The nature and effects of the largest single known influence, the sun, are poorly understood. There may be other significant influences, not yet apprehended. The models are basically curve-fitting exercises based on precedent, faith and luck. There is no reason to suppose that any model that depends on our present state of knowledge will have reliable predictive power. It is reasonable to conclude that these horrendously expensive ongoing exercises are simply a waste of resources.
No doubt, the modellers will disagree. However, I seem to recall another serious, hotly contested debate about how many angels could dance on the head of a pin. A century ago respectable intellectuals happily agreed on the natural inferiority of the Negro. How about the briefly fashionable ‘runaway global warming’? (We’ll end up like Venus!) There may be one or two people here who would prefer to forget their involvement in that one.
It seems to be an unfortunate fact that in intellectual discourse (as in most other things) fashion, rather than rigour, is the prime determinant of of an idea’s ‘correctness’.

Worrying about the precision of calculations is a waste of time. Sure; the calculations will be more accurate but the results, however deviant, are still dependent upon a description of initial state where each parameter has in the range of 6 to 10 bits of “resolution” – before homogenisation – which tends to “dampen” the information content, reducing effective bits of state information.
Internally, the models tend to be a swill of ill-conceived code apparently written by undergraduates with little appreciation of software engineering, let alone the physical systems that they’re “modelling”. Physical “constants” that are e.g. only constant in ideal gases, are hard-coded to between 10 and 16 bits of imprecision. Real constants such as π, are not infrequently coded as a decimal (float) value with as few as 5 digits. (I knew that that was very wrong when I was an undergraduate.) Other formula constants are often lazily hard-coded as integer values. Typical for multiplication or division of DOUBLE (or better) PRECISION variables can have different results; often determined by the order in which the equations are written and subsequently compiled.
Superficially, that doesn’t make much difference. But as the GCM are iterative models, it’s the changes in parameters that determine the behaviour of the simulated system. i.e. Lots of nearly-equal values are subtracted and the result is used to determine what to do next.
Subtraction of “nearly-equal” values is also a numerical technique for generating random numbers.
I’ve previously commented on the credibility of ensembles of models and model runs.

None of them are close to the observations, so why should we care if they are close to each other? They are all wrong.

The models work just fine. They produce the propaganda they are intended to produce.

DirkH says: July 27, 2013 at 2:33 pm
… The Mandelbrot equation is not very complex yet chaotic.
The Mandelbrot equation terms are complex numbers, which should qualify it as complex.

One should read Donald Knuth’s “The Art of Computer Programming” Section 4.2.2 “Accuracy of Floating Point Numbers” in Volume 2 to get a feel for how bad it can be. It turns out it is worse when there is addition and subtraction operations on floating point than for multiplication or division and in fact the normal axioms of arithmetic do not hold, for example the associative law does not hold, eg (a+b)+c is not always the same as a+(b+c), depending on the actual values of a, b and c.
Knuth also writes about random numbers, and they are an exceptionally difficult thing to produce.
So, when I did look at some of GCM source code a few years ago, I could see that there was no attempt to track errors from the underlying floating point system and that random numbers are used widely (which by itself is an indication that the “models” are suspect). I found instances where repetitive iterations used floating point additions, and it was clear to me that these models would quickly degenerate into nonsense results.

“Climate model” is an oxymoron. There is not and never will be a valid model of the Earth’s climate. What climatologists create are climate emulators, conglomerations of algorithms that produce an output with short-run climate-like properties. The result is not Earth climate, but meaningless academic exercises whose primary use is to deceive people, including themselves.

Mike McMillan says:
July 27, 2013 at 9:01 pm

“DirkH says: July 27, 2013 at 2:33 pm
… The Mandelbrot equation is not very complex yet chaotic.
The Mandelbrot equation terms are complex numbers, which should qualify it as complex.”

….rimshot.

This error wouldn’t be possible outside of academia.
In the real world it is important that the results are correct so we write lots of unit tests. These are sets of tests for each of the important sub routines in the program where we pass in a wide range of papmeters and compare the result to the expected result. So if you moved your program to a different computer that produces slightly differing results your unit tests would fail.
Unit tests also ensure that when you make changes to the program that you haven’t broken anything. A lot of software professionals that I know wouldn’t trust the results of any program that didn’t have a comprehensive suite of unit tests.

cliff mass says:
July 28, 2013 at 8:52 am
“This is highly relevant to an initial value problem, like weather forecasting. Climate modeling is something else…it is a boundary value problem…in this case the radiative effects due to changes in greenhouse gases.”
You are claiming that there are no tipping points. “Tipping points” have been the number one mainstay of climate science for years, and still are; they continue talking about a “point of no return”.
Thanks for clarifying that official climate science now believes something else; namely a simple one to one relationship between greenhouse gases and temperature.
In that case, why do they run 3 dimensional models at all? A handkerchief should provide sufficient space for extrapolating the temperature in the year 2100.
Are you saying they -cough- are wasting tax payer money by buying themselves unnecessary supercomputers? Oh. Yes that is exactly what you are saying.

This sort of thing is very difficult to control for. See, for example Floating-Point Determinism:

Some of the additional things that you may need to worry about include:
* Compiler rules for generating floating-point code
* Intermediate precision rules
* Compiler optimization settings and rules
* Compiler floating-point settings
* Differences in all of the above across compilers
* Different floating-point architectures (x87 versus SSE versus VMX and NEON)
* Different floating-point instructions such as fmadd
* Different float-to-decimal routines (printf) that may lead to different printed values
* Buggy decimal-to-float routines (iostreams) that may lead to incorrect values

Blarney says:
July 28, 2013 at 9:18 am
“If you think that climate models are wrong, just don’t make the mistake of buying in any proposed explanation of why it is so, because this instead of making your argument stronger, it makes it weaker.”
I haven’t even begun to cite the holes in the physics in the models because I didn’t want to derail the debate.

cynical_scientist says:
July 28, 2013 at 9:31 am
“The more important question is how do you validate a model of a chaotic system? ”
-get as much data of the real system as possible for a time interval.
-initialize the model with a state that is as close to the state of the real system at the start of your reference time interval as possible.
-run the model.
-compare with what the real system did.
And that’s what climate scientists obviously never did. They do their hindcasting but they initialize with a random state. Now, is that incompetence or malice or both?

“Nope. you are forgetting that re-ordering non-dependant floating point operations can give you different rounding error. That is not deterministic.”
The compiler may well change the sequence of floating point instructions for better performance (e.g. changing a*b+c*b into (a+c)*b to eliminate a multiply), and thereby produce different results to those you expect. I’m not aware of any CPU that will do the same.

If you are writing heavy duty code, in any language, to be used for a serious purpose, then there a few tips that people need to be aware of:
Others have and still are, doing this job now, so copy the best methods.
Use approved compilers and only use subsets of your language of choice.
A typical example is MISRA C. It is not a compile but a set of ‘tests’ that you chosen compiler/CPU will ALWAYS WORK THE SAME WAY EVERYTIME.
What does this mean in practice? Common programming tricks are banned because the ‘system’ can not guarantee it will give the same result every time it runs. (Heard that one before).
MISRA C is now the accepted standard to use if you are coding for automobiles etc.
http://www.misra.org.uk/misra-c/Activities/MISRAC/tabid/160/Default.aspx
If you want to code for aircraft flight control systems, it just gets worse, from the programmers perspective, (but safer for the passenger).
DO-178B (changing to ‘version C soon) is much tougher, contains many more rule/procedure based checking details.
To prove that you have correctly used MISRA C or DO-178 (the programming rules) you use a software QA toolset. http://www.ldra.com/ produce tools that analyse your code before you even run it, getting rid of many bugs at the beggining.
If i were involved in spending 1 million on producing a GCM I would have expected a few professional programmers, a test engineer, a QA engineer to prove to the customer that what we have produced fits the bill and can be seen to do so.

For the ins and outs of rounding read this:
http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
I also not that toolset provider http://www.ldra.com/index.php/en/products-a-services/availability/source-languages do not support FORTRAN, implying that GCMs contain wholly unverified code.

Mark Negovan says:
July 28, 2013 at 6:03 am
This is one of the most important posts here at WUWT on the subject of using GCM crystal balls and is one of the first ones that I have seen here that shows the true computational insanity of all of the climate models.
…
errors and uncertainty at any time step is [are] unbounded and the error bound will increase to the extremes of where the climate has been in the past in a very short time.

Concisely! Thanks for this, Mark.
The tendency or practice of the AGW cult to hand-wave away uncertainty as fuzz that can be averaged out of existence and consideration is the fundamental falsehood and sham on which their edifice rests. In fact, the entire range of past variation is equally likely under their assumptions and procedures. Which means they have precisely nothing to say, and must be studiously and assiduously disregarded and excluded from all policy decisions.

I read all the posts until jorgekafkazar at (July 28, 2013 at 12:00 am) and finally got some insight. My understanding is that the climate emulators are run multiple times to generate a set of results from which a statistical average and standard deviation envelope are generated. As DirkH pointed out, it could also be called simulation.
Years ago I was shocked when I read references suggested on RealClimate wherein climate modelers would effuse over some physical phenomenon like ENSO or monsoonal flow when it emerged spontaneously in the model. Shiny! The use of such phenomenon has been used for countless model “validations” as well as “predictions” of the future of such phenomena. But it is 100% crap.
The climate modelers recognize that their emulations are highly dependent on parameter settings, so they choose (mostly cherry pick) a few parameters and ranges to demonstrate that the statistics don’t change very much. But AFAICS there is never any systematic validation of model parameterizations, and it may be impossible since the climate measurements used for hindcasting are quite poor in many cases.
I always believed that the demonstrated non-skill of modern ENSO prediction invalidates the climate models. Need to eliminate the inconvenient global MWP? Just “emulate ENSO” and voila, Greenland and Europe stay mostly warm but persistent La Nina compensates by cooling the rest of the globe for 100’s of years. But they are not emulating ENSO, but emulating persistent La Nina. In a narrow sense it can be validated, but only for an artificial skill in hindcasting; thus it can’t be used for predictions. ENSO is just one of the factors (albeit an important one) without which global atmospheric temperature cannot be understood.