From Lockdown Sceptics
Stay sane. Protect the economy. Save livelihoods.
An experienced senior software engineer, Sue Denim, has written a devastating review of Dr. Neil Ferguson’s Imperial college epidemiological model that set the world on a our current lock down course of action.
She appears quite qualified.
My background. I wrote software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects.
She explains how the code she reviewed isn’t actually Ferguson’s but instead a modified version from a team trying to clean it up in a face saving measure.
The code. It isn’t the code Ferguson ran to produce his famous Report 9. What’s been released on GitHub is a heavily modified derivative of it, after having been upgraded for over a month by a team from Microsoft and others. This revised codebase is split into multiple files for legibility and written in C++, whereas the original program was “a single 15,000 line file that had been worked on for a decade” (this is considered extremely poor practice).
She then discusses a fascinating aspect of this model. You never know what you’ll get!
Non-deterministic outputs. Due to bugs, the code can produce very different results given identical inputs. They routinely act as if this is unimportant.
This problem makes the code unusable for scientific purposes, given that a key part of the scientific method is the ability to replicate results. Without replication, the findings might not be real at all – as the field of psychology has been finding out to its cost. Even if their original code was released, it’s apparent that the same numbers as in Report 9 might not come out of it.
Ms. Denim elaborates on this “feature” quite a bit. It’s quite hilarious when you read the complete article.
Imperial are trying to have their cake and eat it. Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”, but at the same time, they’re fixing such bugs when they find them. They know their code can’t withstand scrutiny, so they hid it until professionals had a chance to fix it, but the damage from over a decade of amateur hobby programming is so extensive that even Microsoft were unable to make it run right.
Readers may be familiar with the averaging of outputs of climate model outputs in Climate Science, where it’s known as the ensemble mean. Or those cases where it’s assumed that errors all average out, as in certain temperature records.
Denim goes on to describe a lack of regression testing, or any testing, undocumented equations, and the ongoing addition of new features in bug infested code.
Denim’s final conclusions are devastating.
Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.
On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.
Would the same (or very similar) comments apply to Dr Mann’s work product – if anyone can find it?
M&M uncovered Mann’s amateurish attempts at PCA and statistics, to be charitable. No doubt Mann’s coding skills 20 years ago were on the same level… amateurish if even existant. Nowadays, he can hire a knowledgeable coder to create his spaghetti code. But what any major software project beyond a few 10,000 lines of code needs is a qualified software architect at the beginning to stop the spaghetti factory from getting started, and inject process, traceability, and regression testing on bug fixes/changes.
If you are going to test it you need a verifiable historical record of the last pandemic. That must include all factors. After the design phase in the specification which should include such things as known person-to-person transfer and so on. For instance you cannot know if person-to-person transmission is decreased by social distancing unless you have confirmed data in the past. I doubt we have any of that and since all models must be tested before you can rely on them this model becomes a guess.
Excerpted from article:
“WOW”, ….. now that revelation really surprises me.
And here I was sure that those “coronavirus contagion” models …… were being produced/generated by those “climate modeling” computer programs.
When the “upgraded” and “improved” code is still a piece of crap, you know the original was really bad. Frankly I’ve dealt with such code a few times in my career. Ultimately I ended up determining what the data and processing requirements were and rewriting from scratch. Some things just can’t be fixed.
Even worse than a GIGO analysis using bad data would be a model that does “Good data In, Garbage Out”….a GIGO that seems to apply to this model
Bingo!
Accurate and well stated, Paul.
I have had several similar experiences.
No amount of searching and correcting ever finds all of the bugs/errors/’incorrect mathematical formulas’. Especially as authors of spaghetti code never bother to set up proper test data to ensure the program runs correctly.
Ergo, “just run it a lot of times and take the average”; the epitome of GIGO.
The problem is compounded thousandfold when amateurs include shortcuts where program control is diverted based on some local program condition.
This is commonly called spaghetti code because there is no organization or rationale to program logic, process and order.
Spaghetti code programs get worse over time as said amateurs patch bugs/fudge errors and hardcode datums into the software as fixes. Never throwing away the bad code in favor of properly designed software/databases/’test data’.
Yes.
“Academic efforts don’t have these people, and the results speak for themselves.”
Ouch! That’ll leave a mark.
That’s why the term ‘but that’s academic’ is used to point out when something isn’t real.
Is there a “Harry – Read Me” file ?
No but there’s an “Antonia – F*** Me” file.
This comment definitely apply to ModleE …
“that’s not a problem, just run it a lot of times and take the average”
“that’s not a problem, just run it a lot of times and take the MAX”
There, fixed it…
In an article at National Review, John Fund exposes a litany of failed alarmist predictions by Neil Ferguson. (https://www.nationalreview.com/corner/professor-lockdown-modeler-resigns-in-disgrace/).
What particularly caught my attention was… “Ferguson declined to release his original code so other scientists could check his results. He only released a heavily revised set of code last week, after a six-week delay.” A very Mannly approach to rigorous science.
As an expert in vascular biology said of Ferguson, “I’m normally reluctant to say this about a scientist, but he dances on the edge of being a publicity-seeking charlatan.” Again, very Mannly.
The article concludes, “So the real scandal is: Why did anyone ever listen to this guy?”
What is it about the media that leads them to promote alarmist predictions made by “scientists” shown by experience to be serial failures? It can only be that “journalists” of the modern, post-truth era care nothing about being recognized for their insightful analysis, but are focused merely on grabbing the attention of the public, which is where the money is: click-bait and eyeballs. Unfortunately 99% of journalists give the remainder a bad name.
“Unfortunately 99% of journalists give the remainder a bad name.” Good one. –AGF
The problem? Every single journalist (and climate scientist) believes himself to be in the 1%, not the 99%. That is the central problem.
“I’m normally reluctant to say this about a scientist, but he dances on the edge of being a publicity-seeking charlatan.”
A person who does this sort of slap dash thing really doesn’t do a good job of his core ‘expertise” either. I hope there aren’t innocent souls out there who have to rely on his vascular biology skills.
Steve McIntyre, a very gentlemanly and modest fellow, having been complimented on his takedown of the hockey stick said of the consensus climate “Team” after dealing with them and their work methods:
“It seems to me that most famous “amateurs” from the past were highly professional in their field. Nor do I find invocation of their stories very relevant since the sociology of the science enterprise has changed so much.
In my opinion, most climate scientists on the Team would have been high school teachers in an earlier generation – if they were lucky. Many/most of them have degrees from minor universities. It’s much easier to picture people like Briffa or Jones as high school teachers than as Oxford dons of a generation ago. Or as minor officials in a municipal government.
Allusions to famous past amateurs over-inflates the rather small accomplishments of present critics, including myself. A better perspective is the complete mediocrity of the Team makes their work vulnerable to examination by the merely competent.
– Steve McIntyre, Climate Audit Aug 1, 2013
Similar to academics and climate scientists applying thermodynamics and heat transfer they “studied” on Wiki or a 45 minute YouTube.
https://en.wikipedia.org/wiki/Earth%27s_energy_budget
The referenced NASA energy budget is linked above.
According to the diagram 163.3 W/m^2 made it to the surface.
18.4 W/m^2 leave the surface through non-radiative processes, i.e. conduction and convection.
86.4 W/m^2 leave the surface through latent processes, i.e. evaporation and condensation.
That leaves 163.3-18.4-86.4-0.6 = 57.9 W/m^2 leaving as LWIR.
That’s it!
The energy balance is closed!
Fini!!
But what about!?
LWIR: 398.2 total upwelling – 57.9 from balance – 0.6 absorbed = 340.3??
An “extra” 340.3 W/m^2 have just appeared out of thin air!!!???
So where does this 398.2 W/m^2 upwelling “extra” energy come from?
Well, actually the 398.2 W/m^2 is a theoretical “what if” S-B heat radiation calculation for an ideal, 1.0 emissivity, Black Body with a surface temperature of 289 K or 16 C.
The SINGLE amount of LWIR energy leaving the surface has just been calculated by TWO different methods!! and then combined to effectively double the amount!!!!
398.2 is THEORETICAL!!!!!
340.3 is NOT REAL!!!
340.3 VIOLATES conservation of energy!!!!!
And, no, it is NOT measured except by the amateurs who don’t understand how IR instruments work or emissivity and assume 1.0 when it is in fact 57.9/398.2=.145.
There is no 398.2 upwelling “extra” energy, there is no 340.3 “trapping” and “back” radiating “extra” energy, no RGHE, no GHG warming and no CAGW.
BSME CU ’78 where thermo and heat transfer are formally taught, studied and applied.
And 35 years of power generation where thermo and heat transfer turn fossil fuel into electricity.
And where not understanding thermo and heat transfer equals loud unpleasant, expensive noises and unemployment.
The whole basis for global warming is where that extra energy comes from. No alarmist has ever shown where it comes from. As you say, it does not exist in reality.
The ‘extra’ energy comes from the feedback model which assumes an infinite, implicit source of Joules powering the gain. This, along with strict linearity, are the only two preconditions for using linear feedback amplifier analysis. both of which were ignored by Hansen and Schlesinger when they misapplied Bode’s analysis to the climate.
They then try to claim that the average not accounted for by the incremental analysis is the power supply. That the effect of the average is different from the incremental effect necessitating this bastardization of the theory presumes non linearity which precludes using feedback analysis in the first place! Two wrongs don’t make a right.
Not to mention the idea that you can somehow convert energy into temperature without invoking the gas law. CAGW is all a house of cards.
I want more information on Sue Denim. She gives her bio but a simple search is only getting hits on this particular article.
I am definitely curious about the history and identity of the author “Sue Denim” as well.
Say Sue Denim quickly.
Pseudonym. 🙂
Doh!
😄 Hi, cBob. Thanks for letting me know that you, too, didn’t figure that out. That was gracious of you. Hope all is well with you Mr. NOT a commie (hm…. that could be….. Mr. Natakami (hope that doesn’t mean something horrid in Japanese, lol)).🌷
I didn’t figure it out, though, clever Charles. I was very seriously searching in multiple ways for that scholar “Sue Denim.” “Susan Denim” didn’t work at all for some reason, heh.
Thanks Charles – I was a bit slow.
Crossed my mind as well….Pseudonymically speaking.
“Sue Denim” appears to be qualified to critique Ferguson’s code (i.e., her/his remarks, unless blatant lies, appear to be legit — at least, they are technically saavy enough to be worth considering), however, I don’t think that is her or his real name… .
The reason I think this is that I just found this….🤨
Sue DeNim is a pen name for a government employee who aspires to writing fiction.
July 16, 1979
https://books.google.com/books?id=oX2Qc-GVSAYC&pg=PA42&lpg=PA42&dq=Sue+DeNim,+software&source=bl&ots=laQ6AW45pB&sig=ACfU3U0Dv0NA5I9LGA–ZUKS2fu6wOfvdg&hl=en&sa=X&ved=2ahUKEwivrKK52aDpAhVLvZ4KHeUXBmQQ6AEwBXoECAkQAQ#v=onepage&q=Sue%20DeNim&f=false
Obviously a play on pseudonym. The hostage crisis has existed for years among people that want to keep their jobs, even POTUSes themselves.
Even Pierre Delecto…
Perhaps Sue Denim will identify herself or himself through these columns.
SF writer Harry Turtledove used to identify himself in his early fiction as “Tak Hallus” – the Arabic word for something like ‘pseudonym’!
Sue Denim – pseudonym?
You’ll have to be quicker yet to outpace Janice Moore, I’m always fashionably late to the party so I can see what other folks think. 😎
Pop! Am I glad to see you! :). You mentioned not feeling well awhile back and ever since, I have prayed for you. Glad to see you are, at least, well enough to comment. Better late than never! 🙂
And, thanks for the encouragement.
Take care and enjoy all those pretty Midwest wildflowers.
Cool shades, btw
In the last paragraph https://lockdownsceptics.org/code-review-of-fergusons-model/
where this crique came from,
the person did actually say
“My identity. Sue Denim isn’t a real person (read it out).”
So it required one to read the article, to grasp this fact.
That’s not his/her real name.
What is truly amazing it has taken all this time to actually have someone review the code that has cost the world trillions of dollars and it turns out it is garbage. I have not reviewed the code nor will I unless paid to do so. Yes I have over 50 years of writing code and I still get paid to write code. I still believe the focus should be on the treatment and analyzing the best possible plan for each patient. But it seems that the only thing that counts is the optics of politics. Let the bad outcomes fall where they may.
Terry Bixler
May 6, 2020 at 6:22 pm
The code itself is innocent, the guy behind no so much.
And there is more than one “guy” behind and upfront of that code.
many many many…. (It seems his girlie too)
cheers
“This revised codebase is split into multiple files for legibility and written in C++, whereas the original program was “a single 15,000 line file that had been worked on for a decade” (this is considered extremely poor practice).”
I think the common term is “spaghetti code”? Like unravelling a bald eagle’s nest I imagine, a shitty and tedious job that no one wants to do. I bet she got paid to review it lol
It was a certainty.
Great post. Thanks. Maybe we can turn her loose on climate science and its dark statistics mysteries.
https://tambonthongchai.com/2018/12/03/tcruparody/
British MPs are taking notice.
https://platform.twitter.com/widgets.js
Sue Denim – I wish she would review the climate models, i bet she would find the same or worse!
You need access to the climate models raw code – just as Mark Steyn how little forthcoming Mr Mann has been.
Harry already has – http://di2.nu/foia/HARRY_READ_ME-20.html
Surprising that this did not kill the veracity of climate models nor the many critiques of how the models break most of the rules modellers are supposed to live by.
While politicians follow the advice of the likes of Ferguson there seems little hope of sanity prevailing
“[The model] attempts to simulate households, schools, offices, people and their movements, etc.”
Taking into account how little we know about the epidemiology of the virus itself, the “model” introduces a whole additional set of arbitrary parameters. Not a way to achieve usable results. A post-normal “science” in all its ugliness.
Article Summary:
What the actual F#@% have we been doing, using this model?
500,000 deaths in the Imperial world but 30,000 or so in the real world.
Ferguson has £ Billions to a answer for along with the significant and growing number of deaths from people untreated or too frightened to seek treatment for life-threatening, non-Covid diseases.
far as i can ascertain this model was used and supported by the prof king or whatever he was that died last week to make the decision to murder healthy rare breed and peoples pets and livestock in the foot n mouth outbreak as well
animals recover and get well although it is ugly and painful while the scabs are there
what the nation lost was damned near unrecoverable bloodlines and lifetimes of breeding
his deathtolls then were orders of magnitude wrong just like now
sorry that ws lord may and ex aussie we were well rid of!
“that’s not a problem, just run it a lot of times and take the average”
Where have we seen that before? Let me think………
Head in oven + feet in liquid helium=comfy person!
One article I’m not going to drill down into. Still have PTSD from my years in the trenches…
I will just reiterate my previous response here.
The excuse for the model’s non-deterministic behavior is that the model is stochastic. OK then.
If the model is truly stochastic, it should be backed up with reams and reams of data. You can tell if a program is properly stochastic. I have to assume that Sue Denim knows the difference.
Describing a buggy apparently useless model as stochastic is simply brilliant. Don’t they say that a well developed sense of humor is the sign of a great intellect? My hat is off to whoever thought to call that model stochastic.
They use the word “stochastic” when they should use the word “non-deterministic”.
The former will give values conforming to a statistical distribution when input parameters are varied slightly. That’s ok.
The latter will give different results even for the same inputs. That is not ok. There can be non-determinism due to multi-threading but in this case it is due to bugs. That is, the code is WRONG.
I’ve written code for 30 years. I still get paid (handsomely) to code. I wrote scientific code for a PhD thesis in Astrophysics. I reviewed and modified lots of other institutions code as well. Scientific code is garbage compared to what the Free Market/industry produces using modern software engineering techniques.
I looked at the Github repository for the code even after Microsoft had come through to clean it up. Absolutely terrible and I’d fire people who produced that.
Worst of all, not only zero unit test but the author seems not to know that that has been a minimum standard for 15 years.
The politicians and bureaucracies were too incompetent to know how bad the software is – especially given the stakes involved for Billions of people and Trillions of dollars.
Anyone who did understand software engineering who promoted anything based on this garbage should be prosecuted for criminal conspiracy to defraud.
Yesterday I downloaded the source code from GitHub and fed it into Microsoft Visual Studio Community 2019. To my surprise, it compiles with only a few warnings. A quick inspection of the code reveals a number of issues, probably noted by the “Sue Denim” group as well. A couple of note:
1) There are a number of compiler / IDE warnings about type conversions (double to float, 4 byte to 8 byte) that could lead to overflow (arithmetic errors).
2) Tons of hard-coded char array declarations, i.e. “char buf[65536]” that can lead to memory management issues, and is a rather lazy coding practice. (VS2019 presents warnings to that effect, no need for the programmer to search for the problem.)
I didn’t do a thorough code inspection, as it can be difficult to evaluate code written for an unfamiliar project at the best of times, and the cluttered code makes my eyes glaze over. But it seems that a first step should be deal with the warnings from the IDE and the compiler.
This simulation could probably make for a nice science project, but it’s in no condition to be used as a basis for government policy.
Perhaps not so surprising if the following is true:
“What’s been released on GitHub is a heavily modified derivative of it, after having been upgraded for over a month by a team from Microsoft and others.”
It would be interesting to run it through an AQSC analysis. At my company we are required to do this every time we modify code before checking it back in. A lot of the time it’s just nit-picky, but from time to time it finds a real problem.
When I get some time, I’ll examine the code and conclusions of another group called “The London School of Hygiene & Tropical Medicine”. The city of Ottawa is using their models and on April 8 stated “the number of cases in Ottawa is likely between 11,000 and 34,000.”
https://ottawa.ctvnews.ca/ottawa-s-top-doctor-can-t-say-when-covid-19-will-peak-as-models-suggest-up-to-34-000-cases-in-ottawa-1.4888091
Today, May 7, the reported total in Ottawa is 1,579 laboratory-confirmed cases, and the medical officer is claiming we’re “in the post-peak period.”
It looks like the LSHTM model’s predictions are much too high:
https://www.lshtm.ac.uk/newsevents/news/2020/estimates-relative-impact-different-covid-19-control-measures-uk
I certainly would not hold up Microsoft as an example of a company which can fix buggy code.
I don’t have time now (and probably won’t have) to do a proper read of this. What I will say is that stochastic models are a Good Thing for certain problems. They are used in certain disciplines (the wikipedia article you reference is about insurance, but that’s only one of the disciplines).
You run a stochastic program multiple times, and it randomly chooses data points each time (or to be precise, pseudo-randomly, since computers can’t do real random choices). Because you choose different sets of data points on each run, you *expect* the results to differ.
Why would you create such a program? One possibility is that there is too much data to input all the data points and compute something; the program would go on forever. Another possibility is that the complete set of data points itself represents a sampling of the real world, and using the whole set would give you results that might depend too closely on what that original sample was.
Another way to think of stochastic modeling is that you have some idea what the Brazilian butterfly did, but you don’t know *exactly* what he did. So you try a bunch of different flutterings of that butterfly’s wings, so as to get an idea of the range of effects you might see, how closely or how widely they cluster around a single outcome, and just how unlikely certain events are.
Let me amplify that last point. It may be that the butterfly in Brazil doesn’t have much effect on storms in Montana. If that’s the case, then a good stochastic model should give nearly (but not exactly) the same result every time. But if that butterfly really does have a substantial effect, then the model ought to give substantially different results each time. If that happens, it’s not an indication that the model is bad, it means the initial conditions strongly affect the outcome–and you really do want to know that.
So as I said, stochastic models are a good way to understand certain kinds of situations. Whether they were in this case, I have no idea. But simply saying that the program is wrong because it gives variable results is a Bad Misunderstanding of what stochastic models are. If they give the same result all the time, then something is wrong with the program.
But — following the article, the complaint was NOT about stochastic calculations. Valid stochastic models give the same results each time if the inputs are exactly the same, no? The discussed model is reported as giving different results over multiple runs, each using the exact same input values.
I first became aware of stochastic models in the 1970s and then managed to completely ignore them so I have no first hand experience. That said:
So, a stochastic model doesn’t appear to be the same as just using statistics. If I use statistics to calculate how many phone lines are needed between two cities, I’ll get the same results every time. link
The point is that these “stochastic” models are not random. They are pseudo-random. The inputs are known and controllable – only the outputs are created by the model.
They run the model over and over with slightly different starting parameters and the output provides an envelope of scenarios.
If the envelope is narrow then the model is predicting low sensitivity to starting parameters.
If the envelope is wide then the model is predicting high sensitivity to starting parameters.
So far so good. The models are investigating our understanding of the real world and so can be compared to the real world.
BUT… this is different. The variation of up to 80,000 deaths is not due to the starting conditions. It’s due to which computer you happen to run the model on.
Now that’s the problem. Because the real world is not run on a computer (or at least, no-one is swapping the computer over day to day).
The model output is not reflecting our understanding of the real world. It is confounded by gibberish.
A related aside, is anyone using TRNGs or QRNGs to drive the random functions in statistical modelling?
Had you done a proper read, you might have saved yourself the embarrassment of writing your comment. The author clearly understands the difference between a stochastic model and a buggy mess. In a proper stochastic model, if you feed it the same random number seeds each time it will give the same result. For the code under question, if you feed it the same data each time, it gives different results.
Considering that everyone is doing their best, there is always the maxim:
“No operation extends with any certainty beyond the first encounter with the main body of the enemy.”
=======
“There is no instance of a nation benefitting from prolonged warfare.”
― Sun Tzu, The Art of War
Ironic that the crowd who scolds the rest of us about denying science seem to be the primary advocates and supporters of a methodology that fails basics standards of reproducibility.
From what I understand, the good doctor has generated hugely erroneous studies before. Someone needs to file a class action lawsuit to put him out of business.
Yes. Nobody cared to vet the doctor, the model or the results before publishing and sensationalizing the “prediction.”
That our media as a whole are incurious is not questionable. However, it appears this study influenced the lock downs in the US and UK. If this proves to be true, the government “experts” have a lot of explaining to do.
“vet the doctor” ….. very droll Jeffery; and very apposite. when you say doctor it is just PhD ..
…… Ferguson first came to notoriety in the UK foot and mouth problem. His coding caused the death of perhaps 10 million healthy animals: he was running around urging “kill the cows, kill the cows ..”
The veterinary authorities were appalled at the way Ferguson; and his then boss, RM Anderson, elbowed their way into an animal health issue; with coding; they knew nothing of veterinary issues.
The lack of subject knowledge of course has now inhibited him in any way in this current business.
Gets very funny with the latest post at “Lockdown Sceptics”
” Ferguson has been carrying on an affair with a married mother-of-two during the lockdown. Talk about breaking the social distancing rules! And the icing on the cake is that the name of his 38 year-old mistress is Antonia Staats! ”
https://lockdownsceptics.org/
“Antonia Staats, senior climate activist of avaaz.org”
https://avaaz.org/page/en/highlights/
revealed by a comment from Petit_Barde here
https://realclimatescience.com/2020/05/shutting-down-incompetent-academic-modelers/
”
Antonia Staats, senior climate activist of avaaz.org, wants to end imaginary fossil fuel subsidies. By a “staggering coincidence” she met Ferguson, the gravedigger of the world economy.
The climate clown show is a very small world.
Sue Denim is quoted: “Due to bugs, the code can produce very different results given identical inputs. They routinely act as if this is unimportant.”
I am dubious (and I’ve been programming computers for 55+ years). Programs are deterministic — if the machine they run on does not fail, the answers do not differ from run to run on “identical inputs.” (I say this in full knowledge that the origins of Chaos Theory stem from the discovery that “same inputs” may result in different results if the same inputs are “not the same” due to rounding errors — see Edward Lorenz and his study of weather in early 1960s.)
If Ferguson’s model suffers in the same way as did Lorenz’s, then this is old news. The same problem plagues also ALL weather models. If this is the main gripe against Ferguson, then I yawn and move one.
“If Ferguson’s model suffers in the same way as did Lorenz’s”
Unlikely. It may incorporate a non-linear differential equation, but is unlikely to include the kind of attractor that Lorenz showed with multiple interacting variables. I too am dubious about the “same inputs” story. And even then, the question would be “how much different?”. There is a lot missing from this account.
Agreed, there is a lot missing. Someone should get hold of the original code, and scrub it.
I, too, am skeptical about the idea of identical inputs producing dramatically different results. It is certainly possible if a random seed generator is buried somewhere in the code – and the generator is a hardware derived seed (like a disk drive randomness source). But this review doesn’t allow one to judge whether that is the case.
The differences resulting from running on single core or multi-core processors may make some sense, but the rest of the core discussion does not make sense to me.
Uninitialised stack variable used before set.
You beat me by one minute, Harry!
+1 Harry!
I can think of a few software errors I’ve made over the years that did it. Mostly not cleaning up after myself.
“…I too am dubious about the ‘same inputs’ story…”
Probably invokes a random number generator at some point, e.g., when utilizing a genetic algorithm. Code is fubar so while the results should always match with the same inputs, a coding error ruins it. Or that they thought they would be smart to include randomness rather than have a hard-wired model which was overly constrained.
“…And even then, the question would be ‘how much different?”…”
Well the article says “very.” But honestly, ANY difference is a sign of failure unless the randomness were intentional.
To quote the linked article:
Delta 80,000 deaths in 80 days is suspiciously round.
But if true then I would agree it was “very” different.
Yes but there is a possibility the code was giving cool results due to the creative use of some random number function. Makes it seem more real. I remember some things sort of like that on a CDC 6600.
No, historically even single-threaded programs have not always been deterministic, if they are badly written. At least, if you were writing in a language in which it’s possible to fail to initialize variables, and in an environment where the contents of a block of memory you are handed by the operating system (e.g. stack space or dynamically allocated memory) is not guaranteed. Modern programming languages handle these issues better than older ones did; but Ferguson was writing in C, which has been around for a long time.
Apparently significant portions were the result of running FtoC on ancient card deck-style FORTRAN.
You are right, for one-threaded programs.
For multi-threaded routines, you can easily have different results in two cases :
1. if the result integrator routine is bugged. If the order in which partial results arrive for integration is important, and you can’t guarantee a particular order, then you get different results.
2. If shared variables (global or not) between processes are used and their state is important. You can get different results dempending from which point of time the various processes modify and read them.
In both cases, it’s poor programming practice.
It appears there are cases hidden in the code where the seeds for random number generation may be set by e.g. the machine clock, rather than as definitive inputs. Also, multi threaded behaviour is clearly different. That suggests that there is insufficient care in setting up the PDE integration regimes.
The article offers examples of how deterministic code can produce non-deterministic results.
If multiple unmanaged threads are modifying shared memory, each run will reflect the time-slice that the OS provides for each thread.
There was also the mention of using a database. Databases are required to persist and return data based on queries, unless there is an explicit requirement for grouping/sorting, the order in which database rows are returned are at the discretion of the database software provider. That is, I can persist ten thousand rows in a particular order. If I select those same rows without specifying an order, the rows returned may or may not reflect the order those rows were originally stored – now add the fun of multiple threads contesting to write to the same database (yet another form of shared memory).
Looking through some of the code, there were some choice made to run certain algorithms based on available memory. C/C++ code is notorious for allowing neophyte programmers to create memory holes. If a 100 units of memory are available for allocation and a particular process requires 10 units of memory but there is no block of contiguous memory of 10 units or larger available, then that particular process would fail to execute due to lack of memory.
The amount of memory available may have differed due to competing processes or even the OS having no memory to give – part of the reason why running on different computers yields different results.
NeedleFactory posted: “Programs are deterministic — if the machine they run on does not fail, the answers do not differ from run to run on “identical inputs.”
Uhhh . . . would that statement apply to code (a “program”) that has buried within one or more calls to a random number generator function?
Well, no, but it’s a rather elementary blunder to use a PRNG without first seeding it, except in those cases when you truly want different numbers from run to run. If you DO want different results, it’s still usually best to seed the RNG with different seeds from run to run, giving you reproducibility so you can debug or understand unexpected or weird results.
I am reminded by others here that using a value before setting it can be an obscure source of non-determinism. Modern systems make this more rare; and therefore even harder to locate when it happens. One could argue that if it’s possible for a program to use a value without initializing it, either (a) the computer or (b) the program language or (c) the program-development-system on which it runs is at fault.
The issue of different results with the same inputs is likely unintentional and is a bug.
Many intelligent people who are proficient in their fields think they can write useful code without knowing more than the basics. Experience tells us amatures do not do well with in their attempts to code for complex problems.
GIGO rules!
I wish that I’d used the term ‘stochastic’ when writing a simulator to train Air Traffic Controllers! I just went a long way to set up GOOD pseudo-random number generators and a ‘normal’ distribution model for radar detection. The boss didn’t approve of me ‘messing with maths’ while I was testing the generators!
Would someone please help me understand how a software model can output different results from the same inputs? Unless there’s a random number generator in there, it should give exactly the same result for a given input — every time.
Sure. See my comment just prior to yours, then google for Lorenz, weather, chaos and butterfly. You’re in for an intellectual treat!
Yes but you qualified that the inputs were not the same in the case of Lorenz.
Exactly so.
Thus, either Ms. Demim is mistaken about so-called “same” inputs, or she is not. Either way, her allegation implies nothing noteworthy.
Thanks for your reply.
What if the model is designed as a deterministic model?
E.g., Moa above writes:
“They use the word ‘stochastic’ when they should use the word ‘non-deterministic’.
The former will give values conforming to a statistical distribution when input parameters are varied slightly. That’s ok.
The latter will give different results even for the same inputs. That is not ok. There can be non-determinism due to multi-threading but in this case it is due to bugs. That is, the code is WRONG.”
https://wattsupwiththat.com/2020/05/06/brutal-takedown-of-fergusons-model/#comment-2988042
If “Ms. Denim” is not mistaken about the inputs being the same, but rather “she” is correct that buggy code is producing non-deterministic results from the same inputs, shouldn’t that considered noteworthy on a deterministic model?
According to my readings of history, Lorenz’s simply realized (or possibly was the first to discover?) that very small input value variations, i.e. beyond the 12th decimal point, will lead to major differences in results in a chain of many calculations, each based on the results of the earlier calculations.
“Discovered” seems unlikely considering how well this is understood today for so many kinds of useful processing. Surely it was understood, as least in theory, before Lorenz was running his first simple weather model. Perhaps, however, as computers were fairly new, and with rather limited memory, he was the first to become aware of it with a practical application.
Yet another lifelong software industry professional here.
While I think the poster is spot on that it is junk code given all the flaws cited, there are legitimate cases where different results from identical inputs can be correct. Say that at one point in the model the code needs to simulate what number of people in a given locale decided to go a store on a given day. The model can’t pick an exact number, so it may, indeed, use a pseudo-random number. But a correct model cannot pick just ANY pseudo-random number. There needs to be a valid statistical basis for the range and distribution within which the number falls, and so it is not so much pseudo-random as probabilistic. Each run may then get a different results from the original identical inputs, but the reliability will tie back to how valid the math is behind each such choice. I suspect if this is what they were trying to do, there probably was just wild guessing at it at best, resulting in it being completely unfit for use.
RANDSEED (X) where X is defined, and not e.g. taken from milliseconds since boot time.
Then make sure your calls to RAND follow consistent threads of random numbers.
You can easily, if you’re in a multi-threaded environment, where the order of partial results is important.
Quite. Part of the fun with the Limits to Growth models was the order in which the 4th order Runge Kutta iterations were applied to the simultaneous higher order PDEs implicit in the model.
Uninitialized local variables will be whatever the last process left in that memory location. Depending on your program and operating system, the “last process” might not even be a part your program.
MarkW,
Quite right. To my mind, this is the most likely reason a single-threaded process will exhibit this type of behavior. I saw this exact thing recently where I work. The tester couldn’t understand why the results varied even though they ran the test the “exact” same way each time. From their description I was able to find the bug in just a few minutes because I was sure the problem was an uninitialized buffer. I was right.
And if the code was decent C++, for example, the buffer creation would have a constructor that automatically initializes the buffer to default values, eliminating the problem and the need to track down the culprit.
Emphasis on “decent”. I’ve seen really horrendous C++ and Java code too. I’d rather work on well written C code than poorly designed C++ or Java code any day.
The average of bad data and bad models is still bad
“Take Down and Pin – Susan Denim!”
I love the way ‘a team from microsoft’ immediately jumps in to try to ‘update’ this steaming heap of code….
Well, why shouldn’t they? They have the best experience.
They’ve been doing exactly that since they pinched original code from Digital Research (more than once).
M$ can’t even fix their own code. Lets not talk about Hyper-V. The company I work for want out of Hyper-V and want in with VMWare. I support that move!
This problem makes the code unusable for scientific purposes,
≠=======?==.
This exact same problem exists in climate models. Indeed they typically employ a random seed to ensure each run is different. This of course makes it impossible to test if the model is programmed correctly.
Which is in fact quite a useful feature if you are writing/hiding buggy code.
“. . . makes it impossible to test if the model is programmed correctly.”
Maybe, or maybe not:
Substitute a single defined numeric value for the output number coming from the random number generator. Run the program with this modification many times . . . see if the output value(s) is (are) the same for all runs: YES=good, NO=trash the code.
” Indeed they typically employ a random seed to ensure each run is different.”
Not in any of the codes I have examined. Climate models are usually fully deterministic. Typically, each model lab runs a pre-industrial control run (“PIControl”) for several hundred years. (This is prescribed by the CMIP.) When a prediction run is required, the initial conditions are chosen to be slightly different in each run by the simple expedient of sampling different kick-off times from the stored PIControl state data, say at intervals of every 5 years. The end-state minus the initial-state is then the reported result for each run. The high sensitivity of the models to initial conditions (aka chaotic mathematics) on its own is sufficient to ensure that that the small changes in initial conditions from the sampling of kick-off times translates into a spread of outcomes for the same imposed forcing drivers and parameters.
And now for the brutal takedown of Dr Fauci & friends.
Dr Judy Mikovits PHD, lays bare the dirty side of Dr Fauci. Circulate this widely.
https://banned.video/watch?id=5eb3062575314400169f3e6c
The original video posted to YouTube had over 1 million views before it was deleted for contradicting the World Health Organization and the NIH.
TRM May 6, 2020 at 7:38 pm
——————-
oh dear INFO WARS
where you can buy (this link contains adverts)
Leading the charge in super high quality nascent iodine, Infowars Life Survival Shield X-2 will help boost your health beyond expectations. *
A Stronger Formula From 7,000 Feet Below The Surface
Is not iodine, iodine or are we getting into the memory of water!!!!!!!!!!!
Do you have any intelligent critique of Dr Judy Mikovits’ statements?
Do you have any intelligent critique of the points raised in the documentary?
If you don’t like getting it from the “banned.video” site you can get it from “plandemicmovie.com”.
Many here wishing someone would review the GW models, however….the modelers will not allow a proper investigation and hide their inputs claiming copy write etc. I remember a loooong time ago David Evans wanted to run a few warmists highly confident assumptions, inputs and outcomes through his 1st class models and….they refused. I guess they were not so much confident as fraudulent.
I haven’t heard from Evans for a while so had a quick look…and here is an article that contains criticism of modeling amongst many other criticisms as well. An interesting article would be raw data next to homogenized “data” especially the deleted stations should be examined. I think we would all find that very interesting.
https://www.skepticalscience.com/David_Evans_arg.htm
As the tabloids noted, he was forced to resign due to unlawful entries into his model 😉
6 May: Breitbart: Who Would Buy a Used Computer Model from ‘Bonking Boffin’ Neil Ferguson?
by James Delingpole
Professor Neil Ferguson is currently the most reviled, scorned and mocked man in Britain.
Such a pity, though, that it’s for all the wrong reasons…
Being an arrogant, sexually incontinent, hypocrite is a fairly venial slip, after all, when compared to destroying the world’s fifth-largest economy – and with it the livelihoods, job prospects and prosperity of a nation of 65 million people…
Few men in recent British history, I would argue, have done more long-term damage to their country. It’s about time this was recognised by two important groups: by the politicians who, up till now, have been taking him seriously as their guru; and by the general populace, a significant portion of which has been terrified by his doom-laden prognostications into a hysteria out of all proportion to the nature of the actual threat posed by coronavirus.
Ferguson, let us not forget, has a long track record of failure…
https://www.breitbart.com/europe/2020/05/06/would-buy-a-used-computer-model-from-bonking-boffin-neil-ferguson/
7 May: Politico: What COVID-19 scientists can learn from their climate change colleagues
The epidemiologists and virologists helping governments are now thrust into the public eye — which also makes them target
by RICHARD BLACK
(Richard Black is director of the Energy and Climate Intelligence Unit. He was formerly BBCscience and environment correspondent for 12 years)
The late climatologist Stephen Schneider titled his memoirs “Science as a Contact Sport” — and for him and his colleagues, either side of the explosive 2009 U.N. climate summit in Copenhagen, life was exactly that…
And hence the defenestration of Imperial College London’s Neil Ferguson by newspapers whose comment pages speak to an abhorrence for lockdown policies. Newspapers happy to call a scientist whose advice probably saved many thousands of lives “the bonking boffin” and “Professor Lockdown.”…
Climate scientists have been at this a lot longer than their coronavirus peers. So what can the latter usefully glean from the formers’ experience?…
The detractors of climate science no longer have currency anywhere it matters (outside the White House) because their claims, whether “climate change is all natural” or “reducing emissions is economic suicide,” have been clearly shown to be wrong.
COVID-19 science, as Ferguson has just found out, may currently be a contact sport. But the experience of climate science suggests it is a sport that good scientists will eventually win.
https://www.politico.eu/article/coronavirus-what-covid-19-scientists-can-learn-from-their-climate-change-colleagues/
The Covid-19 panic-demic/scam-demic is as fake as climate change.
NY, NJ, MA, IL and CA together have more Covid-19 cases than the ENTIRE rest of the US.
NY, NJ, MI and MA together have more Covid-19 deaths than the ENTIRE rest of the US.
The top ten states have 84% of the cases and 90% of the deaths.
New York state is in sixth place among global deaths. Good job!!
Why?
BLUE states and BLUE cities with BLUE administrations.
Coincidence is cause, is that not correct?
Covid-19 is NOT a US national problem.
The US, Italy, Spain and UK together have more confirmed Covid-19 cases and more Covid-19 deaths than the ENTIRE rest of the world combined.
Covid-19 is NOT a “global” pandemic.
Shutting down the entire US and global economies was driven by the speculative and erroneous theories of “experts” that Covid-19 spread exponentially when the DATA!!!! was clearly second order.
Much like IPCC’s RCP 8.5, a computer model stacked on assumptions with a 0.1% connection to reality.
Then the public and politicians were stampeded to the precautionary principle by a fake news MSM propaganda machine intent on getting Trump.
Remember that come November, unless before then the lying, rabble-rousing, fact free, shit stirring fake news MSM left-wing propaganda machine goes “all in” and deposes Trump by fomenting a coup.
they just released some info
MORE – like 60+% of the NY cases were people who HAD stayed inside at home!
6 May: US Spectator: Neil Ferguson’s remarkable fall from grace
It’s time our tribal chief found a more sensible adviser. This witch doctor, like so many before him, has turned out to have feet of clay
by Toby Young
I originally had Neil Ferguson down as a kind of Henry Kissinger figure. The professor of mathematical biology at Imperial College London seemed to have bewitched successive prime ministers, blinding them with his brilliance. Whenever a health emergency broke out, whether it was mad cow disease or avian flu, there he was, PowerPoint in hand, telling the leaders of the United Kingdom what to do. And they invariably fell into line…
But it turns out to be less a case of Dr Strangelove than Carry On Doctor. On Tuesday night, we discovered that the furrowed-browed scientist, who has been at the Prime Minister’s side throughout this crisis, is in fact Austin Powers in a lab coat…
Truth be told, though, I’m not that bothered about the double-standards. My hope is it will be an emperor’s-new-clothes moment, breaking the spell this Rasputin figure has cast over Boris and the cabinet. God knows, there’s enough evidence that his computer model, which predicted 250,000 would die if the government didn’t place the country under house arrest, is about as reliable as Paul the Octopus.
Exhibit A is the team of scientists at Uppsala University who plugged Ferguson’s numbers into their computer in early April, at a time when many in Stockholm were hoping to frighten their government into imposing a lockdown. Imperial had provided the ammo: if the Swedish authorities continued to pursue its ‘reckless’ mitigation strategy, it was argued, the healthcare system would be overwhelmed 40-fold and approximately 96,000 would die of COVID-19 by the end of the year. By May 1, the model predicted, the death toll would be 40,000. In fact, at the time of writing, Sweden’s death toll from coronavirus is 2,854 and its hospitals are nowhere near the projected collapse.
It’s time our tribal chief found a more sensible adviser. This witch doctor, like so many before him, has turned out to have feet of clay.
https://spectator.us/neil-ferguson-remarkable-fall-grace/
Hey, Paul the Octopus had a much better record!!!!
A Henry Kissinger figure, as in ‘I wonder who’s kiss inger now.”
a new model fit for mass hysteria:
3 Apr: Nature: Special report: The simulations driving the world’s response to COVID-19
How epidemiologists rushed to model the coronavirus pandemic.
by David Adam
An earlier version of the Imperial (College London/Neil Ferguson) model, for instance, estimated that SARS-CoV-2 would be about as severe as influenza in necessitating the hospitalization of those infected. That turned out to be incorrect…
The true performance of simulations in this pandemic might become clear only months or years from now…
“Forecasts made during an outbreak are rarely investigated during or after the event for their accuracy, and only recently have forecasters begun to make results, code, models and data available for retrospective analysis,” (John Edmunds, who is a modeller at the LSHTM) and his team noted last year in a paper6 that assessed the performance of forecasts made in a 2014–15 Ebola outbreak in Sierra Leone…
Media reports have suggested that an update to the Imperial team’s model in early March was a critical factor in jolting the UK government into changing its policy on the pandemic. The researchers initially estimated that 15% of hospital cases would need to be treated in an intensive-care unit (ICU), but then updated that to 30%, a figure used in the first public release of their work on 16 March…
Ferguson says the significance of the model update might have been exaggerated…
https://www.nature.com/articles/d41586-020-01003-6
Ferguson’s old group not lockdown enough for David King & The Guardian?
4 May: Guardian: Public’s trust in science at risk, warns former No 10 adviser
Ex-chief scientific adviser sets up rival panel of experts over Covid-19 ‘lack of transparency’
by Hannah Devlin
Prompted by growing concern about the lack of transparency around the government’s Scientific Advisory Group for Emergencies (Sage), ***Prof Sir David King has convened a panel of experts that he says will act as an independent alternative.
The group, which will broadcast live on YouTube and take evidence from global experts, will hold its first meeting on Monday…
The group, which includes a range of leading scientists working across public health, computer modelling, behavioural science and intensive care medicine, aims to present the government with “robust, unbiased advice”.
King argues that the official Sage is compromised by the fact that 16 of the 23 known members of the committee, including the prime minister’s strategist Dominic Cummings, are employed by government.
The Independent Sage meeting will cover seven areas, including the criteria for lifting lockdown, testing and tracing and quarantine and shielding policies for vulnerable groups. It will formally submit its recommendations to the health and social care select committee, placing pressure on the government to explain the advice behind its lockdown exit strategy, parts of which are expected to be unveiled in the coming week.
King said that the biggest potential pitfall in weeks ahead would be to relax lockdown measures too soon and that he believes the government’s so-called five tests for whether it is safe to ease restrictions are inadequate.
“My own feeling is that the extent to which the virus is still in the population means we are not yet close … to coming out of lockdown,” he said. “Undoubtedly the biggest potential pitfall is removing lockdown too early and too quickly.”
A second peak, he said, could not only increase casualties, but could also lengthen the overall period of time before the country is able to fully exit lockdown. “If you go into a second peak, it just becomes more and more difficult to end the pandemic,” he added.
King previously held the chief scientific role, now occupied by Sir Patrick Vallance, between 2000 and 2007 and served as the UK’s climate envoy from 2013 to 2017.
The independent advisory group will include some vocal critics of the government’s Covid-19 policies, such as the global public health expert Prof Anthony Costello, as well as ***former and existing Sage experts.
https://www.theguardian.com/world/2020/may/03/publics-trust-in-science-at-risk-warns-former-no-10-adviser
***will Ferguson be joining?
This left wing nonsense SAGE group, is being chaired by the same Sir David King (adviser to Tony Blair) who predicted we would all be living in Antarctica due to run away man made global warming.
He was barmy then and clearly things have not changed.
Here is the full story from 2004.
https://www.independent.co.uk/environment/why-antarctica-will-soon-be-the-only-place-to-live-literally-58574.html
David King is the kind of “scientist” who gave morons a bad name.
Sir D King :
– are we supposed to believe such a clown ?
https://notalotofpeopleknowthat.wordpress.com/2017/04/19/does-sir-david-king-still-believe-the-drivel-he-spouted-in-2004/
Well Petit Barde the climate alarmists will believe anything their religion tells them to, so there are those people who will “believe” him. The safe option is to remain part of the realists group or normal people who study science and allow the warmists/alarmists to worship just about anybody daft enough to agree with their flawed understanding of what is actually going on in real life.
“poor praxis” are all Microsoft products with no exemption.
Gmail, chrome, and Google maps are not much better.
Neil is a scientist, not a programmer.
Two fatal errors (of several — and, of course, one is enough…):
1. Propagation of Error
(Dr. Pat Frank has explained this vis a vis the IPCC’s climate computer simulations here: https://www.youtube.com/watch?v=THg6vGGRpvA&t=7s (“Propagation of Error and the Reliability of Global Air Temperature Projections” discussed here: https://wattsupwiththat.com/2019/09/07/propagation-of-error-and-the-reliability-of-global-air-temperature-projections-mark-ii/)
2. Groupthink Due to Management’s Order to Repair Instead of Re-create
Sue Denim (https://lockdownsceptics.org/code-review-of-fergusons-model/ )
***********************
And one more thing… an all-too-human lack of moral courage by the techs:
“Baker: Did you ever have any qualms about what you were doing? … did you ever think of saying, ‘I do not think this is quite right.’ …
Porter: Yes, I did.
Baker: What did you do about it?
Porter: I did not do anything.
Baker: Why didn’t you?
Porter: … because of the fear of the group pressure that would ensue, of not being a team player.
,
(The Mind of Watergate, Leo Rangell, M.D., http://leorangell.semel.ucla.edu/published/books/mind-watergate-1980 )
well..on one side you have a wrongly observed mortality rate, on the other hand a guess about possible spread of epidemic..
it gives a raw number of deaths..
and you can play with models..with simulate social interactions , population and dynamic of epidemic..
it helps a bit to show how effective a measure to fight epidemic can be..
what amazes me is people thinking models are the actual world..
a well coded program would not really help..because actual uncertainty are huge..
and guessing is important..
if seems obvious that public transportation helps spreading epidemic..may be school …thing like that..
let make a model to confirm or quantify that…
It is worth noting a couple of things.
Firstly the author does not appear to have read or analysed the code.
Their analysis appears to be based solely on other people’s comments on github rather than an actual analysis. No examples are given of actual errors in the code, i.e. stating which lines in which files are wrong.
Secondly the main issue appears to be the non-determinisic nature of the code. Which the authors argue is a feature not a bug since the code trys to model what people will do and that is always uncertain so the best thing anyone can do is to provide an estimate plus the associate error. Plus the non-deterministic nature appears to be due to the code-writers using un-seeded random number generators (again judging from the comments in the essay). Whether or not this is a bug is open to debate but it is certainly not best practice.
Finally the estimates given using the code are in line with estimates from other modellers and also from
simple back of the envelope calculations. In the USA there are currently about 1 million cases and 70 thousand deaths. The population of the USA is about 330 million so most people have still not been infected or exposed and if we assume that eventually 10% get infected (less than on various cruise ships where the infection rate was 20%) then there will be 30 times more cases 2.1 million deaths. This is to be compared with the estimate of less than 1.5 million deaths produced by Ferguson’s model. And who is to say that a bad model will produce an over estimate of the number of deaths.
A must See Fauci should be arrested/fired immediately
https://kfyi.iheart.com/content/2020-05-06-coronavirus-watch-dr-anthony-faucis-ex-employee-tell-all/
already blocked from view
The main problem may be one that the piece doesn’t consider, that is, the excessive quest for detail under the impression that this is related to accuracy.
You see this in business models all the time. You are trying to forecast takeup of some new product type. The department starts out with a back of the envelope one liner of total sales and price.
As the investment required rises, in the effort to satisfy senior management that they are thinking rigorously, the department breaks every significant parameter down further. So they end up forecasting by product type, by region, by single, married, city, rural…. etc.
By the time this process get through, and this happened in the notorious wireless spectrum auctions during the dotcom bubble, you have a model in Excel covering hundreds of pages with extensive use of macros. Now the code will probably be unstructured and uncommented. But that is not the real problem with it.
The real problem is that it has become impossible for decision makers to have a sensible argument about the key parameters. They no longer have anything on which they can bring their knowledge and experience to bear.
Whereas at the start they could sit around a table and argue whether those sales estimates were reasonable, and draw on examples from experience, they could happen in this or that mix or this or that blend of different prices, now they sit staring at the output from a model they have not seen and cannot understand while some bright spark from Finance or Marketing explains to them that this is what the model shows. And offers of proof of its legitimacy that everything has been carefully modelled down to the last detail.
As in Ferguson’s model – its modelling, apparently, hotels differently from other vectors. When you read that, you know immediately that we are in the realm of arbitrary assumptions at an excessive level of detail. But you can’t see how arbitrary all the assumptions are, because they are buried somewhere in pages of code, and you can’t question the result without having to question all the detail, which you can’t get at. Certainly not in a meeting or reasonable length.
Any model which is going to be used as the basis for public policy should fit on one A4 and have no macros or VB in it. Then managers can actually bring their experience and intuition to bear on the key drivers. The way its usually done, and the effect of the Ferguson model, is to turn experienced and qualified people into Yes/No switches.
Its tough for generalists to get their heads around this stuff. A friend of mine tried to overcome the difficulty by using Crystal Ball. It did not help. The problem was it required management to have intuitions about the shape of the probability curves of various parameters. They were very competent people, quite used to arguing through the case for various outcomes. But they didn’t have intuitions expressed in this form. He dropped it and went back to a simple spreadsheet model, expressed in terms they were used to.
If by the way anyone wants to do what CB does, but without the huge outlay, you can do something similar in Gnumeric. A lot more technical, but it can be done.
A very good insight into the problem, michel, thanks.
Dr. Neil Ferguson and his gang have been way off in near all of their work, look how accurate they were with Foot and Mouth (killing large numbers of uninfected and unexposed animals), Bird Flu (going by their figures most of us should be dead already), Swine Flu (same as Bird Flu) CJD (we should all be infected to some degree) and several others how and why the UK Government still uses Imperial College makes one wonder what friends in high places or what information they hold over people?
Going by their track record (similar to global warmists) they’ve missed the mark consistently. In business they’d be broke within days.
James Bull
I have been a programmer for 30 years as well, and I’m struggling to understand how a program can give different outputs for identical inputs. One of the things I had to do was debug code, and the one thing we could rely on was the fact that if you took a copy of the original input, and made sure the system date was set to the date of the original run, the output would be the same. The code is the same, no matter how buggy, and programs are deterministic, so the same input values must create the same output values each time. In fact, had such a thing happened, where everything was identical except for the results, I would have to conclude that the laws of physics are not invariant over space and time. Crazy stuff.
As an aside, I know that climate models often give hugely divergent results from small differences in starting values. But that reinforces my point: the starting values had to differ.
How about a random number generator somewhere using the cpu clock time as a seed?
In that case, how would the random number generator output not be considered an input? In other words, not all the inputs would be the same from one run to another.
One varying value can produce widely differing results if there are enough later steps making progressive calculations based on earlier steps’ results.
If they are using a random number generator against an assumed probability distribution, this wouldn’t be so. There may be no one starting value. If one of the inputs variables is defined as a random pick from a normal distribution with mean and SD of a given value, every time you run it, it will have different inputs. Now, run it long enough, and then successive runs should converge on the same output values. These will be probabilistic of course. But every run, especially with lower numbers of trials, may be different.
The potentially valuable thing about this approach is that it will allow you to model how the probability distributions of the variables interact in a total scenario. But its very hard to get generalists, politicians or managers, to have reliable intuitions about probability distributions. And if all they do is accept your suggestions, your modelling has actually subtracted value.
Vincent,
Uninitialized variables and/or buffers. This is quite easy to do in C, and since local variables are on the stack, they are not set to zero when the program starts. The stack contents are typically not initialized either, so depending on what processes were/are running on the machine, the stack area can literally contain anything and can be different run from run. I’ve seen this exact failure more many times in my career.
In one part of my career I wrote test software used to test various digital electronics. As with all human endeavour from time to time we had to fix. My boss never liked my response to his usual question of “Is it working now?” of “It seems to be”. I could have added until we find the next bug but that would really have upset him.
One fault in a compenent from one manufacturer took about 18 months to identify, thanks to a chance remark that it “only does it when”. But what was interesting was how difficult it was to persuade the manufacturer that his component was faulty. Americans going into meltdown and banging tables a feature of one high level meeting. Demonstrating to engineers was a lot easier.
The end result was a lot of expensive scrap and a different set of people to deal with. Hopefully, at least one person less inclined to shoot the messenger
Stop using Imperial garbage !
Adopt the International System of Units !
🙂
I just read the original thread on this by Sue Denim.
On that thread (and here) are people with misconceptions about how stochastic models run in computer code.
The stochastic model element runs via a random number generator. However, a random number generator is not truly random. It is an algorithm that generates a sequence of numbers that appear to be random when subjected to statistical tests. The output sequence is usually referred to as “pseudo-random”.
However, and this is the key point for software testing and reproducibility, the random number generator algorithm is initiated be a seed number (typically a large, odd, integer eg 385671). When given the same start seed, the random number generator produces the same sequence of pseudo-random numbers.
So for testing code which contains random number generators, it will produce the same results if the same seed is used. This is critical for checking the code. Ferguson’s code appears to be producing different results even when the same seed is used for initialisation. This is very bad.
If the code is running correctly, then multiple runs of the code with either different start seeds (not ideal, but commonly used) or looping many times from the same initial seed/generator combination (much better) produces multiples stochastic simulations, an “ensemble” which can then be analysed statistically. As an aside, generating random seeds is a bad idea as it prevents both reproducibility and tracking down bugs. It also makes the simulation (technically, usually trivially) less random than using a single seed and a long sequence generator.
My background is in stochastic seismic inversion simulation – I have designed and built several very large scale and mathematically complex 3D spatially coupled stochastic simulation models using geostatistical techniques. Similar techniques are used in Oil Industry stochastic (static) reservoir model schemes for simulating geology, again an area in which I have a high level of expertise.
Ironically, I also taught geostatistics as a Visiting Lecturer at Imperial College for 14 years. So when I see Ferguson’s code from Imperial cannot reproduce the same output from the same input seed red flags are out immediately.
The similarities with the mess of climate models and code is also apparent.
“The similarities with the mess of climate models and code is also apparent.”
somebody has not looked at GCM code..
Go get MITGCM.
you wont
Best-written GCM codes still can’t predict air temperature.
Admit that.
You won’t.
That reminds me, Steve and the other trolls quite frequently declare that only those they recognized as having expertise in an area have a right to comment.
Based on that logic, what business does anyone who doesn’t have a degree in computer science have trying to write code?
In the area of modelling , garbage in , garbage out is a well known idea .
But there is another one , good date in treated like garbage results in garbage out.
So even when you feed a model good data , if the manner in which you use it is poor , you lose rather than gain value from the results.
And these ideas are true of ANY MODELLING approach no matter what is used for , even the born perfect ‘science ‘ of climate doom and we known this partly how often they been proved WRONG.
I have great problems with the Imperial team’s modelling. However I have not seen a single criticism of the modelling in Sue Denim’s paper.
You have got to distinguish between the model (an attempt at a mathematical idealisation of reality, probably including some randomness) and the code, which is an attempt to implement that model.
It is one of the major fallacies of the 21st century to confuse these two.
Indeed, I would argue that “ensemble modelling”, as perpetrated by climate scientists, is, in part, an example of that fallacy in practise.
From the Lockdownsceptics page is an interesting comment by commenter Anne which rather obviates the concerns about Ferguson’s code. Previous modelling predictions from Ferguson’s code are quoted as:
Bird Flu = prediction = 200m globally. Actual = 282
Swine Flu prediction = 65,000 UK. Actual = 457
Mad Cow prediction = 50-50,000 UK. Actual = 177
On Mad Cow, the late Christopher Booker in “Scared to Death” pointed out that more farmers died by committing suicide over government policy to tackle Mad Cow than actually died from the disease itself.
The real problem for Ferguson and the government (and now the population at large) is that the ability to model these diseases appears to be lacking and unproven and yet huge decisions are based on “stochastic ensemble” output of the computer simulations. Climate models anyone? Academics are very good at dressing up any old crap as pseudo-science and government Ministers and Civil Servants are stupid enough to believe it.
Just had a quick look, its 99% C code that is compiled as C++ and uses multithreading and multidimensional fixed arrays- collectively a guaranteed way to shoot both feet off with one bullet every time.
There are these things called scaling frameworks that take care of the fiddly stuff in ensuring that things run in parallel without depending on the phase of the moon to complete reliably & repeatedly. The original coder had only to go down the corridor to the Comp Sci Dept and ask… Cobbling together your own thread handling framework and making data exchange work reliably is something the best of us have trouble getting right.
The code should be put out of its misery and deleted. The original coder should take up knitting.
Like the National Academy of Sciences recommendation that the writing of climate science papers should include collaboration with a decent statistician in order to eliminate some of the many statistical faux pas often found therein?
That must have been before Mann was elected to the NAS?
even more exaggerated than the Imperial College model was the one Harvard’s Marc Lipsitch spread around via a media call, as follows. 2% fatalities from 70% of US population infected would have been approx. 4.6m deaths, tho no media ever bothered to calculate this.
22 Feb: WaPo: Coronavirus outbreak edges closer to pandemic status
By Carolyn Y. Johnson, Lena H. Sun, William Wan and Joel Achenbach; Min Joo Kim in Seoul, Amanda Coletta in Washington and Chico Harlan and Stefano Pitrelli in Rome contributed to this report
Harvard epidemiologist Marc Lipsitch estimates that 40 to 70 percent of the human population could potentially be infected by the virus if it becomes pandemic. Not all of those people would get sick, he noted. The estimated death rate attributed to covid-19 — roughly 2 in 100 confirmed infections — may also drop over time as researchers get a better understanding of how widely the virus has spread…
https://www.washingtonpost.com/health/coronavirus-outbreak-edges-closer-to-pandemic/2020/02/21/03afafc0-5429-11ea-9e47-59804be1dcfb_story.html
shortly after the above was published, Lipsitch was asked by Harvard’s Feldman, a Democrat, (who made an appearance at the Trump impeachment inquiry) if WaPo had quoted him correctly as to 40 to 70% of the global population getting infected; Lipsitch said it was roughly correct, but he should have said of the adult population and he was making amends, but didn’t say where or how:
AUDIO: 25m49s: 28 Feb: Stitcher: Deep Background with Noah Feldman (Harvard)
The Coronavirus Isn’t Going Away
Marc Lipsitch, an epidemiologist at Harvard University, predicts that between 40 to 70 percent of adults in the world will become infected with the coronavirus.
https://www.stitcher.com/podcast/pushkin-industries/deep-background-with-noah-feldman/e/67663436
nonetheless, Lipsitch was still pushing the “world’s population” meme days later:
2 Mar: CBS: Coronavirus may infect up to 70% of world’s population, expert warns
by Jim Axedrod
CBS News spoke to one of the country’s top experts on viruses, Marc Lipsitch from Harvard University, who cautions that 40-70% of the world’s population will become infected — and from that number, 1% of people who get symptoms from COVID-19, the disease caused by the coronavirus, could die…
If it really does spread as widely as that projection says, and that’s what I think is likely to happen, then there are gonna be millions of people dying. And I don’t think there’s any way to get around that…
https://www.cbsnews.com/news/coronavirus-infection-outbreak-worldwide-virus-expert-warning-today-2020-03-02/
same date he was tweeting:
Tweet: Marc Lipsitch, Harvard T.H. Chan School of Public Health
Because I am now less certain of where the R0 will end up (and how it may vary geographically) I am going to revise downward the range of outcomes I consider plausible to 20%-60% of adults infected. This involves subjectivity about what range of R0 may turn out to be true.
3 Mar 2020
To preempt the critique that the earlier figures were alarmist: I update my beliefs when the available data change, as any rational person would do. The available data are pointing to a different (and better) outcome than before. So I’m updating…
3 Mar 2020
Tweet: Summary: Should have said 40-70% of adults in a situation without effective controls.
25 Feb 2020
Postscript: My original quote was in the @wsj which I thought had huge circulation. Around the same time I said the same to the @TheAtlantic. The WSJ article made some ripples, but the Atlantic one went completely viral. Not what I expected.
25 Feb 2020
https://twitter.com/mlipsitch/status/1234879949946814464?lang=en
The Atlantic article that went viral:
24 Feb: The Atlantic: You’re Likely to Get the Coronavirus
Most cases are not life-threatening, which is also what makes the virus a historic challenge to contain.
by James Hamblin
The Harvard epidemiology professor Marc Lipsitch is exacting in his diction, even for an epidemiologist. Twice in our conversation he started to say something, then paused and said, “Actually, let me start again.” So it’s striking when one of the points he wanted to get exactly right was this: “I think the likely outcome is that it will ultimately not be containable.”…
Lipsitch predicts that within the coming year, some 40 to 70 percent of people around the world will be infected with the virus that causes COVID-19…
https://www.theatlantic.com/health/archive/2020/02/covid-vaccine/607000/
27 Mar: UK Independent: Coronavirus: Dr Deborah Birx making ‘fundamental scientific errors’ in rush to reopen US, warns expert behind White House data
Marc Lipsitch condemned Deborah Birx for presenting a best-case scenario as likely
by Andrew Naughtie
A leading US epidemiologist has accused one of the doctors on the White House’s coronavirus task force of “false reassurance” after she said a model he helped develop to predict the spread of the virus overstated the number of people likely to develop Covid-19 – when in fact it referred to something more like a best-case scenario.
Marc Lipsitch, a professor of epidemiology at Harvard University, has previously criticised the US government for a “feckless” response that has failed to slow the epidemic’s progress, and called for intense social distancing policies coupled with a “massive expansion” in testing capacity…
In response, Professor Lipsitch wrote that “Our modeling (done by @StephenKissler based on work with @ctedijanto and @yhgrad and me) is one of the models she is talking about” – and that on that basis, he found Dr Birx’s explanation misleading…
Johns Hopkins public health academic Tom Inglesby: “Anyone advising the end of social distancing now, needs to fully understand what the country will look like if we do that. COVID would spread widely, rapidly, terribly, could kill potentially millions in the year ahead with huge social and economic impact across the country.”…
https://www.independent.co.uk/news/world/americas/coronavirus-harvard-scientist-deborah-birx-scientific-error-a9429516.html
don’t know if this involves the above modelling or not:
14 Apr: Science: Report: Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period
Stephen M. Kissler1,*, Christine Tedijanto2,*, Edward Goldstein2, Yonatan H. Grad1,†,‡, Marc Lipsitch2
1Department of Immunology and Infectious Diseases, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
2Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
https://science.sciencemag.org/content/early/2020/04/24/science.abb5793
30 Mar: Wired: The Mathematics of Predicting the Course of the Coronavirus
Epidemiologists are using complex models to help policymakers get ahead of the Covid-19 pandemic. But the leap from equations to decisions is a long one.
by Adam Rogers, Megan Molteni
That’s where Chris Murray and his computer simulations come in.
Murray is the director of the Institute for Health Metrics and Evaluation at the University of Washington. With about 500 statisticians, computer scientists, and epidemiologists on staff, IHME is a data-crunching powerhouse…
You can see how fiddling with the numbers could generate some very complicated math very quickly. (A good modeler will also conduct sensitivity analyses, making some numbers a lot bigger and a lot smaller to see how the final result changes.)
Those problems can tend to catastrophize, to present a worst-case scenario. Now, that’s actually good, because apocalyptic prophecies can galvanize people into action. Unfortunately, if that action works, it makes the model look as if it was wrong from the start. The only way these mathematical oracles can be truly valuable is to goose people into doing the work to ensure the predictions don’t come true—at which point it’s awfully difficult to take any credit…
Speaking at a White House briefing on Thursday, Deborah Birx, response coordinator for the Coronavirus Task Force, admonished the press against taking those models too seriously, even as New York governor Andrew Cuomo begged for federal help with acquiring ventilators and protective equipment for health care workers. “The predictions of the models don’t match the reality on the ground,” Birx said.
Responding to Birx in a thread on Twitter, Harvard infectious disease epidemiologist Marc Lipsitch said Birx had been talking about work from his lab, which the federal government had asked for two days prior. In a preprint (so not peer-reviewed), his team had used an SEIR model with numbers tweaked to simulate the tightening or loosening of social distancing measures, as well as a potential flu-like seasonal variation in Covid-19 infections. He was varying R0, essentially. In the model, putting a stop to strict social distancing (without something like a vaccine or a cure coming along) allowed infections to climb right back up to their peak of about two critical cases per 1,000 people—which could be 660,000 Americans getting seriously ill or dying. And even with the strictest lockdown-type measures lasting from April through July, his team’s model finds that the disease surges back in autumn…
If Lipsitch’s team is right, the characteristics of Covid-19 might require a cyclical flux between strict social distancing and viral resurgence, on and on, perhaps until 2022…
https://www.wired.com/story/the-mathematics-of-predicting-the-course-of-the-coronavirus/
Lipsitch criticises IHME’s less exaggerated model in mid-April, because Trump!
17 Apr: StatNews: Influential Covid-19 model uses flawed methods and shouldn’t guide U.S. policies, critics say
By Sharon Begley; Helen Branswell contributed reporting
A widely followed model for projecting Covid-19 deaths in the U.S. is producing results that have been bouncing up and down like an unpredictable fever, and now epidemiologists are criticizing it as flawed and misleading for both the public and policy makers. In particular, they warn against relying on it as the basis for government decision-making, including on “re-opening America.”
“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington…
The IHME projections were used by the Trump administration in developing national guidelines to mitigate the outbreak. Now, they are reportedly influencing White House thinking on how and when to “re-open” the country, as President Trump announced a blueprint for on Thursday.
The chief reason the IHME projections worry some experts, Etzioni said, is that “the fact that they overshot will be used to suggest that the government response prevented an even greater catastrophe, when in fact the predictions were shaky in the first place.” IHME initially projected 38,000 to 162,000 U.S. deaths. The White House combined those estimates with others to warn of 100,000 to 240,000 potential deaths…
Believing, for instance, that measures well short of what China imposed in and around Wuhan prevented a four-fold higher death toll could be disastrous…
There are two tried-and-true ways to model an epidemic. The most established, dating back a century, calculates how many people are susceptible to a virus (in the case of the new coronavirus, everyone), how many become exposed, how many of those become infected, and how many recover and therefore have immunity (at least for a while). Such “SEIR” models then use what researchers know about a virus’s behavior, such as how easily it spreads and how long it takes for symptoms of infection to appear, to calculate how long it takes for people to move from susceptible to infected to recovered (or dead).
“The fundamental concept of infectious disease epidemiology is that infections spread when there are two things: infected people and susceptible people,” Lipsitch said…
IHME uses neither a SEIR nor an agent-based approach. It doesn’t even try to model the transmission of disease, or the incubation period, or other features of Covid-19, as SEIR and agent-based models at Imperial College London and others do…
https://www.statnews.com/2020/04/17/influential-covid-19-model-uses-flawed-methods-shouldnt-guide-policies-critics-say/
when Stanford University epidemiologist John Ioannidis published this:
17 Mar: StatNews: A fiasco in the making? As the coronavirus pandemic takes hold, we are making decisions without reliable data
By John P.A. Ioannidis
At a time when everyone needs better information, from disease modelers and governments to people quarantined or just social distancing, we lack reliable evidence on how many people have been infected with SARS-CoV-2 or who continue to become infected. Better information is needed to guide decisions and actions of monumental significance and to monitor their impact.
Draconian countermeasures have been adopted in many countries. If the pandemic dissipates — either on its own or because of these measures — short-term extreme social distancing and lockdowns may be bearable. How long, though, should measures like these be continued if the pandemic churns across the globe unabated? How can policymakers tell if they are doing more good than harm?…
The data collected so far on how many people are infected and how the epidemic is evolving are utterly unreliable…
https://www.statnews.com/2020/03/17/a-fiasco-in-the-making-as-the-coronavirus-pandemic-takes-hold-we-are-making-decisions-without-reliable-data/
Lipsitch was ready to pounce the very next day:
18 Mar: StatNews: We know enough now to act decisively against Covid-19. Social distancing is a good place to start
By Marc Lipsitch
In a recent and controversial First Opinion, epidemiologist and statistician John Ioannidis argues that we lack good data on many aspects of the Covid-19 epidemic, and seems to suggest that we should not take drastic actions to curtail the spread of the virus until the data are more certain.
He is absolutely right on the first point…
We spoke by phone on Tuesday, not long after his article appeared, and found that we had more in common than it appeared when I first read it…
https://www.statnews.com/2020/03/18/we-know-enough-now-to-act-decisively-against-covid-19/
the following day CBC weighed in:
19 Mar: CBC: Prominent scientist dares to ask: Has the COVID-19 response gone too far?
Leading epidemiologists publish duelling commentaries, igniting debate on social media
by Kelly Crowe
It’s a clash of titans — an epic battle between two famous scientists over the world’s response to the COVID-19 pandemic.
In one corner, influential Stanford University epidemiologist John Ioannidis, who wrote a commentary asking whether taking such drastic action to combat the pandemic without evidence it will work is a “fiasco in the making.”
Across the mat, prominent Harvard epidemiologist Marc Lipsitch punched back with a defiant response titled: “We know enough now to act decisively against COVID-19.”
Watching from the sidelines? Everybody else…
https://www.cbc.ca/news/health/coronavirus-covid-pandemic-response-scientists-1.5502423
CBC definitely weights the article against Ioannidis.
this may be of interest
https://www.sciencedirect.com/science/article/pii/S1755436518300306
BBC pandemic experiment
Would be intersting for insurance companies to offer products related to ‘climate change’, ‘coronavirus’ etc.
In that way we could be sure of professional actuarial standards of modelling, and there would be a gold standard to which journos could cross-check these essentially self-serving academic models.
Another way would be for bookies to offer odds on academic prognostications ….. or at least for journos and/or the general public to ask academic authors to take wagers on their claims.
Old code of more than 15000 lines. Sounds like a good old Fortran job to me with legacy stuff full of GOTO statements and labels. Transcribing such a muddle to an object language like Java or C++ is an absolute nightmare (I speak from experience). Better to redesign the whole thing from scratch. Takes less effort and is much more transparent (also here speaking from experience). Should have been done 30 years ago. You need well-documented algorithms though and that may have been the problem.
Just maybe, this time, perhaps, politicians and civil servants will start to understand that the science “community” is no more or less venal or corrupt than any other and take the “facts” that “the science” produces with the same caution and initial scepticism they would naturally apply (one would hope!) to estate agents and second-hand car dealers.
I live in hope but I’m not holding my breath. (I might add that if dealing with King or Deben the caution should be the same as they would apply to snake oil salesmen!)
Now I know who to blame for google maps and mail.
Your sense of humor is obscure at times. I didn’t get this until I reread the article.
Blame Google.
The original code in “C”? I would have thought in Fortran. Lots of legacy codes are patched up with pieces of newer languages, but may have flaws still residing from decades back. I got some old codes from “COSMIC” at the University of Georgia in the early 80’s and found flaws in the original Fortran dating from more than a decade before. Remember the Assigned GoTo? Or block common? Errors on different machines with different word lengths. Or something uninitialized, but no one ever notices?
I write code professionally using a range of languages. One of the worst things to hear, especially from academia, is: “the original program was ‘a single 15,000 line file.'”
In the 21st century, programmers are still writing all their code in a single file. What a crying shame.
Any time I have a chance to talk to academics in computer science, mathematics, statistics, and engineering, I try to interject the recommendation that academia learn from the business world good software development practices. Things like, test-driven development, separation of concerns, refactoring, and so forth, all of which are battle-tested practices that produce superior code and code that developers can demonstrate their confidence in. Academia should not be stuck in ~1980 software development practices.
More like pre-1980, maybe even pre-1970.
Who would have thought that modeling in epidemiology was so stunted?
Where are all of the other models and their predictions/results?
Thus far the best model I’ve seen is from a civil engineering professor with a water resources background applying a pond storage model.
Respectfully I doubt a programmer is adept epidemiological modelling. I come across IT techs and coders every day who seem to suggest they know subjects of engineering, modelling, and many other areas better than professionals in those fields. I often feel embarrassment for them and it appears to be an issue of ego. The code structure is rather irrelevant and perhaps speaks to a broader issue of under investment into this area of epidemiology. I doubt the authors of the model ever insisted draconian measure were to be taken based the outputs. Negative consequences and misrepresentation of this information is the mistake of the political class and media.
Leaving aside the question of whether a given coder is skilled at epidemiological modeling,
I would suggest that you clarify the quoted remark. Until you do, you will appear to be ignorant of what matters in computer simulations called “models.” The coding (along with the underlying equations being coded) is more than relevant, it is virtually EVERYTHING.
I too develop software for money. In my experience, there sure are situations when a program with the same input doesn’t always produce the same output. Without random number generators or multithreading.
Remember the pentium bug? Or working with float data type? Sometimes 1.0 actually is 0.99999 or 1.000001. That’s no problem for the software you develop for the local birdwatcher club, but not good enough for banks, insurance companies or science.
Personally I don’t write modelling software, but I think they often use the output from say simulated year one as input for simulated year two and so on. These errors are going to add and multiply like crazy, and not always in the same way. Fun stuff.
My wife, programming back in the 1980’s, had a really ‘orrible bug in a FORTRAN program she was working on. After days of debugging (this was in line editor /compile / test run /read the printout days), she found that a provided subroutine was CHANGING THE VALUE OF ONE! The subroutine writer was unaware how FORTRAN passed its call data, and was using an input parameter as ‘scratch’, thereby changing the value of ‘1’ in the calling routine! And I myself found that a brand-new ICL FORTRAN 77 compiler created incorrect code when you turned the debugging off! (That required assembly-level debugging!)
Ah – the fun we had!
I don’t think that we can pass judgment until the code is released and it is analysed forensically (If it is ever released).
It is likely that there are errors in the code. Speaking as an academic I know that academics write terrible code and that there are errors in the code.
I think that the errors will stem from assumptions about the processes in the model and their parameterisation.
This is what happens, and this is the parallel to Climate.
You start out with a clear and simple idea of how its going to work. CO2 levels, for instance, are going to drive warming. Your intuition is expressible in a simple formula.
But obviously this is not good enough to be convincing. So you now model in exhaustive detail. You are now splitting the world into thousand, hundreds of thousands, million, billions of cells, and doing each one separately. You now treat CO2 as an output of modelling every country and every fuel. Same with takeup.
The next stage is invisible to everyone including the developer: your model in its early versions now does not give output which matches your simple formula. So, with or without realizing what you are doing, you tweak it till it does.
You’d have been better off staying with the simple formula, and testing it and refining it. If the energy had gone into validation and examination of the parameters, the result would at least have been something people could profitably argue about.
This is probably what has happened to Ferguson, several times in succession.
Link to the original article now broken.
I hope “Miss” Pseudonym hasn’t been found out and sacked from his/her present position.
Interesting. The software was released on GitHub and Microsoft upgraded it. I’d like to know how this unfolded.
Microsoft was once run by Bill Gates, a billionaire who is concerned with over population and climate change. I wonder if he pulled any strings at Microsoft to get them to ‘upgrade’ the code?
Meanwhile, Gates is getting involved on finding a vaccine for the virus while his Windows OS, which has been upgraded eleventy-zillion times with virus patches over a span of several decades, still gets viruses.
I can see the future, we will be getting notified every couple of months to install (inject) new virus vaccinations or update patches.
What, me worry? Ha!
kramer
“I can see the future, we will be getting notified every couple of months to install (inject) new virus vaccinations or update patches.”
Ha. Yes. And we will pay through the nose for the license and the upgrade service and…
Unbelievable. Science literally depends on being able to replicate and analyze results, but these guys have an amazing “Just do it, it will work out” perspective that trascends anything and everything. So… all of those papers contributing to the alarm regarding climate change were based on a faulty model? Sincerely it’s hard to believe most if any thing nowadays man.
Sue Denim – Pseu Donym: Nice One There! Brett Keane (the real one haha).
As I said above.
Spaghetti code — aauugg!
I had to modify code in the Space Shuttle simulation programs I was using that had been written originally back in the 60’s for Apollo and had been added to over and over again through the years.
Of course, we had massive testing requirements done by multiple independent companies and, of course, many users were still finding bugs in them constantly.
These guys have none of that rigor at all.
The consequences of using the modeling for the corona virus from Wuhan, as well as the modeling of climate forecasting are much more devastating to humanity than possibly losing a Space Shuttle and the astronauts.
These guys are playing around at simulation and modeling and playing around with billions of people’s lives.
I am not sure that this is the takedown that some think. Commenters dr_t and earthflattener had pretty good takedowns of Sue Denim aside from Sue Denim’s anonymity.
“a single 15,000 line file that had been worked on for a decade” (this is considered extremely poor practice).”
And she then links to John Carmack’s comments on twitter (https://twitter.com/ID_AA_Carmack/status/1254872369556074496). Yes, he says it is 15000 lines of code, but: “It turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren’t enabled or hit. ”
I’m sure I would’ve grumbled if someone handed me a single file containing 15k lines of code, but it would be downright silly for anyone to pick an argument with Carmack on this topic. I’ve never heard of Sue before. John Carmack OTOH is a living legend (for 30-odd years).
In refusing to produce his code is Ferguson the UK’s very own Mann?
“Sue Denim” is a pseu-donym of author Dav Pilkey: https://en.wikipedia.org/wiki/Dav_Pilkey
I think he was inspired by one of Bruce Sterling’s running-mates at the original Cheap Truth zine, https://en.wikipedia.org/wiki/Cheap_Truth
— where “Sue Denim” as in pseu-donym, in this case for Lewis Shiner. Oh, those SF/F guys!