The headline is a hat tip to The Daily Bayonet, who we miss, still in hiatus. Excerpts from the Los Angeles Times:
June 19, 2012, 12:33 a.m.
Faulty computer modeling caused the equipment problems that are expected to keep the San Onofre nuclear plant dark through the summer, federal regulators said Monday.
Officials from the U.S. Nuclear Regulatory Commission gave their first public account of the initial findings of their investigation into the plant’s problems at a meeting in San Juan Capistrano.
What they did not give was any indication of how long the plant is likely to remain out of service, saying there are still questions plant operator Southern California Edison needs to answer and more inspections the NRC must do.
…
NRC officials said it appears that simulations by Mitsubishi underpredicted the velocity of steam and water flowing among the tubes by a factor of three or four. The high rate of flow caused the tubes to vibrate and knock against each other, leading to the wear.
It was not clear why the computer modeling was so far off. Mitsubishi had no representatives at the meeting and could not be immediately reached for comment. Collins, however, said that ultimate responsibility lies with Edison.
Full story here
=================================================================
If a linear system like steam in pipes can’t be properly modeled, it makes you wonder how well the chaotic non-linear system of global climate does in computer models like GISS Model E.
Did someone say there is a problem with the model?
That’s some pretty faulty logic used in the conclusion.
Models used for weather forecasts are quite reliable a few days in advance even though they are pretty useless for simulating steam flows.
REPLY: With weather models, the designers get feed back in a few days that helps them make the model better. With climate models, you have to wait 50-100 years, so its modeling without immediate feedback for improvement. Bedsides, global scale is far more complex with orders of magnitude more surface area and variables than synoptic scale modeling. Climate modeling is mostly an open ended shot in the dark on a large scale with no immediate feedback to test it.- Anthony
Oh what a beautiful ironing.
Lemme guess…. the failure of the models is either by a factor of 2.54 or 3.94…
That is so, so stupid!..There is absolutely no reason that engineering models should be that far off. These were replacements for steam generators that were designed in the 60s and installed in the 70s — and engineers can’t get it right today????!!!! What about the validations of the calcs???!!!
There is an unfortunate propensity for people in all fields to trust what the computer says.
A gross error such as the one described should have been spotted by an Engineer doing manual checks on the numbers to ensure that the results still made sense. Ideally, that should be an Engineer at arms’ length from the one running the computer simulations as they are both likely to make the same assumptions and mistakes if working together.
My hunch is that one (or more) of the boundary conditions for the tube calculations isn’t in fact the same as that used for the calculations.
I find the “acceptance” that tubes normally wear by rubbing against support structures to be quite disturbing. Can’t engineer the things so that that cannot happen? Seriously?
California SanOnofre + Mitsubishi Japanese Company = fail
Japan Fukushima + General Electric American Company = fail
Oh the Nuclear Ironing….
2 international nosebleeds.
DJ says:
June 20, 2012 at 8:35 am
Lemme guess…. the failure of the models is either by a factor of 2.54 or 3.94…
3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679 8214808651328230664709384460955058223172535940812848111745028410270193852110555964462294895493038196
4428810975665933446128475648233786783165271201909145648566923460348610454326648213393607260249141273
724587006606315588174881520920962829254091715364367892590360011330530548820466521384146951941511609
Ironingly, gator69’s video reveals nothing but girls in underwear behind the model curtain.
I’m looking forward to all those windmills and mirror plants being shut down because of faulty climate models!!! 🙂
Mailman
Pity that our government is unwilling to build an experimental LFTR Liquid Fluoride Thorium Reactor) and continue to explore the potential for that technology. The current LWR (Light Water Reactor) technology seems to have all sorts of inherent problems.
Always frustrating when replacements last 1/10 as long as originals. Root cause probably due to a lack of information sharing between originals and replacements, miscommunication between English and Japanese, or some sort of English to Metric Mars-probe type conversion error. Ah the joys of outsourcing our manufacturing …..
Factors can only be out by 3,1415926535 etc if its on a Euclidian plane, tubes are a little more `curved` than `fla` so can I allow a factor of 2 or 4, in fact knowing its an Amercian reactor can I use `taxi cab` geometry and get an even more disturbed factor
Maybe old Mr Reynolds was wrong in the frist place.
Not only that, but if the shell side volume is reduced by 4 and the diameter by 2, how did they cram enough tubes in to get the required temperature approach?
Blond model…
From what I’ve seen much of the plant level design of many of the older nuclear plants was done mostly by hand calculation, and I assume much of the component level design such as steam generators was the same. Those old nuclear plant designers were something else, they were true professionals and artists, jack of all trades, and their work was just wonderful examples of applications of basic engineering and common sense. If it didn’t require the horsepower of a computer they didn’t use it. But recently things seemed to have trended to computer runs or canned programs and something was lost in that transition. For one, it’s just hard to pick up a stack of older computer results and determine if the calculation is correct. The format is just plain lousy compared to the logical textbook style of doing a calculation in engineering school, and the program may be out of date or lost and not easily modified for changes to the design. Second, people will tend to trust the computer result more than the hand calculation result and may even skip or gloss over checking the sanity of computer results. When you have a complicated problem the hand check is usually not easy (I’ve done several) and there is no glory in it. You are considered a lesser engineer if you do not get to say you have experience running XYZ canned program.
But the biggest problem of all with canned programs imo is you just don’t get that holistic view of what matters and what doesn’t when all is hidden in several black boxes and stacks of computer paper output. I’ve always felt you never really understand a mechanical design unless you write all the code yourself or do the hand calculations yourself. Then you see how everything fits together like a puzzle and you know the strengths and weaknesses of the design. How is this even possible these days with so many people from so many countries (many guarding their own turf, job security, and company secrets) all working on a single complicated and expensive component? Someone may think they understand but understanding 99% is not good enough when one little oversight can sink you.
I guess what I’m saying is that when you type in inputs and push a button and go with the results many times its sort of like putting a TV dinner in the microwave then calling yourself a master chef. It’s just not the same.
I can shed some light on this.
The issue is not with the one phase liquid flow inside the tubes. The issue is with the two-phase water flow on the outside of the tubes. The flow is three-dimensional, with cross-flow having the greatest influence on fluidelastic vibration of the tubes. We use ATHOS to model the flow on the secondary side of steam generators.
It is a complicated model.
However bad the secondary side flow was modelled, it doesn’t explain how tube-to-tube contact occurred in both steam generators.
I am unaware of the assumptions that MHI used in their model. However, I do know that MHI has never built steam generators as big as those at SONGS and there may some issues with their experience in scaling up
Engineer’s equivalent of, “Patient died because his heart stopped.”
When you don’t know what you’re modeling…get a bigger computer.
Flame virus is among the most sophisticated and subversive pieces of malware to be exposed to date.
So did some green person open up the site to the internet?
Ya know just to see if the flame would work?
Their failure was not using the “Crane Manual” (aka “Flow of Fluids”, Crane Technical Paper 410)
http://www.flowoffluids.com/
One of the all time best documents for basic fluid flow calculations, and one of the very few books I took with me to my professional engineering exam many years ago. Computer and software not required, slide rule optional…
A failed statistical model of O-ring characteristics lead to the ultimate destruction of the Challenger space shuttle.
Our climate system is chaotic by virtue of being unwieldy and incompletely characterized. At present, it is only possible to describe an envelope for its behavior, and there is reason to believe that only our estimation of this envelope will improve with time. I suppose the question we should ask is what level of accuracy is sufficient in order to properly conduct risk management.
She is probably too young to understand just how safe even old nuclear technology is and the extraordinary sacrifice which would be required if we were to universally embrace solar technology. Not to mention the environmental damage that follows from recovery of resources necessary to construction of solar panels and their later disposal. Fortunately, little girls with cute signs do not set policy.
Finally, why are we outsourcing a critical process to Mitsubishi or anyone else?
The problem is that they didn’t have enough models to average out.
Well, depending what you’re solving for you may still need to do a square root for the final or intermediate answer. Although I was taught how to do that long hand, and it’s not that onerous, I’d rather use my trusty Pickett N4-T.
Since they are talking about steam and water, the problem must be at least partially on the turbine side of the heat exchanger and involve 2-phase flow in an upward direction. This flow is chaotic and unpredictable, even statistically, with 6 different flow regimes that it can transition between depending of temperatures of the steam, water, and surface, pressure profile, gas/liquid volume fractions, and velocity.
I was at Berkeley when they presented the results from the large scale loss of coolant reflood test experiment. The results of which were radically different from the predictions used for the designing the systems. The conclusion was that the formulas would have to be completely rewritten. One of the professors that developed those formulas was in the back of the room and exploded in protest that they must have misapplied the formulas. The presentation pretty much ended there as the 2 argued and over 100 grad students in between with their heads whipping back and forth like participants at a tennis match.
Turned out that they had neglected to fully account for the effects of the water flash boiling and wetting the heat exchange tubes.
Bernd Felsche says:
June 20, 2012 at 8:37 am
I find the “acceptance” that tubes normally wear by rubbing against support structures to be quite disturbing. Can’t engineer the things so that that cannot happen? Seriously?
Any tube capable of vibrating at all should be attached to a support using an isolation mount, and if the tube is vibrating so much that it *does* wear thin is an indication that a pump is seriously out-of-balance.
Bottom line — more people than the modelers eff’ed this one up. Badly.
Why all the trust in computer models? I call it the “Star Trek” effect.
We can model many things quite well. Engineering models work well because of the centuries worth of base knowledge that goes into the field and because the models are constantly improved by real-world feedback.
Two-phase flow is tricky. The basic problem, however, is that some engineers and scientists put far too much faith in mathematical models. It is, as the French might say, a déformation professionnelle.
The very models of all those modern major-generals.
D.J. Hawkins: “I’d rather use my trusty Pickett N4-T”
PICKET? Only pinko industrial engineers used Pickets. Real engineers used Dietzgen Microglides, with rosewood base and teflon grooves!
Ah, for the days of the religious wars over slide rules! 🙂
How good do you think those models are for projecting climate effects for the end of the century? By the way the Met Office abandoned long range seasonal forecasts for the public because the ‘weather’ model was complete and utter crap. So there. 😉
It’s not the first time Mitsubishi equipment caused big problems for Americans.
Kasuha says:
June 20, 2012 at 8:18 am
That’s some pretty faulty logic used in the conclusion.
Models used for weather forecasts are quite reliable a few days in advance even though they are pretty useless for simulating steam flows.
REPLY: With weather models, the designers get feed back in a few days that helps them make the model better. With climate models, you have to wait 50-100 years, so its modeling without immediate feedback for improvement. Bedsides, global scale is far more complex with orders of magnitude more surface area and variables than synoptic scale modeling. Climate modeling is mostly an open ended shot in the dark on a large scale with no immediate feedback to test it.- Anthony
============================================================
Another thing to consider is that the “input” is only days away. A front moving east from Illinios will soon impact Ohio. Before there were such things as computer models there was a growing understanding of weather patterns thanks, in large part to the invention of the telegraph, telephone and radio. Real time data could be collected about where the front had been, where it was now and then relayed to where it was expected to go next. As more data was collected and technology advanced (radar, satelites) improving the quality and reliablity of the data, more accurate forecast could be made as these things made better hypothesis possible. Calculators replaced sliderules. Computers replaced calculators. Models replaced spreadsheets. But all of the theories and models could be shown to be good or bad very quickly, as Anthony pointed out. “Tweaks” could be made and tested.
Climate model “forecast”? The “real time data of where the ‘front’ has been” is based on broken hockey sticks in the models used by the CAGW crowd. The “long range forecast” that were made by those such as Hansen? They’re already wrong in just the short term.
Yet the CAGW crowd and the UN still are trying to spoil our picnic.
“It was not clear why the computer modeling was so far off”
Ummm, it’s called GIGO!
Re Andrew Russell says:
June 20, 2012 at 12:20 pm
D.J. Hawkins: “I’d rather use my trusty Pickett N4-T”
Pickett N4-ES (eyesaver yellow) is better for those of us with fading eyesight. Wood is better on the fingers when it is cold and damp, but wood swells and sticks in the rain. For field work, a 6-inch Pickett N200-ES is great.
Seriously, I just received a “Dear Customer” letter from Southern California Edison warning us to be prepared for power outages this summer, thanks to the situation with their steam generators:
http://www.sce.com/info/poweroutages/rotatingoutages/outagemaps/default.htm
polistra says: —- It’s not the first time Mitsubishi equipment caused big problems for Americans.
JK——————And they got nuked for it!
Thanks
JK
Well, some days a “sanity check” look at the data utput is needed. Hard to do in a complex system.
The example that pops into my conscious mind at this instant is Morton Thiokol’s stress analysis of the shuttle booster rockets was backward, as could have been discovered if anyone had envisioned a ballon with rubber bands around it.
According to World Nuclear News
“22 February 2011
Southern California Edison (SCE) announced that the reconnection to the grid of unit 3 of the San Onofre Nuclear Generating Station (SONGS) “signals the safe, successful completion of a massive ten-year construction project – replacing the plant’s largest components, its steam generators.” The company decided to replace the steam generators of the 1070 MWe unit 2 and 1100 MWe unit 3 pressurised (sic) water reactors at the SONGS plant after a cost-benefit study showed that the modernization would save customers some $1 billion during the plant’s current licence (sic) period, which runs until 2022. The steam generators of unit 2 were replaced in 2009. “
“15 March 2012
A small number of steam generator tubes have failed pressure tests at unit 3 of Southern California Edison’s San Onofre Nuclear Generating Station (SONGS) in California as the company investigates the cause of a leak in the recently installed components.”
Reference:
http://www.world-nuclear-news.org/results.aspx?sparam=San%20onofre
Tube wear after a year of operation? What an engineering disaster! If a design model failure caused the design flaw, there was a way too great a dependence on the computer models. Didn’t the head of the design department check to see that the steam generator design “looked right.” If it was the first of its kind, some testing of mock-ups should have been done. This was more than a failure of a model, it was a failure of good engineering practice.
Its ok, they probably had a team of system engineers working on it. I’m sure the documentation was beautiful 😀
How far must one have fallen, when one turns to children to spread their propaganda.
It defies reason.
u.k.(us) says:
June 20, 2012 at 6:03 pm
“How far must one have fallen, when one turns to children to spread their propaganda.
It defies reason.”
SOP in every major religion.
Steam and water in pipes is FAR from linear.
The reliability of weather models even a day out is pretty poor.
3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679 8214808651328230664709384460955058223172535940812848111745028410270193852110555964462294895493038196
4428810975665933446128475648233786783165271201909145648566923460348610454326648213393607260249141273
724587006606315588174881520920962829254091715364367892590360011330530548820466521384146951941511609
is that Humble Pi
[Moderator’s Note: I was going to snip this for being off-topic, I really was, but then I thought, “I’ve never seen pi calculated to so many digits…. but rather than being humble, it sure looks pretty ostentatious from where I’m sitting…. -REP]
As a fully fledged member of “Concerned Scientists”, shouldn’t Kenji weigh in on this? He could maybe wear a sandwich sign.
@Galane
“Feynman innocently questioned a NASA manager about the o-ring temperature issue. As the manager insisted that the o-rings would function properly even in extreme cold, Feynman took an o-ring sample he had obtained out of a cup of ice water in front of him. He then took the clamp off the o-ring which was being used to squish it flat. The o-ring remained flat, proving that in fact, resilliancy was lost with a temperature drop.”
http://creepyoldguys.wordpress.com/2007/10/02/feynman-solves-challenger-mystery/
u.k.(us) says:
June 20, 2012 at 6:03 pm
How far must one have fallen, when one turns to children to spread their propaganda.
Betcha nobody’s told her the sun’s just a big, self-sustaining, nuclear fireball, either.
Contrast the steam generator replacement at San Onofre with the history of replacements world wide. There have been a very large number of replacement projects implemented successfully, giving new life to aging nuclear plants and allowing them to extend their operating lives from 40 to 60 years. And, now, we have efforts beginning to extend from 60 to 80 years.
Looking back on 40 years of operating with problematic steam generators, before the highly successful replacements most plants have undergone, there are many options that have been implemented over the years that may be appropriate for SONGS. These have included such things as anti-vibration bar installation, plugging smaller radius bend tubes, in situ tube repair, etc., etc.
I suspect this issue will end with implementation of coping strategies similar to those implemented for aging steam generators in the late 80’s and 90’s in parallel with development of new replacement steam generators funded at MHI’s expense, at least partially, if not fully.
Oh, the ironing! ha ha classic!!
“Foul unbeliever!! Feel, thou miscreant, the sting of my K&E Log Log Duplex Decitrig (Pats. Pend.)!!!” :-))
I have one thing to say about this:-
STUXNET!
Mmmmm, just in case you’re serious, not very likely. A steam generator is basically a big tube with a lot of little tubes inside it. You can control the steam flowrate on the outside of the little tubes and the water flow rate inside the little tubes and that’s about it. Not a lot of room for mucking around.
Maybe not. Those ballons are tricksy. Much better to use balloons.
Measurements are a very good idea, much easier to do with today’s sensors and computers. But they still cost money, takes judgement to say “we must test”.
Indeed, some old designers were very good, as “scott” says. They built something like the Avro Arrow interceptor airplane and its Iroquois engine in a short time. However, there were some duds in those days. (Some of those cases likely program management failures, which also delays programs that eventually work out – like the 787 airplane, probably quite good now but at a huge cost penalty from the delays. A common problem is what I call “schedule push”, not getting fundamentals right at the beginning because work is rushed.)
As for slide rule calculators, Walter Shawlee in the Kelowna BC area has a collection from which he might sell or trade. He tells me that one of the large wooden ones has recently been produced in India (but perhaps mis-represented as original vintage). I have my treasured Pickett metal slide rule stored somewhere, with its case, purchased with money earned working in a grocery store during high school.