By David Wojick
Failure mode analysis is a standard engineering practice. It means figuring out the basic ways a system can fail, each way being a mode of failure. Once you understand these failure modes, you can do things like watch for them, or prepare for them, or take steps to keep them from happening.
Way back when I was a junior engineer, I did mathematical failure mode analysis for large dams. There were just 3 to 5 basic modes, depending on the design. Earth dams had more than concrete ones, but I digress.
We now have repeated, even urgent, warnings that the electric power grid is increasingly prone to failure. Some of these warnings have come from people who actually oversee the grid, including FERC, NERC, and the independent system operators (ISOs). My fellow skeptics have also been vocal about this growing threat of disaster.
Everyone talks about blackouts, but I have not seen a detailed analysis of the various ways these might occur. I suspect there are several different basic ways, each calling for a different approach. So here are some starter thoughts.
First of all, there are deliberate rolling blackouts versus uncontrolled blackouts. ISOs and utilities may well have internal plans, or perhaps rules, for running rolling blackouts. If so, it would be very helpful to know what these are. For example, emergency service groups at all levels of government could have rolling blackouts response plans.
Uncontrolled blackouts may be unpredictable, but they can still be planned for to some degree. I live way out in the country, and we get blackouts several times a year, so we have a well-prepared routine for dealing with them if they do not last too long.
Of special importance are the size and duration of blackouts, as both features deeply affect planning. Should we expect a lot more small blackouts or just a few more big ones? How about really big ones, a few of which have occurred in the past?
It also matters how hot or cold it is, especially for large, long-lived blackouts. Severe cold is really dangerous. The Texas disaster killed a lot of people, and PJM almost went that way last Christmas.
The first question is how much failure mode analysis can already be done using existing computer models? The ISOs and utilities do a lot of modeling. For example, the ISOs already can determine what system upgrades will be needed before a new large generator can be connected to the grid. The wind and solar people complain about this because it sometimes makes their remote projects very expensive.
If the ISOs can do that kind of detailed flow analysis, they ought to be able to see where things are likely to break and what that might do to the system. I am reminded of The Wichita Lineman line saying, “if it snows that stretch down south will never stand the strain”.
It may well be that they are already doing this sort of failure mode analysis; they just don’t want to tell us about it, lest it worry us. But given all the warnings, we clearly need to worry and to take steps to address that worry.
It also may be true that they cannot do the kinds of failure mode analysis I am describing. The growing threat is new, after all, so the software simply may not exist. I seem to recall both the New England ISO and PJM saying they really could not evaluate the impact on the reliability of this attempted transition.
If this is true, then given the huge amounts of potential damage, including deaths, we should be developing that software as fast as possible. Not knowing the impact of the near-term transition on reliability is a true emergency. We may be flying blind into the wall of impossibility.
There is another form of failure that also needs to be addressed, namely wholesale power price spikes. We have already had a few of these and are still struggling to figure out how to pay for them. When unit power prices go from tens of dollars to thousands, there is a lot of damage, even if it is fiscal, not physical.
FERC and NERC reliability standards require various forms of analysis. Publicly accessible failure mode analysis should be among them. Vague warnings are not good enough.
Author
David Wojick, Ph.D. is an independent analyst working at the intersection of science, technology and policy. For origins see http://www.stemed.info/engineer_tackles_confusion.html For over 100 prior articles for CFACT see http://www.cfact.org/author/david-wojick-ph-d/ Available for confidential research and consulting.
Climate science brain stormed for the failure mode of all their climate models and they didn’t come up with any.
It’s one big fail, just wait till that reality bites
Excellent idea, where do you start CA, TX?
Better to let CA descend to its eventual, guaranteed crisis – that will show the world how corrupt and inhumane nut zero really is
“”good failure mode analysis””
In the age of post-modern everything??
That strikes me as an oxymoron
I taught Senior Capstone Design from 2014-2021, and before my time I am not certain that FMEA was a regular part of that course. Engineers are pretty well educated people, but no one should be surprised that graduate engineers have some “holes” in their knowledge. A lot of engineering is learned on the job from more experienced people.
The state of tertiary education speaks for itself
Back when I worked fir a UK defense supplier every project that supplied hardware had to produce an FMECA, https://en.m.wikipedia.org/wiki/Failure_mode,_effects,_and_criticality_analysis
which effectively produced a figure MTBF figure for every part of the equipment and the whole system. This would build up to, will the system survive the required number of battle days in given scenarios.
These calculations were updated regularly to ensure contract requirements were met.
Anyone with a wide technical background would expect this to be done with all aspects of the electrical supply system but we are talking about political design changes so perhaps not.
It is so sad that our technological society is beginning to crumble by design or stupidity.
It is by design.
Do not underestimate stupidity; especially when practiced by know-nothing regulators and administrators who command huge amounts of taxpayer money — think Jenifer Granholm.
It was certainly done for the design and safety analysis of nuclear power plants.
In my engineering career, I sat through many SIL (safety and integrity) meetings. Revisiting a decade later, most of the results weren’t as sound as the design the experienced design engineer came up with originally….plus added complexity and equipment that eventually introduced unforeseen initial start-up dangers. In fact, the SIL request was often a disguised attempt to cut costs using instrumentation rather than pay for adequate relief valves, venting systems, or pay for enough real estate to have adequate fire spacing.
But that is the basis of the whole climate alarmism / nut zero con – they ignore science, engineering, technology, economics and reality because it does not support their warped objectives
Simply compiling and categorizing an extensive list of all semi-modern power outage or regional blackout causes would be a good start. Weather-induced (snow, ice, wind, flood, lightning strikes/fires), Human-caused (lack of maintenance eg. tree trimming, human-caused wildfires, lack of inspection or routine maintenance, vehicle crashes), Grid instabilities (surge in supply, surge in demand, lack of available backup, etc.
Someone has probably already assembled such a list. Maybe even assigned frequencies to the events.
Add rolling blackouts shutting down natural gas supply to the generating stations.
Of course, one of the ironies is that by switching to weather dependent wind & solar
sources they are making failure more likely.
Good article.
“When unit power prices go from tens of dollars to thousands, there is a lot of damage, even if it is fiscal, not physical.”
This has bothered me a lot about grid-connected battery storage. The implication is that revenue to provide a return on investment for a battery installation must come from high prices over short durations. This is terribly unwise and bad for ratepayers. The problem originates with intermittent wind and solar developers being excused from providing their own backup, or contracts for backup.
Same for other short-duration storage concepts, not just batteries.
Buckle up. Rough road ahead.
“Who’s gunna pay fer dat?” is always an appropriate question to ask.
If you start by building a system that is inherently high cost it has to be paid for. You can make choices about who pays how much, and when they pay. So you can pay a contingency fund to spread the cost of price spikes at wholesale. You can subsidise your choice of poor or industrial consumers by loading bills on everyone else.
The supply elements also have to be remunerated to pay for their cost. Those with low utilisation can either charge a peak price or secure a regular availability income. Distorting markets to try to avoid peak pricing will likely result in a very suboptimal and therefore more costly system.
The solutions to these problems that are being proposed in fact end up favouring one set of interests over another, especially when low emissions objectives are nominally given primacy.
The existing battery storage schemes are not at grid transmission levels and only at HV connections via step up transformers – a lot of expensive, loss inducing kit, for little benefit
“Publicly accessible failure mode analysis should be among them. Vague warnings are not good enough.”
Should include war. What if Ukraine was now 100% dependent on wind and solar? It wouldn’t take many drones to knock out the entire system. I bet Israel will never bet on net zero since it’s surrounded by enemies.
What is the chance a solar storm knocks out the power grid?
100%
It already happens – a direct, strong CME hit not only affects satellites orbiting the earth and radio etc systems on earth
A geomagnetic storm three times smaller than the Carrington Event occurred in Quebec, Canada, in March 1989. The storm caused the Hydro-Quebec electrical grid to collapse. During the storm, the high magnetically induced currents damaged a transformer in New Jersey and tripped the grid’s circuit breakers. In this case, the outage led to 5 million people being without power for nine hours.
How Climate Scientists solve problems.
“ ISOs and utilities may well have internal plans, or perhaps rules, for running rolling blackouts. If so, it would be very helpful to know what these are…”
It seems to me that the Texas blackouts in February ’21 were exacerbated by the fact that they really couldn’t “roll” anything. There were folks who simply got blacked out and other unaffected. So, on a system wide basis there was enough generation to prevent anyone from having to suffer more than a few hours, but there was no way to apply this “average” to the entire grid.
The death toll was terrible, but in addition The Dallas FED estimated the costs of that two weeks of dodgy to non-existent power as around $80-130 Billion direct and indirect. Unreliable energy is very expensive.
This is an aside, but in reading about various current rate increases in jurisdictions across the world, I see that people, utilities and maybe regulators, are constantly pointing a finger of blame at rising fossil fuel prices, even when there is no evidence of a sustained price rise, especially under the guise of net power costs (NPC) which involves a lot more than just fuel price. It is a classic misdirection to deflect and develop a disgust at fossil fuels — makes the medicine of cheap power do down better.
In the worst case we may be headed toward something like a holodomor which starvation was brought upon by policies based on the crackpot agronomical ideas of Trofim Lysenko; an unfortunate confluence of state power and pseudoscience. This time we citizens will be the unfortunate peasantry and there is no end of people eager to play the role of Stalin or Lysenko.
During the summer of 2022 ISO New England and the governors of New England sent a letter to Granholm citing high natural-gas prices as a reason to waive the Jones Act and allow for domestic LNG imports to the region. They also requested more coordination with the federal government to ensure energy reliability and help modernizing New England’s heating-oil reserve. The ISO argued that lack of a natural gas pipeline into New England could cause rolling blackouts during the winter.
It seems that the New England grid is on the verge of instability, similar to California, but for entirely different reasons. Federal and state policies have created self inflicted distortions and shortages in the energy market. The end result is an electric grid which is much more sensitive to seasonal weather patterns.
See New England Risks Winter Blackouts
Failure Mode Analysis is looking at the components of the system – poles, towers, cables, transformers, turbines etc.
But ultra-reliable components are no good if you don’t have any electricity or fuel to make any electricity.
Fail Mode Analysis will not make the wind blow or cause the sun to shine in reliable fashions
Actually there is an engineering course entitled “systems engineering”. Its focus is exactly how the interconnection of components impacts the performance of systems made from those components. Unfortunately it isn’t offered as a course in all engineering schools. It’s another of those things that many engineers will have to learn from their more experienced brethern.
Surely, the lack of wind/sun are the primary failure mode for wind/solar generators. We all know that there is only one solution – turn on reliable generation equipment. Of course once the crazy climate clowns have destroyed the reliable (ie fossil fuel) stuff the solution will simply be to get used to not having power until the weather changes.
My response – install a wood stove, have 2-3 cords of seasoned firewood on hand and buy a gas powered generator capable of running the fridge and well pump.
I’m sure this has been done already and considering all the variables associated with intermittent power generation I bet they’ve all been hidden because of the negative outcomes. Telling someone there’s a 50% chance of rain doesn’t have the same impact as telling someone there’s a 50% chance they won’t have electricity for lighting, heat, AC, hot/any water, TV, computer, or ability to charge their car.
Normal grids run calculations of forecast power flows (both active and reactive – which is effectively reverse power flow for part of every cycle because voltage and current are not in phase) given forecast demands at substations that feed local networks and plant estimates of generation. This is used to fine tune generation to ensure that supply and demand are matched without inning too close to grid constraints that would lead to thermal overload.
A further round of analysis is then run, looking at the impact of losing each component of the system in turn: each generator, each grid link and each demand substation. A so-called N-1 evaluation. It looks at how the grid rebalanced for each loss, and if any of those contingencies result in risks of overload on parts of the system then the pattern of generation is further adjusted to reduce those risks.
This is feasible at the level of a few thousand grid components where the chances of simultaneous spontaneous failures is very small provided that the operation is not being stretched to its limits.
Moreover, in a conventional grid the challenges are mitigated by siting generation interspersed with centres of demand. Consider a circular grid with 30MW demand centres A,B,C and D with a generator supplying 30MW between A and B, B and C, C and D and D and A. Each town is fed by the generators either side of it, and each generator supplies 2 towns, so the loading on each grid segment is 15MW. Now work out what happens when we lose say the generator between D and A. The other generators need to boost their output by a third to 40MW to compensate for the loss. Both A and D now need to get 100% of supply from their nearest functional generator link, since there is no point in sending power between them. So the AB generator will have 10MW spare to supply B, which will get 20MW from BC which will in turn supply 20MW to C, which gets 10MW from CD, which keeps the lights on in D with the other 30MW. The worst cases are a couple of grid links increased from 15MW to 30MW, and the remaining power stations boosting from 30MW to 40MW.
With a renewables grid there are lots of extra challenges. For a start, most of the generation is remote from demand. So our example might see all the 120MW of generation supplied by wind between A and B. 60MW would have to go via A, and 60MW via B, dropping off 30MW at each of A and B to feed C and D, with no flow on the link between C and D. Now what happens if we lose the link between the wind farm and A? All 120MW must go via B, progressively dropping off 30MW as it goes through C and D to feed A. The grid must be supersized to cope. If it isn’t, we must curtail the wind and start up other generators.
But that isn’t the end of the story. No longer is an N-1 analysis of potential failures adequate. Weather fronts can move through in many different orientations, knocking out or starting up renewables generators in different locations almost simultaneously. If we have to analyse N-2 and N-3 configurations the number of evaluations required starts increasing combinatorially, and it becomes infeasible to explore them. We probably have to rely on more heuristic approaches, and they risk going wrong.
Renewables adversely affect both grid inertia and reactive power (voltage control) hence the need for STATCOMS and other expensive units to manage active / reactive power flows and ensuing voltage support
This is a necessary additional cost for renewable generation, often ignored in cost breakdowns when deceitfully spouting how good and cheap renewables are
Bloomberg’s green energy research team estimates the cost at $US 200 Trillion to stop warming by 2050. That is $US100,000 per household for the 2 billion households in the world. Ninety percent of the households in the world can’t afford anything else. That means $US1 million per household in the developed countries over 27 years. Working families can’t afford anything like that or they would all be living in million-dollar mansions. Given the choice, probably 99% of families would prefer having $US 1 million dollars and one or two degrees of warming.
The major problem of random energy is built in. It has a very short MTBF given that solar fails every night and wind every few days. Their “fixes” for this are based on pixie dust and other assorted applications of hoping. That’s what comes from art majors making technical policy decisions. The people who who don’t understand a screwdriver.
David,
Yes this is a good article that makes valid points, but we all might keep in mind that some engineering problems, including some major disasters, resulted from designs and specifications that were far outside engineering experience. Examples include the Tacoma Narrows bridge (a design more slim by a factor of nearly 3 than any similar bridge built), the Columbia shuttle disaster (simulation software being used far outside its envelop of validation). These are circumstances that Rumsfeld would have probably referred to as “unknown unknowns”. I fear a lot of this proposed energy system will be another such example.
The blob are doing a great job of planning for blackouts – the increasing grid saturation by renewables will ensure that – grid physics is a speciality of mine following 40+ years in power generation & distribution – take it from an experienced, competent Engineer, its looking uglier by the day
story tip: Vermont power company wants to install batteries in customer’s homes: https://www.nytimes.com/2023/10/09/business/energy-environment/green-mountain-home-batteries.html
For those who don’t have a paid NYT subscription, here is a “gifted link” to the article.
GMP is our electric provider and is planning on burying the lines on our two lane gravel road. They admitted it was all new technology for them, however if done correctly it will make a huge difference as the tree canopy over the road has become very dense (probably from all the plant food in the atmosphere). It is quite beautiful, especially at this time of the year.
The battery idea is beyond stupid, but the GMP CEO thinks it’s a cool idea…
Wait till the insurance companies hear about this…
The same kind that explodes in EV cars and emits poisonous gases when it does?
The same kind that explodes
It appears to be.
Came across it on LinkedIn. Lots of people loving the idea, no clue of the hazards. How many house fires until it sinks in?
“It may well be that they are already doing this sort of failure mode analysis; they just don’t want to tell us about it, lest it worry us. But given all the warnings, we clearly need to worry and to take steps to address that worry.”
What a load of BS! No I do not need to worry because chicken little says the sky is falling.
The main reason people die when there is a major power outage is that such events are so infrequent that they are not prepared.. The US grid is very very very very reliable. That is four nines reliable.
For example, an ice storm can take out the power lines causing a major outage. It has happened twice to me in 30 years. You did not hear about it because it was before the internet. Sure ice storms happen every year someplace but utilities are ready to restore power.
I can think of only two major cascading grid failures in 40 years. I had to read about them because the people did the fight things keeping the grid on line. I also read the technical report. One factor that could have prevented the failure is maintaining right of ways.
The 2000/2001 Califonia rolling blackout was not a grid failure. It was know for many years that places like Texas and Califonia needed to build new power plants.
Greyout Davis got fired by the voters and has not been heard of since. Governor Bush got the job by building fossil plants and small amount of wind and solar. He is better know for being POTUS.
For those who do not know, the purpose of wind power is to suck money out of California.
David Wojick,
Not quite what you are looking for but close enough as to be of interest, I think, from the UK.
Loss of Mains (ROCOF) (nationalgrid.com)
A major factor in the blackout of 9th August, 2019 – and despite the same issue having been flagged a decade earlier. It kicked off a lot of renewables embedded generation, and the Grid had no idea it was happening because they do not see live data on embedded generation.
Having installed many diesel & gas generator peaking plants, at 11 & 33kV connections, LoM (islanding) protection, via ROCOF and Vector Shift schemes – ROCOF has become the desired protection scheme due to its better sensitivity – all Grid support peaking plants went through a process of having to tighten ROCOF settings following a severe outage involving a large off shore wind farm (Hornsea 1) – Orstead & RWE were both fined £4.5Mn for their part in the blackout in 2019
Failure mode analysis should be mandatory and public. As for wholesale power price spikes the power generators responsible for the power shortfall should have to pay for the power they couldn’t deliver.
Why? You want to know what to hit for target practice.
I read this somewhere “The grid EV-type energy storage batteries they will need will have to store the equivalent of a few nuclear bombs. When they blow it will devastate everything in a 100-mile radius.”
It may well be that they are already doing this sort of failure mode analysis; they just don’t want to tell us about it, lest it worry us.
There are those tasked with dancing on political eggshells whilst struggling technically with the growing fallacy of composition problem-
Solar switch-off a must for all states, says AEMO, to control “seven Erarings” of rooftop PV | RenewEconomy
Naturally the AEMO are the messenger rooftop solar investors want to shoot with inverter curtailment as the upshot is while they are curtailed they can’t even use their own solar generation.