Guest essay by Eric Worrall
An alleged computer hardware design defect in 158,000 Tesla Model S & X vehicles will cause important systems such as windshield de-fogging, turn signals, and the rear view cameras to malfunction after five to six years, according to the US Department of Transport.
Watchdog urges Tesla to recall 158,000 Model S, X cars to fix knackered NAND flash that borks safety features
Firmware updates aren’t enough to tackle worn-out memory
Katyanna Quach Thu 14 Jan 2021 // 23:40 UTC
The US Department of Transport has recommended Tesla recall 158,000 Model S and Model X vehicles after an investigation found worn-out NAND flash memory can cause the cars’ rearview cameras to fail.
The dept’s National Highway Traffic Safety Administration (NHTSA) launched a probe into Tesla’s failing digital storage in June. In November, it concluded Model S and Model X vehicles built between 2012 and 2018, are at risk of several issues, ranging from the rearview cameras blacking out and an inability to defog windshields, as well as the loss of turn signal chimes and other audio alerts.
The problems all stem from the hardware that powers the car’s infotainment system, also known as the media control unit (MCU): it includes an Nvidia Tegra 3 system-on-chip with 8GB of eMMC NAND storage. Over the course of five or six years, according to the NHTSA, this consumer-grade flash reaches its program-erase cycle limit. They will be unable to reliably store data, and this renders some of the car’s functions inoperable when the MCU fails.
…
“ODI has tentatively concluded that the failure of the media control unit (MCU) constitutes a defect related to motor vehicle safety,” the office’s director said in a letter addressed to the electric car maker’s legal department.
“Accordingly, ODI requests that Tesla initiate a recall to notify all owners, purchasers, and dealers of the subject vehicles of this safety defect and provide a remedy, in accordance with the requirements of the National Traffic and Motor Vehicle Safety Act, 49 U.S.C. §§ 30118-30120.”
…
Read more: https://www.theregister.com/2021/01/14/tesla_nand_recall/
The US Department of Transport investigation was triggered by 16,000 consumer requests for service from people experiencing the problems described, so this seems to be more than a hypothetical problem.
Flash memory retains its data even when the power is switched off, but there is a limit to how many times that data can be changed. The alleged problem occurred because Tesla apparently opted to use cheap consumer grade flash memory, instead of more expensive but more durable alternatives.
Although Tesla is not legally obliged to action the US Department of Transport’s recall request, there will likely be a lot of pressure on Tesla to comply. I cannot imagine tort lawyers ignoring this recall recommendation, if someone gets hurt because of a safety failure.
That defect is definitely dangerous. Tesla cars have been rated as just about the most unreliable by one consumer rating agency. I remember one horror story of the repair costs for an insignificant dent in the wheel well – two days labor and over $10,000 for the repair, which an insurance company refused to pay.
If you want to drive a Tesla, and you can do your own repair work, I suspect that you can do the old army jeep thing. Buy two of them and make one of them. They are quite cheap in the post accident market due to the repair difficulty. The insurance folks just plain total them and sell the carcass.
The “defect” is a well known property of flash memory. You only get so many write cycles on a cell before the cell goes bad. SSD disks get around this to some degree why having the usual spares and then swapping them in as tracks start to get errors. It was totally predictable when the design was made. It should require the replacement of a inexpensive part, but if that part is soldered into a rather tough thing to replace, it will cost lots of labor.
Those things are designed to not have “user serviceable” parts to allow rapid assembly.
Many times MCU units are buried up inside the dash making them very labor expensive to replace.
Certainly some parts of early 90s Chrysler Caravan vehicles manage to bury things, in the dash and sidewalls and doors.
Uncorrectable bit errors are analogous to driving under the influence. Massively distributed errors are analogous to brain damage. Progressive errors are analogous to dementia with a forward-looking prospect of catastrophic neural collapse. Then there are the deceptive sensory states and erroneous inferences.
Single bit error detection and correction (EDAC) has been available for decades at the cost of 2 extra bits per byte. This could have extended the life of the chip substantially at relatively low cost.
Less than a year ago the Business Insider published an analysis of Tesla cars’ safety, worth reading if you consider buying one.
https://www.businessinsider.com/why-tesla-cars-catch-on-fire-2019-4?r=US&IR=T
How about a freeze on their use of tax credits “out of an abundance of caution” as they say in NY.
I thought they had hit the 200,000 mark for no more Esubsidies.
Usage of credits has not matched earn rate each year. They were also selling large amounts of credits to others to offset their tax liability too.
Philo, there’s more than one credit. They indeed hit the mark for the elimination of the federal consumer tax rebate (the Esubsidy you seem to be thinking of) however, they get a ton of money from selling zero-emission vehicle regulatory credits to other car companies (you may have heard that tesla has had several “profitable” quarters in a row lately. Those profits didn’t come from selling cars, their cars cost more to make then they get in sales. No, those profits came from the sale of those regulatory credits – take away the government mandated regulatory credit scheme and Tesla is a money pit). Not to mention other tax incentives that they qualify for at the state and local levels.
Now imagine if one of the “important systems” was the driving computer.
And people wonder why I’m so against the idea of self-driving cars.
REMOVE ALL SUBSIDIES…LET THE SUCKERS BUY THESE THINGS. Progressive Norway is the no. one EV market percent wise and Tesla is not real popular…maybe 10% of market……would Tesla really want his name being used on this thing?
Ok, ok, ok; I get this isn’t a good think for Tesla to have done.
However, it’s funny to see “loss of turn signal chimes” classed as an Armageddon event.
On the other hand, the windshield defogger may fail. Pretty dangerous if you can’t see in front of you.
If the defogger fails it can be bad news. If the audio alert from a collision/lane alarm fails it can be bad news. If the car is purchased with advanced driver safety features and the driver gets used to them being there then failure can cause catastrophic situations.
I’m the only person on the planet who uses turn signals so a missing chime is basically nothing.
now here is someone who makes real-life observations 🙂
“Chime” or “no chime” wouldn’t tell me a single thing. The signals are supposed to “click.” (They do on everything that I own.) That’s from the days when it was a relay and a capacitor that ran the things.
If I don’t hear a click, I roll the window down and use the hand signals. Yes, those ARE a thing.
These days, the younger generation doesn’t know what the hand signals mean. They probably think you’re attempting some kind of a rude gesture and will give you one of their own in return!
so you’re the guy signaling for half a block before his turn.
I’ve seen plenty of vehicles driving down the road where the driver forgot to cancel their turn indicator. At the very least a potential for confusing other road users, never a good thing. I agree the potential for defogger failure is a far worse problem.
Or the backup camera failing at just the time a kid on his skateboard is about to enter its field of view.
Of course, that isn’t a problem for me, either – I’ll glance at it when I change gears, but then still look around to check out the back window and the mirrors. That camera only gives you the tiny field – it does NOT see the idiot who just turned into the cul-de-sac at thirty miles an hour.
Most of those people are probably just “Going Around the World to the Left”
When I was young in Paris, I noticed that significantly more often, looking at turn indicators gave incorrect info:
Out of all the possible failures that could result from this problem, turn signal chimes was the one you chose to highlight.
Geez. My original post stipulated what Tesla did was inappropriate.
Then I commented that Western civilization could survive without “turn signal chimes” (which, in a Tesla, can be set to a fart noise. Just saying). I made no comment about other failures that would actually be dangerous (no window defogging).
I get 10 responses correcting me for a single snarky, sarcastic observation?
The not so subtle point is if there is real danger, why don’t the regulators have the balls to require a fix before people get hurt? THAT’S WHY THEY EXIST!
10 is a bit of an exaggeration. At the time you posted that, there were only 5 in direct response to your post with only a couple of more posts under those (which were more in response to the contents of the 5 rather than of your own post).
Pedanticism aside, I do agree that “turn signal chimes” isn’t a serious “Armageddon event” issue. Of all the things that could fail due to this problem, “turn signal chimes” would seem to be among the least worrying.
Because that was the “funny” one. The point Javert seems to be making is that of all the serious things that could go wrong from this, turn signal chimes seems rather lacking in serious importance.
So, they recalled all Teslas in US? If the recall is worldwide the number might get close to 159,000.
This is such a basic design concept, that it wouldn’t surprise me to find out that all the computers that are made by Tesla have this same problem.
Ford has a recall on the backup camera on my new Escape. Wonder if they used the same supplier.
Sounds like the failure is in the device that determines whether all these accessories should be turned on or not, and what mode they should be running in. It’s not the peripherals, it’s the device that is controlling all of the peripherals.
I’m getting older, and my nonvolatile memory is failing, as well. 😒
I hope he makes a more reliable spaceship…
In other news, NASA says 2020 was the hottest year….says last decade was hottest since 1880s…and beware….the ice is not only getting scarcer…it’s getting thinner – stay off the ice. NASA has warned you.
Eric,
Have you driven one yet?
No. I tried to arrange an EV test drive recently, not a Tesla because they don’t have a dealer anywhere near me, so I could report on the experience.
The head office politely explained they couldn’t help me, because EV sales aren’t really a thing in Australia. Most sales are going to countries with substantial government subsidies, like Norway. I was straight with them that I wanted to report on the experience, not buy one.
That’s a shame. I thought Tesla did did test drives in Australia, but I’m not sure. I borrowed a friends and holy sh*t it was amazing.
I’m sure EVs will be a major part of the future. I have no problem switching to an EV if they fix the cost, range and recharge time issues. Its the climate crusade side of the issue which bothers me, trying to force people to give up gasoline vehicles before EVs are good enough to replace them (price / performance / range / convenience).
Eric; you write….”I have no problem switching to an EV if they fix the cost, range and recharge time issues.”
Surely these are the “unfixable” things and the reason that EVs are as dud as the MCUs that are fitted to the Tesla.
Which also happen to be the hardest issues to fix. EVs haven’t managed to fix them in the over a century since the first electric car was ever made.
They’re toys for now. Expensive ones for woke people
Expensive toys shouldn’t be filled with cheap parts.
Australia is a big place. Tesla may well have done some test drives in Australia, that doesn’t mean any of them where anywhere close to Eric’s part of the continent.
That’s disappointing Eric.
I was hoping you’d say that you took one out and thrashed it like a red-headed step-child.
HEY. I’m red headed…
Yeah, but that’s not the reason you routinely get thrashed around these parts.
I’m not sure word “urges” should apply here, as malfunctioning windshield defogger could be serious. Probably not as bad as the exploding Takata air bags, but I would expect to have the bad flash memory replaced at the next scheduled service.
I think randomly exploding airbags would be worse. In principle the driver could make the sane decision not to drive, if their de-fogger failed – and you would probably be able to drive for long enough with a failed de-fogger to find somewhere sensible to pull over.
Failing NAND flash is a problem not limited to Tesla, it is potential problem for semiconductrs used in infrastucture, such as: traffic signals, pipeline controls, smart meters, building climate controls, intrusion monitoring systems, etc. Most embedded NAND flash has been sorted to commercial grade. The gate leakage increases with temperature (charge on the isolated gate deceases over time, allowing the bit to become less definite.)
A reliable NAND flash system will use ECC, RAID, age leveling, and regular refreshing. Most commercial NAND flas are reliable to 50K hours (5 yrs,) under commercial spec. Automotive conditions may exceed mil. spec. conditions, which may lead to the semiconductors becoming “creative” in the behavior.
NAND flash is not only used in the MCU (brain), but FPGAs (spinal chord) that connect the MCU to the motors and sensors.
I wonder if the failing NAND issue will soon appear in other vehicles. New cars are computers with wheels that move you from Point A to Point B. Compared to the 2008 car it replaced, my new Kia is loaded with tech, from the entertainment system to vehicle control subsystems.
I’m wondering if we are headed to a new “Y2K” problem…
Except, the “y2K” problem, while real, wasn’t much of a problem for the most part (certainly not to the extent it was hyped). There were only a relatively few systems for which the issue was a truly serious one, and it was known about far enough in advance that those systems were fixed long before the “problem” would have kicked in. As some one who worked on fixing some of the “Y2K” bugs, I have to say it really was a damp squib of a problem (at least in comparison to the hype).
It needn’t be a problem provided you buy quality parts and design with automatic fail over in mind.
And design with the ability to relatively easily and cheaply replace such key components when they do fail (as all parts eventually will). If you have to pay 10K in labor (to borrow a number from upthread) just to replace the part, that’s a big problem.
Perhaps this is Telsa’s computer saying “I’m sorry Elon, I’m afraid I can’t do that”
Makes me real confident in tesla. That is a very rookie Firmware mistake!
It’s not necessarily a Firmware mistake. The system architect or hardware engineer that specified using that part (or possibly the cost accountant that downgraded the selection) has the responsibility of notifying the firmware engineer of its restricted usage. Then someone had the responsibility of determining where the fix should be implemented. Of course all of this should have been determined during a proper design review and the results included in the system specifications. I hate to think it, but this implies that the design of the Tesla was a garage-shop operation and this is but one of many design and implementation failures to come. I do hope his space program is better organized.
Since I see a number of Tesla’s here in Fort Lauderdale, I was curious and did some reliability research. They are awful. The problems sort into three categories:
how Tesla stock got so high is beyond me. More valuable than Ford?!? with their F series pickup franchise? Strange times.
The main that I have heard about Tesla (aside from the truly impressive acceleration) is the extremely long waits for repairs.
I marvel at any company that puts a safety-critical function in the “infotainment” system. And uses low-grade parts.
Sadly, I think that is a problem that is becoming industry wide.
It’s no so much safety-critical functions are being put into the “infotainment” system, it’s that all functions (safety-critical and “infotainment” alike) are increasingly being put into interconnected computer control (not just Tesla, across most all modern manufacturers)
As Montgomery Scott said:
They saved money with a quick design. And saved even more money with the cheapest` cost parts. The part should have at least a 150K erase cycles. But Tesla set the design specs too low (cheap). When tesla gets sued for the failure of the MCU while driving. Will Tesla pay attention then?
Not really, Tesla has been built on everyone else’s money.
It’s business model is funded by Govt schemes, it was funded by investment.
They literally do not care and as they are *cough* ‘green’ they have Govts tripping over themselves to offer them more lucrative reasons to be in their country.
If a Tesla is damaged and then sold on as legitimate salvage they disable all software updates and Fast Charging. This is a monopoly position. This monopoly instead of being torn down is being propped up by global Governments.
Tesla is sadly the poster boy for what will become a very challenging future of global monopoly.
I’m not sure you understand the meaning of the term monopoly. Tesla disabling software on scrapped Tesla’s isn’t a “monopoly position”, it’s an action that only affects Tesla’s own products, it in no way hampers the competition from offering competing vehicles, and there are dozens of other car manufacturers out there that offer competing vehicles. And it’s effectively the equivalent of removing useful and/or environmentally toxic parts out of a car on its way to the scrap heap, which happens all the time with “legitimate salvage”, with the only difference is in doing it remotely/virtually instead of physically.
I also don’t think you quite understand what “legitimate salvage” means, it’s not cars going directly to the second hand market, it’s cars designated for going to the scrap heap (ie cars so “damaged” that they’re considered not worth repairing). What does a trashed car, sitting in a pile of other junkers, awaiting it’s date with the compactor need with software updates and fast, or even slow, charging? assuming it even still had a working battery in it.
Now it’s true that some small percentage of cars that get sent to the scrap heap later get “rescued” and rebuilt (usually with parts taken from several junked cars to make one “whole” car). In theory such vehicle rescuers could request Tesla turn those things back on for the rebuilt Tesla (I say in theory, because, to my knowledge no one has yet tried to rebuild a trashed Tesla – so it remains to be seen if Tesla would do so when asked).
After some searching, I do see there is at least 1 guy (a youtuber) who has turned a hobby that started with rebuilding a salvaged Tesla into a business. More power to him. (Tesla isn’t too fond of him, to say the least)
I’d be curious to know what percent of Tesla’s sold have been recalled compared to other makes and models.
I found this: https://www.rd.com/article/car-brands-most-fewest-recalls/
this is a 150 dollar part replacement. It sounds like an entertainment issue. btw, Tesla cars are rated the best cars safety-wise.
Best cars for safety?
Are you kidding?
What happens when you really seriously crash one and short out the big battery? !burn burn burn!
If you can’t get out, you burn with it.
https://mashable.com/article/tesla-electric-vehicle-car-crash-fire-reignite/
https://electrek.co/2019/02/25/tesla-crash-burning-car-door-handles/
https://www.boston.com/cars/car-news/2019/07/12/electric-car-batteries-can-catch-fire-days-after-an-accident
(I walked away from a massive accident between my Jaguar and a 7.5T fully loaded truck in Italy, despite my car having a brim full fuel tank)..as did my wife and children.
Cheap Nand memory? The tip of a big iceberg!
The resale value of a 5yr old Tesla will be close to zero.
I predict a great market in spare parts, and dead lithium batteries.
Maybe I should open a scrap yard?
Another safety issue has to do with extrication – if that car is overturned and/or you’re trapped, it is MUCH more difficult for firefighters to get you out (I’m a volunteer and this is something covered in our extrication training).
I was just about to post something on that. I had a next door neighbor a few years back who was a full-time firefighter. He told me about the then new training they had to undertake about EVs involved in a crash. Of special note was the fact that they had to know on EVERY EV where the power line (600V!) was, in case you had to use the “jaws of life” to cut open a wreck. Some manufacturers routed the power line in the roof, alongside the top of the door lines.
You are pretty much the only person who thinks that way.
Many well documented videos and facts about quality and reliability issues with Tesla vehicles. The new battery that was hyped is still not being made along with the cyber truck and semi-trailer. Tesla does not make any profit on the cars. It makes money selling “green” credits Tesla has in spades to other car makers to “offset” the making of conventional ICE vehicles. That all changes this year when real car makers start to compete in the EV market. IMO Tesla won’t last long unless something revolutionary happens.
I have more faith in SpaceX. It’s more exciting, too.
I have a Tesla model S that had this issue back in September it cost $2500 to replace the MCU. When the MCU was replaced the mileage on the car said 35,XXXX miles but I’m near 80K now so I assumed they put a used unit in. They said they did not and promptly corrected the mileage to what I remember it when I brought it in for the original repair.
A few weeks after I had the issue I saw stories about this issue and when the 12V battery was dying and affecting the update process. I informed the repair person that I should be refunded the MCU repair the tech said that Tesla is working on how they are going to deal with this and I heard somewhere in Feb I should check back.
I’ve had a lot of fun in the car and made a cross country trip once (and only once 🙂 ) . It’s a great car and I look forward to the Cyber Truck.
(The Tesla is my carbon offset, I have two other 1984 diesel pickups which I do the service work on).
“Although Tesla is not legally obliged to action the US Department of Transport’s recall request, there will likely be a lot of pressure on Tesla to comply.”
Including from the government itself. It can MANDATE that a recall be conducted.
Well the US GOV did a fat lot of mandating while 737MAX was crashing and burning.
Don’t expect any GND government to mandate anything more than F-all.
If you can’t do jack-sh..t when people are dying in fossil fuelled a/c that are supposed to be “safe as houses” but there..we go back to the good old mantra.
It takes plenty of people to die before Boeing actually started to take notice of what they termed a “simple software” problem.
That means, I ain’t trusting any “software” from anything stuck firmly to terra firma, and takes an effort to trust it for anything flying too.
Remember, the pilot of a Harrier aircraft that told it to retract the undercarriage on the ground, which it promply did, and got court martialled for the damage……until someone sensible pointed out it shouldn’t happen!
Instead we have idiots like Musk attempting to run a car company, using the customers as guinea pigs to find out the bugs as the cars wear out, while being subsidised by the poor to fund the wealthy.
Cut off the subsidies.
Cut out the conflicts of interest.
Simple.
The more there is on a car, the more there is to go wrong. Windscreen defoggers? My utterly reliable 25-year old Mazda doesn’t have air conditioning, just an old fashioned and simple to use heater which blasts the windscreen with hot air as needed. Rear view camera? Totally unnecessary – just park carefully! Infotainment? My car has a radio which I never use, and ditto the CD player. Nor does it have telephone connections – the car’s a place for quiet meditation as far as I’m concerned.
There aren’t even any gas struts to support the open hood and trunk – a prop and an ingenious spring mechanism respectively do the job.
What I truly like is the computerised engine management system – far superior to the old fashioned carburettors (been there, done that).
It’s seemingly impossible to buy a simple new car, one that isn’t laden with unwanted electronic clutter. I’m keeping my ‘oldie’ as long as I possibly can.
I’ve still got a couple of Corvairs. And I know how to work on them.
Corvairs? Wow – I remember reading about them in my young years! There aren’t many, if any, in the UK where I live,
Ah, chip wearout – a known problem.
Tends to effect
Some people I knew got caught in wearout of ‘dual port ram’, that they used to store values temporarily. They faced a regulatory requirement regarding probability of incorrect functioning of the device. Fortunately they were able to simply change software to slash refresh rare, designers hadn’t thought about the how often that memory needed to be refreshed.
Say what you will about Elon Musk’s Tesla cars, I am reluctant to buy ANY new car. The have become computerized to the point where one needs the equivalent of an EE degree to fix anything. The days of getting your (disabled) car to run with a nail file, some electrical tape, and baling wire are long gone.
Now, if your car stops, you gotta “call the hook”. All the accessories are “connected”, so any repair will cost thousands- even something like your turn signals.
you should see how much it costs to change a door handle on an Aston Martin!
Or how about a BMW X5.
Most of them ended up at the scrap yars because a ECU in the A/c unit costs about 2.5K.
This controls such a multitude of functions if it goes down the car becomes unsaleable.
In our area there were scores of them in the scrapyard, because it was more economic to write the car off than face the giant bill of changing the ECU.
I have 3, 1980s cars I use on a daily basis.
They are about 300kg LIGHTER than their modern equivalents cos they have no bloat, and no fancy electronics stuffed in every corner.
When anything goes wrong I have the spares, and invariably what goes wrong is never a show stopper.
As a result of the lighter weight they actually use less fuel than a “modern”, brake better and handle better without needing giant tyres, huge brakes, and none of the so called electronic gizmos that the ultrabloats need to try to achieve the much vaunted straight line performance they actually measure on paper.
Put my 30yr old V6 honda powered car against a modern SUV, on any mountain road, and it becomes hilarious, despite them having all the so called “superiority”inc 3x the power.
I would imagine a Tesla with its gigantic bloated weight would be even worse.
Heavy cars DO NOT go round corners, and they don’t brake especially on snow & ice.
You can’t defy physics, as Colin Chapman proved beyond doubt with his Lotus cars.
Here are some facts:-
Tesla model S weighs 2.107T. while models 3 and X weigh 1,611T and
(a massive) 2,458T respectively.
Model S has a weight distribution of 48% and 52% for front and rear.
So it’s rear heavy which would make it a nightmare on snow and ice…
What doesn’t seem to occur to the greenie brigade is we have been going backwards for 30years.
The pinnacle of vehicle technology happened in 1985-1990, and we have been just adding bloat ever since which has made cars less efficient, and use a lot more energy to start and stop them.
I largely agree with Jim Randle’s observations who sadly died last year.
The future was lean burn, and even in small turbine engines.
He (tech director at Jaguar cars) was a strong admirer of innovation, and said an astonishing innovator citroen lost their way.
“The 2CV is an example of something which could not have been done better.”
It’s small, light, handles like a dream, will go anywhere in snow and ice, has tiny inboard brakes, air cooled and is feather light, and uses hardly any fuel.
Try using a Tesla at our current -25C here and see how far you get!
For a start how are you going to get it warm?
Next what do you do with a battery that only produces 15% of its stated power without stopping dead on the side of the road at those temperatures?
Unless your 30yr old V6 honda is an SUV or equivalent (you didn’t say it is, and indeed referred to it as a “car” rather than as an “SUV”, so there’s no reason I should assume it is) you aren’t exactly comparing like for like. The appropriate comparison is with another car in the same category (truck to truck, sedan to sedan, sports car to sports car, SUV to SUV, etc) not with a completely different category of vehicle. Now I grant you, finding an equivalent to some classes of cars is getting more difficult, what with the big auto makers push to phase out smaller cars in favor of bigger more expensive SUVs and “crossovers”, but that still doesn’t change the fact that you should either compare like for like or explicitly and unambiguously admit upfront you are not comparing like for like.