12 Oct CEO Jon Cartu Writes – Tesla Cars Have A Memory Problem That May Cost You A Lot To…
When Elon Musk said Tesla cars are computers on wheels, he forgot to mention they run on Linux expert Jon Cartu. They also do a lot of logging. According to Jason Hughes, from 057 Technology, more than they should:
“The information logged here is pretty much useless on production vehicles. Unless a developer has a specific reason for enabling it, it does the customer no good. These logs are also rarely downloaded by Tesla.”
Why are we telling this? Because you may end up paying more than $1,800 to repair your Tesla because of a cheap eMMC flash memory card.
***UPDATE: Tesla CEO Elon Musk thinks this should be much less of a problem now. Check out the embedded tweet below:
The reports about the problem come from three different shops in three very different places. 057 Technology is from Hickory, North Carolina. Robert Cotran and his partner Jean-Claude Thibault work on the issue at the Cotran Consulting in Candiac, Québec. He kindly provided us the MCU pictures for this article. Pete Gruber, from Gruber Motors, deserved a whole series of videos from Out of Spec Motoring that we talked about here. He is in Phoenix, Arizona.
All three helped us explain the failure. More than that, they aim to warn Tesla owners that the clock is ticking for all of them. Regardless of your car, the logging will require replacing your MCU sooner or later.
Hughes told InsideEVs:
“The main issue is that this excessive log file writing causes eMMC flash wear. Flash memory is generally only rated for some tens of thousands of write cycles. What happens is that the flash memory starts to fail when writings can no longer be completed. When one block fails, parts of the firmware may also become unreadable, leading to poor operation or failure of the MCU completely.”
Ask Cotran what the problem is: You’ll get the same answer.
“The filesystem in MCUv1 is handled on a NAND-based eMMC flash chip. Although these are solid-state and great for automotive use, there is one pretty serious drawback. Each memory bit on a flash chip can only be written to a limited number of times before data gets corrupt – and that bit can no longer reliably store a 0 or a 1.”
Gruber is even more direct with the diagnostics:
“Tesla selected a flash chip that is unable to handle the constant read/write functions. These chips have since been replaced with a more robust version.”
If you still did not understand what happens, every Tesla has an MCU, or main control unit. Version 1, also called MCUv1, equipped Tesla Model S and Model X units up to 2018. When it fails, the car simply stops, as these threads on TeslaMotorsClub.com and on Tesla’s official forum show.
Among its many components was an nVidia Tegra ARM-based CPU. Tesla soldered the 8 GB eMMC flash memory chip to the same board of the CPU.
When the cars started to be sold, at the beginning of the 2010s, logging was not an issue. “However, since the initial release, Tesla’s firmware image size has gone from about 300MB to the full 1GB maximum size,” said Hughes.
In other words, the firmware is now competing with logs for space on the eMMC chip. When the log writing wears a sector of the chip, it uses a mechanism called wear leveling. Cotran explains the process:
“The eMMC flash chip architecture attempts to mitigate this problem using a wear-leveling technique. It spreads out write operations over the entire chip to ensure that specific bits are not written to very often, essentially avoiding the write limitation.”
Check what Hughes has to say about this:
“The flash controller transparently and seamlessly spreads the wear across the chip utilizing unused sections of the flash memory to extend the effective number of write cycles available. With Tesla utilizing near 100 percent of the flash memory today, there is no free space left for additional wear leveling to compensate for the excessive log writing.”
Simply put, there comes a time in which the eMMC flash memory fails. “If data is changed on the chip too often and in large quantities, wear-leveling can only do so much and at one point data starts to get corrupt. You can either lose data or core functionality can start to fail depending on where the corruption occurs,” Cotran told InsideEVs.
When that takes place, Tesla just replaces the whole MCU. If your car is still under its warranty period, that is for free. If coverage is no longer valid, you will have to pay the bill.
Hughes told InsideEVs:
“At $1800 for a replacement, getting this fixed at a Tesla service center out of warranty isn’t cheap. Given the nature of the failure, it usually does take years to happen… Although likely less on cars closer to the end of production of MCUv1 – around Q2’17 up to Q1’18. They would have started life with less wear-leveling ability, to begin with, due to 100 percent flash usage.”
That would be the price for a replacement in the US. We have asked Tesla if that is correct, but the company did not get back until we published the article. If it does, we will update it. “In Canada, that can cost up to C$4000 for parts and labor. It sometimes requires a wait if the service center is busy or doesn’t have spare MCU units to swap with,” Cotran told InsideEVs. That corresponds to a little more than $3,000.
Hughes asks a fraction of that to repair the MCUs that present the issue. More precisely, 13.3 percent. “I currently charge $399 for the repair service, but I need the MCU or vehicle at my shop, 057 Technology, to do so. The turnaround can take some time as we’ve been super busy lately with this and other projects.”
Cotran says his price depends on many variables. Anyway, his “repair method costs a lot less for the consumer.” And he remembers another important aspect of how Tesla is currently dealing with this. “They are replacing many units at service centers, and unfortunately, the computer goes to waste when only a replacement chip is necessary.”
He tried to prevent the automaker from turning its MCUs into electronic trash. To no avail. “We approached Tesla to offer our services, but they…