How a Software Update Could Add Another Tesla Stunning Feature

In that sense, a very interesting feature developed by Andrea Ranieri, a Deep learning researcher at CNR-IMATI (@4ndr3r), and published on Twitter yesterday, could potentially turn Tesla on board cameras into additional, different kinds of sensors, without any need to add other electronic equipment to the vehicles: it would be just a software update (that would obviously require some tweaks and parameter adjustments beforehand). In this case, we are talking about a Deep Learning AI, trained with @fastdotai as Ranieri explains, that basically uses regular cell phone cameras to detect potholes and cracks on cement roads as you walk by. More details will be available as soon as he publishes a paper he will release: code, data and pre-trained models.

OH MY GOD! This one turned out really well!(Trained with @fastdotai btw)#DeepLearning #segmentationhttps://t.co/1nkSvGMhKA pic.twitter.com/oz703vE0MY— Andrea Ranieri (@4ndr3aR) November 5, 2021

It caused immediate reactions on twitter: "...the application for detecting structural damage to buildings, bridges, dams, etc caused by earthquakes, hurricanes, etc could be game-changing. Consider the recent condo collapse in FL. This could save countless lives. Read up on: Nondestructive Testing techniques" commented Dr. JT Kostman (@jt_kostman); to which Ranieri replied "...we would like to work on this too! We are submitting requests for funding on this topic but so far they have all been rejected. In fact, with a bit of re-training, the model would be able to detect damage even on concrete and other materials."

When asked if the technology would work on online or offline video, he replied " ...it works on images, so it can work both on offline videos or streaming videos. This one is an offline video and it's processed at 7fps on my RTX 2070. But the inference script is still unoptimized, there is a lot of margin for improvement!" How many layers did he use? To which he answered "...it's a DeepLabv3+ with a ResNet-101 as backbone. Both training and video images are 540x540 px, resolution matters when you want to see cracks on the asphalt."

Asked if by chance will it GPS tag the potholes, Andrea said that "…no GPS at the moment, but we're working on a semi-permanent installation on a bus that should provide a GPS fix." Another Twitter user, @GiorgioMantova, asked him if it would "...work so well also on dark asphalt? (Dark road, dark pothole).", to which he replied that "...Nope, on dark asphalt the model has some more "uncertainty". I will need to add more dark asphalt images or decrease the brightness as data augmentation."

Ranieri said that he would be writing a paper on this, and then he can do a summary/tutorial to share on twitter; and that we will have to be patient. Finally, he said that the demo he published is a video from a normal smartphone but that he would be also glad to try fusing information from a LIDAR, but that he doesn´t have the hardware at the moment; "...maybe for the next project."

Just imagine applying that technology on millions of Teslas (Model 3, Model Y, S and X, and the upcoming Cybertruck and Roadster) on the road, providing 24/7 information on a variety of categories (that is, not only potholes or cracks on the cement/asphalt roads), and creating a series of giant AI data-sets that could then be retrained for a whole gamut of different uses in Machine Learning: that would be heaven on earth for Tesla AI engineers, I bet.
What do you think? Please let us know in the comment section below.

All images in this article Courtesy of Tesla, Inc.

Nico Caballero is the VP of Finance of Cogency Power, specializing in solar energy. He also holds a Diploma in Electric Cars from Delft University of Technology in the Netherlands, and enjoys doing research about Tesla and EV batteries. He can be reached at @NicoTorqueNews on Twitter. Nico covers Tesla and electric vehicle latest happenings at Torque News.