California just released its 2017 report on autonomous vehicle testing in the state. What does it include and what can we learn from it?
Follow us today...
Each year California releases a report outlining the previous year’s results of autonomous testing performed on the state’s public roads, by all vehicle manufacturers which have registered for this ability. The California Autonomous Vehicle Testing Regulations require every manufacturer authorized to test autonomous vehicles on public roads submit an annual Autonomous Mode Disengagements Report, to provide the quantity of and details surrounding each incident that required autonomous mode to be disengaged, “due to either failure of the technology or when the safe operation of the vehicle required the test driver to take immediate manual control of the vehicle”.
For 2017, 19 manufacturers provided results. Faraday and Future was also granted a permit but did not file a report.
Here’s a look at some statistics from the larger companies Like Waymo, Tesla, Ford and Nissan:
Company
No. Vehicles NR=Not Reported
Autonomous Miles
Events
Miles Per Event
Waymo (Google)
75
352,545
63
5,596
GM Cruise
94
131,676
105
1254.1
Drive.ai
7
6,572
151
43.5
Nissan
5
5,007
24
208.6
Zoox
11
2,244
14
160.3
Bosch
NR
2,052
598
3.4
Delphi
1
1,811
81
22.4
Mercedes Benz
3
1,088
842
1.3
Nvidia
2
505
109
4.6
Volkswagen
0
0
0
N/A
Honda
0
0
0
N/A
BMW
0
0
0
N/A
Ford
0
0
0
N/A
Tesla
0
0
0
N/A
Can we glean anything from the data? On the surface, it would appear Waymo is far and away the leader, with both the largest number test miles driven, as well as the most impressive results with over 5000 miles per disengagement. Unfortunately, we cannot make any hard and fast conclusions from this data, for several important reasons:
1. The dataset is far too small. Additional states have stepped up to also invite testing, such as Michigan and Arizona. Waymo for one, has already moved some of its testing to other states. Additionally, many of the companies are global, and at least one is already testing extensively in other countries and US states, as reported by Bosch.
2. This data represents only results from testing in actual public road conditions. Many, if not all, of the companies have dedicated (private) test tracks and simulated road condition test beds. Since we do not have access to these results, we can only assume the California results represent but a fraction of their experiential data.
3. California’s reporting requirements allowed for some leeway in their interpretation. As a simple example, it was not made clear whether disengaging the system at the end of the test should be counted as an event. Oddly enough, Nvidia did count those, while Nissan did not. We don’t know how other companies interpreted this.
4. Companies have devised greatly differing strategies in tackling the self-driving goal. Waymo has created a “database of more than 20,000 scenarios tested on our private test track, including evaluating our system’s core driving skills and its ability to avoid common pre-crash scenarios, and multiplying those real-world tests with billions more miles tested in simulation”. On the other hand, Tesla gathers real world data from its “fleet of hundreds of thousands of customer-owned vehicles that test autonomous technology in “shadow-mode” during their normal operation”, observing how its Autopilot system would have interpreted each real world situation.
Suffice it to say, the reader is strongly advised to take the data for what it is, merely a small snapshot of the testing that is taking place. One thing we can conclude however, is that manufacturers are getting on board in increasing numbers, adding California as another available test bed. The first of these annual reports covered the testing performed in 2015 by the initial 7 manufacturers. As of January 11, 2018, a total of 50(!) permits have been issued. No doubt companies are scrambling to reach for finish line.
Don't Miss: When Will Autonomous Vehicles Become a Reality and How Will They Really Be Used
Please, let us know in the comments section below what you think about this report for discussion.
Follow us today...
This is surprising since the
This is surprising since the self-driving Nissan I drove didn't work in the rain.
There's no order that I can
There's no order that I can discern in the table rows for Autonomous Miles=0 (the last 5 rows). Volkswagen could just as well have been "last" and probably should have been (sort on manufacturer name secondarily with the primary sort on Autonomous Miles). As Tesla's report notes, their testing covered billions of miles, but "Tesla did not test any vehicles on public roads in California in autonomous mode, as defined by California law," so it properly reported zeros. I wonder about that, however, since we've all seen videos of Tesla's fully autonomous driving tests, which were clearly conducted in California -- why didn't those tests meet California' law's definition?
Good question! I noticed
Good question! I noticed that also, but do not have a good answer. And yes, you are correct about the table. It is sorted purely by miles driven. So, any of the companies reporting zero could have been last, but that is not indicative of where they are in development, which really was the point. Thanks for reading!
I really hope that 100% self
I really hope that 100% self driving happens while I am still on this Earth. The idea of getting into a car, speaking or typing in a destination and just riding would be a great thing. Don't get me wrong, I really love driving but there are a lot of times where it would be fantastic to just climb in and have the computer do the work.