Waymo 1st, Tesla Last in CA Self-Driving Car Report, But 4 Reasons We Can't Make Hard Conclusions

California just released its 2017 report on autonomous vehicle testing in the state. What does it include and what can we learn from it?

Each year California releases a report outlining the previous year’s results of autonomous testing performed on the state’s public roads, by all vehicle manufacturers which have registered for this ability. The California Autonomous Vehicle Testing Regulations require every manufacturer authorized to test autonomous vehicles on public roads submit an annual Autonomous Mode Disengagements Report, to provide the quantity of and details surrounding each incident that required autonomous mode to be disengaged, “due to either failure of the technology or when the safe operation of the vehicle required the test driver to take immediate manual control of the vehicle”.

For 2017, 19 manufacturers provided results. Faraday and Future was also granted a permit but did not file a report.

Here’s a look at some statistics from the larger companies Like Waymo, Tesla, Ford and Nissan:

Company 
No. Vehicles NR=Not Reported
 Autonomous Miles
 Events
Miles Per Event

 Waymo (Google)
 75
 352,545
 63
 5,596

 GM Cruise
 94
 131,676
 105
 1254.1

 Drive.ai
 7
 6,572
 151
 43.5

 Nissan
 5
 5,007
 24
 208.6

 Zoox
 11
 2,244
 14
 160.3

 Bosch
 NR
 2,052
 598
 3.4

 Delphi
 1
 1,811
 81
 22.4

 Mercedes Benz
 3
 1,088
 842
 1.3

 Nvidia
 2
 505
 109
 4.6

 Volkswagen
 0
 0
 0
 N/A

 Honda
 0
 0
 0
 N/A

 BMW
 0
 0
 0
 N/A

 Ford
 0
 0
 0
 N/A

 Tesla
 0
 0
 0
 N/A

Can we glean anything from the data? On the surface, it would appear Waymo is far and away the leader, with both the largest number test miles driven, as well as the most impressive results with over 5000 miles per disengagement. Unfortunately, we cannot make any hard and fast conclusions from this data, for several important reasons:

1. The dataset is far too small. Additional states have stepped up to also invite testing, such as Michigan and Arizona. Waymo for one, has already moved some of its testing to other states. Additionally, many of the companies are global, and at least one is already testing extensively in other countries and US states, as reported by Bosch.

2. This data represents only results from testing in actual public road conditions. Many, if not all, of the companies have dedicated (private) test tracks and simulated road condition test beds. Since we do not have access to these results, we can only assume the California results represent but a fraction of their experiential data.

3. California’s reporting requirements allowed for some leeway in their interpretation. As a simple example, it was not made clear whether disengaging the system at the end of the test should be counted as an event. Oddly enough, Nvidia did count those, while Nissan did not. We don’t know how other companies interpreted this.

4. Companies have devised greatly differing strategies in tackling the self-driving goal. Waymo has created a “database of more than 20,000 scenarios tested on our private test track, including evaluating our system’s core driving skills and its ability to avoid common pre-crash scenarios, and multiplying those real-world tests with billions more miles tested in simulation”. On the other hand, Tesla gathers real world data from its “fleet of hundreds of thousands of customer-owned vehicles that test autonomous technology in “shadow-mode” during their normal operation”, observing how its Autopilot system would have interpreted each real world situation.

Suffice it to say, the reader is strongly advised to take the data for what it is, merely a small snapshot of the testing that is taking place. One thing we can conclude however, is that manufacturers are getting on board in increasing numbers, adding California as another available test bed. The first of these annual reports covered the testing performed in 2015 by the initial 7 manufacturers. As of January 11, 2018, a total of 50(!) permits have been issued. No doubt companies are scrambling to reach for finish line.

Don't Miss: When Will Autonomous Vehicles Become a Reality and How Will They Really Be Used

Please, let us know in the comments section below what you think about this report for discussion.