The Autopilot system of Tesla Model S is a technology and regardless of its sophistication level watchfulness is always needed because several things may always go wrong with any technology.
Update on this story, June 21, 2017 from Reuters. "Tesla driver in fatal 'Autopilot' crash got numerous warnings: U.S. government."
Dallas Morning News reports that, according to the May 7th Tesla Model S crash survivor, the owner killed in crash was watching ‘Harry Potter’ while using car's autopilot. As you may already know the NHTSA has started an investigation probing Tesla's Autopilot system and its safety.
In my opinion, regardless of Tesla's Autopilot system's safety and reliability one should always be watchful. Especially now, when the system is still in its infancy and lacks decades of testing and data gathering.
Autopilot works when sensors work right. The sensor may simply become defective. What if a sensor was broken or defective? You can't ignore that because you put your life on the line.
Another reason that requires driver's alertness and watchfulness. Chips are made from silicon and one or more bits may fail.
Third reason, the technology, like any computer can always possibly get bugs: simple software bugs that happen with any other technologies, computers, smartphones and so on. This technology is still infant.
In a blog post, Tesla responded to NHTSA's investigation, writing that "Autopilot is getting better all the time, but it is not perfect and still requires the driver to remain alert. Nonetheless, when used in conjunction with driver oversight, the data is unequivocal that Autopilot reduces driver workload and results in a statistically significant improvement in safety when compared to purely manual driving."
My heart truly goes out to the owner of this Tesla, who lost his life in this crash, but I wrote this article to alert the current Tesla Model S and Model X owners to be alert and stay watchful when driving in Autopilot. Perhaps it can help to prevent future crashes.
Also See: Reason Why Tesla May Curtail Model S Autopilot Capabilities in The Upcoming Update