Phil Koopman’s new book How Safe Is Safe Enough? dropped over the Labor Day weekend, arriving as a market thus far faithful to highly automated vehicles begins to entertain second thoughts.
Just last week, a class-action lawsuit was filed on behalf of investors in an autonomous truck company, TuSimple Holdings. Complaints include the company’s overstated safety claims, its rush to beat competitors to AV testing, and the company’s inadequate safety culture.
Koopman is a renowned safety expert and associate professor with the Carnegie Mellon University department of Electrical and Computer Engineering. “How safe is safe enough” is a question directed not just at designers and developers of highly automated vehicles, but also investors in AV companies. Koopman’s book also addresses federal safety agencies and state/city authorities. If these regulators are indeed serious about keeping vulnerable road users safe on public roads, they must ask tough questions in developing AV policies instead of simply trusting AV companies.
Technology enthusiasts on social media – eager to shape the AV narrative with or without evidence – should heed Koopman. His book describes where widely accepted AV myths go wrong.
North Star
For the media, How Safe Is Safe Enough? is a reference guide and dictionary, useful for curbing our enthusiasm and clarifying our thought processes. I’m most inspired by Koopman’s North Star – his insistence that we ponder the basic idea of safety (“What do people mean by ‘safe’?”), that we debate when to deploy AVs (“An approach to setting deployment criteria”), and that we question who makes that decision and based on what criteria (“Governance of AV deployment”).
No passage hit me harder than the following in Koopman’s chapter on ethics:
The most pressing AV ethical issue is whether the people in charge of a company – and who stand to profit enormously from winning the race to autonomy – should be able to decide when an AV is safe enough to deploy without substantive, informed input from the stakeholders.
By stakeholders, Koopman means all affected by AVs – companies, shareholders, investors, insurance companies, regulators, legislators, technology suppliers, and most important, “the people who will be affected by it [the deployment on AVs on public roads] but may not have visibility or even a voice.”
“To me, this is an ethical issue. People need to think hard, when they are making value judgments that involve other people’s lives,” said Koopman in a recent podcast with the Ojo-Yoshida Report.
Scope
The book comes with sufficient technical meat to satisfy the engineering community. It defines “a risk framework and metrics that can both predict safety and provide a traceable path to the AV design and validation,” and includes details on “the use of Safety Performance Indicators, as originally conceived in the ANSI/UL 4600 safety standard,” as Koopman noted.
How Safe Is Safe Enough? written in plain English, is accessible and even entertaining partly because Koopman refers to real-world events and dilemmas typically overlooked by AV designers training their AI-driven vehicles. (#didyouthinkaboutthat)
Koopman acknowledges: “The book is more of a discussion and not an academic review paper… It even has a bit of snark to lighten things up.”
Crucially, the book addresses the non-technical audience.
Anyone with a passing interest in AVs is already exposed to, or might have already bought into, a slew of misguided talking points. For example: “Computers won’t drive drunk so they will be safer than human drivers,” or, “More miles in test driving prove AV safety,” or, “Disengagement during the test drives is a key indicator of the maturity of AVs.” And so on. The book helps debunk these tidbits of misinformation.
Recommended:
Their Credibility Shaken, Can AV Companies Recover?
Unknown unknowns
Koopman writes bluntly: “In reality, nobody yet knows if AVs will be as safe as human drivers when deployed at scale.”
Reading lines like this, my first thought was: If this is not a letdown, what is?
“The day you deploy AVs, you’re going to have uncertainty, and that uncertainty is in the form of the unknowns. It is not possible to get rid of all the unknowns,” Koopman said. He also cautioned that we should not “just throw up your hands say, ‘Well, there’s nothing we can do.’” He believes there is a way.
Mileage-based bootstrapping approach to safety ‘amounts to a strategy to plan to get lucky, with the day of reckoning delayed until the first severe crash.’
Phil Koopman
Koopman is focused on “measuring and predicting autonomous vehicle safety.” While raising myriad questions about AV deployment, the author also offers answers.
Many AV companies, under extreme pressure to deploy AVs, float “bootstrapped arguments of safety.” Koopman explained, “A bootstrapping approach to safety assurance based on system-level outcomes starts with a small pilot fleet and grows confidence in safety over time to permit the operation of larger fleets.”
Here’s one example.
Although “the reasoning that exposure increases over time can be seductive,” Koopman writes, “such a process does not assure acceptable safety during the bootstrapping process itself… Rather, it amounts to a strategy to plan to get lucky, with the day of reckoning delayed until the first severe crash.”
Bootstrapping based on miles alone is a weak approach. The reason, Koopman writes, is because it strives to determine whether an AV is safe without confirming an engineering-based preconception of safety.
The alternative Koopman suggests is “bootstrapping confidence in the soundness of the safety case.”
In other words, “If the safety case starts with high initial confidence due to the use of engineering best practices, that gives us a reason to have strong prior belief in safety before bootstrapping even begins.”
The author acknowledges that “this is a speculative idea that might not work out in practice.” He adds, “But it is presented here in the hope that it provides a starting point for a viable hybrid approach to establishing a responsible belief in safety up front, followed by using a bootstrapping approach based on real-world experience to build confidence in the safety case over time.”
Koopman’s book is a comprehensive and conscientious discussion of AV safety, introduced and moderated in an engaging, accessible voice. Engineers, AV enthusiasts, public officials and even ordinary drivers need Phil Koopman’s insights to both set the safety record straight on safety issues while pausing to consider what must be done before unleashing AVs on our streets.
The book is available for purchase on Amazon here.
Junko Yoshida is the editor in chief of The Ojo-Yoshida Report. She can be reached at junko@ojoyoshidareport.com.
Copyright permission/reprint service of a full Ojo-Yoshida Report story is available for promotional use on your website, marketing materials and social media promotions. Please send us an email at talktous@ojoyoshidareport.com for details.