RAND releases commentary on ODD policy after research with AAA Northern CA

RAND and AAA Northern California conducted research into operational design domains (ODDs), which informed RAND’s commentary in support of state and local policies for AV safety.

Authored by Xantha Bruso

October 5, 2020

Last year, AAA Northern California convened a workshop on automated vehicle (AV) safety metrics, which was facilitated by the RAND Corporation (RAND) and Securing America’s Future Energy (SAFE). Individuals from 20 different AV-related companies, academic institutions and non-profits participated. The goals of this workshop were to:

1) Create alignment among industry – or at least among a critical subset – on the value of collaboratively supporting targeted research on AV safety metrics

2) Communicate the potential for roadmanship, as articulated by RAND in a 2018 report,[1] and/or related metrics to describe AV safety performance, and

3) Gather industry feedback and suggestions for ideas to improve understanding of these metrics and their role in communicating about AV safety.

One of the topics that emerged from the workshop was the need for a more rigorous and standardized definition of operational design domain (ODD), which was both required for and would advance several different AV safety metrics. Based on this finding, the project team (AAA Northern California, RAND, and SAFE) conducted a literature review and individual interviews with several workshop participants to further define the scope of a project to investigate ODD conceptualization. The hypothesis that emerged was that defining ODDs in a consistent manner would support the analysis of AV data such that ODD-specific safety metrics could be calculated, which would help illustrate the performance of different AVs (or different versions of the same AV) in a more accurate way.

Using this hypothesis, the project team reviewed additional literature and conducted more detailed stakeholder interviews[2] to develop the whitepaper found below, recently published on RAND’s website, which 1) describes how ODDs are currently used, and 2) explores how ODD conceptualization could be expanded or modified to support ODD priorities.

Since an ODD forms the basis of an AV’s operating model, which is used to specify driving tasks and requirements and to verify and validate ADS behavior, ODDs are a key element of AV safety. A main point of this report is that since ODDs help convey an AV’s capabilities and limitations, and since customer trust is earned by living up to expectations, clearly communicating to consumers when, where and how an AV will function is critical to building trust in this emerging technology.

Another point is that while ODDs can have innumerable categories and subcategories (helpfully contextualized by SAE’s recent best practice document on an ODD conceptual framework and lexicon),[3] an ODD can never be completely comprehensive to everything an AV may encounter. Therefore, an AV can find itself in situations for which it is unprepared. As such, testing to ensure an AV can perform safely within its ODD, recognize when it’s inside and outside its ODD, and respond safely to being outside its ODD is a critically important part of making an AV’s safety case.

To support an AV’s safety case, the report proposes an ODD verification function – a feature that can help an AV decide whether and how to devolve to a minimum risk condition. Creating such a feature would support a safety case by providing another layer of redundancy that checks on an AV’s ODD by relying on the vehicle’s sensors to assess the compatibility of the AV’s environment against its basic operational requirements.

To do this, an AV’s ODD must be characterized in a way that is detectable by an AV, technology neutral, and able to be tested by a third-party. By focusing on concepts like “visibility” (e.g., the AV’s ability to perceive an object 100 feet away with 95% accuracy while at 35mph) instead of whether visibility is impaired by rain, snow, leaves or a sensor malfunction, an ODD verification function can test against an AV’s ODD and provide an additional layer of safety, reassurance, and information to developers and consumers alike.

Whitepaper on ODD

[1] Fraade-Blanar L, Blumenthal MS, Anderson JM, Kalra N. Measuring Automated Vehicle Safety: Forging a Framework (RR-2662). Santa Monica, CA; 2018. https://www.rand.org/pubs/research_reports/RR2662.html

[2] To understand how ODD is currently used, conceptualizations and categorizations were drawn from the technical literature, policy guidelines, standards including ISO 26262, SOTIF, UL 4600, and Voluntary Safety Self-Assessments. To develop concepts further, subject matter experts from ADS developers, academia, and other sectors were consulted.

^[3] SAE. AVSC Best Practice for Describing an Operational Design Domain: Conceptual Framework and Lexicon, AVSC00002202004, 2020: https://www.sae.org/standards/content/avsc00002202004/