Guest blog: Taking a statistical approach to driverless car safety standards

With the rapid advancement of automated driving technology come new automotive safety standards. The BS EN ISO 26262 series on functional safety has defined how safety risks from malfunctioning vehicle electronic or electrical systems should be addressed, but with automated vehicle technologies come new sources of risk.

ISO 26262 defines guidelines to demonstrate that automated vehicles will perform safely within their intended real-world settings. But this standard only covers safety hazards that result from a system malfunction; it does not cover safety hazards that arise from the performance limitations or technological shortcomings of the system. New standards were required to bridge this gap.

Setting new standards

In July 2022, BS ISO 21448 was introduced to guide the assurance of safety of the intended functionality (SOTIF), with input and guidance from HORIBA MIRA. It requires engineers and designers to adopt a safety assurance approach to system development, and to provide evidence of the design, verification and validation (V&V) phases that demonstrate the absence of unreasonable risk of hazards resulting from functional insufficiencies, or from reasonably foreseeable misuse.

With new E/E systems taking over some of the responsibilities traditionally performed by the driver and implementing this functionality using novel sensing technologies and complex decision algorithms, SOTIF must be designed into a vehicle's systems to ensure functionality and safety. BS ISO 21448 defines a process by which the absence of unreasonable risk can be justifiably argued based on observed evidence and adherence to processes.

For instance, environmental conditions could trigger a system to react in a way that is hazardous: the automatic braking system could react to a reflection of a vehicle rather than to a physical vehicle. We need to identify and analyse these unintended behaviours, and what could trigger them, to improve the system – by adding a new sensor or revising the decision-making algorithms, for example – and then repeat the process to assert that the risk has been sufficiently reduced and that no other risks have been created. It's an iterative process that has the potential to be incredibly time and labour intensive.

Determining acceptability

In functional safety, Automotive Safety Integrity Levels (ASILs) and compliance with the appropriate requirements in ISO 26262 are used to argue the absence of unreasonable risk. By contrast, SOTIF does not use ASILs: setting the acceptance criteria is a bespoke part of the process itself.

The automotive industry is still getting to grips with the scale of the V&V task required by these standards. Considering the almost infinite number of scenarios an automated driving system could encounter on public roads, the ambition of SOTIF provides a significant new challenge for engineering teams.

A statistical approach

The functional safety and ASSURED CAV teams at HORIBA MIRA have collaborated, garnering their breadth of knowledge and experience, capabilities, and facilities to create a systematic route to SOTIF assurance. The result is a state-of-the-art CAV SOTIF framework that leverages statistical concepts, but also draws clearer distinction between two streams of evidence: analysis, and verification and validation.

CAV test programmes are extremely broad and much wider‑ranging than those of conventional vehicles. The long-standing CAV paradigm that we will never be able to fully expose any system to all possible scenarios during testing has led us to develop and apply statistical techniques. This yields maximal coverage of the relevant test cases for minimal time and effort. The evidence we collect must be aggregated and analysed as a collective body of proof that a system is not only safe, but that a high level of confidence in said safety has been determined and asserted.

During the framework's analysis phase, hazards and hazardous behaviours are identified and their associated risks are evaluated. Functional insufficiencies and triggering conditions are assessed; the system can then be modified according to recommendations based on its present performance and requirements of the operational design domain (ODD). The outputs are sets of system- and vehicle-level acceptance criteria, and a validation test strategy.

This test strategy can then be executed: scenarios and test parameters are identified and defined. A sample of test cases is selected and iteratively refined upon using statistical methods that maximise efficiency, minimise uncertainty, and optimise for sensitivity and specificity of the tests. Testing is performed in both (virtual) simulation and in (physical) proving ground settings. Analysis of the results provides evidence of compliance, or otherwise, with the aforementioned SOTIF acceptance criteria.

Throughout the process, data analysis happens in real-time. This hybridised approach, whereby analysis is conducted in parallel with testing rather than sequentially, can provide significant time- and cost-saving benefits for those taking automated driving systems to market.

By taking a statistical approach we target the most meaningful and revealing of tests; all the time balancing our test programme to ensure its sensitivity to any SOTIF concerns, while maintaining a strong level of specificity, which are vital if the evidence gathered is to be genuinely informative and of significant inferential value. Real-time analysis is not outlined in BS ISO 21448, but it can inform an early, rapid response if preliminary test results indicate that assurance thresholds will never be met, and therefore reduces time spent on iterative development and V&V costs. We always keep the safety case argument in mind: that’s why we perform V&V tests in the first place.

HORIBA MIRA's framework is now supporting OEMs and automated system developers to evaluate and assess their product performance to comply with SOTIF, provide safety assurance and make roads and all road users safer.

Michael Orgill, Connected and Autonomous Vehicles (CAV) Project Engineer at Horiba Mira

HORIBA MIRA is an active member of the working group for standards such as ISO 21448. Its detailed knowledge of the standard's intent and future direction enables it to provide guidance and support for organisations looking to BS ISO 21448. Find out more about SOTIF and Functional Safety here.