In the late 1800s, it was unlikely that Robert Louis Stevenson had autonomous cars in mind when he said ‘nothing made by brute force lasts.’ But this sentiment is proving to be apocryphal for engineers seeking to design cars that can drive by themselves.
So-called autonomous vehicles are adding substantial complexity not just to the process of automotive design, but also to the cost of testing and validating these new technologies. Indeed, these testing costs are becoming so prohibitive that, without some form of intervention, this innovation could cease to be commercially viable for car makers.
Faced with this exponential growth in the cost of testing new vehicles, a group of engineering companies, car makers, simulation experts, academics and research agencies are currently proving out a solution to solve this problem. The aim of the project which has been titled ‘CERTUS’ is to strip out 40% of the testing costs of new automated vehicle technologies. For a car with level 4 automation, these testing costs can reach $400m. For even more complex and fully autonomous vehicles such as the robotaxis being developed by Waymo and Cruise, this testing budget can reach £1.6b, accounting for 50% of the overall vehicle development cost.
CERTUS is creating a new approach to testing automated driving technologies. Many motorists are familiar with automated emergency braking (AEB) and lane-keep, while the next generation of driver aids to reach the market such as traffic jam pilot represent the next step towards the progressive transition from human to automated driving. Improving the way testing is done will enable these technologies that remove the monotony of manual driving - for instance in stop-start traffic - as well as significantly improve road safety.
Mileage accumulation
Much of the cost of testing automated driving technologies stems from two problems. The first of these is the reliance on a process of mileage accumulation on the public highway, which is an inefficient approach to testing that can extend to millions of kilometres in a bid to test the safe performance of automated driving systems in all possible eventualities. With no ability to control the scenarios encountered, engineers are forced to apply ‘brute force’, which in this context means driving millions of kilometres in order to increase the possibility of testing every possible circumstance an automated system might potentially encounter.
The second and connected problem with mileage accumulation is that it is an open-ended testing process. Thus this approach can never satisfactorily answer an engineer’s question of when enough testing has been done to validate a system for use on the road. It may just be that the system simply hasn’t encountered a particular set of permutations during the mileage accumulation process.
Certainty accumulation
CERTUS has set about reducing this reliance on brute force testing alone and providing engineers with some form of certainty about the completeness of the testing of automated driving systems. The project is one of the first to hybridise real world testing with simulation technologies. The resulting mixed reality or multi-pillar testing allows simulated scenarios to be applied to real-world vehicles – for instance a simulated pedestrian stepping out in front of a real vehicle on the proving ground to assess the performance of automated emergency braking. This capability to generate and test a series of scenarios quickly and methodically closes down the problem of open-ended real world testing and allows engineers to far more efficiently work through a defined set of tests.
This approach is now enabling engineers to more efficiently develop ground truth – this is in essence a dataset that is used as a benchmark for the performance of an automated driving technology under certain conditions.
Extrapolation
The next step in the CERTUS solution to speed up testing is applying the ground truth dataset to a set of oracles. Oracles provide a mathematical method to exponentially reduce the size of the problem space of scenarios in which the automated system needs to be assessed. For any given driving scenario, the oracles assess the performance of the automated system and then intelligently select the next series of tests to avoid over-testing in areas where the performance is already good and not under-test in areas where the system might fail. It is this focused approach that allows CERTUS to yield the cost and time-savings that are critical to car makers.
When to stop
Smarter multi-pillar testing and the optimisation of the search space using oracles is a key feature of the CERTUS approach to reducing testing cost and time. It also provides quantification of the test coverage to define, for instance, that an automated driving system can be assessed to be effective in 70% of operating conditions. This allows stakeholders from car makers to national transport agencies and importantly insurers to make informed decisions about the introduction of automated technologies onto the public highway by understanding the limits of testing coverage and what residual risk remains.
By overlaying test results over a national GIS mapping dataset of the UK road network, CERTUS will, at the completion of the project in the next 12 months, not only be able to quickly and more cost-effectively answer the broad question of how effective an automated driving system is, but also where the technology can be safely deployed on the domestic road network.
Alejandro Trueba is Senior Engineer for Connected and Autonomous Vehicles at HORIBA MIRA. CERTUS is a collaborative project from HORIBA MIRA, partnered by Polestar, IPG Automotive, Connected Places Catapult and Coventry University.
Five ways to prepare for your first day
If I may add my own personal Tip No. 6 it goes something like this: From time to time a more senior member of staff will start explaining something...