Verification and validation

Safety verification and validation of autonomous vehicles (AVs) remaine unresolved despite the rapid advancement of perception and planning algorithms. Criticality metrics are necessary to evaluate the safety of AVs in knonwn unsafe scenarios to conduct safety verification, while discovering unknown unsafe scenarios and statistical proof of safety are keys to achieve safety validation. To deliver holistic safety arguments for AVs, we are developing novel safety verification and validation methods.

1, Silent Testing

As a part of the German project VVM, we have developed the silent testing method to validate the safety of AVs. The idea of silent testing is:

Test partial functions or capabilities of AVs in the background without intervention in the physical vehicle

We implemented this method to address the validation of AV perception, utilizing a lane marking detection algorithm (LMDA) as a case study. Meanwhile, we demonstrated the method to validate AV systems botn in urban and highway scenarios. Our research was also reported in the VVM midterm event.

The left figure shows the principle of silent testing to validate AV perception, while the right figure illustrates the architecture to validate AV systems.

2, Safety assessment

For the safety assessment of AVs, criticality metrics play an important role. They are not only essential for collision avoidance systems but also play a vital role for verification and validation of automated vehicles. With respect to the first application, criticality metrics should be real-time capable and applicable in various traffic situations. For thesecond application, holistic safety evaluation by criticality metrics is desired. However, existing criticality metrics hardly meet these two requirements. They are either only applicable in post-processing or only assess the safety of maneuvers in longitudinal direction.

We solve this problem by proposing an acceleration-based criticality metric, which takes various evasion maneuvers into account when dealing with critical situations, enabling more objective and holistic risk assessment. In addition, we are modeling careful and competent driver models for assessing the safety of AVs. An AV is to be blamed if it occurs collisions in a scenario where a careful and competent driver does not. Meantime, if a careful and competent driver has no collison in a scenario, an AV shall have the same result in this scenario.

Reasonably foreseeable scenarios from a defined ODD within the open world are fed to an AV and a reference driver model. By comparing whether they lead to a collision or not in the same test scenario, AV safety verification is performed.