Introduction

Autonomous emergency braking is an advanced active safety system that helps drivers avoid or mitigate collisions with other vehicles. The test space for autonomous driving functions is incredibly large, and it is not feasible to validate large numbers of scenarios with real-world driving tests. This example enables you to perform scenario-based testing of an AEB system by generating multiple scenario variants from a seed scenario.

The example shows how to vary ego speed and collision point parameters to generate multiple variants of the CPNC test scenario. The example also shows how to use iterative testing methodology in Simulink® Test Manager to test the generated variants. In this example, you:

Explore Test Bench Model

This example reuses the AEBTestBench model from the Autonomous Emergency Braking with Sensor Fusion example.

To explore the test bench model, load the autonomous emergency braking project.

openProject("AutonomousEmergencyBraking");

To reduce Command Window output, turn off model predictive controller (MPC) update messages.

mpcverbosity("off");

Open the system-level simulation test bench model.

open_system("AEBTestBench")

This test bench model has these modules:

For more details on these components and simulating the test bench model, see the Autonomous Emergency Braking with Sensor Fusion example.

In this example, your focus is on automating the simulation runs for multiple variants of a seed scenario. You create the seed scenario using the Driving Scenario Designer app.

Review Seed Scenario and Generate Variants

The helperCreateSeedScenario function generates a cuboid scenario that is compatible with the AEBTestBench model. This is an open-loop scenario containing two stationary target vehicles on a straight road and a child pedestrian. This scenario is inspired by Euro NCAP CPNC scenario as per the Euro NCAP Test Protocol - AEB VRU systems version 2.0 in which the ego vehicle collides with a pedestrian child who is running behind the obstruction from the nearside. The frontal structure of the vehicle hits the pedestrian.

Plot the open-loop scenario to see the interactions of the ego vehicle and the pedestrian child target.

hFigSeedScenario = helperPlotScenario("helperCreateSeedScenario");

The ego vehicle, which is not under closed-loop control, collides with the pedestrian child. The goal of the closed-loop system is to avoid or mitigate the collision with the pedestrian actor. In the AEBTestBench model, the ego vehicle has the same initial speed and initial position as in the open-loop scenario.

Close the figure.

close(hFigSeedScenario)

Generate Variants of Seed Scenario The helperCreateSeedScenario function creates a seed scenario in which the speed of the ego vehicle is 60 km/h, and the collision point is at the front-left corner of the ego vehicle. To test this AEB CPNC scenario per Euro NCAP test protocols, you create multiple variants of the seed scenario by varying the ego speed and collision point. Use the helperCreateSeedScenario function to create the seed scenario.

[seedScenario,egoVehicle] = helperCreateSeedScenario;

Use the getScenarioDescriptor function to create a scenarioDescriptor object that has the properties of the seed scenario. You can modify the properties of this object to generate scenario variations.

Get the scenario descriptor from the seed scenario

scenarioDescriptor = getScenarioDescriptor(seedScenario);

Define the IDs of the ego vehicle and the target actor that collides with the ego vehicle.

egoID = egoVehicle.ActorID;
targetID = 2;

To generate scenario variations, the ego vehicle in the seed scenario must satisfy these criteria:

To generate variants of the seed scenario, use the generateVariants function. The function uses these input arguments: the scenarioDescriptor object, the ego vehicle ID, the target actor ID, a new speed for the ego vehicle, and a new collision point.

Specify the new ego speed in m/s. To convert 20 km/h to m/s, divide by 3.6. This results in a speed value of approximately 5.5 m/s.

egoNewSpeed = 20 * 1/3.6; % m/s

Define the new collision point, as specified in the Generate Scenario Variants for Testing AEB Pedestrian Systems example, which is required for scenario variant generation. For a collision point at the center of the front of the ego vehicle front, specify newCollisionPoint as 0.5.

newCollisionPoint = 0.5;

To generate a scenario variant, create an empty variationProperties object, which stores actor and coliision property variations. Then, add a speed variation to the ego actor by using the varyActorProperties function of the variationProperties object.

variation = variationProperties;
varyActorProperties(variation,egoID,Speed=egoNewSpeed);

Add a variation for a collision point between the ego and target actors by using the varyCollisionProperties function of the variationProperties object.

varyCollisionProperties(variation,egoID,targetID,Actor1CollisionFraction=newCollisionPoint);

Generate a variant scenario descriptor from the seed scenario descriptor using the generateVariants function.

variantDescriptor = generateVariants(scenarioDescriptor,variation);

Get a variant scenario, as a drivingScenario object, from the scenario variant descriptor object by using the getScenario function. In the variant scenario, the ego vehicle travels at a speed of 5.5 m/s and hits a pedestrian at the center of the front of the vehicle.

scenarioVariant = getScenario(variantDescriptor,Simulator="DrivingScenario");

Review the generated variant scenario using the Driving Scenario Designer app.

drivingScenarioDesigner(scenarioVariant);

Notice that the generated variant has an ego speed of 5.5 m/s, and the collision occurs at the center of the front of the ego vehicle. You can use this scenario with the AEBTestBench model to perform closed-loop testing.

Simulink Test enables you to automate the generation and testing of scenario variants with the closed-loop model. Using Simulink Test, you can generate a large number of variations from a seed scenario and catch the failures.

Perform Iterative Testing with Scenario Variants

This example includes a Test Manager configuration for automating testing of the AEB application with the generated variants. Open the AEBScenarioVariationTest.mldatx test file in the Test Manager.

sltestmgr
testFile = sltest.testmanager.load("AEBScenarioVariationTest");

The Simulink scripted iteration framework enables you to automate scenario variant generation and closed-loop testing using the test bench model. The iterative testing script in the AEBScenarioVariationTest.mldatx file describes how to combine scenario variant generation and closed-loop testing of the AEB application.

The setVariable function sets these parameters for each iteration:

The addIteration function adds test iterations. For more information on creating scripted iterations, see the Test Iterations (Simulink Test).

After simulating the test case, the Test Manager uses the helperPlotAEBResults function from the CLEANUP callback to generate the post-simulation plots. For more information on these plots, see Autonomous Emergency Braking with Sensor Fusion example.

Run and Explore Results for CPNC Scenario Variants

To test the system-level model with the generated variants from Simulink Test, use this code:

resultSet = run(testFile);
testFileResults = getTestFileResults(resultSet);
testSuiteResults = getTestSuiteResults(testFileResults);
testCaseResults = getTestCaseResults(testSuiteResults);
testIterationResults = getIterationResults(testCaseResults);

Visualize Results and Review Generated Report

The testIterationResults output contains the results for all the iterations. To plot the summary of results in the form of a color grid, use the helperPlotAEBVariantResults function. This helper function plots the grid results with specific colors for each cell that indicate the pass or fail status of closed-loop simulation of the AEBTestBench model with the generated scenario variants.

helperPlotAEBVariantResults(testIterationResults,egoNewSpeeds,newCollisionPoints)

In the result grid, the columns represent the collision point values of 0.2, 0.3, 0.5, and 0.7. The rows represent nine ego speed values ranging from 10 km/h to 50 km/h in steps of 5 km/h. For each value of ego speed, there are four possible collision points, which results in a total of 36 variations. The right table in the figure shows the color lookup table for the VUT Test Speed vs VUT Impact Speed Range. VUT test speed is the initial speed of the ego vehicle and VUT impact speed is the speed of the ego vehicle at the time of the collision. The difference between the test speed and the impact speed is known as speed reduction. This lookup table shows the speed reduction to infer AEB system performance at the time of the collision. The green column represents significant speed reduction and the red column represents lower speed reduction of the ego vehicle.

Each cell of the grid shows the final ego speed using a color that corresponds to the amount of speed reduction. Notice that all cells of the grid are green, which indicates that the algorithm in the AEB test bench passed all 36 variations of the CPNC scenario.

Use this code to generate the report, for further analysis of specific iterations:

sltest.testmanager.report(testIterationResults,"Report.pdf", ...
Title="Automate Testing for CPNC Scenario Variants of AEB System", ...
IncludeMATLABFigures=true, ...
IncludeErrorMessages=true, ...
IncludeTestResults=0, ...
LaunchReport=true);

Review Generated Report

Examine Report.pdf. Observe that the Test environment section shows the platform on which the test is run and the MATLAB version used for testing. The Summary section shows the outcome of the test and duration of the simulation, in seconds. The Results section shows pass or fail results based on the assessment criteria. This section also shows the plots logged from the helperPlotAEBResults function.

Enable MPC update messages.

mpcverbosity("on");