Generating Useful Accident-Prone Driving Scenarios
via a Learned Traffic Prior

Davis Rempe1,2 Jonah Philion2,3,4 Leonidas Guibas1 Sanja Fidler2,3,4 Or Litany2
1 Stanford University2 NVIDIA3 University of Toronto4 Vector Institute
Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Paper

Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, these scenarios must be realistic and challenging, but not impossible to drive through safely. In this work, we introduce STRIVE, a method to automatically generate challenging scenarios that cause a given planner to produce undesirable behavior, like collisions. To maintain scenario plausibility, the key idea is to leverage a learned model of traffic motion in the form of a graph-based conditional VAE. Scenario generation is formulated as an optimization in the latent space of this traffic model, perturbing an initial real-world scene to produce trajectories that collide with a given planner. A subsequent optimization is used to find a "solution" to the scenario, ensuring it is useful to improve the given planner. Further analysis clusters generated scenarios based on collision type. We attack two planners and show that STRIVE successfully generates realistic, challenging scenarios in both cases. We additionally "close the loop" and use these scenarios to optimize hyperparameters of a rule-based planner.

Scenario Generation for Rule-based Planner

From an initial nuScenes scenario (left) STRIVE uses an adversarial optimization to attack a given planner (middle). Furthermore, a solution optimization (right) ensures the generated scenario is "solvable" and thus useful for downstream tasks. In all examples, the planner and adversary are colored accordingly.

Original replay
STRIVE attack
STRIVE solution
Original replay
STRIVE attack
STRIVE solution
Original replay
STRIVE attack
STRIVE solution
Original replay
STRIVE attack
STRIVE solution

Collision Scenario Clusters

Generated collision scenarios are clustered and labeled based on the relative position and heading of the adversary and planner at the time of collision. Representative examples from these clusters are shown below.

T-Bone Right
Merge Right
Head On
Front Right
Cutoff Right
Behind
T-Bone Left
Merge Left
Front Left

Adversarial Optimization Comparison on Replay Planner

Qualitative examples of scenarios generated by STRIVE compared to the Bicycle baseline which does not use the learned traffic model in any way. In each example, the original scene used for initialization is shown on the left followed by the output of Bicycle adversarial optimization and ours. Note the unrealistic attacks containing sharp turns from Bicycle.

Original replay
Bicycle attack
STRIVE attack
The adversary turns sharply to block the planner in a seemingly malicious fashion.
The adversary suddenly changes lanes in a more realistic way, since STRIVE uses the learned prior.
Original replay
Bicycle attack
STRIVE attack
Original replay
Bicycle attack
STRIVE attack
The adversary unnecessarily loops around to collide, since only its dynamics are regularized in optimization.
The adversary instead slows down to cause a rear-end collision, which is more likely under the prior.
Original replay
Bicycle attack
STRIVE attack
Our adversarial optimization chooses a different vehicle to collide than the heuristically pre-chosen attacker of Bicycle, resulting in a more plausible collision.

Second-order Effects

One interesting ability of adversarial optimization is to produce scenarios that cause collisions with the planner using so-called ``second-order effects", where multiple adversaries act in conjunction to cause a collision. In the specific examples below, the planner is obstructed by one adversary, causing the planner to slam on its brakes and get hit by a different vehicle.

Original replay
STRIVE attack
Original replay
STRIVE attack
Original replay
STRIVE attack

Pedestrian and Cyclist Adversaries

We train our traffic model on all categories in the nuScenes dataset and use it to generate scenarios for the Replay planner. In the examples below, a specific pedestrian or cyclist adversary is chosen before optimization to cause the collision.

Pedestrian adversary
Pedestrian adversary
Cyclist adversary

STRIVE in 3D Simulation

We used STRIVE within NVIDIA's DRIVE Sim simulator to generate multiple collision scenarios based on a scene reconstructed from a real-world video. For more details, see the GTC keynote talk.

Citation
@inproceedings{rempe2022strive,
    author={Rempe, Davis and Philion, Jonah and Guibas, Leonidas J. and Fidler, Sanja and Litany, Or},
    title={Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior},
    booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2022}
}            

Acknowledgments
This project page template is based on this page.
Contact
For any questions, please contact Davis Rempe drempe@stanford.edu.