WiseBench: A Motion Planning Benchmarking Framework for Autonomous Vehicles

Rapid advances in every sphere of autonomous driving technology have intensified the need to be able to benchmark and compare different approaches. While many benchmarking tools tailored to different sub-systems of an autonomous vehicle, such as perception, already exist, certain aspects of autonomous driving still lack the necessary depth and diversity of coverage in suitable benchmarking approaches - autonomous vehicle motion planning is one such aspect. While motion planning benchmarking tools are abundant in the robotics community in general, they largely tend to lack the specificity and scope required to rigorously compare algorithms that are tailored to the autonomous vehicle domain. Furthermore, approaches that are targeted at autonomous vehicle motion planning are generally either not sensitive enough to distinguish subtle differences between different approaches, or not able to scale across problems and operational design domains of varying complexity.\ This work aims to address these issues by proposing WiseBench, an autonomous vehicle motion planning benchmark framework aimed at comprehensively uncovering fine and coarse-grained differences in motion planners across a wide range of operational design domains. WiseBench outlines a robust set of requirements for a suitable autonomous vehicle motion planner. These include simulation requirements that determine the environmental representation and physics models used by the simulator, scenario-suite requirements that govern the type and complexity of interactions with the environment and other traffic agents, and comparison metrics requirements that are geared towards distinguishing the behavioral capabilities and decision making processes of different motion planners. WiseBench is implemented using a carefully crafted set of scenarios and robust comparison metrics that operate within an in-house simulation environment, all of which satisfy these requirements. The benchmark proved to be successful in comparing and contrasting two different autonomous vehicle motion planners, and was shown to be an effective measure of passenger comfort and safety in a real-life experiment. The main contributions of our work on WiseBench thus include: a scenario creation methodology for the representative scenario suite, a comparison methodology to evaluate different motion planning algorithms, and a proof-of-concept implementation of the WiseBench framework as a whole.