Scenario Driven Development for Open Source Autonomous Driving Stack

The development of an autonomous driving stack (ADS) is challenging since it requires rigorous testing at each step. Whether the stack is modular, semi-modular, or end-to-end, the testing pipeline always follows a simulation to real-world testing hierarchy before any new feature is deployed on the autonomous vehicle (AV). On an abstract scale, it is difficult to keep track of what improvements have been made to the stack and what changes degraded the overall performance of the autonomy. This problem worsens when multiple research teams are actively contributing to the development. In this paper, we discuss a scenario-driven development approach that guides the development of the autonomy stack. We use Autoware Mini, an internally developed modular autonomy stack (currently in its early stages of development), as a case study and evaluate its performance as development progresses with Software-In-loop (SIL) and holistic testing. We use CARLA Leaderboard and Scenario Runner to evaluate the stack with automated benchmarking and discuss the evaluation feedback, specifically w.r.t. perception and prediction module of the autonomy stack.