Synthetic Scenario Generator to Train Autonomous Driving Systems on Edge Cases (syn-TEC)

With the rapid evolution of autonomous driving systems (ADS), the demand for realistic training data has surged. Although many existing datasets contain large quantities of recorded driving data, some road scenarios, called edge cases (ECs), are rare and dangerous to encounter. Not only is there less real data for these ECs due to the inherent dangers of encountering them, but they are also very useful for training driving systems as they are more likely to contain situations where the vehicle does not know how to behave. Thus, a popular alternative is simulating these ECs to create synthetic training data. However, current simulations require the time-consuming creation of 3D models and suffer from a lack of realism, reducing the reliability of ADS trained with them once applied to a real environment. This project, “Synthetic Scenario Generator to Train Autonomous Driving Systems on Edge Cases (syn-TEC),” offers a novel approach to the generation of synthetic driving data. It relies on recent advancements in neural radiance fields (NeRF), a technology that allows the generation of photorealistic new views of a given scene based on real recorded visual data. We aim to create a generator of synthetic data that would create a NeRF model based on real normal traffic data, then modify the scene by editing other agents to recreate ECs. Although NeRF models used to struggle with scene editing and dynamic scenes, recent publications have shown significant improvements in both areas. We could then extract synthetic training data directly from this NeRF and use it for ADS. Our method differs from existing driving simulation frameworks because radiance fields models allow much higher visual quality than manually created models and are much less time-consuming. Unlike similar existing works, we do not rely on a complex physical model to represent the realistic movement of other agents. Instead, we rely on a simpler motion prediction model that is sufficient to generate realistic vehicle motion, allowing us to focus on maintaining our high standard for visual quality while editing radiance fields.