Multi-view object detection using NeRF and YOLO

Object detection is one of the fundamental tasks of computer vision with numerous applications in various fields. Existing object detection methods mostly focus on enhancing single-view detection for occluded objects or operate directly on 3D object representations. However, in contemporary applications, multiple RGB input views are often available, and detecting objects obscured by partial or full occlusion is crucial. This holds particular significance in autonomous vehicles and driver assistance systems, where pedestrians and other road users frequently appear from hidden positions. In this paper, we present a new model for detecting partially or fully occluded objects by integrating NeRF (Neural Radiance Fields) with YOLO (You Only Look Once) object detection. Our approach leverages inputs from multiple viewpoints to detect objects, even if they are not directly visible from the original viewpoint. We accomplish this using pixelNeRF, which reconstructs 3D scenes and generates novel views from a small set of input images. We replace the conventional RGB output with YOLO features and render these features onto one of the input images, facilitating object detection utilizing multiple input views. By implementing this approach, we achieve a superior F
1
score compared to the state-of-the-art single-view YOLO network. We accomplish this by detecting occluded objects and therefore increasing recall. We train and evaluate the model using a custom synthetic dataset featuring urban road intersections, specifically designed to simulate scenarios with numerous occluded objects.