Granular Analysis of Pretrained Object Detectors

Object detectors have become the fundamental building blocks of many real-world machine learning applications. Even though different problem domains require their own unique object detector specifications, it is common practice to take a pretrained object detector off the shelf and either use it as-is or fine-tune it with limited amounts of labeled training data. However, the image distribution that such object detectors are trained on is more often times than not different from the targeted problem domain of interest. In this work, we scrutinize whether existing state-of-the-art object detectors have the ability to generalize across different domains. Specifically, we evaluate whether widely used pretrained state-of-the-art objectors such as Faster-RCNN and YOLOv3 generalize to images sampled from an autonomous vehicle application. For this purpose, we evaluate the performance of detectors on localizing humans and vehicles on images from the KITTI dataset and report results of detailed subgroup analysis on multiple factors. Our analysis shows that the detectors exhibit different levels of performance on varying levels of object-object occlusion and object size. Moreover, we report the performance drop of the object detectors with different image-altering hazardous factors.