R3-Net: A Deep Network for Multioriented Vehicle Detection in Aerial Images and Videos

Vehicle detection is a significant and challenging task in aerial remote sensing applications. Most existing methods detect vehicles with regular rectangle boxes and fail to offer the orientation of vehicles. However, the orientation information is crucial for several practical applications, such as the trajectory and motion estimation of vehicles. In this paper, we propose a novel deep network, called a rotatable region-based residual network (R³-Net), to detect multioriented vehicles in aerial images and videos. More specially, R³-Net is utilized to generate rotatable rectangular target boxes in a half coordinate system. First, we use a rotatable region proposal network (R-RPN) to generate rotatable region of interests (R-RoIs) from feature maps produced by a deep convolutional neural network. Here, a proposed batch averaging rotatable anchor strategy is applied to initialize the shape of vehicle candidates. Next, we propose a rotatable detection network (R-DN) for the final classification and regression of the R-RoIs. In R-DN, a novel rotatable position-sensitive pooling is designed to keep the position and orientation information simultaneously while downsampling the feature maps of R-RoIs. In our model, R-RPN and R-DN can be trained jointly. We test our network on two open vehicle detection image data sets, namely, DLR 3K Munich Data set and VEDAI Data set, demonstrating the high precision and robustness of our method. In addition, further experiments on aerial videos show the good generalization capability of the proposed method and its potential for vehicle tracking in aerial videos. The demo video is available at https://youtu.be/xCYD-tYudN0.

I. Introduction

A long with the now widespread availability of airplanes and unmanned aerial vehicles (UAVs), the detection and localization of small targets in high-resolution airborne imagery have been attracting a lot of attentions in the remote sensing community [1]–[6]. They have numerous useful applications, to name a few, surveillance, defense, and traffic planning [7]–[11]. In this paper, vehicles are considered the small targets of interest, and our task is to automatically detect and localize vehicles from complex urban scenes (see Fig. 1). This is actually an exceedingly challenging task, because of: 1) huge differences in visual appearance among cars (e.g., colors, sizes, and shapes) and 2) various orientations of vehicles. Fig. 1.

Examples of multioriented vehicle detection produced with the proposed network, over two scenes taken from DLR 3K Munich Data set. Best viewed zoomed in.