Circular Convolutional Neural Networks for Panoramic Images and Laser Data

Circular Convolutional Neural Networks (CCNN) are an easy to use alternative to CNNs for input data with wrap-around structure like 360° images and multi-layer laserscans. Although circular convolutions have been used in neural networks before, a detailed description and analysis is still missing. This paper closes this gap by defining circular convolutional and circular transposed convolutional layers as the replacement of their linear counterparts, and by identifying pros and cons of applying CCNNs. We experimentally evaluate their properties using a circular MNIST classification and a Velodyne laserscanner segmentation dataset. For the latter, we replace the convolutional layers in two state-of-the-art networks with the proposed circular convolutional layers. Compared to the standard CNNs, the resulting CCNNs show improved recognition rates in image border areas. This is essential to prevent blind spots in the environmental perception. Further, we present and evaluate how weight transfer can be used to obtain a CCNN from an available, readily trained CNN. Compared to alternative approaches (e.g. input padding), our experiments show benefits of CCNNs and transferred CCNNs regarding simplicity of usage (once the layer implementations are available), performance and runtime for training and inference. Implementations for Keras with Tensorflow are provided online².

I. Introduction

Convolutional Neural Networks (CNN) are a widely used and powerful tool for processing and interpreting image-like data in various domains like automotive, robotics or medicine. Compared to layer-wise fully connected networks, CNNs benefit from weight-sharing: Instead of learning a large set of weights from all input pixels to an element of the next layer, they learn few weights of a small set of convolutional kernels that are applied all over the image. The goal is to achieve shift equivariance: A trained pattern (e.g. to detect a car in a camera image) should provide strong response at the particular location of the car in the image, independent of whether this location is, e.g., in the left or right part of the image. However, this goal is missed at locations close to the image borders, where the receptive field of the convolution exceeds the input. Typically, these out-of-image regions are filled with zero-padding. During the repeated convolutions in a CNN, this zero-padding occurs at each layer and the effect of distorted filter responses grows from the image borders towards the interior. While this seems inevitable for imagery from pinhole-model cameras, this is not the case for panoramic data. In panoramic data, there is at least one dimension with wrap-around structure and without an inherent image border. However, feeding panoramic data as 2D images to a standard CNN artificially introduces such borders. Fig. 1 shows the example of object recognition using a car mounted Velodyne - its 360° degree depth and reflectance images are practically important examples for panoramic data. Ignoring the wrap-around connections by using standard CNNs creates blind spots near the image borders where we are unable to interpret the environment.