Determining on-shelf availability based on RGB and ToF depth cameras

In this paper, a method of calculating the occupancy of a shelf will be presented. A vision pillar composed of two RGB cameras and two ToF depth cameras will be used to scan a shelf and determine the percentage of emptiness for the products. Since the cameras will scan a small section of the shelf at the time, a stitching method will be applied on both depth and RGB stream. In order to identify categories of products, along with their spatial delimitation within a shelf, a segmentation neural network is used. By combining the segmentation output with the depth information, the distance from the pillar to the product can be determined. Since this type of information does not help a human operator or a supervisor in understanding the state of the shelf or in making an appropriate decision, such as restocking, a method for computing the shelf occupancy is proposed. Optimization must be done to the processing algorithms to enable them to run on an embedded platform, which will allow for the implementation of the proposed architecture on a platform deployed in a production environment. All the computation is done on board of the mobile robot without transmitting the data to an external server.