Predicting Part Affordances of Objects Using Two-Stream Fully Convolutional Network with Multimodal Inputs

For a robot to manipulate an object, it has to understand the functions and the actions that can be subjected to the object. This set of information is known as affordance of the object. Affordances are generally defined by the geometrical structures and physical properties of the objects. In this paper, we present an affordance detection network (ADNet) for detecting object affordances using multimodal input i.e., RGB-D data. The method is based on the state-of-the-art fully convolutional network with two encoding streams and one decoding stream. In the presented formulation, the network learns powerful discriminative features independently from the RGB and depth images, which enables it to abstract rich photometrical and geometrical properties of the objects. The multimodal encoding is combined at multiple stages of the network using the late-fusion strategy and used is for predicting the potential affordances of the objects.