Recalling Multiple Object Manipulation Candidates by Learning Based on Observation

When robot learns object manipulations based on observation of human's daily tasks, only one action is available for one object in an observation, because people have no intention to teach the robot but just use it. This paper proposes a method for predicting possible actions for any objects by learning the relationship between observed actions with object shape similarity. This method learns the code descriptions of object shapes by a variational auto-encoder (VAE) with an object image as input data, and the code descriptions of actions by a conditional VAE with object shape as a condition and an action as input data. Since the action is unknown recall target, it is desirable to obtain the code description of the action from only the object shape during recalling. The distribution of the code description of actions conditioned by input object shape on the obtained code description space is obtained by marginalization of the distribution learned by the encoder part of Conditional VAE. However, since this marginalization is difficult to operate analytically and numerically, a deep regression model that “imitates” this marginal distribution is trained by using a maximum likelihood method based on sampling. Common actions of similar-shaped objects are shared among the similar objects in this “marginalization by imitation” process. Various possible actions for the input object shape can be recalled by repeatedly sampling from the imitated marginal distribution. This paper describes the results of experiment using actual object images and manipulation actions and demonstrates the effectiveness of the proposed method.