JoTyMo: Joint Type and Movement Detection from RGB Images of Before and After Interacting with Articulated Objects

This paper presents a CNN-based network architecture aimed to classify and detect joint types within Articulated objects, specifically the Push-P joints, P-joints, R-joints and objects lacking joints. The movement modeling of the detected objects is explored by processing pre and post interaction RGB images within the SAPIEN PartNet-Mobility dataset. In order to achieve this, an architecture is proposed to leverage consecutive CNN encoders based on the VGG architecture, in order to classify joints based on pre and post-interaction images. Additionally, in order to detect the effect-point and movement vector, a convolutional encoder is applied for each joint type. Moreover, extensive evaluation is made to showcase the results, achieving 96% accuracy in joint classification and 94% accuracy in regression on the considered dataset. Elaborate details regarding network architectures, training procedures, and testing methodologies are available at source code repository.