Contrastive Learning for Action Assessment Using Graph Convolutional Networks With Augmented Virtual Joints

A fine-grained detection of posture problems for action assessment has a wide range of applications for health care, sports, and rehabilitation. However, there exist many design challenges, e.g., the difficulty of detecting subtle deviations in actions from standard ones, lack of annotated datasets, and even multiple posture problems that may be present in a single action. In this paper, we propose a contrastive learning framework leveraging graph convolutional networks to address these challenges. We introduce Augmented Virtual Joint which is a learned position in space where its associated graphs provide a holistic view of spatio-temporal dynamics of body joints, offering a flexible and generalized representation of actions. Next, we propose Degraded Negative Contrasting, which judiciously contrasts incorrect action samples for effective discrimination of incorrect actions from correct ones. We also propose Frame-Selective Pooling which provides a simple yet effective selection of important frames from action clips. Experiments show that, as compared with the state-of-the-art architectures, the proposed model consistently achieves the best performance under a lack of training data and in the presence of multiple posture problems, which demonstrates its efficacy for fine-grained evaluation of actions.