MemRank: Memory-Augmented Similarity Ranking for Video-Based Depression Severity Estimation

Deep learning-based methods have shown substantial promise in visual depression severity estimation. Nonetheless, their effectiveness is limited by the scarce availability of labeled depression data, potentially leading to overfitting during representation learning. One feasible approach to address this issue is to incorporate, in the training objective, regularization that considers the unique characteristics of depression data. Typical regularization includes the similarity ranking through ordered consistency between visual features and their target scores. However, previous ranking methods are limited to using only samples within a mini-batch, resulting in a decreased regularization effect in depression representation learning. To address this limitation, we propose MemRank, a global similarity ranking method that operates not only on mini-batch samples but also on a well-designed feature memory, which stores smoothed and dynamically updated feature prototypes at diverse levels of depression during training. Furthermore, we show that incorporating the feature memory in the regression loss enhances the stability of training a deep regressor, leading to improved depression predictions. Empirically and analytically, we show that our MemRank outperforms alternative ranking methods and achieves state-of-the-art results on two benchmark datasets.