Spectral Representation of Behaviour Primitives for Depression Analysis | IEEE Journals & Magazine | IEEE Xplore

Spectral Representation of Behaviour Primitives for Depression Analysis


Abstract:

Depression is a serious mental disorder affecting millions of people all over the world. Traditional clinical diagnosis methods are subjective, complicated and require ex...Show More

Abstract:

Depression is a serious mental disorder affecting millions of people all over the world. Traditional clinical diagnosis methods are subjective, complicated and require extensive participation of clinicians. Recent advances in automatic depression analysis systems promise a future where these shortcomings are addressed by objective, repeatable, and readily available diagnostic tools to aid health professionals in their work. Yet there remain a number of barriers to the development of such tools. One barrier is that existing automatic depression analysis algorithms base their predictions on very brief sequential segments, sometimes as little as one frame. Another barrier is that existing methods do not take into account what the context of the measured behaviour is. In this article, we extract multi-scale video-level features for video-based automatic depression analysis. We propose to use automatically detected human behaviour primitives as the low-dimensional descriptor for each frame. We also propose two novel spectral representations, i.e., spectral heatmaps and spectral vectors, to represent video-level multi-scale temporal dynamics of expressive behaviour. Constructed spectral representations are fed to Convolution Neural Networks (CNNs) and Artificial Neural Networks (ANNs) for depression analysis. We conducted experiments on the AVEC 2013 and AVEC 2014 benchmark datasets to investigate the influence of interview tasks on depression analysis. In addition to achieving state of the art accuracy in severity of depression estimation, we show that the task conducted by the user matters, that fusion of a combination of tasks reaches highest accuracy, and that longer tasks are more informative than shorter tasks, up to a point.
Published in: IEEE Transactions on Affective Computing ( Volume: 13, Issue: 2, 01 April-June 2022)
Page(s): 829 - 844
Date of Publication: 30 January 2020

ISSN Information:

Funding Agency:

Citations are not available for this document.

1 Introduction

Major Depression Disorder (MDD) is a psychiatric disorder defined as a state of low mood with a significantly higher level of duration/severity. It negatively impacts one’s day to day life, causing people to become reluctant or unable to perform everyday activities, which can negatively affect a person’s sleeping, sense of well-being, behaviour, feelings, etc. [1]. In extreme cases it can lead to suicide, which is the leading cause of death for men under 50 in the UK [2]. Depression is currently the most prevalent mental health disorder and the leading cause of disability in developed countries. A correct and early diagnosis can be vital to provide the right mental health support at the right time. It facilitates communication between (potential) patients and health professionals about the support and services they need [3] and is the key to choosing the correct intervention for treating patients.

Cites in Papers - |

Cites in Papers - IEEE (42)

Select All
1.
Eric Hsiao-Kuang Wu, Ting-Yu Gao, Chia-Ru Chung, Chun-Chuan Chen, Chia-Fen Tsai, Shih-Ching Yeh, "Mobile Virtual Assistant for Multi-Modal Depression-Level Stratification", IEEE Transactions on Affective Computing, vol.16, no.2, pp.611-623, 2025.
2.
Jiaqi Xu, Hatice Gunes, Keerthy Kusumam, Michel Valstar, Siyang Song, "Two-Stage Temporal Modelling Framework for Video-Based Depression Recognition Using Graph Representation", IEEE Transactions on Affective Computing, vol.16, no.1, pp.161-178, 2025.
3.
Rahul Islam, Tongze Zhang, Sang Won Bae, "MoodCam: Mood Prediction Through Smartphone-Based Facial Affect Analysis in Real-World Settings", 2024 IEEE Smart World Congress (SWC), pp.599-607, 2024.
4.
Muhammad Hamza Khan, Muhammad Maiid, Aamir Arsalan, "Exploring the Diagnostic Potential of Heart Sound Spectrograms in Depression Identification", 2024 19th International Conference on Emerging Technologies (ICET), pp.1-6, 2024.
5.
Hanzhe Xu, Xuefei Liu, Cong Cai, Kang Zhu, Jizhou Cui, Ruibo Fu, Heng Xie, Jianhua Tao, Zhengqi Wen, Ziping Zhao, Guanjun Li, Le Wang, Hao Lin, "Temporal Shift for Personality Recognition with Pre-Trained Representations", 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP), pp.446-450, 2024.
6.
Batuhan Sayis, Rui Gao, Siyang Song, Hatice Gunes, "Learning Graph Representation for Predicting Student Mental Wellbeing in Robot Assisted Journal Writing Context", 2024 12th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp.238-246, 2024.
7.
Zihan Wang, Siyang Song, Cheng Luo, Songhe Deng, Weicheng Xie, Linlin Shen, "Multi-Scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition", 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.1270-1280, 2024.
8.
Yuchen Pan, Junjun Jiang, Kui Jiang, Zhihao Wu, Keyuan Yu, Xianming Liu, "OpticalDR: A Deep Optical Imaging Model for Privacy-Protective Depression Recognition", 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.1303-1312, 2024.
9.
Haotian Shen, Siyang Song, Hatice Gunes, "Multi-modal Human Behaviour Graph Representation Learning for Automatic Depression Assessment", 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pp.1-10, 2024.
10.
Haifeng Lu, Zhiyang You, Yi Guo, Xiping Hu, "MAST-GCN: Multi-Scale Adaptive Spatial-Temporal Graph Convolutional Network for EEG-Based Depression Recognition", IEEE Transactions on Affective Computing, vol.15, no.4, pp.1985-1996, 2024.
11.
Xiaodong Duan, Haoyong Li, Feng Yang, Bo Chen, Jingshuai Dong, Yuangang Wang, "Multimodal Automatic Personality Perception Using ViT, BiLSTM and VGGish", 2024 5th International Conference on Computer Engineering and Application (ICCEA), pp.549-553, 2024.
12.
Mingyue Niu, Ya Li, Jianhua Tao, Xiuzhuang Zhou, Björn W. Schuller, "DepressionMLP: A Multi-Layer Perceptron Architecture for Automatic Depression Level Prediction via Facial Keypoints and Action Units", IEEE Transactions on Circuits and Systems for Video Technology, vol.34, no.9, pp.8924-8938, 2024.
13.
Rongfan Liao, Siyang Song, Hatice Gunes, "An Open-Source Benchmark of Deep Learning Models for Audio-Visual Apparent and Self-Reported Personality Recognition", IEEE Transactions on Affective Computing, vol.15, no.3, pp.1590-1607, 2024.
14.
Shiqing Zhang, Xingnan Zhang, Xiaoming Zhao, Jiangxiong Fang, Mingyue Niu, Ziping Zhao, Jun Yu, Qi Tian, "MTDAN: A Lightweight Multi-Scale Temporal Difference Attention Networks for Automated Video Depression Detection", IEEE Transactions on Affective Computing, vol.15, no.3, pp.1078-1089, 2024.
15.
Tao Chen, Yanrong Guo, Shijie Hao, Richang Hong, "Semi-Supervised Domain Adaptation for Major Depressive Disorder Detection", IEEE Transactions on Multimedia, vol.26, pp.3567-3579, 2024.
16.
Md Taufeeq Uddin, Lijun Yin, Shaun Canavan, "Spatio-Temporal Graph Analytics on Secondary Affect Data for Improving Trustworthy Emotional AI", IEEE Transactions on Affective Computing, vol.15, no.1, pp.30-49, 2024.
17.
Yuchen Pan, Yuanyuan Shang, Zhuhong Shao, Tie Liu, Guodong Guo, Hui Ding, "Integrating Deep Facial Priors Into Landmarks for Privacy Preserving Multimodal Depression Recognition", IEEE Transactions on Affective Computing, vol.15, no.3, pp.828-836, 2024.
18.
Ruibin Wang, Jing Guo, Jiashun Wang, Lang He, Yun Yang, "A Multi-Frame Rate Network with Attention Mechanism for Depression Severity Estimation", 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp.2679-2686, 2023.
19.
Misha Sadeghi, Bernhard Egger, Reza Agahi, Robert Richer, Klara Capito, Lydia Helene Rupp, Lena Schindler-Gmelch, Matthias Berking, Bjoern M. Eskofier, "Exploring the Capabilities of a Language Model-Only Approach for Depression Detection in Text Data", 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), pp.1-5, 2023.
20.
Euodia Dodd, Siyang Song, Hatice Gunes, "A Framework for Automatic Personality Recognition in Dyadic Interactions", 2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp.1-8, 2023.
21.
Felipe Moreno, Sharifa Alghowinem, Hae Won Park, Cynthia Breazeal, "Expresso-AI: An Explainable Video-Based Deep Learning Models for Depression Diagnosis", 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII), pp.1-8, 2023.
22.
Christian Nash, Rajesh Nair, Syed Mohsen Naqvi, "Machine Learning in ADHD and Depression Mental Health Diagnosis: A Survey", IEEE Access, vol.11, pp.86297-86317, 2023.
23.
Daegil Choi, Gengjia Zhang, Da Eun Kim, Jaehyo Jung, "Depression Diagnosis Algorithm Based on 2-stream CNN Using Facial Image", 2023 IEEE/ACIS 23rd International Conference on Computer and Information Science (ICIS), pp.43-47, 2023.
24.
Zihan Wang, Siyang Song, Cheng Luo, Yuzhi Zhou, Shiling Wu, Weicheng Xie, Linlin Shen, "Spatial-Temporal Graph-Based AU Relationship Learning for Facial Action Unit Detection", 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.5899-5907, 2023.
25.
Lan Zhang, Jian Zhao, Lang He, Jian Jia, Xianjia Meng, "An Improved Global–Local Fusion Network for Depression Detection Telemedicine Framework", IEEE Internet of Things Journal, vol.10, no.22, pp.20230-20240, 2023.
26.
Chuang Yu, "Non-verbal Facial Action Units-Based Automatic Depression Classification", 2023 11th International Conference on Bioinformatics and Computational Biology (ICBCB), pp.169-174, 2023.
27.
Constantino Álvarez Casado, Manuel Lage Cañellas, Miguel Bordallo López, "Depression Recognition Using Remote Photoplethysmography From Facial Videos", IEEE Transactions on Affective Computing, vol.14, no.4, pp.3305-3316, 2023.
28.
Siyang Song, Zilong Shao, Shashank Jaiswal, Linlin Shen, Michel Valstar, Hatice Gunes, "Learning Person-Specific Cognition From Facial Reactions for Automatic Personality Recognition", IEEE Transactions on Affective Computing, vol.14, no.4, pp.3048-3065, 2023.
29.
Bochao Zou, Jiali Han, Yingxue Wang, Rui Liu, Shenghui Zhao, Lei Feng, Xiangwen Lyu, Huimin Ma, "Semi-Structural Interview-Based Chinese Multimodal Depression Corpus Towards Automatic Preliminary Screening of Depressive Disorders", IEEE Transactions on Affective Computing, vol.14, no.4, pp.2823-2838, 2023.
30.
Md Azher Uddin, Joolekha Bibi Joolee, Kyung-Ah Sohn, "Deep Multi-Modal Network Based Automated Depression Severity Estimation", IEEE Transactions on Affective Computing, vol.14, no.3, pp.2153-2167, 2023.

Cites in Papers - Other Publishers (33)

1.
Yongtang Bao, Yuzhen Wang, Yutong Qi, Qing Yang, Ruijun Liu, Liping Feng, "Emotion-Assisted multi-modal Personality Recognition using adversarial Contrastive learning", Knowledge-Based Systems, pp.113504, 2025.
2.
JIE YANG, SHUO WANG, "THE BEHAVIOR ANALYSIS OF DEEP LEARNING MODEL IN THE TREATMENT OF DEPRESSION RESEARCH", Journal of Mechanics in Medicine and Biology, 2025.
3.
Cui Cao, Lang He, "LOGLformer: Integrating local and global characteristics for depression scale estimation from facial expressions", Review of Scientific Instruments, vol.96, no.3, 2025.
4.
Zhenyu Liu, Bailin Chen, Shimao Zhang, Jiaqian Yuan, Yang Wu, Hanshu Cai, Xin Chen, Lin Liu, Yimiao Zhao, Huan Mei, Jiahui Deng, Yanping Bao, Bin Hu, "MPDRM: A Multi-Scale Personalized Depression Recognition Model via facial movements", Neurocomputing, pp.129669, 2025.
5.
Xiaoming Cao, Lingling Zhai, Pengpeng Zhai, Fangfei Li, Tao He, Lang He, "Deep learning-based depression recognition through facial expression: A systematic review", Neurocomputing, pp.129605, 2025.
6.
Mengting Ma, Yizhen Jiang, Mengjiao Zhao, Xiaowen Ma, Wei Zhang, Siyang Song, "Deep spatial-spectral fusion transformer for remote sensing pansharpening", Information Fusion, pp.102980, 2025.
7.
Daniel Harlev, Shir Singer, Maya Goldshalger, Noham Wolpe, Eyal Bergmann, "Acoustic speech features are associated with late‐life depression and apathy symptoms: Preliminary findings", Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, vol.17, no.1, 2025.
8.
Mengyuan Yang, Yuanyuan Shang, Jingyi Liu, Zhuhong Shao, Tie Liu, Hui Ding, Hailiang Li, "LMS-VDR: Integrating Landmarks into Multi-scale Hybrid Net for Video-Based Depression Recognition", Pattern Recognition and Computer Vision, vol.15040, pp.299, 2025.
9.
Haijun Lin, Jing Fang, Junpeng Zhang, Xuhui Zhang, Weiying Piao, Yukun Liu, "Resting-State Electroencephalogram Depression Diagnosis Based on Traditional Machine Learning and Deep Learning: A Comparative Analysis", Sensors, vol.24, no.21, pp.6815, 2024.
10.
Keshan Yan, Shengfa Miao, Xin Jin, Yongkang Mu, Hongfeng Zheng, Yuling Tian, Puming Wang, Qian Yu, Da Hu, "TCEDN: A Lightweight Time-Context Enhanced Depression Detection Network", Life, vol.14, no.10, pp.1313, 2024.
11.
Ningya Xu, Hua Huo, Jiaxin Xu, Lan Ma, Jinxuan Wang, "Automatic diagnosis of depression based on attention mechanism and feature pyramid model", PLOS ONE, vol.19, no.3, pp.e0295051, 2024.
12.
Siyang Song, Yiming Luo, Tugba Tumer, Michel Valstar, Hatice Gunes, "Loss Relaxation Strategy for Noisy Facial Video-based Automatic Depression Recognition", ACM Transactions on Computing for Healthcare, 2024.
13.
Shanliang Yang, Lichao Cui, Lei Wang, Tao Wang, Jiebing You, "Enhancing multimodal depression diagnosis through representation learning and knowledge transfer", Heliyon, pp.e25959, 2024.
14.
Wheidima Carneiro de Melo, Eric Granger, Miguel Bordallo Lopez, "Facial expression analysis using Decomposed Multiscale Spatiotemporal Networks", Expert Systems with Applications, vol.236, pp.121276, 2024.
15.
Jian Zhao, Lan Zhang, Yihai Cui, Jia Shi, Lang He, "A novel Image-Data-Driven and Frequency-Based method for depression detection", Biomedical Signal Processing and Control, vol.86, pp.105248, 2023.
16.
Biswajeet Sahu, Kumar Palo, Mahesh Chandra, "Modeling of human mood states from voice using adaptively tuned neuro-fuzzy inference system", Serbian Journal of Electrical Engineering, vol.20, no.1, pp.13, 2023.
17.
Yuhao Wang, Zepeng Li, "Depression Detection with Dynamic and Static Visual Features", Journal of Circuits, Systems and Computers, vol.32, no.18, 2023.
18.
Mohana Shanmugam, Nur Nesa Nashuha Ismail, Pritheega Magalingam, Nik Nur Wahidah Nik Hashim, Dalbir Singh, "Understanding the Use of Acoustic Measurement and Mel Frequency Cepstral Coefficient (MFCC) Features for the Classification of Depression Speech", Current and Future Trends on Intelligent Technology Adoption, vol.1128, pp.345, 2023.
19.
Huiting Fan , Xingnan Zhang , Yingying Xu , Jiangxiong Fang , Shiqing Zhang , Xiaoming Zhao , Jun Yu , " Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals ", Information Fusion , pp. 102161 , 2023 .
20.
Daniel Highland, Gang Zhou, "A review of detection techniques for depression and bipolar disorder", Smart Health, vol.24, pp.100282, 2022.
21.
Emna Rejaibi, Ali Komaty, Fabrice Meriaudeau, Said Agrebi, Alice Othmani, "MFCC-based Recurrent Neural Network for automatic clinical depression recognition and assessment from speech", Biomedical Signal Processing and Control, vol.71, pp.103107, 2022.
22.
Lang He, Chenguang Guo, Prayag Tiwari, Rui Su, Hari Mohan Pandey, Wei Dang, "DepNet: An automated industrial intelligent system using deep learning for video?based depression analysis", International Journal of Intelligent Systems, vol.37, no.7, pp.3815, 2022.
23.
Lang He, Chenguang Guo, Prayag Tiwari, Hari Mohan Pandey, Wei Dang, "Intelligent system for depression scale estimation with facial expressions and case study in industrial intelligence", International Journal of Intelligent Systems, vol.37, no.12, pp.10140, 2022.
24.
Lang He, Jonathan Cheung-Wai Chan, Zhongmin Wang, "Automatic depression recognition using CNN with attention mechanism from videos", Neurocomputing, vol.422, pp.165, 2021.
25.
Zilong Shao, Siyang Song, Shashank Jaiswal, Linlin Shen, Michel Valstar, Hatice Gunes, "Personality Recognition by Modelling Person-specific Cognitive Processes using Graph Representation", Proceedings of the 29th ACM International Conference on Multimedia, pp.357, 2021.
26.
Linxi Chen, "Deep Learning Static and Dynamic Movie Attributes for Box Office Prediction", 2021 5th International Conference on Computer Science and Artificial Intelligence, pp.402, 2021.
27.
Joy O. Egede, Dominic Price, Deepa B. Krishnan, Shashank Jaiswal, Natasha Elliott, Richard Morriss, Maria J. Galvez Trigo, Neil Nixon, Peter Liddle, Christopher Greenhalgh, Michel Valstar, "Design and Evaluation of Virtual Human Mediated Tasks for Assessment of Depression and Anxiety", Proceedings of the 21th ACM International Conference on Intelligent Virtual Agents, pp.52, 2021.
28.
Meng Li, Changyan Lin, Lixia Shu, Xin Pu, Yu Chen, Heng Wu, Jiasong Li, Hongshuai Cao, "Crossmodal Matching Transformer based X-ray and CT image registration for TEVAR", 2021 6th International Conference on Biomedical Signal and Image Processing, pp.85, 2021.
29.
Andry Chowanda, "Separable convolutional neural networks for facial expressions recognition", Journal of Big Data, vol.8, no.1, 2021.
30.
Muhammad Muzammel, Hanan Salam, Alice Othmani, "End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis", Computer Methods and Programs in Biomedicine, vol.211, pp.106433, 2021.

Contact IEEE to Subscribe

References

References is not available for this document.