Preliminary Results of Emotions Detection with AMIGO App

AI agents' emotional intelligence (EI) is crucial for fostering empathetic human-computer interactions. This study introduces AMIGO, an Emotionally Aware AI (EAAI) Agent trained on the AMIGOS dataset to enhance emotion recognition. The research compares four fusion approaches: unimodal (facial video frames only), early fusion (combining facial and physiological features at input), mid-fusion (separate encoding of modalities before merging via self-attention), and late fusion (final combination of different modalities). Preliminary experiments confirm that integrating physiological signals with facial data, alongside attention mechanisms, improves the agent’s robustness and reliability. Adding depth video frames should enhance the results. These findings highlight the potential of multimodal AI systems in applications such as mental health support, social robotics, and educational tutoring.