Swin Transformer Can Recognize the Emotion of Fear Through Brightness Contrast in Abstract Visual Artworks

This study investigates whether a Swin Transformer model can be trained to recognize emotions in abstract visual artworks. We fine-tuned a Swin Transformer model using the ArtEmis dataset and found that it demonstrated the ability to identify the emotion of fear in abstract visual artworks. GradCAM heatmaps were generated for artworks classified as “fear” by the model, from which we discerned a tendency for the model to focus on regions with abrupt changes in brightness. Correlational analyses between the entropy of the Grad-CAM heatmaps and color-related features of the artworks further supported this observation. Specifically, two significant positive correlations were identified: one between Grad-CAM entropy and brightness variability in the HSV color space, and another between Grad-CAM entropy and contrast in the L channel of the CIE LAB color space. This alignment between the model's attention patterns and findings from psychological studiesindicating that strong brightness contrasts evoke fear and anxiety in humans-suggests that the model effectively captures a psychological principle underlying fear responses to chromatic features in abstract visual artworks.