Enhanced Multimodal Chain-of-Thought with Visual Self-Contrastive Distillation

Chain-of-thought (CoT) reasoning research has predominantly focused on language modality, neglecting the intricate interaction of multiple modalities crucial for real-world reasoning, such as visual question answering. Current methods primarily concentrate on modal conversion and feature fusion to enable language models to utilize CoT in a multimodal environment. However, these methods have inherent hallucination limitations. They tend to excessively rely on the prior knowledge of language unimodal and lack guidance in learning visual differences, often resulting in content incongruent with the images. This study introduces a novel approach, visual self-contrastive distillation (VSCD). The proposed method equalizes the status of language and vision modalities by freezing both encoders, enabling more balanced learning. Furthermore, we use contrastive decoding to enable the model to learn image differences during distillation, enhancing its understanding of visual nuances. The comprehensive experiment on the ScienceQA dataset demonstrates the superiority of the proposed VSCD method across various categories of multimodal CoT. Code and data are released at https://github.com/zgMin/VSCD.