Multimodal AI for Seamless Communication: Integrating Computer Vision and Speech Translation with LLMs

Multimodal Artificial Intelligence (AI) dramatically enhances linguistically and visually AI-aided communication. This study proposes a new framework integrating computer vision, speech translation and large language models (LLMs) for context-sensitive and real-time communication. The proposed system employs computer vision for scene comprehension and text capturing, speech translation for real-time multilingual conversation, and LLMs for comprehension and response generation. Combining these technologies enhances interaction between machines and humans, provides assistive technology solutions for people with disabilities and aids communication in different languages in various settings. In this work, the framework utilises a multi-stream fusion strategy to enable coherent and pertinent responses to be generated during information processing across various modalities. A system benchmark evaluation confirmed the system's effectiveness in speech-to-text, image-to-text and multilingual voice recognition tasks instead of classical single-modality methods. Benchmark results show a BLEU score of 41.24, outperforming mBART and the Transformer baseline, while remaining competitive with M2M-100. The results emphasise the role of multimodal AI in overcoming communication barriers and their drastic use in educational purposes, as well as in healthcare, customer relations, and international settings and collaborations.