Multi-Modal Architecture for Cricket Highlights Generation: Using Computer Vision and Large Language Model

Generating highlights for cricket matches is a labour-intensive task that necessitates a high level of both cricket and video editing knowledge. Creating a coherent video with smooth transitions involves sorting through hours of video information, identifying key moments, and merging clips. Sports video summarization has gained a lot of traction in recent days. In this study, we provide a multi-modal framework designed for efficiently producing cricket highlights. We focus on identifying key events while utilizing information from commentary text and visual data. We make use of cues like replays, bowler and umpire positions, and commentary to do so. Starting by splitting the target video into its building blocks (non-replay deliveries), the commentary is transcribed using Automated Speech Recognition (ASR). The textual commentary is then preprocessed so as not to alter the context of the extracted speech. Based on the preprocessed text, a Large Language Model (LLM) is used to predict whether an event occurred after a particular delivery. Two computer vision models—one designed for bowler detection and the other focusing on replay identification—work at the heart of this architecture. These models perform admirably, as evidenced by their respective F1 scores of 0.97 and 0.99. Using BERT LLM exceptional F1 score of 0.96 is achieved. Notably, the architecture’s large-scale training data (CricPulse) includes cricket matches from both the Indian Premier League (IPL) and Pakistan Super League (PSL), demonstrating its adaptability and robustness. In short, our study addresses the challenges of highlights generation by introducing a comprehensive framework for cricket match summarization. We help to accelerate this complex task by utilizing multi-modal inputs and cutting-edge transformer-based models, thereby improving viewing experiences for cricket lovers around the globe.