Advances in Electrical Grid Assets Inspection: Exploring Multimodal Large Language Models

Electrical grid asset inspection is crucial for ensuring infrastructure reliability, preventing failures, and optimizing maintenance strategies. Traditional methods rely heavily on manual labor, making inspections time-consuming, costly, and prone to inconsistencies. Recent advances in deep learning, particularly CNNs and ViTs, have shown potential for automating visual inspection tasks. Vision models are evolving towards multimodal, large-scale architectures. While MLLMs trained on diverse internet-sourced datasets show strong performance, many industrial tasks require models fine-tuned on proprietary data to capture the nuanced features necessary for specialization. This work explores the capabilities and limitations of multi-modal large language models for electrical grid asset inspection, comparing pre-trained and fine-tuned versions of the Florence-VL model across tasks like detailed captioning, object detection, and segmentation. Ground truth labels for caption-related tasks were generated using GPT-4, while annotations from a dataset of insulator defects were used for region-related tasks. Results indicate that fine-tuning improves class recognition and object localization, though captions often remain simplistic and lack domain-specific detail. Despite GPT-4 providing accurate captions, it lacks batch-processing capabilities, limiting its scalability. These findings underscore the importance of domain-specific fine-tuning and high-quality data to enhance MLLM performance.