Selective Prediction for VQA: Enhancing Trust in MLLMs Through Normalized Edit-Distance Conformal Calibration

Visual Question Answering (VQA) is a challenging multimodal task that requires models to generate accurate, freeform answers based on both visual and textual inputs. While Multimodal Large Language Models (MLLMs) have achieved strong performance on open-ended VQA tasks, they often lack calibrated confidence estimation, which limits their reliability in high-stakes applications. In this work, we propose a lightweight, post-hoc calibration framework using Conformal Prediction (CP) to quantify the trustworthiness of MLLM outputs without any model retraining. Our method introduces a normalized edit distance–based nonconformity score that measures the agreement between predicted and ground-truth answers, enabling finegrained control over prediction acceptance. Applied to Qwen-VL7B and evaluated on the OK-VQA benchmark, our approach significantly improves answer reliability within the selected highconfidence subset, achieving exact match on retained examples under moderate confidence thresholds. We further demonstrate competitive overall performance compared to state-of-the-art VQA systems and conduct category-level analysis to highlight the benefits of CP in enhancing model trustworthiness and interpretability.