Adapting Vision-Language Models for Precision Agriculture: A Study on Crop Segmentation based on UAV Remote Sensing Data

With increasing global food security challenges, agricultural remote sensing image analysis has become particularly important for improving production efficiency and resource utilization in precision agriculture. However, traditional crop segmentation methods for UAV remote sensing images rely heavily on large amounts of labeled data and specialized deep learning architectures, which pose significant challenges for practical agricultural applications due to high annotation costs and limited model generalization. Although general vision-language models (VLMs) have demonstrated remarkable capabilities in natural image understanding, their effectiveness in processing specialized agricultural remote sensing data, particularly for precise crop segmentation tasks, remains largely unexplored.This research proposes a novel approach that adapts multimodal large language models for crop segmentation in UAV remote sensing images by reformulating the segmentation task as a conversational format. Our method transforms traditional pixel-level segmentation into a text-based coordinate prediction task, where segmentation masks are converted to polygon coordinates and represented in XML format. To address the unique characteristics of agricultural remote sensing data, we designed two types of conversational prompts: general object segmentation and crop-specific segmentation.Methodologically, we introduce a coordinate-to-text conversion strategy that transforms segmentation labels into structured XML tags containing polygon coordinates, enabling VLMs to perform segmentation through natural language generation. We systematically evaluate four state-of-the-art vision-language models (Qwen2-VL-7B, Qwen2.5-VL-7B, LLaVA-1.5-7B, and LLaMA3-LLaVA-Next-8B) using Low-Rank Adaptation (LoRA) fine-tuning techniques that selectively optimize only the language model components while keeping the visual encoders frozen.Experimental validation demonstrates that adapted VLMs can effectively perform crop segmentation tasks with varying degrees of success across different crop types. Qwen2-VL achieves the best overall performance with an F1-score of 49.16% and IoU of 46.61%, significantly outperforming other models. Notably, tobacco segmentation shows superior results (F1: 78.42%, IoU: 75.64%) compared to corn and barley, indicating crop-specific adaptation capabilities. The results reveal that while VLM-based approaches may not yet match specialized segmentation models in absolute accuracy, they offer unique advantages in few-shot learning scenarios and provide a promising foundation for developing more generalizable agricultural remote sensing solutions.