Multispectral Image Segmentation in Agriculture: Evaluating Deep Learning Models with Train-Test Split and Cross-Validation Strategies

In agricultural robotics, the integration of multi-spectral image processing and deep learning (DL) has become the state-of-the-art (SOTA) in crop monitoring, yield estimation, and efficient land management. This work addresses the impact of different DL-segmentation models and evaluation protocols on multispectral imagery datasets collected by a UAV over vineyards. In terms of evaluation protocols, we have considered train-test split, standard k-fold cross-validation, and group k-fold cross-validation. While the first two assume that the training and test data are drawn from the same underlying distribution, the group k-fold cross-validation protocol assumes that each fold represents distinct distributions. Most works either adopt train-test split or k-fold cross-validation under the assumption that both the training and test sets are drawn from the same distribution. However, this assumption is rarely met in real-world applications. Therefore, the objective of this study is to evaluate and compare different evaluation protocols within the context of a real-world agricultural task, highlighting their limitations and weaknesses. Two SOTA DL-based segmentation models, SegNet and DeepLabV3, are employed to perform semantic segmentation on datasets of three Vineyards. The models have been trained and tested considering single-modality representations. In addition to the RGB modality, models trained on NDVI, GNDVI and early fusion are also evaluated. The performance of the models are evaluated using the IoU metric across different dataset configurations. The results indicate that the early fusion representation achieves the highest performance across the various splitting protocols, compared to the single-input representations. The results also show that the train-test and random k-fold splitting approaches report similar results. However, when employing group k-fold the performance drops consistently across both models and the modalities. This indicates that the models lack strong generalization capabilities to new data and, on the other hand, that the train-test and random k-fold splitting protocols are appropriate to evaluate model within the same distribution but are less adequate for out-of-distribution assessment.