Multi-Modal Fusion Methods with Local Neighborhood Information for Crop Yield Prediction at Field and Subfield Levels

Yield prediction at both field and subfield level poses a significant challenge, yet it holds paramount importance for decision-making and food security within the agricultural sector. Recent efforts, focused on integrating remote sensing data coupled with machine learning models, thereby creating globally scalable models for various crop types. This study underscores the effectiveness of Sentinel-2 and complementary data sources such as weather, soil, and terrain in enhancing machine learning-based yield prediction. We address the limitations of previous works and introduce a framework that incorporates local neighborhood information using convolutional neural networks and geographical coordinates. Additionally, we address the complexity of sensor fusion, showcasing both input fusion and feature fusion frameworks. We highlight that handling modalities with varying spatial and temporal resolutions requires adequate and advanced fusion mechanisms in crop yield prediction. Notably, this study reports an R
²
of 0.86 for soybean in Argentina using a feature fusion scheme with attention mechanism. The results are demonstrated on a large yield dataset for soybean, wheat, and rapeseed distributed across Argentina, Uruguay, and Germany.