Deep Hierarchical Temporal Data Fusion Improves Yield Estimation under Extreme Climate Stress

Accurate and reliable crop yield estimation under climate stress conditions is essential to ensure global food security. The crop yield variation can be characterized by the daily available climate factors and captured by the multi-temporal remote sensing observations. However, the fusion of multi-temporal remote sensing and meteorological data with different temporal resolutions remains insufficient for crop yield estimation. In this paper, we introduce a two-stream model to perform a feature-level fusion of the vegetation indices and climate variables across multiple time scales. Each stream builds upon a pyramid structure that progressively aggregates the input time series into multi-time-scale feature embeddings. The proposed model is tested in the nine Midwestern states within the US Corn Belt at county level over the years 2006-2012. Quantitative results show that the two-stream model outperforms the one-stream model and typical machine learning under climate stress. The higher estimation performance indicates that feature-level fusion is superior to utilizing the multi-source data rather than simply stacking the input data. Spatial visualization demonstrates that our approach effectively learns to capture the spatial pattern of yield loss, particularly in extreme heat and drought condition. Expanding the number of streams within the proposed architecture enables fusing more data sources, holding promise for crop yield modeling at larger scales.