Improving Surface Soil Moisture Estimation with Distributed Deep Learning and HPC

Soil Moisture (SM) is crucial for land surface hydrology, impacting agriculture, ecology, wildlife, and public health. This paper explores the use of Machine Learning (ML) and Deep Learning (DL) models for SM data analysis and estimation, leveraging High-Performance Computing (HPC) for efficient model training and hyperparameter tuning. The research was performed with a commercial partner and SM experts. However, utilising HPC environments requires knowledge of many low-level HPC modules (e.g., various application libraries or vendor-specific drivers like Nvidia's CUDA, NCCL, etc.) and their specific versions needed for interoperability. Such challenges can be overcome using the Unique AI Framework (UAIF) developed in The European Center of Excellence in Exascale Computing “Research on AI- and Simulation-Based Engineering at Exascale” (CoE RAISE) project. The research in this paper contributed to the co-design of several UAIF components using HPC to search for the optimal hyperparameter setup for each ML model. It compares Artificial Neural Networks (ANNs) and Recurrent Neural Networks (RNNs) to a baseline Random Forest (RF) model. Our result demonstrates a significant improvement in accuracy through distributed learning and systematic hyperparameter tuning. The findings suggest that HPC-driven DL can offer scalable, high-resolution SM predictions while leveraging the UAIF by application-domain-specific experts (e.g., SM experts, etc.), enabling easier use of HPC.