Unbiased High-Precision Cloud Detection for Advanced Himawari Imager Using Automatic Machine Learning

Clouds play a pivotal role in the global radiation budget and energy cycle, making accurate cloud detection via satellite remote sensing essential for understanding their evolutions. However, existing machine learning-based cloud detection algorithms have limitations, particularly in data preprocessing and feature engineering, and few are designed to cover the entire geostationary satellite observation region. In this study, we integrate three deviation elimination schemes for cloud cover differences and various surface type characterization modes to develop an unbiased, high-precision cloud detection algorithm for the advanced Himawari imager (AHI) onboard Himawari-8, leveraging automatic machine learning (AutoML) techniques. In addition, we provide a new perspective on model interpretability by incorporating concepts from cooperative game theory. Our results indicate that cloud detection algorithms constructed separately for different surfaces yield better results than those applied uniformly across the entire region by inputting surface feather. The optimal algorithm achieves unbiased cloud detection across multiple surfaces by adjusting the output threshold of the mapping function, allowing for cloud detection results with varying confidence levels as required. The true skill statistic of the optimal AutoML-based cloud detection algorithm is 87.32%, which is 13.72% higher than that of the AHI official cloud mask. The deviation rate of the optimal algorithm is only 0.79%, significantly lower than the 20.48% of the AHI official cloud mask. Long-term cloud frequency (CF) tests show that our optimal algorithm's CF distribution is more consistent with cloud-aerosol lidar with orthogonal polarization results compared with the AHI official products.