Enhancing UAS-Based Multispectral Semantic Segmentation Through Feature Engineering

Deep learning (DL) is one of the key tools for analyzing images beyond the visible light spectrum, such as thermal data, for energy-related inspection and fault detection. However, publications using multispectral data focus on developing specialized models to handle quality issues without considering the imagery itself. This article investigates how feature engineering (FE), the process of adapting raw data to serve as DL training data, can impact performance when transferring prevalent model architectures to combined red, green, blue (RGB) thermal imagery. The popular U-Net is utilized for the common task of multiclass semantic segmentation in remote sensing. A comprehensive ablation study is performed on a novel, uncrewed aircraft system-based dataset from two German cities to detect thermal urban features. Common performance metrics, training, and energy consumption statistics are compared to find the most suitable combination of platform-specific and general enhancing FE while identifying the impact of resolution, channel count, RGB, and color information. The study reveals FE to significantly influence predictive performance, where the choice of ablation parameters are found to have a 7%pt –10%pt impact. Computational resource utilization depends on image size, following a logarithmic growth curve. Importantly, the study demonstrates that in-depth FE of thermal imagery can replace the need for additional RGB data.