Detection of Novel Asset Failures in Predictive Maintenance Using Classifier Certainty

Model reliability is one of the main building blocks of trustworthy AI where models are capable of indicating their prediction certainty given the provided input data. Despite the numerous model calibration techniques available to gauge the reliability of trained models, it has been shown that even well-calibrated classification models cannot properly represent their uncertainty for data points outside the training distribution. Such models, therefore, cannot be used in industrial settings to find previously unidentified failures in an asset, leaving the maintenance crews in the dark. To counter that, we analyze the sequence of the predicted probability values from a condition monitoring classification sub-model of a predictive maintenance solution. We demonstrate that the data points from unknown regions of the asset parameter space can be flagged by simply analyzing such sequences. The flagged failure instances of the asset are consequently sent for thorough inspection to the maintenance crew. Furthermore, we demonstrate on a practical real-world scenario the effectiveness of our proposed method (mean accuracy of 94.35 %) by comparing it with models trained only on pre-processed data (mean accuracy of 27.96 %) or models based on autoencoder reconstruction error (mean accuracy of 51.09%). We finally show how the predicted probability values from different regions of the parameter space exhibit a higher degree of separability between normal working condition, known and unknown failure modes compared to signal readings of the system or reconstruction errors of an anomaly detection model.