What to Check: Systematic Selection of Transformations for Analyzing Reliability of Machine Vision Components

Machine Vision Components (MVCs) are deployed in safety-critical systems, such as autonomous driving, and their reliability must be checked against scene changes, e.g., rain, that may lead to hazardous situations in the deployment environment. Many scene changes leading to hazardous situations may be hard to reproduce on demand, so existing approaches for MVC reliability analysis use synthetic image transformations to simulate such changes. Therefore, the question of how to select the image transformations to simulate specific hazardous situations is essential to MVC reliability analysis. Yet, this problem has not been addressed by the scientific community so far. In this paper, we propose a framework for mapping between hazardous situations and relevant image transformations using their descriptions. Our framework includes a systematic description mapping process DMaP, a method autoDMaP for automating this process, and coverage metrics measuring how well a list of transformations can simulate a list of hazardous situations. We show the applicability of our framework by mapping hazardous situations from an existing checklist, i.e., CV-HAZOP, to a list of synthetic image transformations from a state-of-the-art transformation library, i.e., Albumentation. As part of evaluation, we conducted an experiment and showed that, compared with the manual, ad-hoc mapping produced by image processing experts, DMaP and autoDMaP resulted in better precision and recall. Additionally, using our new coverage metrics, we found that image transformations considered by state-of-the-art libraries and reliability benchmarks are far from fully simulating the CV-HAZOP hazardous situations, and the MVCs that perform best on these benchmarks have significant reliability gaps against these situations.