Avoiding Confirmation Bias in a Safety Management System (SMS)

Confidence in a safety management system (SMS) could be undermined by a tendency to search for, interpret, choose, or recall information in a way that supports a favorable view of how safety risk is managed in an organization. This tendency, sometimes called "confirmation bias", might be unintentional. Otherwise, it might be the result of a misguided loyalty to an organization, a reluctance to "open a can of worms" or even an intention to mislead or deceive. A temptation to rely on Generative AI as a substitute for critical thinking in the development and maintenance of a SMS might also be a source of confirmation bias. This paper proposes the use of Eliminative Argumentation (EA) to avoid confirmation bias in a SMS. Our approach is an adaptation of an innovative method for safety assurance developed by researchers at Carnegie Mellon University. At the heart of the method, confidence can be increased by identifying potential doubts, and then eliminating these doubts where it is reasonable to do so. A hierarchically structured argument for confidence in the SMS can be explicitly documented using a simple notation that mirrors the structure of the SMS. Following the structure described in Federal Aviation Administration (FAA) Order 8000.369C, the main branches of this argument cover the four components of a SMS, namely the Safety Policy, Safety Risk Management (SRM), Safety Assurance, and Safety Promotion. In turn, each of these main branches are further split into subbranches. For example, one of the sub-branches of the SMR branch of the argument addresses the need to track identified hazards and monitor implemented safety risk controls/ mitigations to ensure that they achieve their intended safety performance targets. To this level, the argument is an instance of a template that can be re-used for other instances of a SMS based on FAA Order 8000.369C. Lower levels of this tree-shaped argument are tailored to the specific details of how the SMS is implemented and maintained. For example, there might be doubts, also known as "defeaters", in the argument at this level that challenge the process used by the organization to decide that particular hazards are adequately controlled. Such defeaters are often expressed in the form of "What if ...?" questions. For example, "what if a decision that an identified hazard is mitigated is made by an unqualified person?". Such defeaters are often eliminated by supplementary details - for example, details about the minimum qualifications of personnel in a particular role. Using appropriate tool support, the structured argument for confidence in the SMS can be linked to Key Performance Indicators (KPI) to measure the effectiveness of the SMS. For example, one such KPI measures how many days on average it takes to analyze and resolve safety issues raised by personnel. If and when this measured value exceeds a defined threshold, it will be flagged as a problem for the SMS. Among other influences on a SMS, this paper will also take account of the impact of relying on Generative AI. In summary, the first contribution of this paper is showing how trust in a SMS can be increased by means of Eliminative Argumentation, especially as a means of avoiding confirmation bias. The second contribution of this paper is showing how KPIs can be used to further increase confidence in a SMS.