Automated Transparency Evaluation of Knowledge Graphs for Clinical Risk Governance

In this paper, the transparency of a knowledge graph (KG) generated by a governance, risk, and compliance tool is automatically evaluated for a real-world clinical use case, using a generalisable evaluation method. KGs are increasingly used in AI systems and their transparency has a prominent impact on the transparency of the systems that create and use them. There is a lack of research examining the transparency of KGs. The draft EU AI Act places transparency obligations on systems that interact with humans. In this paper, Seven transparency dimensions are calculated for a KG using a KG quality evaluation tool and three FAIR (Findable, Accessible, In-teroperable, Reusable) evaluation tools. In addition, the priorities of transparency dimensions are investigated for this use case by conducting a survey with system stakeholders. The survey results show that the understandability and inter-pretability are the most important transparency dimensions in our use case. The transparency evaluation results show that the evaluated KG has a high interpretability and is not linked to other KGs. In addition, more detailed analysis shows that some transparency-related information related to understandability, timeliness, provenance, and licensing information exists in the KG that are not detected by the evaluation tools. In this paper, for the first time, the transparency of a KG in a real-world clinical context is automatically evaluated and the importance of transparency dimensions are investigated, from the stakeholders' perspective.