Evaluating Reliability in Explainable Search

Explainability in AI (XAI) is being investigated in various AI driven systems like search engines to promote trust and fairness in AI. In the context of search systems one major XAI goal is to explain the notion of similarity in text to a non-expert user like an avid reader searching for books in a digital library. What is the similarity between book A and B with respect to narrative time? How do the readers’ emotions change? This is often an important criteria for searching books. In this work we extend a recent explainable AI driven book search engine [1] with local explanations that visually support users to comprehend the similarity model. Each book is represented as a feature vector in an interpretable feature space with aspects such as sentence complexity, writing style. In the absence of ground truth similarities and explanations, the system is evaluated on the aspects of reliability of search results and explanations in user studies. Different baselines are created to compare with our back-end ranking model, keeping the user interface same for all. Experimental results on lab based user study with eye tracking mechanism indicate that the explanations help towards making the search system reliable for non expert users.