6 EXPLAINING THE INEXPLICABLE

An AI, at least today, is made not of flesh and blood but of software and computers. We humans invented these things, so, in theory at least, we humans should be able to understand every aspect of an AI. But that turns out not to be the case. A frustrating result of the recent successes in deep neural nets is that people have been unable to provide explanations for many of the decisions that these systems make. Worse, when an explanation finally emerges, it may reveal that the decision is flawed. If an AI is being used to determine prison sentences, mates, targets for counterterrorism, bad credit risks, hiring decisions, or medical treatment, and there is no explanation, no rationalization for its decisions, should we trust it?