Targeting in the Black Box

Artificial intelligence (AI) is poised to become pervasive in military operations worldwide. In the coming decades, AI-based systems will revolutionize logistics, dramatically change targeting, and ultimately power autonomous weapons systems. Unfortunately, many of the most potent AI-based systems are unintelligible to their developers and offer unexplained outputs to their users-a phenomenon called the “black-box problem.” This paper first describes the basic AI architecture that is giving rise to the black-box problem. It then shifts to consider the unexplored question of whether black-box models comport with the international humanitarian law (IHL) principles of distinction, proportionality, and precaution, which are fundamentally rooted in nuanced context and subjective judgment. After describing the mismatch between black-box models and existing IHL principles, the paper compares existing NATO doctrine with emerging “soft law” embraced by NATO member States. Identifying a nascent movement away from explainable AI, the paper concludes by setting out the importance of interpretability and the aspects therein that should be considered by policymakers in constructing future legal norms and by military officials in assessing AI models to be used in future operations.