Auditing Algorithms: Understanding Algorithmic Systems from the Outside In

Algorithms are ubiquitous and critical sources of information online and increasingly act as gatekeepers for users accessing or sharing information about virtually any topic. This includes information about their personal lives and those of friends and family, news and politics, entertainment, and even health and well-being. As a result, algorithmically-curated content is drawing increased attention and scrutiny from users, the media, and lawmakers alike. Studying such content poses considerable challenges as it is both dynamic and ephemeral. One strategy that has proven effective is the algorithm audit: a method of repeatedly querying an algorithm and observing its output in order to draw conclusions about the algorithm’s opaque inner workings and possible external impact. In this work, the authors present an overview of the algorithm audit methodology. They include the history of audit studies in the social sciences from which this method is derived; a summary of key algorithm audits over the last two decades in a variety of domains such as health, politics, and discrimination; and a set of best practices for conducting algorithm audits today. The authors conclude by discussing the social, ethical, and political dimensions of auditing algorithms, and propose normative standards for the use of this method.