4 The Mismeasurement of Quality and Impact

In this chapter, I would like to discuss the implication of the problematic relationship between “gaming the system” and “properly operating in the system” for research evaluation. As Biagioli and Lippman (Introduction, this volume) argue, the current audit culture in academia has created new dimensions of academic misconduct. The performance indicators that are supposed to objectively measure the value of scientific production have become the objects of new forms of manipulation. Because research evaluations have become reliant on proxy indicators that measure only indirectly what they are supposed to represent (either quality, impact, or societal relevance), it is increasingly hard to see the difference between indicator manipulation and authentic high-value research. According to Biagioli and Lippman (Introduction, this volume), “the traditional locus of evaluation—the publication's claims—has become technically irrelevant to metrics regimes based on impact factors.” Reading has stopped being necessary in institutional evaluations, they argue, while the role of metadata has shifted from descriptive to evaluative. While I think the practice of evaluation is still quite messy (in many assessments, reading might still happen), and the role of indicators is not so straightforward (the Journal Impact Factor, though the most popular indicator, is not the only game in town and neither representative for all), I do agree that Biagioli and Lippman very perceptively sketch a portrait of the dominant tendencies in the academic audit culture and of the market enabled by the infrastructure of citation (Wouters, 2014). Indeed, if these trends remain dominant, high performance scores will be made identical to high-value work. Those aspects of academia that are not presentable in indicators may then no longer be supported (by researchers themselves as well as by their employers) and may have to migrate to the area of works of love (amateurs)—or become extinct. Scientific research may in such a scenario still lead to exciting innovations, but the notion of knowledge itself as an important aspect of human culture would no longer be an important leitmotiv of universities. Scientific research will then have been instrumentalized in its totality.