How many sources are needed? : The effects of bibliographic databases on systematic review outcomes

Systematic reviews are an established method to synthesize the current state of research for a specific question to make evidence-based decisions in research, politics, and practice. A key activity of a review approach is a systematic and comprehensive search strategy to find all potentially relevant literature. Although guidelines and handbooks address relevant methodological aspects and recommend strategies, the right choice of databases and information sources is unclear. Specifically in educational research, an interdisciplinary field, with no core database at hand and multiple potentially relevant sources available, investigators lack guidance for choosing the most appropriate ones. The presented study investigates the coverage in terms of scope, similarity and combination efficiency of seven multidisciplinary, discipline-specific and nationally focused databases. The evaluation is based on relevant assessed literature of two extensive recently published reviews in German educational research that serve as gold standard to evaluate the databases. Results indicate distinct variations in the databases, while also detecting databases with equal coverage. The paper contributes to guidance in choosing databases for educational review studies, while stressing that this process depends on a review’s topical and geographical focus. Moreover, general implications resulting from the study refer to the relevance of database choice for review outcomes, the careful consideration of diverse search strategies beyond database search and a rigorous documentation of database inclusion and exclusion criteria. CCS CONCEPTS Information Systems - Information retrieval diversity, Information Systems - similarity measures, Information Systems - Combination, fusion and federated search