Three Pillars of Practical Reproducibility

The ability to reproduce results is important for future experimenters building on existing work. We argue that the ability to do so in a cost-effective manner – what we call practical reproducibility – should be a mainstream method of scientific exploration and as a community we should invest in building an ecosystem of tools that would support it. Since computational research artifacts require some form of computing to interpret, we argue that such ecosystem is likely to be tied to shared experimental infrastructure where unique resources can be found and effectively shared. We propose three interrelated services, solving the problems of packaging for reuse, findability, and accessibility, respectively. We describe how we developed these services in Chameleon, an NSF-funded testbed for computer science research which has supported 8,000+ users, and discuss their strengths and limitations.