Enhancing collaboration in large scientific projects through virtual logbooks

A key feature of collaboration in large scale scientific projects is to have a log of what and how is being done - for private use and reuse and for sharing selected parts with collaborators and peers, often distributed geographically on an increasingly global scale. Even better if this log is automatic, created on the fly while a scientist or software developer is working in a habitual way, without the need for extra efforts. The CAVES - Collaborative Analysis Versioning Environment System - and CODESH - Collaborative DEvelopment SHell - projects address this problem in a novel way. They build on the concepts of virtual state and virtual transition to enhance the collaborative experience by providing automatic persistent virtual logbooks. CAVES is designed for sessions of data analysis using the popular ROOT framework, while CODESH generalizes the same approach for any type of work on the command line in typical UNIX shells like bash or tcsh. Repositories of sessions can be configured dynamically to record and make available the knowledge accumulated in the course of a scientific or software endeavor. Access can be controlled to define logbooks of private sessions or sessions shared within or between collaborating groups. As a typical use case we concentrate on building working systems for analysis of Petascale volumes of data expected soon with the start of the LHC experiments. Our approach is general enough to find applications in many scientific fields.