SQLite File Databases for Science Data Pipelines 3 Use Cases

Complex science instruments require complicated management of ancillary data such as calibration, file list, or timing information. We have found that the widely used, light-weight, single file database SQLite can make these tasks much easier. Since the database is a file, there is no server to maintain or connect with, just a single file that can be copied anywhere it is needed (including to individual end users’ workstations) and COTS tools can be used to browse it or export tables or selections to other formats. It is also fully supported by pretty much any programming or scripting environment. We present three use cases of varying complexity along with lessons learned:–Keeping a master list of file boundaries so data isn’t "dropped" during file reprocessing–Keeping track of millions of individual telemetry files (CFDP files) for collecting input for a science data pipeline–Managing calibration information for hundreds of thousands of defined channels over multiple instrument configuration regimes