Crystalia: Flexible and Efficient Method for Large Dataset Lineage Tracking

Computation over large data sets is now a critical part of most natural and social sciences. As such, having ways to capture the provenance and lineage of the data is a sine qua non for not just reproducibility, but also for these computational approaches to inform public policy. Standards like PROV-O, RO-Crate, IEEE 2791, and ISO-23494 define provenance for computational processes but lack detailed specifications for describing entities beyond simple URIs or linear hash sums. This is usually sufficient for smaller datasets with the size of a gigabyte or two. However, contempory datasets in domains like the life sciences and high energy physics often exceed tens of terabytes in size. For these datasets, linear hash sums are no longer feasible. In addition these large datasets are frequently copied around to bring them closer to computational resources, limiting URL based tracking.In this article, we introduce Crystalia, a novel, flexible, and parallelizable method for computing dataset descriptions using Merkle trees, represented as Knowledge Graphs. Our solution enables efficient tracking of dataset versions and transformations by capturing reproducibility information in a scalable manner. Leveraging Knowledge Graphs facilitates annotation, exchange, and integration with existing provenance standards.