Versioning Complex Data

The Atmospheric Radiation Measurement (ARM) program is collecting and storing data daily since 1992. New and updated data are continuously added to the data store resulting in a history of data changes that are not readily apparent or easy to document. Users are notified when updated data is available but the reason for the update is difficult to find. The ability to assign a version to the data provides a simple handle for adding further documentation. The software versioning processes did not appear to be a good fit because of the volume and frequency of changes. But by assigning a version number to a set of files and establishing conditions for changing the sets, we created a model that appears to be manageable. As a test, we applied it to the historical ARM data set. Our results showed that with some domain specific adjustments we were able to automatically generate a reasonable set of versions for ARM data. Further, we believe this method could be applied in other complex data domains.