Hashing scheme for space-efficient detection and localization of changes in large data sets

One of primary components of computer systems security is preservation of data integrity. In addition to violation prevention, it also includes methods used to detect such violations after they occur. The most common methods for preserving integrity of binary data are based on various hashing functions. However, an inherent downside of using hashing functions is that a single hash can only be used to verify integrity of a single data string as a whole. This results in it being impossible to locate the exact position within the string where the change occurred. Alternative method entails splitting the data string into blocks, each protected by a hash. While this enables more precise location of changes, storing potentially large number of hashes imposes significant space overhead for large data sets. In this paper, we present a space-efficient method for detection and localization of unwanted changes in large data sets. Our method reuses the idea of splitting data into blocks and hashing each block separately, but with certain added properties: logarithmic instead of linear increase of memory space required to store hashes, ease of parallelization, possible application on distributed data, limited self-verification of hashes and efficient recalculation of hashes for dynamically changing data.