Rapid Node Reallocation Between Virtual Clusters for Data Intensive Utility Computing

Utility computing achieves efficiencies by dynamically reallocating shared resources between services operating on virtual clusters. These efficiencies can be hard to realize for data intensive applications; newly allocated nodes must be populated with a large amount of data which impedes rapid node reallocation. We describe a data management architecture that uses disk caches on each node to reduce data copying and speed up node reallocation for data intensive applications. Cache consistency management is simplified by extensive use of copy-on-write techniques. A data-driven scheme is then used to select nodes for reallocation between virtual clusters based on the amount of relevant cached data. These nodes are identified using a novel technique of statistically sampling the contents of caches. We demonstrate the benefits of this architecture using our implementation of an efficient block level caching and copy-on-write target for the Linux device-mapper framework