FastDu: Efficient directory summaries harvest by tracking file system changes

FastDu is a file system service that tracks file system changes by intercepting file system calls to maintain directory summaries, which play important roles in both storage administration and improvement of user experiences for some applications. In most circumstances, directory summaries are independently harvested by applications via traversing the file system hierarchy and calling stat 0 on every file in each directory. For large file systems, this brute-force traverse-based approach can take many hours to complete, even if only a small percentage of the files have changed. This paper describes FastDu, which uses a pre-built database to store harvested directory summaries, and tracks the file system changes by intercepting file system calls, so that new harvesting is restricted to the small subset of directories that contain modified files. Tests using FastDu show that this approach reduces the time needed to get a directory summary by one or two orders of magnitude with almost negligible penalty to application-aware file system performance.