iStore: Towards the Optimization of Federation File Systems

With the growing volumes of data, many organizations are deploying geo-distributed edge servers and building federations atop of edge servers to improve data sharing, effective collaborations, and analytics. Multiple federation file systems are designed to satisfy such needs, but due to application-specific architectures, these federations neglect some important features that can improve the overall federation performance. In this paper, we address the important challenges of federated file systems, in particular, global namespace, optimal data placement and analysis, efficient data migration across edge servers, and metadata optimizations. To further investigate these challenges, we prototyped the federation file system iStore to emulate the federation and showed the significance of the afore-mentioned key challenges in the federation. The iStore provides unified global namespace atop of geo-distributed edge servers with a generic job and resource-aware data storage and placement algorithm (JRAP), which minimizes the job execution time by considering resources at each edge server. Furthermore, to enable effective data migration, we employed direct channel file layout-aware data transfer and designed a batch-based metadata scheme for federations to reduce the metadata contention with increasing clients. We evaluated the efficacy of various big data applications from data generation to storage and analysis using the iStore on real testbed and simulation.