By Topic

CHMasters: A Scalable and Speed-Efficient Metadata Service in Distributed File System

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

4 Author(s)
Min Xu ; Dept. of Comput. Sci. & Technol., Univ. of Sci. & Technol. of China, Hefei, China ; Junrui Zhou ; Wei Zhou ; Hong An

Distributed file system (DFS) is playing important roles of supporting large distributed data-intensive applications to meet storage needs. Typically, the design of DFS, such as GFS in Google, DMS in Cisco and TFS in Alibaba, is driven by observations of specific application workloads, internal demands and technological environment. In such systems, the metadata service is a critical factor that can affect the file system performance and availability to a great degree. Five requirements have been summarized for the metadata service: location transparent file service, smart director, efficient speed, strong scalability and friendly collaborator. In this paper, we present metadata service module called CH Masters in our DFS. Consistent hashing protocol is used to relieve potential hot spots on name servers. Files' metadata and master nodes are mapped into the same hash space by consistent hash function. And then files' metadata are scattered to master nodes by clockwise "closest" principle. Chunk server acts as a client when report its chunks info. Only a small proportion of files' metadata will be rehashed when master nodes state change. A new scalable file mapping strategy is also proposed to map file sizes from few MB to several GB efficiently. After intensive experiments, it shows CH Masters is satisfying the above five requirements.

Published in:

Parallel and Distributed Computing, Applications and Technologies (PDCAT), 2011 12th International Conference on

Date of Conference:

20-22 Oct. 2011