By Topic

Design and implementation of parallel word occurrence frequency based on message passing interface

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Hongmei Xie ; Sch. of Electron. & Inf., Northwestern Polytech. Univ., Xi''an, China ; Chunli Chen

For large files and files distributed all around the internet, this paper proposed and realized parallel word occurrence frequency counting using C and message passing interface programming. And to solve the specific word occurrence problems like how to extract word precisely, traverse file completely, eliminate redundant words, adapt to any text files, and count the number of letters and word. An improved serial word counting algorithm was designed and implemented. Then the parallel partition strategy based on both balanced file size and balanced task were discussed and a combined partition strategy was designed and implemented using C programming and MapReduce message passing interface (MR-MPI) library. After that the counting of word occurrence frequency for relatively small and very large text file were realized. And last we executed the same task in parallel computing environment. The test showed that the algorithm proposed implements word counting for files of large data set, both in parallel high performance computing and cloud environment with short runtime and high accuracy. Last, some discussion on the parallel implementation of the word counting problem in Google's cloud environment is given out.

Published in:

Mechanic Automation and Control Engineering (MACE), 2011 Second International Conference on

Date of Conference:

15-17 July 2011