Parallel generation of inverted files for distributed text collections

The authors present a scalable algorithm for the parallel computation of inverted files for large text collections. The algorithm takes into account an environment of a high bandwidth network of workstations with a shared-nothing memory organization. The text collection is assumed to be evenly distributed among the disks of the various workstations. Compression is used to save space in main memory (where inverted lists are kept) and to save time when data have to be moved across the network. The algorithm's average running cost is O(t/p) where t is the size of the whole text collection and p is the number of available processors. The authors implemented their algorithm and drew experimental results. In a 100 Mbits/s switched Ethernet network with 4 PentiumPro 200 MHz, 128 MB RAM on each processor they were able to invert 2 GB of TREC documents in 15 minutes. Further they also proposed an analytical model for the algorithm execution time.