Design and evaluation of a distributed scalable content discovery system

A content discovery system (CDS) allows nodes in the system to discover contents published by some other nodes in the system. Existing CDS systems have difficulties in achieving both scalability and rich functionality. We present the design and evaluation of a distributed and scalable CDS. Our system uses rendezvous points (RPs) for content registration and query resolution, and can accommodate frequent updates from dynamic contents. Contents stored in our system can be searched via subset matching. We propose a novel mechanism that uses load balancing matrices (LBMs) to balance dynamically both registration and query load across nodes in the system to maintain high system throughput even under skewed load. Our system utilizes existing distributed hash table (DHT) mechanisms for CDS overlay network management and routing. We validate our system's scalability and load balancing properties using extensive simulation.