Crawling the IPFS Network

IPFS is a distributed data storage service frequently used by blockchain applications and for sharing content in a censorship-resistant manner. Data is hosted by an open set of peers, pointers to both are distributed using a Kademlia-based distributed hash table (DHT). In this demo, we present a crawler for the IPFS overlay network (ipfs_crawler) that can be used to study and monitor the network's structure. Therefore, ipfs_crawler is an important building block when assessing the state and health of the network, as the overlay network significantly influences the robustness and performance of IPFS. Specifically, ipfs_crawler systematically traverses the Kademlia DHT of IPFS to enumerate peers in the network and a subset of the connections between peers. Since network communication in IPFS is carried out through the libp2p networking library, ipfs_crawler can easily be adapted to crawl other libp2p-based networks.