Population Control for Multi-agent Based Topical Crawlers

The use of multi-agent topical Web crawlers based on the endogenous fitness model raises the problem of controlling the population of agents. We tackle this question through an energy based model to balance the reproduction/life expectancy of agents. Our goal is to simplify the tuning of parameters and to optimize the use of resources available for the crawling. We introduce an energy based model designed to control the number of agents according to the precision of the crawling. We present some experiments that show that the size of the population remains under control during the crawling.