Tengu: An Experimentation Platform for Big Data Applications

Big data applications have stringent service requirements for scalability and fault-tolerance and involve high volumes of data, high processing speeds and large varieties of database technologies. In order to test big data management solutions, large experimentation facilities are needed, which are expensive in terms of both resource cost and configuration time. This paper presents Tengu, an experimentation platform for big data applications that can automatically be instantiated on GENI (US federation of test beds) and Fed FIRE (EU federation of test beds)compatible test beds. Tengu allows for automatic deployments of several data processing, storage and cloud technologies, including Hadoop, Storm and Open Stack. The paper discusses the Tengu architecture, the Tengu-as-a-service approach and a demonstration of an automated instantiation of the Tengu experimentation suite on the Virtual Wall, a large-scale Emulab testbed at the Minds research institute in Europe.