A Big Data Framework for Cloud Monitoring

Elasticity is a key component of modern cloud environments and monitoring is an essential part of this process. Monitoring demonstrates several challenges including gathering metrics from a variety of layers (infrastructure, platform, application), the need for fast processing of this data to enable efficient elasticity and the proper management of this data in order to facilitate analysis of current and past data and future predictions. In this work, we classify monitoring as a big data problem and propose appropriate solutions in a layered, pluggable and extendable architecture for a monitoring component. More specifically, we propose the use of NoSQL databases as the back-end and BigQueue as a write buffer to achieve high throughput. Our evaluation shows that our monitoring is capable of achieving response time of a few hundreds of milliseconds for the insertion of hundreds of rows regardless of the underlying NoSQL database.