Managing cloud-generated logs using big data technologies

Cloud computing is a fast-growing paradigm that has forcefully emerged and established itself in the next generation of IT industry and business, performing massive-scale and complex computing. As cloud technology adoption continues to progress, massive growth in the scale of data generated through cloud computing has been observed. A large part of that data has been machine-generated log data; it is generated at every layer and component within the cloud information technology ecosystem that spans a range of IT functions from storage and computation to networking and application services. Log files are essential for enterprise-level monitoring, troubleshooting, security, debugging, compliance, etc. We argue that cloud computing is in need of a new methodology for recording, managing, storing and analyzing the log data that is generated by the cloud infrastructure, based on big data technologies, allowing for the access to a wealth of knowledge that can contribute to its own advancement. Our paper discusses and expands existing standard log management solutions to design new methods of exploiting the unstructured log data in order to gain unprecedented insights.