A Framework for Real-time Sentiment Analysis of Big Data Generated by Social Media Platforms

Sentiment and Opinion analysis have been of significant interest with the possibilities of creating more meaningful business analytics from using data sources such as social media creating a large-scale implementation using Big Data. There has been a range of implementation, typically focusing on one social media platform and user entered text as input. Recently, efforts have been made to make a real-time implementation of such a sentiment system using API and data streams from social media platforms. There exists a need to create a system that uses multiple input sources from social media in real-time. We present an architecture using existing Big Data technologies to implement a real-time multi-social media input source with a central sentiment extraction and analysis component. The proposal uses Apache Kafka for the ingestion layer, lexicon-based classifier and Spark for the analytical layer, YARN clusters for the tasks execution management, and MongoDB database for the storage layer. The performance of the proposed framework is measured based on different quality metrics.