The technical hashtag in Twitter data: A hadoop experience

The continuously growing wealth of data has radically changed the data science landscape. At the same time, Big Data tools have known important progress in terms of optimising performance and scalability. However, applying them into practical deployment settings is still a challenging task that is highly dependent on the particularities of the data. In this paper, we present our experiences with implementing a Big Data analytics pipeline with the purpose of extracting value from Twitter data. We acquire and process nearly 60 million tweets that capture the recent outbreaks of the Ebola and Zika viruses. Our processing pipeline first extracts useful information from tweets and then applies a topic modelling technique, provided by Mahout, a Hadoop-based machine learning library. We further extend our Twitter analysis with the study of temporal evolution of daily sentiment toward an important topic, as expressed through the social platform. We highlight at each level, the technical challenges originating from the specific nature of Twitter data and the lessons drawn from our work.