Implementation of a Social Media Crawler Based on Microservices Architecture

Nowadays, a government needs to get the support of the public, and it can stand as a government only if it gets the trust and attention of the public. Therefore, the government needs to keep in touch with the opinions of the citizens. Almost everyone uses online social networks like Facebook and Twitter to keep up to date with their daily events. Their emotions, their opinions on the government, their favorite topics and trends (movies, food, animals, etc.), and the things going on in their minds are often displayed on their social media platforms. The main objective of this paper is to design and deploy a crawler system based on the microservices architecture that can be independently scalable as demand for an application increases and reduce downtime through fault isolation. This proposed architecture is intended for the crawler application that can be deployed independently, as needed, enabling continuous improvement and faster application updates for searching and analyzing online information. This system can be made to form analytical and statistical reports based on the data received (publications, comments, likes, reposts, views). This paper is intended as a bridge for the improvement of communication between the government and the public.