A distributed framework for information retrieval, processing and presentation of data

The Internet is ever growing and the information is highly unstructured. Finding the best priced product when shopping online is a difficult task, but shopping search engines help the user identify the desired product. Starting from this reality, this paper presents a distributed framework for information retrieval, processing and presentation of data, which has three main components: a data server cluster, a distributed crawler with a product extraction feature, and a web server for presenting the processed data. The considered use case consists of extracting information about board and card games from specialized websites and presenting the data in a user-friendly website. Efficiency is determined by means of performance evaluation metrics that include crawl and extraction times, and a discussion in terms of scalability. The high modularization of the framework allows it to be an effective teaching tool in classrooms; each module can be easily swapped with a custom implementation. The proposed framework offers students and researchers an environment for developing and testing a wide range of algorithms and programming solutions, e.g., load balancer, crawler, reverse engineering a web template, product extractor from web pages, data normalization, notification services, web APIs, custom database.