Streaming Big Data with Open-source: A Comparative Study and Architectural Recommendations

This study focusses on providing state-of-the-art infrastructure for data pipelines in e-Commerce sector, especially for online stores. With people going digital and also latest impact of Covid-19, daily e-Commerce companies are dealing with large amount of data (terabytes to petabytes). With growing Internet of Things, systems of computing devices which are interrelated. The inter-relation may be between mechanical and digital machines, objects or people. The interrelated objects will be provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction. Growth of big data poses several challenges and opportunities in every field of its usage. Realtime analysis of data and its inference gives a competitive edge over its partners in every business field especially in e-commerce. Recent advances in technology and tools have exposed new opportunities to get actionable insights from historical data like market data, customer demographics, along with real-time data. Advancement in distributed streaming technology makes it important to investigate existing streaming data pipeline capabilities in eCommerce sector with a focus on online stores. This study analyzes the published research works on streaming data pipelines in e-commerce sector also to facilitate e-commerce’s variety of data streaming applications requirement. A state-of-the-art lambda architecture for streaming is proposed completely based on open-source technologies. Challenge in proprietary owned streaming platforms are vendor lock-in, limited ability to customize, cost, limited innovation & support. Proposed reference architecture will address many streaming use cases compared to its competitors, it has support of large open-source community in providing the inter-operability between streaming & related technologies like connectors, apart from providing better performance apart from other open-source based product advantages.