Literature Review : A Comparative Study of Real Time Streaming Technologies and Apache Kafka

For data ware housing projects, there are multiple licensed ETL (Extraction Transformation & Load) tools available in the market. To process and pass the data between two applications industry is using ETL tools like IBM Info-sphere Data Stage, Informatica, Ab Initio etc. These tools are exceptionally costly and has recurring enterprise licensees, and processed data is not available in real time, data is getting processed in batches and is available during pre-defined time intervals or on demand. Industry has started adopting the Open Source technologies to avoid the huge licensing cost and that also includes the complete end to end IT infrastructure cost. Open Source technologies and frameworks enables users to run projects with best in class performance and within the budget.In this literature survey paper, all possible technologies have been studied and evaluated, available in the market capable of real/ “near-real-time” streaming. All licensed and open source products which are utilized and evaluated by various IT organizations and which are also evaluated by researchers have been included in this survey. There is a need of a distributed scalable technology that enables the users to ensure availability of data from one end point to another in real time with good throughput, performance and low latency. To study this, a detailed comparative survey of an open source technology Apache Kafka has been done and it compared with the other available technologies capable of doing real time streaming.