Coordinate Checkpoint Mechanism on Real-Time Messaging System in Kafka Pipeline Architecture

The real-time messaging system is the critical thing in computing based on time-critical decision making in many organizations. In the real-time messaging system, fault tolerance is the key challenge of developing reliability requirements. Apache Kafka is a popular framework for consuming data stream into the processing platforms. However, there are many challenges in the replication process because of a server failure in Apache Kafka. To develop fault tolerance in Apache Kafka, this paper focuses on defining a fixed checkpoint interval to reduce the recovery time and lost messages in server failure. Then, we measure the cost of checkpointing, cost of rollback and total time cost of overheads due to the fixed checkpointing intervals. The system shows the drawback of real-time processing depend on the number of lost messages on various partitions and server failure processes. The experimental results emphasize the checkpoint interval method to reduce recovery time and lost messages. Conforming to the experimental results, the performance of the total time cost with checkpoint interval saves time approximately 30% than without checkpoint interval.