Loading [MathJax]/extensions/MathMenu.js
Global Message Ordering using Distributed Kafka Clusters | IEEE Conference Publication | IEEE Xplore

Global Message Ordering using Distributed Kafka Clusters


Abstract:

In contemporary distributed systems, logs are produced at an astounding rate, generating terabytes of data within mere seconds. These logs, containing pivotal details lik...Show More

Abstract:

In contemporary distributed systems, logs are produced at an astounding rate, generating terabytes of data within mere seconds. These logs, containing pivotal details like system metrics, user actions, and diverse events, are foundational to the system’s consistent and accurate operations. Precise log ordering becomes indispensable to avert potential ambiguities and discordances in system functionalities. Apache Kafka, a prevalent distributed message queue, offers significant solutions to various distributed log processing challenges. However, it presents an inherent limitation: while Kafka ensures the in-order delivery of messages within a single partition to the consumer, it falls short in guaranteeing a global order for messages spanning multiple partitions. This research delves into innovative methodologies to achieve global ordering of messages within a Kafka topic, aiming to bolster the integrity and consistency of log processing in distributed systems. Our code is available on GitHub - https://github.com/aryan-jadon/Distributed-KafkaClusters.
Date of Conference: 14-15 November 2023
Date Added to IEEE Xplore: 25 December 2023
ISBN Information:

ISSN Information:

Conference Location: Al Ain, United Arab Emirates

Contact IEEE to Subscribe

References

References is not available for this document.