A Discrete Bayesian Network Model For Diagnosing Latency Growth In Apache Kafka Cluster Within Information Systems For Building Construction Projects

The performance of information systems for building construction projects is critically dependent on the low end-to-end latency exhibited by Kafka clusters. This study introduces a novel discrete Bayesian network model designed explicitly for diagnosing changes in end-to-end latency within Apache Kafka clusters due to modifications in cluster configuration parameters. The development of this Bayesian network model was grounded in an in-depth analysis of Kafka cluster architecture, which facilitated the creation of a network capable of evaluating cluster efficiency based on latency metrics. Observations from testing scenarios were utilized to establish prior probabilities, and Kafka cluster performance metrics further used to learn of network parameters via the Maximum Likelihood Estimation algorithm. Additionally, the methodology incorporated the discretization of performance metric values using hierarchical clustering to adapt continuous data for discrete Bayes network. The ability of the Bayesian model to diagnose latency issues was validated through empirical tests. Due to its demonstrated capabilities, the developed Bayesian network model is recommended for optimizing Kafka settings and improving cluster performance for information system that need real time data.