Bayesian Hierarchical Clustering: Advantages, Algorithm, and Optimization

Hierarchical clustering is a method in data mining and statistics used to build a hierarchy of clusters. Traditional hierarchical clustering relies on a measure of dissimilarity to combine clusters, but it does not ensure the optimal number of clusters. Bayesian Hierarchical Clustering (BHC) addresses this limitation by utilizing a probabilistic model and Bayesian hypothesis testing to determine the appropriate tree depth. This paper explores BHC, employing the Dirichlet Process Mixture Model (DPM) for unsupervised data clustering. The BHC algorithm merges clusters based on the highest posterior probability, enhancing accuracy over traditional methods. The computational challenges are also discussed and optimizations necessary for implementing BHC in Python, comparing its performance on simulated and real datasets with traditional hierarchical clustering techniques. Despite its computational complexity, BHC offers significant advantages in defining predictive distributions and determining the number of clusters.