Improvement to GLASS/Maximum Tree Method of Species Tree Inference Using Measurement Error Modified Single Linkage Clustering

The Global LAteSt Split (GLASS)/Maximum Tree (MT) method of species tree inference, implemented in the software STEM, performs well and is statistically consistent when inferring a species tree from known gene trees. When input gene trees are estimated from DNA sequences, it can perform poorly, with possibly unrealistic sufficient conditions for statistical consistency. We propose a modification to STEM, called genX, which replaces its estimated pairwise coalescence times with randomly generated realizations from an estimated distribution of the true coalescence times. This adjusts for measurement error as described by Su, Reedy, and Carroll in 2018, or using a Bayesian approach. As with STEM, the minimum of these realizations is taken over all loci for each pairwise distance, forming a distance matrix, and the species tree is inferred using single linkage clustering. In simulations and in two empirical datasets, the new methods outperformed STEM in terms of Robinson-Foulds distance from the true species tree, though the Bayesian approach requires more computation time and is prone to convergence issues. In our simulations, overcoming some gene tree estimation error improved the performance of STEM.