Improve Spark-based Application Performance Using Bayesian Hyperparameter Optimization

Metagenomics sequences assembly has an insurmountable obstacle in computational and memory resources limitations. SpaRC, a scalable sequence clustering application using Apache Spark, provides a scalable solution for clustering billions of reads from the next-generation sequencing(NGS) technologies. However, optimizing parameters to achieve better accuracy at this scale becomes very challenging and expensive. Here we explored Bayesian hyperparameter optimization to alleviate this problems by tuning the parameters to achieve near optimal cluster accuracy, without expensive parameter exploration. We report cluster performance on both short reads (Illumina) and long reads (PacBio) sequencing platforms. Results indicates that the methods offers great improvement in clustering performance. We hope insights we gained in this experiment can be generally applied to similar genomics applications based on Spark.