Characterization of QIIME 2 with Intel Technology on-premises and in Microsoft Azure Cloud

The study of characterization of the interaction between software and hardware has the purpose to determine the requirements of the resources necessary for its execution. In addition, it allows testing new technologies on systems to compare heterogeneous products, such as those from AMD and Intel. In this study, we aimed to characterize the application QIIME 2 in the CICIMA cluster of the University of Costa Rica and the Microsoft Azure Cloud to determine the most efficient configurations for rRNA 16S amplicon taxonomic classification studies. The QIIME 2 pipeline has 78.95% of commands executed sequentially and 21.05% in parallel. We analyzed 240 samples (1235 MB) in the cluster to compare hyperthreading (HT) technology and Intel multithreading (MT) technology. Cloud performance data was collected using Microsoft Azure Monitor. The comparison between HT and MT based on metrics such as CPI (cycles per instruction), retired instructions, and execution time showed that MT was more efficient. System utilization clearly and concisely shows a drastic drop with an increasing number of processors. Going from 98% utilization with one processor for both configurations (MT and HT) to a significant loss for MT of 63.60%utilization, while for HT, the loss is 49.90% over system saturation. This is a penalty due to the synchronization of the logical threads and a reflection that QIIME 2 is a compute-bound algorithm. The selected virtual machine in the Microsoft Azure cloud had four virtual processors, and we observed a utilization of 25.5%, similar to what we observed on-prem. In conclusion, we determined that the MT system showed better performance running the QIIME 2 algorithm. However, it was evident that CPU efficiency was lost after two CPUs due to the high proportion of sequential workflow. This knowledge is relevant to optimize the use of cluster and cloud resources. Therefore, we suggest performing similar studies with other applications that require higher computational resources, such as metagenome analysis.