An independent study of two deep learning platforms - H2O and SINGA

Two open source distributed machine learning/deep learning platforms, namely H2O and Apache SINGA, compared their deep learning performances using multilayer perceptron on the classic MNIST database for hand written digits recognition. However, the results reported by both parties differ and neither of them can repeat the results reported by the other side. This paper is an independent study of the performances of H2O and SINGA on deep learning, considering both testing accuracies and time required for model training. We reproduced the performance benchmark, then we designed our experiments to test the performances using a 1-node and a 4-node cluster. We repeated the test for multiple runs and checked the difference in accuracy with a paired t-test. Our study showed that H2O generated stable and accurate performance. SINGA could be trained more efficiently in a short time but the accuracy deviates a lot from the expected if training details were changed.