Evaluation of Tools and Frameworks for Machine Learning Model Serving

Machine learning (ML) models are ubiquitous, as ML expands into numerous application domains. Despite this growth, the software engineering of ML systems remains complex, particularly in production environments. Serving ML models for inference is a critical part, as it provides the interface between the model and the surrounding components. Today, a variety of open source tools and frameworks for model serving exists which promise ease of use and performance. However, they differ in terms of usability, flexibility, scalability, and their overall performance. In this work, we systematically evaluate several popular model serving tools and frameworks in the context of a natural language processing scenario. In detail, we analyze their features and capabilities, conduct runtime experiments, and report on our experiences from various real-world ML projects. Our evaluation results provide valuable insights and considerations for ML engineers and other practitioners seeking for effective serving environments that seamlessly integrate with the existing ML tech stack, simplifying and accelerating the process of serving ML models in production.