Take Your VNF to the Gym: A Testing Framework for Automated NFV Performance Benchmarking

A VNF is a software entity to be run in diverse execution environments with variable configuration options and capabilities (e.g., hardware acceleration) impacting performance. NFV resource multiplexed infrastructures can impose hard-topredict relationships between VNF performance metrics (e.g., latency, frame loss), the underlying allocated resources (e.g., units of vCPU), and the overall system workload. Characterized by many-fold platform configuration and environment variables, the evolving scenario of NFV calls for adequate testing methodologies embracing modern continuous development and integration practices and leveraging open source tools and mindset. To this end, we introduce Gym as our proposed testing framework and methodology for automated NFV performance benchmarking. We present our design principles and the outcomes from a practical validation on a vIMS scenario. A discussion of the lessons learned and the overall NFV performance testing landscape are further contributions of this article.