XFBench: A Cross-Cloud Benchmark Suite for Evaluating FaaS Workflow Platforms

Functions-as-a-Service (FaaS) is a widely used serverless computing abstraction that helps developers build applications using event-driven, stateless functions that execute on the cloud. Commercial FaaS platforms such as AWS Lambda and Azure Functions offer elastic auto-scaling and invocation-level billing to ease operations. Applications are often composed as a dataflow of FaaS functions that are orchestrated by FaaS workflow platforms such as AWS Step Functions or Azure Durable Functions. However, the proprietary nature of FaaS platforms on public clouds means that their internals are less understood. While benchmarks to characterize FaaS platforms exist, none are available for a principled evaluation of FaaS workflow platforms. Further, they are less configurable and often limited to simple workloads and a single cloud provider. We address this limitation by proposing XFBench, an end-to-end automated benchmarking framework for FaaS workflows that works across clouds, and an accompanying function, workflow, and workload suite. The user provides a generic definition of the workflow and workload for benchmarking, and XFBench automatically deploys the workflows across multiple cloud platforms, generates client requests, and profiles the execution. We validate XFBench with realistic workflows and workloads on AWS and Azure platforms in different global regions to offer early insights into understanding the inter-function communication, function execution time, and cold start scaling.