Automatic Prevention of Accidents in Production

We present a framework for automatically testing functional correctness of back-end servers. We created a pre-production environment where traffic between servers can be reconfigured dynamically. Production requests are sampled and replayed in our framework so thatwe can cover many corner cases of the system without having the developer manually write test cases. We also describe how to handle mutate requests and support checking the validity of back-end rewrites.