Changing microsoft's build: Revolution or evolution

Tens of thousands of Microsoft engineers build and test hundreds of software products several times a day. It is essential that this continuous integration scales, guarantees short feedback cycles, and functions reliably with minimal human intervention. During the past three years TSE's charter has been to shorten this cycle time. We went after this goal in two ways: Evolution via CloudBuild and Revolution via Concord. CloudBuild is a build service infrastructure, now being used by all major product groups in Microsoft, like Azure, Bing, Office, SQL except for Windows. CloudBuild addresses all aspects of a continuous integration workflow, like builds, test and code analysis, but also drops, package and symbol creation and storage. CloudBuild supports multiple build languages as long as they fulfill a coarse grained IO based contract. CloudBuild uses content based caching to run build-related tasks only when needed. Lastly, it builds on many machines in parallel. The speed ups of build and testing range from 1.2x to 10x. CloudBuild aims to rapidly onboard teams and hence has to support non-deterministic build tools and specification languages that under-declare dependencies. CloudBuild, being a reliable build service in the presence of unreliable components, currently achieves service availability better than 99%. Windows went a different path. Their past build exhaust was so massive that building Windows in the cloud and bringing the build results back for testing on corp.-net. was considered infeasible. So they decided to move to a new build language, codename Concord. By construction, Concord guarantees reliable builds, no over-build, and allows for efficient distribution. Adopting Concord has led to immense performance improvements, we have seen up to 100X speedup for Windows builds. But the path has been long and rocky, since it not only requires a substantial rewrite of existing build logic, but also all related developer and build lab processes have to change. Whether evolution or revolution is the right path forward — the verdict is still out.