An Ecosystem for the Large-Scale Reuse of Microservices in a Cloud-Native Context

This article presents an ecosystem that Ericsson developed to systematically practice large-scale reuse of microservices in a cloud-native context. We discuss how various ecosystem aspects facilitated the development and reuse of microservices across Ericsson. We also share lessons learned while developing the ecosystem.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/deed.ast. THE USE OF microservices in delivering software-intensive products has been on the rise during the last few years. 1,2 Many companies have shared their experiences of using microservices to achieve benefits such as faster delivery, scalability, and independent deployment. [1][2][3] Microservices are used primarily as a cloud-native architectural style, in combination with various DevOps practices (for example, continuous deployment) and containerbased solutions, to quickly deliver and scale software-intensive products. [2][3][4] In this article, we share Ericsson's journey of moving toward a cloud-native approach of developing software applications-focusing on how technology, practices, and teams are combined in an ecosystem to collaboratively develop common functions as microservices that can be reused and integrated across all applications within Ericsson. We also share lessons learned in developing the ecosystem, which is referred to as the application development platform (ADP) ecosystem.

An Overview of the ADP Ecosystem
Ericsson developed the ADP ecosystem to support its transition to the cloud-native approach of developing applications and services. Figure 1 provides an overview of the current state of the ADP ecosystem. The ADP ecosystem provides technical, process, and organizational support for developing cloud-native applications (CNAs) or cloud network functions within Ericsson. The ADP ecosystem is based on a modern architectural style-microservices and containers. The ecosystem also supports more large-scale reuse of microservices across applications within Ericsson. Due to their size, bounded context, and loose coupling, microservices are potentially a good unit of reuse and can thus support large-scale reuse across the entire organization. 5 The large-scale reuse of microservices across Ericsson is not possible unless they are accessible to everyone within the company. The ecosystem developed a marketplace to increase the visibility and knowledge about the microservices that are available for reuse. Furthermore, the ecosystem also provides process and onboarding support to the InnerSource (IS) ways of working in Ericsson's journey to collaboratively develop applications.
ADP microservices are frequently released and assembled into different applications. In such a scenario, it becomes important that microservices have consistent mechanisms for handling issues such as configuration, integration, product and artifact handling, and backward compatibility. The ADP ecosystem has codified these consistency requirements on microservices as design rules (DRs). The ecosystem provides continuous integration and delivery (CI/CD) pipelines that support the development, integration, and delivery of microservices and the assembly of those microservices into applications using Spinnaker (https://spinnaker.io/) as a critical component to orchestrate these pipelines and Helm (https://helm.sh/) to perform deployment and upgrades.
The ecosystem was started in late 2017, and the various aspects described here have emerged during the following four years as the needs became apparent. In spring 2020, due to the increase in the number of DRs, the need for providing a clear milestone-driven pathway for achieving maturity arose. In autumn 2020, the concept of the maturity staircase was established, and DRs were connected with different maturity levels. The connection of different DRs with varying maturity levels has made it easy to understand and express in which order DRs should be implemented by services to achieve a certain level of maturity. It also clarified what users could expect from a microservice at a certain maturity level.
While research on open source software ecosystems is extensive, we know little about internal software ecosystems (ISECOs  6 To address ISECO challenges, the ADP ecosystem provides a more holistic collaboration that is not limited to providing a product/platform for reuse but also supports the IS way of the collaborative development of microservices, automated checking of their compliance with DRs, and their CI into CNAs. In the coming sections, we will elaborate on different dimensions of the ADP ecosystem in detail.

Architectural Framework
The architectural framework in the ADP ecosystem centers around the concept of microservices and containers. It builds on the ecosystem of open source projects from the Cloud Native Computing Foundation (CNCF) (https://www.cncf.io/). The architectural framework consists of the following three key components.

Container Orchestration
The ecosystem selected Kubernetes as the container orchestrator and Helm as the packaging and deployment mechanism. The Kubernetes application programming interfaces (APIs) form the portability interface, allowing ADP microservices and applications to use any Kubernetescertified cloud platform. The CNCF provides the support to software and cloud vendors to certify their offerings and also maintains an inventory of Kubernetes-certified offerings (for more details, see Cloud Native Computing Foundation 7 ).

Microservices for Reuse Across Applications
The ecosystem provides various microservices corresponding to different functional areas, such as security, data, network, management, monitoring, and messaging. These services are mostly based on open source projects, and their configuration and lifecycle management are adapted to fit them in the ADP ecosystem. In some areas, their reuse is mandatory to promote consistent user experience and a common way to serve Ericsson customers.

Ecosystem Support Services and Tools
A large number of individuals and teams from different units across Ericsson participate in the ADP ecosystem: for example, to integrate, reuse, and contribute to microservices. In such a scenario, it becomes necessary to establish some ground rules to avoid inconsistent behaviors and enable automated mechanisms. The ecosystem developed common principles and DRs that need to be followed by all services participating in the ADP ecosystem. The DRs are formulated to ensure consistency across microservices: for example, how to ensure that logs and metrics are consistently structured, how to annotate Kubernetes resources, how to name and version images, or how to ensure a consistent security posture across multiple microservices. The ecosystem developed a family of DR checkers, which automatically check the compliance of microservices toward these DRs. The developers can integrate DR checkers into their CI pipeline, and the checkers will inform them if they are violating any DRs. These checkers have brought in muchneeded automation to the otherwise complex and time-consuming compliance testing process.

The ADP Marketplace
The ADP marketplace is an intranet website developed by Ericsson. The marketplace indexes all microservices that are open for use and contributions. In addition, there is a series of handson tutorials and guides attached to the marketplace that guides users on how to use and contribute to microservices and a microservice chassis that developers can use as a starting point for new microservices. As shown in Figure 2, it is possible to view all the microservices and apply filters to search for specific microservices based on their category, area, reusability level, and maturity. Other companies have implemented similar portals for publishing their IS projects. 8 SAP (https://www.sap.com/ index.html), for example, also has a similar portal (https://sap.github.io/ project-portal-for-innersource) where IS projects are shared for reuse and contributions. The ADP marketplace provides various filters for displaying and grouping the microservices, which are discussed next.
Generic Domain-specific services-These are specific to application teams within the same domain.
Application-specific services-These are specific to an application.

Reusability Level
The reusability level indicates to what extent a service is fit for reuse at a given point. The different levels are: Level 0-None: This service is not evaluated for reuse.
Level 1-Candidate: This service is potentially reusable and verified to be consistent with ADP architectural principles.
Level 2-Open for reuse: This service is ready to accept contributions and to be included in application assemblies.
Level 3-Reused: This service is effectively used by more than one application.

Microservice Maturity
The microservice maturity indicates the commercial readiness of a microservice and which DRs need to be implemented to meet each level. The different levels are: Level 0-Idea/PoC: This is the proof-of-concept and experimental stage.
Level 1-Development started: This level indicates development ongoing with the intention to reach higher maturity levels.
Level 2-Ready for integration: This level can coexist with other ADP services and has a CI pipeline that can be connected to an application staging environment.
Level 3-Ready for noncommercial use: This indicates limited release for use in demos or testing at customer site.
Level 4-Ready for commercial use: This level is telecommunications company-grade ready and can be used in commercial deployments.
For each microservice, the marketplace provides the following information: the overview, the documentation, compliance (met/not met/exempted), Helm charts to install the microservice, a link to the source code repositories, the team owning the microservice, discussions related to the microservice, and finally, the list of contributors for the microservice. In addition, the marketplace appreciates  More than 250 microservices have been made available on the marketplace for potential reuse and collaboration by January 2022. These include both functional and infrastructurerelated services (for example, metrics). Seventy of these 250 have reached the highest level of maturity; that is, they are ready for commercial use and are being reused by multiple applications.

CI/CD
The ADP CI/CD strives to enable Ericsson-wide reuse of services developed by any unit. If an application development unit finds a microservice in the ADP marketplace that satisfies one of their needs, they can integrate it to try it out. However, a one-shot integration is of course not acceptable-and so, to enable this kind of reuse, it must be possible to connect onto the CI/CD pipeline of the reused microservice also to receive and integrate new versions of the microservice as and when they become available. It is important to note that the CI and release of microservices are decoupled from the continuous assembly and release of applications. When a new version of a microservice is released, it is pushed immediately to the staging environment of those applications that have integrated it to test if it should be included in the applications baseline in place of the previous version.
There is no central service governance policy that applications must follow. However, the applications are aware of: 1. the benefits of switching to the latest microservice versionsthat is, updates and corrections are delivered in the latest versions 2. the risks of staying with the older versions-that is, older versions are more likely to result in several vulnerabilities in the security scans that are mandatory before release.
The ADP CI/CD uses Spinnaker pipelines to connect the individual microservices with the staging environments of different applications (see Figure 3). The ADP development environment provides Spinnaker pipelines as a service. An application may consume one or more microservices from elsewhere. Each connection point (arrows in Figure 3)-between the individual microservice and the staging environment of an applicationrepresents one reuse instance. There are more than 600 such connection points, demonstrating a large-scale reuse of microservices by applications. Some microservices are used only in one or two applications, but the most reused ones are currently included in as many as 46 applications. These include both functional and infrastructure services.
The use of software flow principles, Spinnaker, and its interfaces is mandated. However, the teams still have a high degree of freedom behind these mandated interfaces to use the most appropriate CI technology in their pipelines. Despite this autonomy, the ecosystem encourages teams to share their experiences of using particular CI/CD technologies to facilitate community-based implementations.

The ADP Program
There is a central organization-the ADP Program, which is responsible for sustaining the ecosystem. The ADP Program includes an architecture team that drives and governs principles, DRs, and guidelines related to microservice architecture and the cloud-native approach, enabling alignment in how microservices are developed and maintained. The governance body is an architecture council where ADP architects meet with an applications' architects to agree on architecture direction and any further improvements related to principles, DRs, and guidelines. The ADP Program also includes the development teams (more than 100 developers in teams of varying sizes) that own the development and maintenance of more than 50 Generic Services that are heavily reused across Ericsson.
The ADP Program includes a unit (the ADP Anchor Unit) that is tasked with devising and evolving the structures and policies of the ecosystem.
The microservice maturity indicates the commercial readiness of a microservice and which DRs need to be implemented to meet each level.
This unit also works to make it easier to create, release, and reuse microservices-the unit can be compared to the role that the CNCF takes in the open source Cloud Native Community. More than 100 practitioners, independent of the development teams working on the centrally funded Generic Services, perform these anchoring activities. The name Anchor was given because this unit "anchors" the ecosystem into the company structure-wherein individual services, applications, and tools are developed, owned, and contributed to by many different organizations across Ericsson. To keep track of the progress, the ADP Anchor Unit collects several metrics, such as the number of IS

IS Roles
The ADP services are handled as IS projects. Each IS project needs to have the following roles.
Guardians: Guardians, being the technical owners of the service, are responsible for providing contribution guidelines, participating in the relevant discussion forums, and reviewing and approving feature requests and contributions. The guardian is normally part of the development unit that created the ADP service and the corresponding IS project.
Trusted committers: Trusted committers, 8 who are more experienced and frequent contributors who have the required knowledge to guide new contributors, participate in discussion forums and the review process for discussing contribution requests. The recognition of active contributors as trusted committers is pivotal in creating a sustainable community for an ADP service.
Product owners: Product owners are responsible for maintaining the

IS Principles
To enable IS contributions to ADP services, the ADP Program has defined some basic principles: • The source code of the ADP services shall be visible through the ADP marketplace, but as a quality control mechanism, patches can be integrated only by the relevant guardians or trusted committers after review.
• The backlog of ADP services shall be open to all so that potential contributors are informed about the planned features in the future releases. • The guardian and trusted committers of each ADP service shall create and moderate a forum to discuss bugs, new feature requests, and contributions. • Before an ADP microservice is included in a shipped product, its legal analysis is performed to ensure compliance with trading regulations and the license requirements of any open source software included.
The ADP Program provides further guidance in the form of examples, templates, FAQs, and tutorials.

The IS Ownership Models
The IS ownership models can be broadly classified into central versus distributed models. In the central model, both the product and technical ownership are with one organizational unit. In the distributed model, the product ownership still rests with one organizational unit, while the technical ownership is shared across multiple units. As the community around the IS project grows, it becomes possible to assign the trusted committer role to a few contributors from other organizational units. The trusted committer could then share the technical ownership of the IS project. The distributed model is particularly important for Reusable Services in the ADP ecosystem to foster a sustainable community of trusted committers who can manage the IS project even when the original guardians are not available.

Lessons Learned
With the help of an exploratory case study (the complete case study, with methodological details and other findings, is to be published separately), we identified the following lessons learned by Ericsson in developing the ADP ecosystem.
IS contributions prevent the ADP ecosystem from becoming a bottleneck: It is important for the teams creating Generic and Reusable Services to be open to receiving contributions. In addition, it is also important to provide automated tools and software development kits that make it easier to contribute.
Tool support for checking the DRs: The ecosystem developed a family of checkers to automatically check the conformance of the DRs. The developers only need to integrate the relevant checker in their CI pipeline and let the checker detect the compliance issues, if any. Most developers appreciate the checkers as they allow them to quickly verify their code-to see if they have done a good job of conformance to the DRs.
Open and proactive communication: synchronizing contributions: It is important for the IS contributors to discuss their contribution plan and make adjustments to align with the purpose and architecture of the service before working on the contribution.
Widespread reuse: The widespread reuse of microservices across Ericsson has been made possible by establishing a CI/CD strategy that allows a relatively quick "plug and play" of microservices and applications' assembly pipelines. The key has been to keep the microservices' lifecycle and pipelines independent from the assembled applications' lifecycle and their pipelines. W e aim to improve IS contributions by investigating a sample of Most developers appreciate the checkers as they allow them to quickly verify their code-to see if they have done a good job of conformance to the DRs.
recently completed contributions for identifying the practices that worked well (or otherwise), bottlenecks, and potential improvements in the existing IS contribution practices and guidelines.