Multi-Domain Federation Utilizing Software Defined Networking—A Review

The global growth of the Internet continues with an increase in accessibility, connected users and new applications in emerging fields like eGames, augmented and virtual reality. Technologies that improve reliability whilst reducing cost and latency are being developed. However, before the Internet can sustain the growth in utilisation and new applications there are challenges to overcome. One of the key challenges remaining is to develop new mechanisms for intra and inter-domain federation that facilitate sharing of reachability and routing information. Software Defined Networking (SDN) is being introduced to address control and management challenges at domain boundaries. This paper reviews the current state of domain federation and discusses the potential approaches that utilise SDN for a next-generation solution.


I. INTRODUCTION
The evolution of the Internet is redefining the way that we utilise technology to create and consume information. The evolutionary process involves developing new platforms to increase the efficiency of the Internet, support new applications and services and improve information security. The Internet today is global, complex and operates utilising a range of technologies, protocols, devices and systems that adhere to standards set by the international standardisation bodies. The global roll out of new technologies has become increasingly difficult to manage and often means the standards require backwards compatibility at the expense of the efficiency gain promised by the introduction of the new technology.
The Internet is composed of Autonomous Systems (AS) that are a set of Internet Protocol (IP) prefixes for one or more networks that are managed by a single organisation or entity. AS are controlled by the administering organisation, which sets internal routing, rules and policies. The Border Gateway Protocol (BGP) is a standardised protocol that is The associate editor coordinating the review of this manuscript and approving it for publication was Chakchai So-In . used to exchange routing and reachability information among the border routers in different AS domains. BGP uses the routing information to maintain a local database of network reachability. BGP then utilises this data to construct a graph of AS connectivity that is used to identify routing paths and to enforce policy at the AS gateway. BGP version 4, the current version, was published as RFC 4271 in 2006 [1] and updated eight times over the past sixteen years. Despite its widespread use, BGP is known to have a number of security concerns [2] and can suffer from slow convergence [3]. Another flaw in BGP is a lack of assurance when validating authorisation for network layer reachability notification messages. BGP's original design did not include the security mechanisms that would prevent the Internet being disrupted either accidentally or by malicious act. To address this problem, a distributed repository system known as Resource Public Key Infrastructure (RPKI) was introduced. However, validation necessitates additional AS implementation stages, which may result in performance and compatibility issues for legacy network infrastructure.
The Internet's current issues are a natural consequence of its evolving architecture. The Internet commenced as a single data communication link and it is now a complex global network. Due to the increased utilisation and new applications integrated service platforms have been developed and introduced to support IoT, Big Data, AI, and cloud services [4]. However, the existing hardware-oriented, static, and manually-operated network environment does not provide the capabilities and flexibility needed to support increased utilisation and new applications. The Internet's classical architecture, which the SDN paradigm [5] is now being applied to, has become etched in stone, a so-called Internet ossification [6]. SDN is a cutting-edge networking paradigm that has introduced concepts and capabilities that can make networks flexible, reliable and programmatically supports the roll-out of new network management and control applications and services. SDN manages message flows in the data layer using the standardised OpenFlow protocol between SDN controllers and the data flow devices. SDN [7] entices network designers to be rethink and re-image aspects of the Internet that have remained static for decades. It should be noted that it is incorrect to use the terms OpenFlow and SDN interchangeably.
While the first targets of SDN were the cloud and campus networks, we should not ignore its wide-ranging deployment, particularly from the Future Internet (FI) Research & Development (R&D) perspective. SDN's deployment has provided programmable interfaces that enable enhanced automation in provisioning network services. Service providers in general and National Research and Education Networks (NRENs) in particular are also dealing with SDN technologies being incorporated into new deployments [8]. SDN can offer various benefits that can efficiently address cybersecurity issues for cloud-based IoT devices [9]. Within the servers with limited resources, SDN can also be used to manage the diversity of service requests [10].
Employing SDN at an Internet exchange point can improve management and control by using programmable control methods and serving as a Software-Defined eXchange (SDX) centre to handle a large number of advertisements in an adaptive manner. In recent experiments [11], SDN techniques have also been shown to increase the control of BGP routing and improve performance. However, due to BGP's global adoption and the resulting political, technical, and economic obstacles, deploying competing protocols or modified versions of BGP is challenging. As a result, any advancements and proposed solutions should be compatible with the legacy BGP. In light of this, SDN principles have been advocated to be used to overcome BGP weaknesses.
SDN is increasingly being implemented over heterogeneous, multi-technology, and large-scale networks by carriers [12], a key remaining challenge is to harness the flexible and programmatic SDN paradigm between AS domains. Domains are controlled and operated independently, which could restrict the potential for new federated control and routing mechanisms. New domains, as it turns out, do not come into being with instant interoperability and federation. The new domain is federated into the Internet through the manual setup of its border router and the implementation of the domain control and routing policies.
An SDN based approach for multi-domains would require a control channel between domain gateways for inter-domain SDN federated communication. The advantages of implementing SDN into multi-domain networking compared to traditional methods are its ability to provide policy-driven network monitoring, agility, traffic management and control, and apply network automation. Multi-domain SDN could make it possible to connect different SDN domains automatically, bring about domain interoperability, and improve the offering of cross-domain services.
By enabling a software-centric, dynamic, automated, and intelligent network environment, multi-domain SDN [13] can be deployed to create a next-generation federation framework. The framework should reduce operating costs and enable the flexible management of traffic flows. Inter-domain traffic will be enhanced by removing the control and management complexity of legacy systems and by introducing advanced network service management, including flexible peering arrangements, bandwidth on demand, and real-time provisioning [14].
The reachability of network nodes in SDN based domains is a current research and development focus. The endto-end access rule provisioning requirements and architectural components have not yet been fully defined. In this paper, we review the SDN literature and present the SDN based multi-domain federation concept, the basis for a multi-domain, vendor-neutral SDN based domain federation and the architectural theory. We also provide an overview of recent developments in federated domains and address research issues and strategies for future implementations.
The remainder of this paper will be structured as follows. The SDN paradigm, architecture and control environment are presented in Section II, followed by different multi-domain communication mechanisms in Section III. Section IV discusses federation, including its advantages, applications, and requirements. Then, in Section V, the building blocks and architecture for SDN based federation are covered. Current SDN based federation test beds are presented in Section VI. VOLUME 11, 2023 Section VII discusses the opportunities and challenges of SDN based multi-domain federation. Section VIII brings the paper to a conclusion.

II. SDN CONCEPTS
Current networks can be vertically integrated, which increases complexity with the control and data planes bundled together as shown in Fig. 1. Despite their widespread acceptance, conventional IP networks are complex and difficult to manage. Configuring the network utilising vendor-specific commands according to predefined policies is complicated, and needs constant updating to respond to faults, load and route changes. The distributed network control and transport protocols are the key technologies that allow information to travel around the world in the form of digital packets [15].
Understanding the complex nature of existing IP networks is a focus for the research community and industry. With the knowledge gained, new concepts have been proposed for the design of future networks [16]. Some of the innovative concepts have been implemented, including programmable networks [17], Named Data Networking (NDN) [18], and ''HTTP as a narrow waist'' [19].

A. SDN EVOLUTION
From a network architecture perspective, the SDN capabilities and features mentioned are not entirely new. There were previous attempts to encourage the programmable networking approach. The history is divided into three stages [20], each with its own contribution: (1) active networks (from the mid-1990s to the early 2000s), which introduced programmable functions in the network, resulting in greater innovation; (2) control and data plane separation (from  around 2001 to 2007), which developed open interfaces between the control and data planes; and (3) the OpenFlow Application Programming Interface (API) and network operating systems (from 2007 to around 2010), which represented the first widest range of open interfaces between the control and data planes. Throughout the history of SDN, network virtualisation has played a key role.
Active networking concept attempts to control a network using software in real-time. SwitchWare [21], [22] is an active network-functioning solution that permits packets to flow through a network and dynamically change network operations. Similarly, software routing suites on traditional PC hardware, such as Click [23], XORP [24], Quagga [25], and BIRD [26], are the basis for extensible software routers utilising programmable network devices. By loading new or changing existing routing software, the behaviour of certain network devices can be changed dynamically.
The last decade has seen an increase in research into how to decouple the data and control planes. Caesar et al. first proposed a Routing Control Platform (RCP) in 2004 [27] in which inter-domain routing is replaced by centralised routing control to reduce the difficulty of fully distributed route computing. The IETF released the Forwarding and Control Element Separation (ForCES) framework in the same year, which separates control and packet forwarding elements within a ForCES network [28], [29], [30], [31]. In 2005 Greenberg et al. suggested a 4D approach [32], [33], [34] implementing a blank slate concept of four planes for the entire network architecture. The planes are, respectively, ''decision,'' ''dissemination,'' ''discovery,'' and ''information'' arranged from top to bottom. In 2006, the Path Computation Element (PCE) architecture was introduced to measure label switched paths separately from real packet forwarding within Multiprotocol Label Switching (MPLS) and Generalised Multiprotocol Label Switching (GMPLS) networks [35]. In 2007, Casado et al. introduced Ethane, where simple flow-based Ethernet switches are complemented by a centralised controller to handle flow entry and routing [36], [37], [38], [39]. Commercial networking devices also adopted the idea of separating the control and data planes. For example, the control plane is decoupled from the data plane and modularised in the Cisco Aggregation Services Routers (ASR) 1000 series routers and Nexus 7000 series switches, allowing for the coexistence of an active control plane instance and a standby instance for high fault tolerance and transparent software upgrades.
Compared with traditional networks, the value of SDN lies in that it offers programmability as well as the decoupling of the control and data planes. In particular, SDN provides a flexible approach to introducing APIs to the controllers. In addition, SDN clearly distinguishes the control and data planes within the network architecture. The plane separation means that network service control and monitoring functions can be added, removed, updated and operated independently of the data plane flow devices, thereby without impact on the traffic flows. SDN has significantly improved the opportunity for dynamic and flexible traffic control and management [40].

B. SDN BUILDING BLOCKS
The Open Networking Foundation (ONF) describes SDN [41] as the ''physical separation of the network control plane from the forwarding plane, and where a control plane controls several devices.'' SDN has the following capabilities and features: 1. SDN decouples the control plane from the forwarding plane [42], utilises OpenFlow for traffic flow control messages passed between the SDN controllers and infrastructure layer data flow devices, and provides a northbound API to a programmatic application layer. The SDN paradigm facilitates network services, hosted in the SDN controllers, that manage and monitor the infrastructure layer devices utilising standardised and non-standardised protocols.
2. In SDN the packet forwarding decisions are based on flows rather than destinations. A flow is usually defined by a collection of packet field values, which serve as a criterion for the match (filter) and a collection of actions (instructions). SDN categorises a flow as a packet series between a source and a destination. At the forwarding devices [43], [44] all packets of a flow receive similar service policy. The flow abstraction allows the behaviour of various types of network devices to be unified, including routers, switches, firewalls, and gateways [45]. Flow programming permits unparalleled flexibility, restricted only to the capacities of the flow tables implemented [13].
3. Control functionality is removed from network devices that transition to become simple message forwarding devices. The control logic is transferred to the so-called SDN or Network Operating System (NOS) controller. The NOS is a software platform running on commodity server technology and providing the essential resources and abstractions to facilitate forwarding device programming based on a logically centralised, abstract view of the network. Its function is similar to a conventional operating system. 4. SDN introduces a network architecture that makes it possible to programmatically control rather than configure computer networks. It is a core function of SDN, considered as its primary value proposition.

C. SDN BENEFITS
SDN tackles the challenges of the legacy network architecture as outlined in Table 1. The benefits are derived from enhanced configuration options, increased performance and innovation in network design and operations.

1) ENHANCED NETWORK MANAGEMENT
Network configuration involves a degree of manual processing due to the heterogeneity of network device manufacturers and configuration interfaces. This manual setup process is both repetitive and vulnerable to error. SDN can help remedy this network management problem. In SDN, the integration of the control plane over all types of network devices enables network devices to be configured from a single point, automatically through software control [46]. As such a network can be programmatically configured and optimised dynamically depending on the network status [47].

2) IMPROVED NETWORK PERFORMANCE
One of the main goals of network operations is to optimise the utilisation of the invested network infrastructure. However, maximising network efficiency as a whole was deemed difficult due to the coexistence of various technologies and stakeholders in a single network. Current approaches concentrate on optimising a subset of network outputs, or the quality of user experience for selected network services. Obviously, the strategies employed may lead to suboptimal efficiency, if not conflicting network operations, based on local information without consideration of cross-layers. SDN creates an opportunity for a global change in network efficiency and explicitly offers a global view of the network including data flow control, management and monitoring across the various layers. It follows that new solutions for classical problems, such as data traffic scheduling [48], end-to-end congestion control [49], load-balanced packet routing [50], energy-efficient operation [51], and support for Quality of Service (QoS) [52], [53], can be developed and deployed to verify their effectiveness in improving network performance.

3) THE EMERGENCE OF VIRTUALISATION
SDN is an exciting paradigm used to facilitate hyper-scale Data Centre (DC) management. DCs raise important scalability concerns, especially with the growth and migration of Virtual Machines (VM). Moving a VM and modifying the address table for Media Access Control (MAC) using the conventional network architecture will disrupt user experience and applications. Network virtualisation is a key technology that is used to optimise infrastructure utilisation in hyperscale DCs. It provides tunnels that can abstract the MAC address from the infrastructure layer, allowing traffic from Layer 2 to run over Layer 3 overlays and simplifying network deployment and migration of VMs [54]. In addition, SDN enables multi-tenant hosting providers to link their physical and virtual servers, local and remote facilities, and public and private clouds to a single logical network. As a consequence, the network operator would have an independent view of each customer. SDN facilitates the orchestration of a virtualisation layer to cloud provider network architectures.

D. SDN ARCHITECTURE
SDN includes three abstractions: specification, delivery, and forwarding. The abstractions are essential tools for architecture design, implementation and future research [55].
The specification abstraction will allow a network application to communicate the desired network behaviour without being responsible for enforcing the behaviour itself. This can be achieved through network programming languages as well as virtualisation solutions. These approaches map the abstract configurations represented by the applications, based on a condensed, abstract network model, into a physical configuration presented by the SDN controller for the global network view.  The delivery abstraction shields SDN applications from distributed state complexities, making the distributed control problem into a logically centralised one. Its realisation requires a common layer of distribution, which resides in the SDN NOS. There are two basic functions to this layer. First, it is responsible for installing the control commands into the forwarding devices. Second, it gathers forwarding layer status information (network devices and links) to provide network applications with a global view of the network.
Ideally, the forwarding abstraction permits forwarding behaviour that is required by the network applications (the control system) while hiding the underlying infrastructure. OpenFlow is one realisation of an abstraction, which can be used as the equivalent of a ''device-driver.'' The SDN architecture is shown in Fig. 2. Key features of this architecture include: Management Plane or Application Layer. The management plane is the collection of applications that take advantage of the functions that the northbound interface provides to implement network control and operating logic. This includes routing, firewalls, access, authentication, load balancing, and tracking. In essence, a management application defines the policies which are ultimately translated into southbound specific instructions that control the actions of the forwarding devices.
Northbound Interface. Application developers have a northbound interface API accessible from the NOS, that is, a common interface for overlay applications. Usually, an API abstracts the low-level instruction sets used to control or monitor forwarding devices via the southbound interface.
Control Plane or Controller Layer. There are various applications running in the controller and abstracted for the Operation & Management (O&M) as well as maintenance of the entire network. In summary, the SDN controller provides and retains a global view of the whole network from which users can create further applications to enhance the efficiency of the network and resource utilisation.
Southbound Interface. The forwarding devices instruction set is specified using the southbound interface API. In addition, the southbound interface also defines the communication protocol between the forwarding devices and control plane elements. This protocol formalises how elements of the control and data plane interact with each other.
Data Plane or Infrastructure Layer. Wired cables or wireless radio channels interconnect the forwarding devices. The network infrastructure includes the interconnected forwarding devices, representing the data plane.
Forwarding Devices. The data plane forwarding devices are network devices that perform a series of elementary operations. The forwarding devices have well-defined instruction sets, e.g., flow rules, used to take actions on the incoming packets that may include port forwarding, packet dropping, interaction with the controller, header rewriting or packet encapsulation.
The SDN controllers interact with the underlying data layer forwarding devices to optimise traffic flows and to manage and monitor the network. The control layer facilitates business logic, applications and services that reside in an application layer connected to the controller via the northbound interface API. Controllers can utilise east-west interfaces as a bridge to adjacent controllers or other network devices. The architecture implemented to support network control can affect localised controller performance and scalability [56].

E. CONTROL ENVIRONMENT
SDN has been applied to intra and inter-domain networking scenarios. The SDN network architecture implemented to control, manage and monitor domains can be centralised or distributed and both can utilise a hierarchy of controllers. SDN has both physical and virtualised control planes. In this section the SDN controller distribution is discussed.

1) SINGLE CONTROL PLANE
Early SDN implementations were developed for single controller single domain settings principally situated in a data centre. This type of control plane is referred to as the single control plane and utilised a basic NOX controller [44]. To control, manage and monitor the domain network, Ethane, an early implementation, adopted a single controller approach. Ethane [38] implementations were designed to control, manage and monitor an estimated 10,000 or more data flow devices using a single controller. For small networks, this solution was suitable but it is unusable for large deployments. To reduce controller loads, the single controller technique utilises two approaches: • Hardware or Multi-core Controller. When SDN was implemented, one fundamental problem was how to identify routing solutions using a single controller. If the controller lacks the capability to manage the flow requests, it will become a bottleneck for the entire network. For this reason, a range of proposals were put forward to improve the single controller's processing capability, such as Beacon [57] and McNettle [58]. Beacon offers a platform to control network devices and achieves high performance by using multi-core processing. In addition, the output can be scaled-up linearly with the number of processing cores. For example, Beacon with 12 cores was able to handle 12.8 million flow requests per second. However, the simplicity of a centralised controller comes at the expense of the control plane having minimal scalability. McNettle is designed to solve this issue as an extensible control device. The control capability can be expanded by adding handlers and background programs that utilise a multi-core processor. Using a single 46-core processor, McNettle was capable of serving up to 5000 switches while achieving a throughput of over 14 million flows per second. Maestro [59] provides the application programmers with a basic single-threaded programming model. In addition, parallelism optimisation techniques are used to increase a controller's throughput. Using an eight-core controller, Maestro can handle 600,000 flow requests per second. Although this approach can boost the capability of a single controller, one controller is inadequate in some network scenarios.
For SDN, the new flow routing paths must be determined by the controller. The time it takes for the controller to assign a routing path when it receives a flow request should be minimal.
• Overhead Reduction. The second approach aims to minimise controller load, as demonstrated by proposals such as DIFANE [60] and DevoFlow [61], by expanding the function of the switch data plane. DIFANE partially uses intermediate switches called authority switches to make transmission decisions instead of relying solely on the centralised controller. In DevoFlow a similar method is also suggested with the collection of different flows to be guided to the controller, while switches manage the other flows. In data centres a centralised controller has limited ability to adjust to changes in the inter-network load.
In the case of the Internet, a single controller is not appropriate due to the broad geographical range, network complexity and the need for organisations to control and manage AS domains. Remote flow request transmission increases delay. SDN based networking implementations today generally utilise multiple controllers. The only implementations that would not use a controller hierarchy are small networks.

2) THE MULTIPLE CONTROL PLANE
Multi-controller implementations are an unavoidable and rational solution that provides reliability, scalability, and security. A single controller within the network is a single point of failure and a promising option for attackers. A malicious or malfunctioning controller can also easily disrupt the entire network. Hence the multi-controller configuration reduces risk. Multiple controller implementations have key issues. The first challenge is the requirement that the controllers have a single network-wide vision. This problem is not solved using static configurations, and it is possible to have individual controllers with a much higher load than other controllers in the network. The routing announcements, path identification and linkage between the control plane and the forwarding plane should be automated. The second challenge is to find an optimum number of controllers to ensure there is a linear scaling up of the SDN network. The final challenge is how to synchronise network operation and routing. Most methods use localised solutions to control, manage and monitor the local neighbourhood.
There are two distinct methods of implementing the multi-controller approach to control, manage and monitor the network as outlined in Table 2.
• Logically centralised. The logically centralised controller hierarchy exchanges information to create a coherent view of the entire network. To centralise the logic, Gao et al. [62] used three approaches; a distributed file system, a distributed hash table, or a precalculation of all possible combinations. The controllers maintain a highly consistent network-wide view. The VOLUME 11, 2023  [68] found that utilising a centralised SDN control plane degrades the efficiency of many applications and the study identified that the control plane state and logic should be physically distributed. Levin et al. also showed that a controller can derive a non-optimal but reasonable flow path, even if the network view is limited in scope. Tam et al. [69] and Schmid and Suomela [70] carried out further study in this field. Tam et al. proposed multiple independent controllers with each controller managing a segment of the network. Schmid and Suomela moved this approach forward with the introduction of improved routing algorithms. ElastiCon [71] proposed a controller pool that grows or shrinks dynamically according to traffic conditions and workload. Pratyaastha [72] proposed an Efficient Elastic Distributed SDN Control Plane; a novel method for assigning SDN switches and SDN device state partitions to distributed controller instances.

F. SDN SECURITY ISSUES
Applying SDN to multi-domain networks introduces security challenges that have been highlighted in the literature [73], [74], [75], [76]. The overview of SDN security issues is shown in Fig. 3. Some of the security challenges arise in the control and application planes. Applications that can be subverted and used to insert malicious configurations into network devices without authentication can decrease network capacity and stability, possibly resulting in network failure. Loss of traceability and transparency of application flows can cause network debugging difficulties. In multi-domain SDN, application flows and network state monitoring and auditing can help track and replay network state or debug a network fault. Information on network activity can also be used to identify patterns of attack [77]. The control plane also exposes network-wide resources to overlay applications, opening a door for malicious applications. Additionally, adopting a single controller may result in a single point of failure that could become an attractive target for Denial of Service (DoS) attacks.
When the configuration-complex, Transport Layer Security (TLS) protocol is not implemented, the lack of an authenticated controller-switch communication channel may result in more serious security threats. Adversaries may start a man-in-the-middle attack by seizing all messages between controllers and switches [78], [79]. Additionally, malicious switches may also initiate spoofing attacks by faking identities, e.g., IP addresses, that could lead to DoS or DDoS attacks [80], [81].
In recent years several proposals have been made [82], [83], [84], [85], [86], [87], [88], [89], [90] to resolve the multi-domain SDN security issues. The proposals tackle particular security issues. In particular, current work providing application flow authentication or flow-safe restriction extend secure modules on a controller rather than support a monolithic secure module in a multi-controller environment. Existing role-based access control systems need to be fine-grained on network-wide resources. The optional TLS protocol and other cryptography-based authentication protocols allow multiple interactions (also called multiple passes) to create a communication channel for controllers. In a network with a physically decentralised control plane, merely combining existing schemes does not solve the common security problems effectively as the secure modules have to operate seamlessly between several controllers. A universal security approach that resolves the common security challenges has not been proposed in the literature.

III. MULTI-DOMAIN COMMUNICATION
In an inter-network spanning multiple administrative domains, a logically decentralised control plane is needed. While the domain operators are not likely to agree to unified control, it may be reasonable to provide a certain level of sharing (e.g., to ensure that service level agreements are met for traffic flowing between and transiting domains). Some architectures on distributed control planes with a logically centralised approach such as Onix, Hyperflow, and Elasticon are currently unable to handle inter-domain flows between SDN domains. According to Egilmez (2014) [91] the fully distributed SDN controller architectures, both vertical and horizontal approaches, may be used for multi-domain SDN communication. A multi-domain SDN architecture refers to the architecture of a network linking multiple SDN domains. SDN domain refers to the administrative SDN domain, which can be a sub-network in a data centre network, or a carrier or business network, or an AS. A multi-domain architecture allows various functional domains to be generated within the SDN model that can be managed by different parties, such as the Internet. Such administrative domains will regulate entirely which information and resources they are presenting to others. Three methods for modelling Controller to Controller (C2C) relationships were suggested for multi-domain deployment [92].
The tree-based model (the vertical approach) has tree-like controllers where parent controllers can delegate some roles and responsibilities to their subordinates but maintain complete visibility of all actions and states. Multi-domain services are coordinated in this configuration by requests being sent to the parent controllers by their subordinate controllers when changes to operations occur. Control tasks are allocated to different controllers in this deployment model, based on selected parameters such as network view and locality requirements. Thus, the controller manages local events and is lower in the hierarchy, and the higher level manages global events.
The chain-based configuration model (horizontal approach) has logically configured controllers to communicate directly only with their neighbours, and communication with other controllers in other domains is enabled by the neighbouring networks relaying requests to the next-hop controller. Multiple controllers are arranged in a flat control plane, in which each controls a sub-set of the network switches. This can be achieved either with replication of the state or without replication of the state. There is no reciprocal regulation of administration within these various administrative realms. The domain is managed by its collection of controllers, and requests for multi-domain services depend on the functionality. To do this, an east/west-bound interface VOLUME 11, 2023 communication protocol must be introduced, present and accessible to controllers within each domain.
Hybrid setup is a mixture where the controllers may also provide a certain level of logical decentralisation. An interesting type of proxy controller, called FlowVisor [93], can be used to introduce a level of network virtualisation to SDN networks and allow multiple controllers to manage overlapping physical switches simultaneously. It was initially designed to allow experimental research to be carried out alongside production traffic on deployed networks, and also facilitates and demonstrates the ease of deploying new services in SDN environments.
A crucial question is how many network abstraction modules can be centralised and configured effectively to serve logically structured control tasks while providing physically distributed protocols at the same time. Consequently, hybrid SDN architectures are being considered as an approach to overcome current challenges. Typically the controllers are arranged utilising a combination of tree and chain structures. Orchestrators in this approach will include standard interfaces, frameworks and policies to manage and communicate in distributed environments with the control planes, and will provide high-availability and fault tolerance capabilities.
The separation of SDN controllers into disjoint peer domains may bring benefits:  [95] recommend modified BGP protocols to improve the reachability and routing update process, while other study suggested reachability and routing information exchange could occur by utilising links between private and secure east-west SDN controller interfaces [96]. Several works, such as SDNi [97], Google [98], Microsoft [99], Noise Lab [95], and ON.Lab (Open Networking Laboratory) [42] will be addressed in the following section to provide examples of research into new network architectures.

1) SDNi
The inter-SDN domain protocol (SDNi) can be used as an interface between the controllers in different SDN based domains. The controllers are able to share topology, routes and path information, energy consumption and QoS-related policies and information. In addition, the SDNi protocol permits sharing information on reachability and facilitates endto-end Service Level Agreements (SLA) over heterogeneous networks. To ensure the extensibility of its transport mechanisms and syntax, SDNi still lacks a semantic network model. Ideally, a completely automated service, from negotiation to order, should be assisted by delivery confirmation and charging [100].

2) GOOGLE B4 (GLOBALLY DEPLOYED SD-WAN)
While utilising conventional WAN architectures, Google B4 was motivated by the finding that Google was unable to achieve the degree of the scale, fault tolerance, cost efficiency and control needed for its network. Google's networking requirements included meeting dynamic demands for bandwidth, reducing website page display times, end application controls, and cost sensitivity. An SDN based approach was used to meet the requirements identified. Google adopted commodity servers and devices rather than utilise traditional vendor products. The SDN ecosystem permits the rapid iteration of novel protocols, ii) simpler testing environments, iii) improved capability planning, and iv) simpler management through a fabric-centred, rather than router-centred, WAN view. Google created an internal distributed data centre network called ''G-Scale'' with the SD-WAN solution. The specification for the B4 sites involves the hardware layer forwarding traffic without the intervention of a complex control program. The site controller layer is composed of Network Control Servers (NCS) which hosts both OpenFlow controllers and applications for network control.

3) MICROSOFT SWAN (SOFTWARE-DRIVEN WAN)
Microsoft chose to develop a new WAN architecture with the goal to improve throughput and to promote flexible networkwide sharing. SWAN aims to provide: • Services, excluding interactive, that inform the SD-WAN controller of current traffic demands between DC pairs.
• The controller, which has an up-to-date global view of the topology and network traffic demands, monitors traffic volumes and route utilisation.
• The controller changes the forwarding statuses of flow devices explicitly based on current network demands. Microsoft's design involves the collective deployment of service hosts and brokers to estimate current network traffic demands and this information is used to manage the network utilising distributed controllers.

4) SOFTWARE-DEFINED EXCHANGE POINT
Chung et al. [101] proposed to change the delivery of wide-area traffic by developing, prototyping and implementing a Software-Defined eXchange (SDX) that addresses four challenges: i) convincing applications, ii) programming abstractions, iii) scalable operation and iv) practical implementation. Virtual SDX switch abstraction allows each AS to run SDN based applications defining versatile traffic drop, change, and forwarding policies, and the SDX can be used to merge several AS policies into a single, coherent physical forwarding policy. The SDX route server enables each user to forward traffic for a prefix to all feasible routes, as it has specific features such as overriding default BGP routes and integrating SDX with existing infrastructure. The SDX controller implementation has two main pipelines: a policy compiler and a route server [95].

5) ONOS
To address the need for an OpenFlow-based carriergrade SDN controller, Stanford University's ON.Lab leads the development of Open Network Operating System (ONOS) [102]. ONOS [103], followed in the footsteps of previous SDN controllers like Onix but was released as an open-source project that relies on the SDN community to contribute, review, and analyse its use and operation. ONOS offers a global view of applications on the network, which is logically centralised even though network state information is spread among the controllers. ONOS abstracts device characteristics such that the core operating system does not need to be aware of the particular protocol used to control a device. It offers an open framework by managing network resources and offering high-level abstractions and APIs that simplify the development of creative and beneficial network applications and services that operate across a wide range of hardware. ONOS adopts a distributed architecture for high capacity availability and scale-out. Global Network View, fault tolerance, low-latency, optimised data model, inmemory topology view, and event notifications are some of the key features of ONOS.

B. LIMITATIONS OF CURRENT WORK
Work has been carried out to consider how BGP might be updated to accommodate SDN based networking. Le et al. suggested an SDN Federation Protocol (SFP) [73] utilising a publisher-subscriber model approach to avoid excessive combinations of packet headers with routing entries. However, the authors did not mention the problems when accommodating non-adjacent network configurations, multiple similar flow queries causing flooding, and accuracy and stability. The current inter-domain SDN based network communication techniques do not address a number of issues, including multi-field prefix matching, end-to-end routing between SDN based domains, and multi-path data transmission [104]. The current research are not applicable to a variety of application domains, including IoT, Big Data, AI, and cloud services, where it is necessary to connect the distributed SDN based domains in order to provide integrated service [105]. The communication between the various controllers of the various SDN based domains was managed by a research project [106] based on a global centralised controller, which allowed the local controllers to quickly exchange control messages with relatively low costs and high data transmission performance by establishing layer-2 (L2) network connections to other controllers. Due to the numerous linkages for the data transfer between global and local controllers, the problems with this approach is that there are significant message overheads and scalability issues. In order to provide low message overheads and high federation scalability, an inter-SDN based network federation scheme [107] was conducted based on the distributed and L2 oriented network architecture using direct and simplified message exchanges between the local SDN controllers (with the global controller excluded). Through the data modelling abstraction of the information from the local SDN based network and the streamlined interactions between controllers using automated setups and specialised application interfaces, this research effort leads to a more effective inter-SDN based network architecture. The main weakness of this study is that it only tested inter-SDN based network federation on a small number of SDN based domains. However, federated SDN based networking should be further explored to examine extensibility across numerous domains.

IV. SDN BASED MULTI-DOMAIN FEDERATION
In the specific context of the SDN initiative, the U.S. Global Environment for Network Innovations (GENI) [108] defines: ''A federation is a collection of agreements between individuals or organisations that represent the policies and conditions under which they trust, collaborate, share resources or participate in other joint activities.'' The federation determines the policies and terms of resource use based on the individuals and organisations involved interests and needs. The controller hierarchy for intra-domain federations is represented by the multiple control plane communication outlined in Section II.
Multi-domain federation requires participating systems to use a signalling protocol that manages ''inter-work.'' A federation automatically calls for a ''promise'' between the participating bodies to a certain degree. One can think of each network or entity that manages a part of the network as an autonomous agent that makes ''promises'' with the other agents about the type of service they can provide [109]. Agents are autonomous, have a view of their local network and not a worldwide view. The agents can be cooperative or non-cooperative in a distributed environment. Another interesting feature of multi-domain federation is that it usually doesn't include a completely consistent distribution of state knowledge. Indeed, agents often do not know (nor need not know) the exact state or utility function of the other agents. The desired result can be accomplished by the agents autonomous activity as opposed to a central core of information that can be used to micro-manage.
The concept of ''super SDN controllers'' that micromanage smaller organisations or networks can be equated with the centralised federation mode; unified knowledge base, homogeneous structures, single administrative jurisdictions, and a ''least common denominator'' offer to some degree. The central SDN controller is constrained by the weakest link capabilities and becomes a bottleneck. It is apparent that the design of a distributed structure in which individual controllers can ''promise'' and provide services to other controllers will lead to a much more flexible system, provided it is properly designed. The use of current federation structures is a smart option for the near future. There is a possibility that a limited distributed control approach is not optimal. This may or may not be the case, however, depending on the distributed system architecture. Although a centralised controller that manages everything is conceptually easier to understand, it is not necessarily scalable or operationally functional.
The ''federating'' networks can utilise overlay applications to gain oversight and to control, manage and monitor the federated networks. Overlay and underlay in a virtualised network work together to deliver the outcome. When integrating the two together, a tight coupling or a loosely coupled arrangement can be selected where the ''underlay'' can make clear ''promises'' to the overlay regarding the types of services it can deliver, and it can also inform the ''overlay'' when it should stop providing services. Moving forward new structures for overlay and underlay interactions are anticipated. This does not imply a centralised controller approach nor the development of overlay applications that limit network service operation. It is envisaged that a network where multiple overlays exist that can use the same or multiple underlays providing various types of services and different service APIs, and where they can work together to achieve a shared end-to-end infrastructure. The way to achieve these goals is neither with super controllers nor super APIs. What is needed is a theory-based model that allows independent agents that reside in overlays and underlays to interact with each other to deliver a service.
While an AS is often referred to as a domain, a federation is a multi-domain system with physical or other relations between different domains as shown in Fig. 4. Operating and coordinating a multi-domain network can have challenges. To illustrate this, let's use the Internet as an example [110]. A network is built from a collection of AS that make independent decisions and exchange routing and reachability information by using a federation mechanism, in this case, BGP. AS provide routes to neighbouring AS, act as transit destinations and provide a ''cost'' for that route. Depending on the relative costs and peering agreements they have with other domains, AS can choose the most appropriate routes to use. AS have an implicit motivation to restrict traffic flows that do not match the AS policies and cost structure.

1) WHY FEDERATE?
The motivation for AS federation is to improve the operation of the Internet and to reduce operating costs. End-to-end delay, latency and packet loss are key factors for new and emerging near real time applications and services. The future operation of the Internet, enhancing quality of experience and reducing the cost of telecommunications are key drivers of the need for technology improvements, including AS federation. The notorious complexity of the current multi-domain routing system is difficult to operate and error-prone, resulting in multiple inefficiencies such as suboptimal inter-domain pathways and increased latency. Up to sixty per cent of all Internet paths today suffer from violations of triangular inequality [111]. The current network ossification, which obstructs the introduction of new solutions, further aggravates the problem.
• Federating networks means sharing network resources among multiple independent but collaborative networks with the aim of optimising the use of these resources, improving the quality of network-based services and reducing service provision costs.
• SDN based federated networks will provide mutual benefits to the participating administrative authorities.

2) WHAT TO FEDERATE?
In a federated SDN based network the network devices and associated edge servers and other devices can be affected by the network control policies spanning more than one domain. Data collection, aggregation and submission to core data centres involving sensors and actuators that communicate with cyber-physical devices [112] and future vehicular networks are likely to be affected by factors the support federation. In a federated network environment, it is possible to identify the key actors as logical entities: resource consumers, service providers and trust providers [113] and their interests as bodies, actions and interfaces. Actions are applied to services to meet user demands for resource reservations. Network federation can involve different implementations and scenarios, e.g., a domain may connect with one or more neighbour domains utilising one or more optic fibres that are owned and operated by the AS operator or a third party. A dark fibre connection or wavelength-based transmission service could be employed. A transit link may be employed to provide a tunnel connection between AS and in this scenario the policies of the transit provider need to be considered when the domain federation is implemented. Infrastructure resources are combined to form elements of the federated network architecture [114]. organisations that operate globally may need to: • deploy tiers of their applications across different time zones • diversify their choice of cloud providers, for a number of reasons

A. BENEFITS OF FEDERATION
Multi-domain federation provides benefits including improved control, management and monitoring, cost savings, and improved end user quality of experience.
• Cost savings, achieved by resource sharing. Reduction of investment and leased circuit costs for the federation participating networks. Because these costs contribute to the overall cost of large-scale core networks.
• Enhanced multi-domain services, by unified management of the federation and improved service resilience. Service provisioning, in particular, can become easier and faster by simplifying the user perspective support structures. For large projects which are funded by several realms within the federation, this aspect is significant.
• Improved user experience. The interface between the federated network and its consumers must be abstract from the domains involved. It means users will not be aware of the advantages of the domain they are using. It makes it much easier to use the federated network from the user's point of view. In an SDN based multi-domain federation users can benefit from improved on-demand provisioning of networking, storage and compute resources according to a pay-per-use business model [115].

B. FEDERATION USE CASES
SDN based programmable networks have been a significant part of current FI research. Researchers worldwide are interested in effective, consistent, practical, and reproducible environments that can be used to validate their proof-of-concept experiments and experiment with new algorithms, protocols, or network functions. This section provides use cases that can be used to: • translate into requirements, • provide a guide to describing the architecture, • assist in the discovery of architectural elements, and • to validate outcomes. There are six different use scenarios [116], which are divided into two major clusters: network resources used to transfer data and the entire infrastructure is used to provide network-based services (including computing and storage resources). The clusters are respectively titled data domain and infrastructure domain. It is worth noting that use cases include both the data and infrastructure domains sharing similar design, trust and security assumptions [117].

1) DATA DOMAIN USE CASES
The data domain use cases focus on the efficient use of SDN technologies to provide linkages across geographically distributed networks with the opportunity to rapidly and efficiently realise data migration. The use cases include virtual infrastructure consisting of SDN islands interconnected with (inter-continental) dynamic circuit-switched networks (multi-domain transit networks). One essential aim is to optimise the use of inter-connectivity to realise data migration using SDN and Network Service Interface (NSI) operations between networks. The focus here is about coordinating caching, processing, and network services rather than the exact caching algorithms to be used, which are within the full range of user priorities and controls. Data must be moved from the origin to its destination endpoint usually on another SDN island. The following subsections describe each use case and explain how the data flows traverse networks.
Data-on-Demand. Distributed data delivery on-demand utilising managed data flows. This use case examines how vast volumes of data collected at diverse and distributed locations are handled. For example, several applications, such as astronomical observations or collaborative research, generate enormous amounts of data typically stored in dedicated storage servers or devices in a nearby data centre. An application or user, i.e. a data processor, may want to run a post-processing algorithm on the data that various data providers collect. In this context, transferring the data from the original sites to the final location is neither necessary nor effective. An SDN based network that has a global view of the entire network might be more convenient and provide an intelligent solution as to when and how data might be moved across the network. The local controllers may create automatic links between different endpoints and guarantee end-to-end communication efficiency, with minimal delay and jitter.
Data pre-processing. The purpose of this use case is to provide near-real-time data, e.g. satellite images, to users located in different and remote locations without incurring the large Round Trip Delay (RTD) values typically found when using the public Internet. In this scenario, a dedicated platform could be strategically placed and the data pre-processed. This approach allocates resources for storage, pre-processing, caching and networking. It could also incorporate on-demand and application-driven network services for different data transfer needs which need well-defined network parameters. This approach [118] will also substantially reduce the amount of data to be transmitted through the transit network and increase the scenario efficiency.
Long distance high-quality media transmission. A rapid evolution of media content delivery is currently underway, especially in the context of ultra-high-definition of video streaming, i.e., 4K and 8K resolution. This development is directly related to demand for improved consumer experience and imposes higher bandwidth and lower network latency constraints. Hardware optimisation is required for data content transmission and reception, particularly in a very longdistance environment. At the same time, streamlining of the network is required in both the transport segments (NSIenabled) and the inter-data centre networks (SDN-enabled). In this scenario, the deficiencies of legacy network control, management and monitoring can manifest in noticeable playback artefacts: jitter, incorrect frame spacing, interruption of transmission, etc. In addition, strict specifications are required to provide the consumer with ultra-high-definition and 3D video.

2) INFRASTRUCTURE DOMAIN USE CASES
Use cases for the infrastructure domain are primarily geared towards the use of a virtual distributed system that can be used to move entire workloads for data processing. Here, resources for networking, computation and storage are considered across the allocated physical space. This approach is in line with network service chaining and the ability to migrate network services, scale-out and scale-up facilities, as well as continuous service provision [119]. The rest of this section discusses use cases for the infrastructure domain and explains how the services can be implemented in a federated environment.
Inter-cloud use case. This use case focuses on the SDN technologies used with cloud systems and network infrastructure for a mission-critical, carrier-grade data mobility service. The distributed nature of complex cloud systems requires high throughput and reliable interconnects to transport data between the data centres. Consider, for example, a consumer heading to a distant location and to reduce latency and ensure that quality of service is maintained there is a need to migrate the user's profile and data to a data centre at the new location. This is a conventional mobility management requirement [120], however, depending on the particular user the workload associated with mobility could be considerable, and depending on the time of year, the number of mobility tasks could be significant.
Follow the sun (or moon). As detailed in [121], Internet usage curves follow a similar daily pattern globally and this means that there is a natural shift in the load on data centres during the day. The opposite is true during the night when data centre workloads involve backups, data migration, and other tasks that are minimised during peak times. This is often referred to as the ''follow the sun (or moon)'' principle. In addition, renewable energy prices are highly dependent on wind and solar energy (green energy) being available. In this scenario, one might move the load from one data centre to another using two different strategies: a) effectively moving the entire workload to a more efficient data centre by re-routing user traffic, or b) managing user requests at less efficient data centres by delegating the workflow to more efficient data centres. It is important to note that both scenarios involve end-to-end dynamic and on-demand connections between the federated data centres. However, a variety of different resources (network, compute and storage) need to be configured when the workload is transferred from one data centre to another.
Disaster recovery. In this use case data recovery via Infrastructure as a Service (IaaS) migration to a remote data centre occurs. Hardware and software solutions provide isolated tenants of physical resources (computers, storage, and network) in a multi-user data centre with a mechanism to ensure that data backups and recovery are available. Under specific conditions, such as a serious disaster, middleware can be used to migrate VMs and containers to a remote data centre to ensure business continuity.

3) WIRELESS NETWORKS
Wireless networks incorporate specific features like mobility management, dynamic channel configuration, and rapid client re-association. The value of SDN based wireless networking lies specifically in its ability to provide new capabilities, such as network slicing, and the creation of new services on top of the virtualised resources in the secure and isolated networks. In addition, one of the most important complementary technologies that can benefit by being SDN enabled is the smart home gateway. Service providers aim to provide home gateways with improved traffic control, management and monitoring and with enhanced security. For example, [122] introduced a Virtual Home Gateway Control (VHC) for seamless mobility, service delivery and home energy management.

C. REQUIREMENTS FOR SDN FEDERATION
Domain federation can require an agreement between one or more AS administering organisations that will specify the terms under which the domain federation will occur, including management of the domain federation, security and what will happen in the event of equipment or system failure.
The following aspects need to be considered when design the federation agreement: • Network topology. Routing and reachability is affected by network design and topology and what type of technologies are used.
• Network coverage. What will be included in the network federation agreement.
• Network utilisation. How the federated network connections will be used and resources allocated.
• Network design and upgrades. How the federated network will evolve over time and how changes might affect the federation.
The following problems may need to be addressed in the federation agreement: 1) End-to-end connection. The network federation framework needs to support multiple administrative domains. Federated networks may need to rely on dedicated inter-domain communication services that can be managed in deterministic (e.g. VPN, MPLS, SDWAN, dedicated leased lines, or optical circuits) methods based on connection-oriented paradigms and QoS [123]. 2) Standard interfaces and protocols. Federation is facilitated when participating organzations use standard interfaces and protocols that manage the interconnection and control, management and monitoring information flow between gateways [124]. 3) SDN. With the recent proliferation of SDN based technologies and solutions, it is likely that a federated network would be enhanced by the utilisation of SDN based technologies and solutions [125]. 4) Dynamic network resource allocation. Aris et al. [102] suggests in order to enable aggregation, measurement and visualisation of applications to the distributed multi-domain federated network environment, different network assets need to be dynamically allocated according to meet the workflow requirements of deployed applications. 5) Automated and intelligent service provisioning. Federated networking scenarios create challenges related to specifications and critical features related to service delivery automation and resource management. SDN based network federation promotes enhanced multi-domain networks between enterprise and ser-vice provider data centres, helping clients access flexible bandwidth for ad-hoc applications, timely multi-domain workload migration and data processing [126]. 6) Vendor agnostic. Network and data centre operators and enterprises have moved away from vendor lock-in to vendor agnostic networking. Federated networking in this scenario utilises standardised protocols and APIs to provide interoperability and to reduce operating costs [127]. 7) Access and transit technologies. The selection of access and transit technologies can affect the operating cost and reliability of enterprise and data centre connectivity to gateways and other locations where network federation occurs [128]. 8) Resource accessibility. Client access to resources in a federated networking environment should be transparent and automated utilising APIs or web services [129]. 9) Privacy and secure data management. In a data centre scenario, VMs and containers should be logically isolated with networking implemented that, depending on identity and access credentials, provides authorised access, privacy and security. Federated networks require automatic trust-based access to resources and storage through a single protected identity management system [130]. 10) Network Function virtualisation. NFV and logical network separation are utilised in enterprise and data centres to facilitate hosting systems and applications that have different access and resource demands [131]. Workload mobility, backups and load sharing federated network resources benefit when NFV is implemented. 11) Policy (provisioning) framework. Current research includes routing and reachability and to a lesser degree end-to-end policy management. This is because AS are controlled, managed and monitored by separate organisations that may or may not share the information needed to gain a global view of the network. It is a crucial challenge to orchestrate policy-based schemes into a federated network layer [132]. Federated networks require a new revolutionary architecture for multi-domain SDN with dynamic parameters. 12) Orchestration of configuration tasks. Existing Configuration & Management (C&M) techniques are based on languages with low-level procedural programming (general purpose or scripting). A controller can accept configuration demands or requests from multiple orchestrators in a federated environment. It is important to design end-to-end and automated (C&M) tasks in order to cope with the various resource requirements at different stages of the networking life-cycle [133]. 13) Management abstraction. A network federation should comprise an abstraction layer that enables seamless access and management of external resources through various operating systems and virtualisation mechanisms [134]. 14) Scalability, Flexibility, Agility. Federated networking provides scalability, versatility, efficiency and cost reductions. To achieve scalability, flexibility and agility an SDN based federated network would maintain a global network view and optimise resource allocation [135]. 15) Guaranteed Network Performance. Generally, variance in network load and performance, are symptoms of the ''best effort'' Internet. It is reasonable to assume that Quality of Service (QoS) specifications (e.g. minimum bandwidth) must be guaranteed for federated network interconnections in order to be able to control and manage traffic flows [136].

V. SDN BASED FEDERATED NETWORK BUILDING BLOCKS
The programmable control, management and monitoring of the infrastructure layer is a fundamental reason why SDN based federated networks are being deployed. A programmable forwarding plane enhances information dissemination, automatically path optimisation based on current traffic patterns, versatility, policies and management that are implemented using overlay applications. This section describes the three basic building blocks for SDN based federated networks: network, operations and services [137] as illustrated by Fig. 5.
• Network: The SDN enabled infrastructure layer is an important building block that provides the foundation upon which the programmability and intelligent applications can enhance the control, management and monitoring of the network and the traffic flows [138]. Management and allocation of the resources available in a network and between AS is a critical factor when the network operation is optimised to provide maximum performance. The increased complexity of the network and traffic flows necessitates improved near real-time management.
• Operations: Each federation member can manage operations and service delivery autonomously in a loosely coupled federation. Network operations for the networks of the federation partners, however, become directly interdependent in a tightly coupled federation [139]. Therefore, it is important to have an established set of defined procedures and workflows set out in an SLA, e.g., for fault handling and management. The federation member Network Operations Centres (NOC) implement and manage the processes and workflows. Operating requirements should be documented, understood by all stakeholders, and presented using an operating model that describes the relationship between the partners, the data sources used, and how traffic flows will be optimised. External SLAs can only be respected if partners implement internal procedures and practices to ensure that infrastructure and other resources are available and optimised. In a federated network scenario, relationships between partners can be both user-network and suppliernetwork relationships. A governance structure for the federated network operations should to be agreed upon, and a dispute resolution mechanisms established. The governance structure should be both efficient and fair; it could, for example, assign voting rights and agree on cost-sharing principles.
• Services: For the advantage of their end-users and transit networking partners, the federated network operators pool assets to develop, provide, and manage internetwork services. Services can be assembled utilising the federated network operators infrastructure, however, there could be a requirement to utilise third party infrastructure and facilities, e.g., DCs and transit links. Network services play a dual role in a federated approach to networking: they are both end-user services, (e.g. an end-to-end Synchronous Optical Network -SONET) light path connecting an instrument to a computational site) and building blocks offered to the federated network by federation partners (e.g. a SONET connection between two federation partner point of presence). Choosing how to deliver services could result in quite different network architectures for the federation members. Service provision is no different in a core network, run as a single entity, from that of a typically built carrier or enterprise network. With this approach, on the level of service delivery, the federated network architecture is not directly visible [140]. On the other hand, with the aggregated infrastructure approach, service delivery itself becomes federated, offering services made up of network resources and systems. Therefore, this option is important for network design.

A. SDN BASED FEDERATION FRAMEWORK
Recognising that the SDN based federation is focused on agreed policies and practices, the underlying infrastructure should be available and optimised. An SDN based federation framework should include the capability to control, manage and monitor the resources available for the inter-network connections. The SDN controllers should be linked using east-west interfaces that provide standardised APIs. The application layer could be implemented utilising applications that maintain visibility of the federation member gateways, traffic flows and resources. Unfortunately, existing SDN frameworks do not embrace multi-domain environments. Recently new research has commenced into SDN based multi-domain solutions. In Table 3 the provisioning (connection-oriented) and federated (sliceoriented) connection approaches [141] are compared.
• The connection-oriented SDN based multi-domain framework: focuses on creating circuits (connections) utilising overlay applications to manage the traffic flows between the federation member networks. Circuits can be set up in several ways -Layer 1 connections (portport), Layer 2 streams, Layer 3 IP flows and Layer 4 TCP or UDP port connections. Management of legacy network connectivity add complexity to the inter-domain connectivity, but is it possible to manage the connectivity using an SDN based solution.
• The slice-oriented SDN based multi-domain framework: focuses on implementing distributed slices of networking resources as shown in Fig. 6. This approach utilises one logical domain to facilitate resource management. The systems that manage a slice-oriented federation include a link management service, a slicing management service (creating and handling slices) and a data centre management service to handle the storage and processing resources allocated to each slice. This strategy is based on an orchestrator that serves as a management platform and manages resources while also delegating the execution of operations to the domain infrastructure.

B. SDN BASED FEDERATION ARCHITECTURE
An SDN based federation architecture is a focus of current research. The SDN based federation architecture will be based on the policies and practices needed to control, manage and monitor the infrastructure used for the inter-network connections [142]. Following are the basic elements in the SDN based federation architecture: • Bodies: Three types of logical bodies, resource users, resource suppliers, and trust providers, are typically involved with the federated network environment as outlined in Fig. 7.
• Actions: Actions are applied to resources to fulfil reservation requests based on resource type and policy. For example, network resources may expose isolated or shared access (e.g. optical cable, layer 2, or layer 3), normal or personalised packet switching, and different levels of service quality. Computing resources may expose physical systems, VMs or containers. Other resources may be utilised including instruments, sensors, and software services,e.g., firewalls, load-balancers, and encryption services.
• Interfaces: In a federated network environment the interfaces need to be standardised and secured. There is a requirement that the network operators implement monitoring and auditing to ensure the access to the interfaces is controlled. As there is a degree of trust VOLUME 11, 2023  granted to the federation network members, it is important that coordinated access and authentication processes be implemented. The development of an architecture that supports the underlying federation principle can be quantified by considering factors including: • What is the right level of abstraction, the minimum collection of functionalities to be followed in order to share resources held by various authorities?
• What is the best-supporting governance model for subsidiarity? suppliers and the Facility itself that controls all of them. Akhshabi and Dovrolis [143] took advantage of the internet architecture experience to create a model. It was based on two principles.
• The Internet ''Hourglass'' model defining the IP protocol as the convergence layer. For the Federation of Test bed Resources, one such convergence layer is required.
• The peering Internet model that depends on Customer Sand Providers and describes peering agreements in such a way as not to have a single control point. This helps to identify Experimenters, Test bed owners or suppliers and the Facility itself that controls all of them. The following abstractions were defined: • Resource: Management of the resources made available for the federated networking • User: User identity guaranteed • Slice: A distributed container in which resources are exchanged (sharing with VMs, in time, frequency, within flow space). It is also the basis for cost and resource accounting. Slice Facility Architecture (SFA) was developed as an international effort to provide the minimum functionality necessary for a global federation trial to provide a stable shared API. As the understanding of the specifications progressed, the basic components for the federation trial were designed incrementally. In addition, a major effort has begun to supplement and fill the architecture with components that are necessary for the whole life cycle of the SDN federation experiment. This includes the following steps [144]:

1) User account & slice development 2) Authentication 3) Resource exploration 4) Resource reservation & scheduling 5) Configuration & instrumentation 6) Execution 7) Results repatriation 8) Resource release
Step 1) is managed by the user's home authority, where the user has registered. Steps 2) to 4) and 8) are applicable to all of the authorities involved. Steps 5) to 8) are not in the SFA, but for this reason, other components such as Operating Management Framework (OMF) were created. OMF is a system for tracking, assessing and controlling.
SFA offers a secure API that allows authenticated and registered users to access the available resources and to delegate the resources that are required to conduct a particular experiment in compliance with the federation policies. SFA is used to federate the heterogeneous tools to be federated that belong to different administrative authorities. This will allow registered experimenters with these authorities to combine the available resources and perform advanced networking experiments involving wired and wireless technologies. The SFA layer includes the SFA Register, SFA Aggregate Managers (AMs) and drivers. It is the role of the SFA Registry to store the appropriate credentials for users and their slices.

C. UNIFIED DATA PLANE CONNECTIVITY
A homogenised connection method across the federated SDN based domains can be provided by the presence of unified architectures, such as SFA. However, this does not address the data plane stitching issue. A problem that federated design does not answer is how one virtual entity in one domain can interact with another virtual entity in a different domain at the data-link (Ethernet) layer. Techniques that enable the construction of virtual networks spanning resources are required for virtual resource utilisation in a networked environment. Typical network services found in a network environment, such as L2/L3 services, e.g. DNS and DHCP, should be provided by virtual networks with additional features, such as elasticity (up-scaling/down-scaling) and migration (rearrangement of resources and reconstruction of arbitrary network topologies over different physical infrastructure). With the presence of heterogeneous network resources that must be managed and configured within the federated environment, the issue of providing virtual networks becomes difficult. Argyropoulos et al. [145] presented a data plane stitching mechanism for federated heterogeneous virtual infrastructure. This technique operates in multi-tenant systems that require isolation, concurrency, programmability, and elasticity and is network virtualisation layer independent. As far as scalability is concerned, the heterogeneity of this mechanism does not impose restrictions on the overall num- ber of concurrent network slices, nor does it result in any appreciable performance degradation in terms of end-to-end delay and bandwidth between virtual entities as compared to physical (substrate) entities. Additionally, the use of widely used network protocols (IEEE 802.1Q VLANs and GRE tunnels) ensures interoperability with a wide range of FI infrastructure.
In addition, the majority of the Internet today is heterogeneous, vendor-specific, vertically oriented, functionally fragmented, and locked behind closed identity and access management systems [146]. As shown in Fig. 8, it is not possible to create composite applications using services from different domains due to the interoperability gaps between the domains, entity interfaces, communication protocols, data formats, data models, and identity providers [147]. A recent effort in this area is Data Spine [148], a federated platform enabler that links and creates interoperability between domains. It offers the capability to federate the identity providers of various platforms, enables Single Sign On (SSO) functionality at the ecosystem level, supports interoperability between the services of various domains, and creates the technical foundation for quick, simple, and code-light development of cross-platform applications. From the perspectives of both service providers and service users, the methods for integrating synchronous (request-response) and asynchronous (publisher/subscriber) type services are also discussed in the Data Spine proposal.

VI. TEST BEDS
In research laboratories around the world, novel networking technologies are being created and demonstrated. The test beds developed and implemented by the researchers utilise software and hardware components and include monitoring and management systems. Research to address the requirements of the FI includes a new architecture that is based on a ''clean slate.'' Federation extends infrastructure reach, both geographically and administratively. SDN federation has occurred across several dimensions for use by representatives of various research organisations. Although specifications and priorities vary, they all have to solve a similar set of problems. Such a federation suggests the fundamental needs of applications to interact across a generic, federated network. This section discusses a variety of current and planned SDN federation research efforts. A proposal for the federation of the test beds indicates that the research teams would be able to carry out distributed and collaborative experiments with FI protocols and applications [149]. Table 4 provides a summary of the recent work on SDN based federation.

A. GENI
Global Ecosystem for Network Innovations (GENI) [150] offers a virtual laboratory for research and education into networking and distributed systems. GENI allows researchers to obtain computing resources from locations across the United States, connect computing resources using the 100 Gbps Internet2 Layer 2 service, install custom software or customised operating systems on the computing resources, control how network switches handle traffic flows in their experiments, and run applications at Layer 3 and above. GENI's architecture incorporates cloud federation. GENI administrators can determine the resources the community has at their disposal and thus a portion of GENI resources are reserved for internal use. The GENI architecture enables researchers to combine computing and storage resources. For full control of networking traffic, researchers may run their SDN controllers on a Layer 2 network domain. This capability is useful for the transfer of large volumes of scientific data.

B. OF@TEIN
At the Asia-Europe Summit (ASEM 3) in Seoul in 2000, the Trans-Eurasia Knowledge Network (TEIN) initiative was commenced. It was one of ASEM's most successful initiatives to connect ICT infrastructures between Asia and Europe. OpenFlow at TEIN (OF@TEIN) is a regional OpenFlow test bed project that began in 2012 with the cooperation of TEIN NRENs in South East Asia (SEA). It aims to promote SDN infrastructure via the TEIN network. The project [151] focuses on the following objectives to create a virtualised OpenFlow-enabled SDN test bed over international sites: • SmartX rack design and checking • OF@TEIN network deployment and test platform • Development and deployment of SDN software at OF@TEIN A unique SmartX rack was first developed for this project, with SDN based switching and remote management functions. The racks were installed at seven OF@TEIN sites (Vietnam, the Philippines, Thailand, Indonesia, Malaysia and two sites in Korea) and connected to allow for SDN experiments. Several SDN tools have been planned and built to support OF@TEIN test bed experiments and management. The project is currently looking forward to extending the OF@TEIN network by combining it with the organised creation of video-based medical cooperation and the global IPTV service.

C. FELIX
Federated test beds for Large-scale Infrastructure Experiments (FELIX) establish an integrated SDN test bed between Europe and Japan. It implements new APIs and logic for heterogeneous SDN and IT islands spread globally, enabling them to exchange information on resources, share overall resource pools, and provide dynamic network inter-connectivity between and within the islands [152]. Using the SDN control and NSI features to build SDN based federation services, FELIX offers new network management standards.

D. GÉANT
The GÉANT project's goal is to provide a pan-European network and related infrastructure for next generation networking research that will meet the connectivity needs of the affiliated research communities. The research needs include both a data-producing transport facility and a networking environment in which experiments can be carried out. At GÉANT, using SDN, the new ''Test beds as a Service'' [153] includes a dynamic packet network test bed service designed to tackle research on the upper network layers and dynamically construct a network consisting of a variety of network resources: routing, switching, computing, circuit and link, storage, and advanced experimental hardware or software.

E. OFELIA
Open Flow in Europe-Linking Infrastructure and Applications (OFELIA) is an experimental network designed to provide diverse OpenFlow-enabled infrastructure that allows experimentation with the SDN paradigm. OFELIA currently consists of ten sub-test beds (the so-called islands), most of which are in Europe and one in Brazil. In [154] a subset of the test bed resources, known as a slice, is made available including the programmable OpenFlow switches. Recently a new network virtualisation method named VeRTIGO was introduced to expand the way in which isolation is accomplished between slices, enabling arbitrary virtual network topologies on top of a physical test bed.

F. Fed4FIRE+
The Federation for Future Internet Research and Exploration (FIRE), Fed4FIRE (2012-2016), merged into Fed4FIRE+ (2017-2021), is an EU project aimed at developing a free, usable and secure platform for the cross-border federation of Internet research infrastructures. Here, sample community domains include optical and wireless networking, SDN, cloud and grid computing and smart cities. To ensure reliability and support of heterogeneous networks and experimentation requirements, heterogeneous communities, including the next-generation optical networking community, are involved in the project. The approach taken [155] includes: • free tools for life-cycle control of experiments • experimental measurements and monitoring • trust and security processes • inter-test bed enhanced communication capabilities

A. OPPORTUNITIES
A list of SDN functionality that could be leveraged in federated environments is provided in [156]. As shown in Fig. 9, the preliminary efforts offer a range of advantages as long as criteria are met. The federation of an SDN framework provides: • rich and scalable traffic management capabilities • a federated Internet that has improved operational efficiency, programmability and automated technology deployment capability • new business models for operators and users, such as meeting the requirements of end-to-end QoS aware services The list can be divided into two groups: firstly, the ability to dynamically build and change network services (automation), and secondly, improved interfaces that provide visibility (control). This section discusses the SDN based federation designs.

1) AUTOMATION, COMPATIBILITY AND INTEGRATION
Through automation, on-demand networks will respond to changes in services depending on the resources and specifications in place at the time. SDN based federation will also have greater interoperability and programmability [157]. A standard programming platform that can be used by interested parties simplifies orchestration and maintenance for operators, companies, independent developers, and users.

2) CONTROL, DISTINCTION, VISIBILITY, AGILITY
Since SDN has a standardised interface between the policy-controller and the lower-level flow devices, it provides the ability to enable, change, delete and transfer the domain resources. Policy-based controls monitor the bandwidth and service levels automatically, enabling consumers to configure specific facilities and access rights based on rules, regional areas and role-based access. SDN federation provides the ability to deliver unanticipated SDN services enabling enhanced network visibility with enhanced network situational awareness clarity and currency [158]. This in turn provides better quality of information about network services to make rapid and efficient decisions.

B. CHALLENGES
SDN based federation imposes demanding requirements and critical features in terms of service delivery automation, resource management or the development of on-demand network connectivity. Challenges and possible future directions are discussed in this section as outlined in Fig. 10.

1) SCALABILITY
A large number of network view requests can result in the control plane becoming overloaded. Routing entries from several domains can impose a strain on both the memory and the lookup capacity of the flow device forwarding engines. A challenge is to build a large multi-domain SDN network that avoids or solves the scalability problems [159].

2) FLEXIBILITY
SDN has the ability to enable the implementation and operation of more effective network applications and services and manage virtualised infrastructure. However, more recently SDN based networking research is tackling the flexibility needed for large network operation. The traditional centralised control mechanisms are difficult to utilise in federated networks and are not suited to the large-scale and increasing service requirements [160]. The SDN based networking control plane architecture is critical to integrated system flexibility. The modularity and flexibility of controller composition are still under investigation.

3) MIGRATION AND SUSTAINABLE DEPLOYMENT
The SDN based and conventional IP networks coexist using gateways. As SDN based networking expands into the remaining legacy networking environments there is a need to ensure that operators update existing systems and provide support for legacy equipment that may remain [161]. Challenges remain with the use of hybrid flow devices in SDN-enabled networks to support legacy systems [162]. Future research is needed to formulate strategies and interaction mechanisms that optimise the SDN based networking benefits while reducing the added complexity of paradigm coexistence.

4) SECURITY
Security is a key issue today and the deployment of new technologies that enhance security in a federated networking environment is important. In the heterogeneous networks, there is a critical need to provide enhanced privacy and protection. Compared to traditional networks, the highly programmable SDN based networking the potential of security threats to disrupt the network is far more serious [163]. Security research is ongoing and the multi-domain SDN security challenge remains.

5) RESILIENCY
In a multi-vendor, multi-tier service-based network the provision of enhanced resiliency is a challenge. Issues with a local device or service can create errors that can impact the entire network. There is a need for increased resiliency knowledge and management capability [164]. One of the key concerns when providing a virtualised network is what happens when a controller fails. Management of this situation typically revolves around a stable fallback configuration, enhanced redundancy and resiliency.

6) RELIABILITY
SDN controllers are used to control flow devices, to verify network topologies and to manage message flows in the event of equipment or system failure. In an SDN based network the controller is a critical element in the strategy to improve network reliability [165].

7) ROBUSTNESS
A functioning SDN based network should continue to work when the network degrades due to device or system failure or when traffic increases [166]. An important challenge is to design and implement systems that manage network degradation to provide robustness.

8) ORCHESTRATION
A challenge for SDN based networking is the provision of orchestration that facilitates solutions, applications and services from multiple vendors [167]. Orchestration has become complicated, with more and more features per end device. There is a growing need for SDN based service orchestration. An open controller northbound API that is standardised among vendors is required that can be used for consistent network management instructions that are deployed over the multi-vendor infrastructure.

9) COST MODEL
Resource sharing should result in cost-sharing for federated services and this a challenging problem [168]. Cost-sharing problems become more complex for on-demand services such as on-demand bandwidth. Costs are incurred for the on-demand services even though customer may not be fully utilising the resource and there is a financial risk of not being paid when offering on-demand services.

10) TECHNOLOGY DIFFERENCES AND A LACK OF STANDARDS
SDN controller and network management solution vendors are not focused on inter-operability in a multi-vendors infrastructure environment. The problem of technical discrepancies on the lower layers has to be resolved when networks are federated. Some domains are Synchronous Digital Hierarchy (SDH) based, others are MPLS based, or Ethernet. When federation happens there is a need to ensure that network and system interoperability is fully achieved [169].

11) PERFORMANCE EVALUATION
SDN based networking introduces performance evaluation challenges, particularly in a federated network environment. SDN performance questions have not been properly investigated [170]. Except for some commonly used time-consuming simulation and costly performance assessment experimental techniques, analytical modelling may permit the definition of a networking architecture that paves the way for network designers to establish performance outcomes. Despite evaluating the network architecture, there are other design aspects to be evaluated as well. Analytical models can include performance metrics related to routing, resource allocation algorithms and networking schemes. They can be used to capture network performance data, such as packet delivery rate, packet delay, jitter, buffer length, network throughput, and network blocking probability.

VIII. CONCLUSION
The introduction of SDN has succeeded in paving the way for next-generation networking. Consequently, multidomain SDN based networking interconnection should become widespread. This paper discusses the benefits of the multi-domain SDN based network federation architecture and offers an understanding of its functionality. Due to ongoing research on the SDN model, some key problems influencing the architecture's effectiveness and full utilisation have been identified, focusing on scalability, flexibility, performance, security, and interoperability. The stakeholders involved in the SDN based network federation value chain face a range of barriers that need to be lowered to ensure that adoption occurs. A software-defined revolution would result from the optimisation of network administration made possible by the SDN based network federation paradigm, which is still being tested and developed across various platforms. It is anticipated that once the issues are dealt with, the SDN based network federation will ''sky-rocket'' to the pinnacle of technological development.