The UK Programmable Fixed and Mobile Internet Infrastructure: Overview, Capabilities and Use Cases Deployment

Leading state-of-the-art research facilities at the Universities of Edinburgh (UoE), Bristol (UoB), Lancaster (UoLan), King’s College London (KCL) and Digital Catapult (DCAT) are interconnected through a dedicated JISC/JANET network infrastructure. Using Software Defined Networking (SDN) and Network Function Virtualisation (NFV) technologies, these distributed test-beds are integrated using a multi-domain NFV Orchestrator. This paper introduces a novel specialist distributed test-bed developed for facilitating the increasingly large and complex experimentation of future Internet system architectures, technologies, services and applications between the geographically dispersed laboratories across the UK. The aim is to enable students, researchers and enterprises to interconnect with and carry out remote experiments using these test-beds. Each one contributes a range of key capabilities for Internet research including optical networks, optical wireless and radio frequency communications, Internet of Things (IoT), SDN, NFV, as well as cloud computing technologies and services.


I. INTRODUCTION
The emergence of fifth-generation (5G) communication systems, cloud services and Internet of Things (IoT) has led to an unprecedented increase in connection capacity requirements and an exponential growth of devices that require reliable dynamic Internet connectivity. The current Internet network infrastructure is somewhat rigid, due to high-level protocols for configuration and adaptation of network policies that The associate editor coordinating the review of this manuscript and approving it for publication was Abderrezak Rachedi. are implemented manually through a command line interface (CLI) [1]. There is also a large amount of installed, purpose-built hardware. This imposes strong constraints on the performance of network applications and on dynamic services provisioning and adaptability. In addition, research on future Internet technology, systems and services requires findings from relevant and representative end-to-end (e2e) experiments.
Experimental research test-bed facilities are being developed in various information and communications technology (ICT) domains across the UK in laboratories specialized in optical [2] and radio frequency (RF) wireless communications [3], [4], signal processing, high-performance computing, quantum secured communications [5], distributed networks and systems [4], [6], the Tactile Internet [7], and also artificial intelligence and machine learning [8]. The limitations of carrying out research in these separate facilities make the current research output very limited in addressing realistic e2e Internet systems. Furthermore, research in telecommunications and networked systems remains largely segregated in independent optical, wireless or computer network research labs, so researchers rarely have the opportunity to experiment across the boundaries between these domains.
The adoption of Software Defined Networking (SDN) and Network Function Virtualisation (NFV) technologies can eliminate the rigidity present in traditional networks and bring in the much needed flexibility in the network deployment and management [9]. Standards have been proposed to develop Management and Network Orchestration (MANO) systems [10] that can support dynamic network service deployment by using SDN and NFV technologies. A MANO-based architecture has been extended to support multi-domain and distributed ICT test-beds while meeting the requirements of e2e network connectivity [11]- [14]. It supports large scale test-beds that are required to validate new Internet network architectural designs.
The National Dark Fibre Facility (NDFF) [15] enables researchers to develop the underpinning communications technologies to remotely configure the future Internet network using SDN. It provides the Aurora network as a platform for network research. The USA's main test-bed GENI (Global Environment for Network Innovations) architecture supports programmable and federated e2e virtualized slices [16]. However, the users of GENI have limited control over the selection and configuration of node resources and lower layers on which experiments run, though the nodes can be programmed on-the-fly using SDN. In contrast, the EU's main test-bed Panlab (Pan-European Laboratory) supports a network domain federation (NDF) concept which enables different test-beds from multiple domains to run together [17]. Users have full control over the selection and configuration of components in their virtual customer test-bed. 5G EVE (5G European Validation platform for Extensive trials) [18] supports validation trials on the state-of-the art 5G technologies: Multi-user multiple-input and multiple-output (MIMO), Massive MIMO, Radio Access Network (RAN) virtualization, latency reduction, intelligent connectivity [17]. The 5th Generation (5G) e2e Network, Experimentation, System Integration, and Showcasing project (5GENESIS) [19], [20] validates 5G key performance indicators (KPIs) for various 5G use cases, in both controlled set-ups and large-scale events. The platforms leverage the evolution of existing test-beds, owned and operated by the 5GENESIS partners, suitable for large-scale field experimentation. 5G-VINNI (virtual network infrastructure) [21] adopts the 'network slice as a service' (NSaaS) delivery model, whereby 5G-VINNI provisions tailored network slices to vertical applications upon request. The assigned slices meet the requirements of application for trialling activities, setting up various use cases and assessing their KPIs under different network conditions [22]. SDN exchanges (SDNxs) enable traditional networks and exchange facilities to support more services and capabilities for network technology integration. Leading research facilities at Bristol, Edinburgh, Lancaster, KCL, and Digital Catapult (London) are interconnected through a dedicated 10 Gbps JISC (Joint Information Systems Committee)/JANET network infrastructure. A SDNx interconnects these research test-bed facilities via OpenFlow (OF)-enabled switches [23], as shown in Fig. 1, which can support SDN and network slicing [24]. There are multiple challenges to developing such a large experimental test-bed facility. First, local optical and 5G wireless test-beds are required to support advanced SDN and NFV technologies that are not mature enough to cope with the dynamics of their data plane to support access for concurrent user experiments. Second, we can identify the major technical challenges for enabling a software-defined data plane of the SDNx-enabled inter-connected laboratories networks, namely: (i) simple OF-enabled switches programming, (ii) predictable services provision performance, and (iii) isolation among multiple concurrent slices and network services. Two other crucial challenges for the experimental facility are broad and long-term user engagement and facility sustainability.
The  [15], [18], the INITIATE test-network does not focus on a specific technology; instead it allows multiple technologies to inter-operate in a seamless manner. All the test sites are connected through the INITIATE network, which provides a plethora of choices to users for experimentation and use cases deployment. Similar to other efforts [18], [20], [21], the INITIATE project uses NFV and SDN technologies to program the deployment of multi-domain and e2e network services across different test-beds.
A user portal has been developed on top of the SDNx to provide remote access to the INITIATE facility and the necessary information required for users to compose customised test-beds that support their research. This supports experimentally driven research that addresses the integration of multi-domain and multi-technology 5G and IoT access platforms with high-speed optical transport networks. End-users are integrated as part of the experimental process to support user-driven scenarios such as mobile edge computing, data visualization and autonomous mobility. Comprehensive experimental control and user access tools will be deployed across the INITIATE platform. These tools mainly rely on SDN/NFV principles, infrastructure virtualization and slicing to offer an open and programmable experimental infrastructure, and to support multi-tenancy for experimenters. This includes the convergence of heterogeneous separate wireless networks: wireless fidelity (WiFi), light fidelity (LiFi), long-term evolution (LTE)/4G, and 5G.
The rest of this paper is structured as follows. Section II discusses the important features and capabilities of ICT test-beds running on the different islands interconnected through the INITIATE test-network. Section III describes the 5GUKex architecture, the multi-domain orchestrator of INITIATE network, and the data plane broker. Section IV discusses the use cases deployed on the INITIATE infrastructure. Finally, Section V concludes this paper and discusses ideas for further research and development.

A. UNIVERSITY OF BRISTOL TEST-BED
The Smart Internet Lab 2 test-bed at the University of Bristol (UoB) runs across a multi-site network connected through a number of dark fibre links with several active switching nodes. The network has a number of primary sites in Bristol: Smart Internet Lab (home of High Performance Networks (HPN) Research Group), We-The-Curious (focusing on curiosity for Science) next to Millennium Square (a large public open space) and M-Shed Museum (focusing on Places, People and Life stories in Bristol). The network core lies in the HPN Lab with edge computing nodes spread across the other sites a few kilometers from Bristol City centre. A central node at the HPN Lab hosts the cloud computing infrastructure and a mix of optical and L2 switching devices as well as cloud radio network base band processing units. Furthermore, the test-bed infrastructure offers a varying set of GPU servers, capable of supporting applications with high compute-intensive requirements for either augmented reality(AR) / virtual reality (VR) or machine-learning use cases. The access technologies for this test network are located at various outdoor and indoor locations in Bristol, as shown in Fig. 2.   [26] infrastructure) terminates at the HPN Lab. It also connects to other test networks across the U.K. and Europe, including NDFF [15] and GEANT [27]. The Smart Internet Lab test-bed (HPN) has been part of multiple 5G trials [28] as well as part of the INITIATE infrastructure, which offers many computing, connectivity and access network features in bringing multiple specialized test-beds together for research and experimentation purposes.

1) RADIO AND ACCESS SOLUTIONS
This test-bed uses licensed and unlicensed bands for connectivity towards user devices as endpoints for communication services, or for device-to-device connectivity. For transport Xhaul solutions, the test network is equipped with dual connectivity on most outdoor access points where the switches at the edge of the network can redirect the access points' connectivity either via a fibre network or the wireless transport network operating in licensed or unlicensed band depending on its deployment.
The access technologies in licensed bands operate in a similar way to above, i.e., they are used either for access to the end user device or for device-to-device connectivity as part of the transport network. The technologies include LTE-A and 5GNR, which provide access to the UE devices, as shown in Fig. 2. The Xhaul technology solution uses proprietary self-organising radios in creating a mesh transport network as an alternative to the fibre network, providing access to the radio cells or fixed network via the L2 switches. The radio spectrum bands for all these licensed and unlicensed bands are also shown in Fig. 3 The test network is equipped by two different vendors for the WiFi access point and its control. Furthermore, the 60 GHz unlicensed band has been used as a wireless transport network alternative to the fibre network, using phased array antennas, creating many-to-many node connectivity as a mesh transport network. This solution has been deployed for the demonstration of use case in the Roman Baths as well as in the Millennium Square for the smart city use-cases. Among the unlicensed access technologies, the test network also provides three indoor LiFi access points for technology demonstration at We-The-Curious as well as a couple of LiFi access points at the HPN for experimentation.

2) OPTICAL FIBRE NETWORK
The heart of the UoB test network lies in the dark fibre connections across the city, providing a high performance and dynamic test-bed for the experimenters. The fibre layout of the UoB infrastructure spans across Bristol city centre, as shown in Fig. 2(a). The fibre links can be configured as a star connection with its centre at the Smart Internet Lab, or a mesh network connecting all the main nodes of the infrastructure. The dark fibre network is used for the fronthaul between the baseband units and the remote head radios for access points. It is also used as 10 Gbps L2 LAN connections between the sites, and as 1 Gbps links within the sites. The Smart internet Lab is also an endpoint of the INITIATE JANET/JISC link that terminates at the VIRTUS data centre hosting the 5GExchange. The link between the HPN and the satellite networks at the Roman Baths was leased as 10 Gbps from BT Openreach as a satellite test network. This fibre network is interconnected using a 10 Gb fibre link connection to the Slough data centre and other locations where projects demand connectivity as part of their use cases demonstration and experimentation.

3) COMPUTE AND VIRTUALIZATION SOLUTIONS
There are two types of network virtualization solution implemented at the UoB testbed: open-source MANO (OSM) [29], and Nokia CloudBand and their Core network EPC along with NSA as a proprietary solution. Similarly for the compute capabilities, there are two solutions deployed at the UoB cloud and edge datacenters: Nokia MEC as proprietary and Openstack as the open-source solution. There are a number of solutions operating in the HPN cloud network to manage the network infrastructure, which include NETOS as SDN controller, a suite of solutions from Nokia: NETACT, Flex-iZone controller, Cloudband, IP service router, vMEC, EPC core and EPC-NSA 5G core. The tenant research projects also bring their own platforms that are hosted on the HPN cloud network. For each project, this cloud network provides a determined VPN access to the project nodes only. The test-bed compute resources are further distributed in such that each node away from the HPN lab presents a fibre patch panel, a small switch connected to a small router and an edge computing resource as a MEC solution, and also if required a PoE+ to an access point. As a result of this, the test network provides a highly flexible cloud network slices for experimenters.

B. UNIVERSITY OF EDINBURGH TEST-BED
At the UoE, the LiFi R&D centre has a number of running test-beds that support various research activities to develop LiFi optical wireless communication systems, LiFi attocellular networks and integration with next-generation RF wireless technologies, focusing currently on WiFi networks. A test-bed platform for a software-defined LiFi/WiFi hybrid network has been developed to facilitate experimental validation of the small cells networking algorithms on handover, tracking, localization, traffic offloading, network load balancing, security and energy efficiency, as shown in Fig. 4. The UoE test-bed platform comprises of LiFi attocells and a WiFi AP, which are interconnected through an OF-enabled switch to a centralized SDN OpenDayLight (ODL) controller. This manages the data plane of the SDN-enabled LiFi/WiFi hybrid network through the southbound interface, while supporting applications via the northbound REST (Representational State Transfer) application program interface (API). The SDN controller has a number of software agents running on the APs. They pre-process the associated user generated traffic information that forms the state of their corresponding APs, and then send it periodically to the controller, as shown in Fig. 5. The SDN applications regularly receive the network state which they process to support intelligent services in the data plane. The software agents inherit from each other the information of some static parameters, with values that do not change its state over time, such as the type of AP (e.g. WiFi or LiFi), medium access control (MAC) and Internet protocol (IP) addresses. A group of software agents only process and then relay the global state of APs to the controller, while another group processes the instructions received from the controller to execute commands regarding the users' association with the APs and users' handover or traffic offloading, as shown in Fig. 5. A LiFi access and traffic engineering (LATE) application has been developed to support the LiFi/WiFi hybrid network monitoring and management, user mobility and load balancing services. Applications require different information, which can be extracted from various network layer parameters. These are processed by different software agents, as shown in Fig. 6. The LATE application supports three main services which have graphical user interfaces to help managers and administrators oversee the network state and measure the impact of network or service policies at the AP or network (SDN controller) level, as shown in Fig. 6. The software agent groups can be extended to further support SDN applications and deliver more services in the forwarding plane.

C. KING'S COLLEGE LONDON TEST-BED
As one of the INITIATE network islands, the test-bed of KCL is connected to the 5GUK exchange (5GUKEx) data centre in Slough via 10 Gbps fibre, as shown in Fig. 1. The local test-bed comprises (i) a virtualized and cloud system distributed in three physical location areas within the University's Strand campus, (ii) the 5G Tactile Internet Lab, where researchers have access to connected devices and test potential use cases and applications, (iii) the data centre, where the main physical servers and the virtual baseband units of 5G base stations are located, (iv) the roof, where the RGE (relative gain enhancement) antennas are located, covering areas of Sommerset House and KCL's Strand campus courtyard. On the air interfaces, a diverse set of technologies and frequencies are supported, such as 5G, 4G and WiFi.
The KCL test-bed is fully run by software, offering a NFV orchestration of a wide range of VNFs of different compute requirements using OSM. All compute nodes are configured similarly with 128 GB of memory and two Intel E5-2699A CPUs of 44 cores each (88 virtual processors) clocked at 2.4 GHz. Furthermore, compute nodes have been configured to dedicate 4 cores (8 threads) to the host OS for hypervisor functions while 40 cores (80 threads) are reserved for virtual machines. The host OS reserves 8 GB of memory for its own processes. Network connectivity is provided by 10 Gb Ethernet for all performance-related functions of OpenStack, and by gigabit Ethernet interfaces for out-of-band management and SSH access to nodes.
Additionally, the test-bed provides Cloud-RAN capabilities for different functional splits of the RAN protocol stack with a packet fronthaul. This setup has been tested with both Ethernet and Optical fronthaul [30], while additional capabilities to support ultra-reliable, latency constrained communication are deployed and tested [31].

D. LANCASTER UNIVERSITY TEST-BED
The Lancaster University INITIATE island connects the University test-beds to the INITIATE network through a 10 Gbps link, as shown in Fig. 8 and Fig. 1. The test-bed is designed to support extensive experimentation in NFV and network programmability, offering a large OSM/OpenStack cluster equipped with over 300 CPU cores, 8 TB of memory and a multi-Tbit network storage service. The cluster is interconnected using 10 and 40 GbE NICs with pass-through and Data Plane Development Kit (DPDK) capabilities, and a multi-vendor SDN network (NEC, Edge Core, Pica8, Corsa) with extensive metering and traffic prioritization capabilities. Test-bed users can access the DataPlane Broker (DPB) [32] network manager service to deploy point-to-point and multipoint overlay network slices with precise QoS control over this physical infrastructure. Furthermore, the Lancaster University test-bed offers access to the production SDN network within Infolab21, offering OpenFlow 1.3 and slicing support. Additionally, the test-bed provides P4 programmability through a range of software, FPGA, NPU and ASIC devices with P4 support.  The Lancaster island can also be extended to connect local external resources. Static VLANs can be set up in the core and exposed through OpenVPN, with access control delegated to an external Certificate Authority (CA) that can be managed by the party responsible for the resource. Custom NSDs can route the static VLANs to the Slough exchange, and therefore connect the local external resource to other islands.
The Lancaster University facility interconnects the INITIATE test-bed with a range of novel 5G application environments, including rural broadband, industrial control systems, and smart grid. It maintains a direct 10 GbE connection with B4RN (Broadband for the Rural North Ltd.), a community-based rural broadband ISP, providing gigabit broadband access to more than 5000 residences in rural areas of North West England. In parallel, Lancaster University hosts a novel TV White Space (TVWS) testbed combining communications infrastructure operating in the TV white space spectrum with distributed NFV compute capabilities at the network edge. The infrastructure allows for experimentation and evaluation of TVWS as the basis for future 5G rural access networks.
Lancaster University also hosts an extensive Industrial Control System (ICS) test-bed. It is extensively used to conduct cyber-physical security. The ICS facility contains three network control systems: a three-stage off-theshelf water treatment training rig from Gunt2, a model factory from Fishertechnik3 and a building management system (Trend IQ4e main controller and sub-field Trend IQ3 PLC). Finally, Lancaster University interconnects the Initiate test-bed with the EasyRES smart grid, a European experimental ICT infrastructure providing secure ancillary services to distributed renewable energy systems across Europe.

E. DIGITAL CATAPULT TEST-BED
The Digital Catapult 5G test-bed is a three-site facility, which is geographically distributed across Brighton and London. It is equipped with commercial and open source protocol stacks.
The three sites are interconnected with Gbps links. In addition, the London node is interconnected with the INITI-ATE test-bed through the 5GUKEx [11]. The main focus of the facility is to offer open access and technical support to third-party companies, primarily Small and Medium Enterprises (SMEs) that want to test and develop 5G-enabled services and applications through the 5G test-bed Accelerator Programme [33].
The three sites are similar both in terms of capabilities as well as setup, following the high-level architecture illustrated in Fig. 9. A range of commercial off-the-shelf (COTS) servers and switches, some of which are SDN enabled, provide the core infrastructure, on top of which a virtualization environment based on Openstack is deployed. Cloud-native applications are also supported through containers. The compute infrastructure offers a cumulative total of more than 1400 compute cores with several Gigabytes of RAM and Terabytes of storage on the premises, which can be exploited through concepts of Multi-Access Edge Compute (MEC).

III. 5GUKex ARCHITECTURE AND ORCHESTRATION FRAMEWORK
A network function (NF) is an infrastructure component that provides a well-defined functional behaviour, such as a media server, a firewall, or routing. It is a piece of software tightly coupled with specific, proprietary hardware, which needs to be manually installed into the network, creating operational challenges and preventing rapid deployment of new network functions. However, a virtualized network function (VNF) implements the NF using software that is decoupled from the underlying, generic, hardware. This VNF can then be deployed on any INITIATE partner site to provide a specific service. A network service (NS) can be created by chaining and then deploying a set of VNFs in a specific order to deliver the intended service [34].
The 5GUKex [11], [35] represents a central point for NFV orchestration in the INITIATE test network. It runs a multi-domain orchestrator (MDO) that orchestrates the deployed network services (NSes) in the different INITIATE network domains. As mentioned before, the 5GUKex [35] connects five test-beds across the U.K. using the layer-2 JANET connectivity, as shown in Fig. 1. We consider that each test-bed runs on a separate island or site. The 5GUKex architecture has a proxy that runs on each island, which maintains communications with a network service broker (NSB) running on the 5GUKex. The NSB receives the information of islands and their supported NSs during the boot-up process, which represents the main point of interaction between the 5GUKex and islands. This allows the experimenters, or other users, to compose the available NSes to create an inter-island NS (iNS) using the network service composer (NSC). The iNS is then deployed by using the NS manager (NSM) that handles its life cycle. The inter-domain connectivity manager (IDCM) creates appropriate network connections between the islands based on the received endpoint information from the islands' Proxies through the NSB.
The 5GUKex is designed to support a scalable orchestration brokering architecture that aims to interconnect heterogeneous network test-beds running on different islands. It is connected with the local orchestrators running on the different islands, which enables experimenters to request services and resources by combining multiple island capabilities.
The brokering functionality interacts with the relevant island orchestrators for managing, coordination, instantiating or termination of the orchestrated network services. The 5GUKex has a unified information model that complies with the European Telecommunications Standards Institute (ETSI) NFV orchestrator standard [10]. This is one of the most widely adopted open-source standards for compute and network resource orchestration, which facilitates the rapid addition of new test-bed islands. It is used to communicate with the network islands for exposing the local test-bed capabilities, requesting experimental resources from other Islands, which also allocates the requested virtual network and computing resources during the orchestration process for the composed NSes. Experimenters can view the performance and status information of their running network services at the 5GUKex.
The 5GUKex can support other solutions for data path connectivity. For example, it can extend the MDO to support the data plane broker (DPB) which allows multi-site mesh topology to deploy NFVs/VNFs. The DPB [32] can be deployed as an IDCM to simultaneously establish layer-2 connections with guaranteed bandwidth between the islands to meet the request requirements of experimenters or users. Within the central switch, it manages a single virtual OpenFlow bridge and the bindings of its virtual ports to VLAN/physical-port tuples. Traffic-shaping is applied per binding to police the bandwidth used by each deployment, which is preceded by admission control to ensure that the total bandwidth allocated to the bindings on a port does not exceed its physical bandwidth. Each virtual port then corresponds to the endpoint of an exchange-orchestrated NS deployment at a given island; and each simultaneous deployment of NSes manifests as a discrete set of virtual ports. When a port set consists of only two elements (i.e. two islands NSs are connected in a single deployment), a pair of OpenFlow rules is deployed to exchange traffic between the two virtual ports. For larger sets (i.e., multi-island deployments), a learning switch is implemented between each set of virtual ports to enable traffic between independent deployments to remains functionally isolated, while traffic-shaping achieves non-functional isolation by bandwidth.

INITIATE partners have interconnected their local test-beds through a test network which is managed by the 5GUKex.
We report the results of five use-cases that show the performance of the INITIATE test network and the capabilities provided by the laboratories involved in these use-cases. The first two use-cases demonstrate the capability of supporting ultra-low latency (ULL) services through the INITIATE test network, focusing on 5G services orchestration that requires low-latency connectivity with guaranteed data rate. The third demonstrates the capability of building VNF-based wide LiFi attocellular networks that receive services from a cloud server and deliver it through the INITIATE network to the wireless users on the islands. It guarantees the required QoS and quality-of-experience (QoE) for wireless user devices and traffic flows. The fourth use case discusses cache load balancing and a high-availability proxy that demonstrates the successful interaction of components across the INITIATE network. The fifth realises a healthcare scenario on provisioning ultra low latency telemedicine services through the INITIATE test network. The following section discusses in further details these use-cases.

A. ORCHESTRATING THE ORCHESTRA
The goal of this use-case was to create an experience where an orchestra is orchestrated by using the OSM across multiple network domains. The aim of the use-case was to allow musicians to compose music while being present at the different physical sites communicating over the low latency and high bandwidth network. The test-bed setup in Fig. 10 shows the three sites: University of Bristol, Digital Catapult and King's College London are connected using the 5GUKex hosting musicians and musical setup at each site. The round-trip network latency (without audio/video codec delays) was approximately 5ms whereas when codecs were included it was in the range of 25-30ms [36]

B. MUSIC FOR ALL-5G MUSIC CLASS
Another use-case trial [37] was completed using the INITI-ATE test network serving the music community. This event was partnered with the music charity ''Music for All'' 6 where the music teacher was located at King's College London and instrumentalists and vocalists were distributed at Bristol and Birmingham. For this use-case, the INITIATE test network was, for the first time, connected to the BT-EE commercial 5G network from KCL to Birmingham as shown in Fig. 11. The major challenge associated with this use-case was to achieve the required latency of under 25ms across the two networks and three sites. Soundjack [38] was the audio mix software and UltraGrid was used to capture and multicast video across the locations. The authors in [37] have discussed the artistic and technological challenges addressed during this showcase. The latency measurements between multiple sites connected in the use-case are shown in Table 1. All the measurements are average Round Trip Times (RTTs). The average network latency between all the sites ranged from 5 to 6ms, whereas the application processes added further latency but remained under 30ms. This use-case shows the versatility of the INITIATE platform as the Birmingham site was connected as the Ad-hoc node and an additional Infrastructure as a Service (IaaS) for this experiment.

C. CLOUD NETWORK-AS A SERVICE
One of the main challenges for the deployment of VNFs is the resource allocation required for the demanded NSs, particularly in wide area small cell networks. This use-case aims to inter-connect multiple LiFi attocellular network test-beds deployed on the different islands interconnected thorough the INITIATE test network. This should enable experimenters/users to access the UoE test-bed and receive services through this wide area LiFi attocellular networks in a reliable transparent manner. To achieve these goals, the UoE implemented a cloud infrastructure and networkas-a-service test-bed in the LiFi R&D centre to support the research activities relating to on-line services provisioning, communication systems development and integration with other wireless technologies in INITIATE.
This use-case runs on the UoE test-bed that comprises the SDN-enabled LiFi/WiFi hybrid access network shown in Fig. 4, PowerEdge Server R440, OF-enabled HP Switch 3810 and other network components, as shown in Fig. 12. The Openstack Pike is installed as a common choice with other INITIATE colleagues. The server (controller) runs the OSM MANO and ODL in virtual machines that are connected to the switch HP 3810, as shown in Fig. 12. The SDN open-source ODL Controller Boron version is configured in the test-bed. It assigns OF rules on the UoE switch and others along the paths established with our partners. The OF rules allow to handle and route appropriately traffic flows through the switches to reach their destinations. When a network service is created between two partners, it creates a network to which virtual machines are attached based on the VNF descriptors in the OSM MANO. They are allocated vCPUs, hard drive (HD) space and memory space, which form a slice of computing, storage, memory and network, as shown in Fig. 13 b. The UoE vision aims to support e2e experimental services provisioning to our INITIATE partners, as shown in Fig. 13 a. For example, a local media server, which provides video contents to the user devices in the hybrid LiFi/WiFi network at the UoE, has been converted to a VNF running in the cloud. It is used to create a NS that enables the user devices in the LiFi attocellular network at the UoB to stream videos from the UoE VNF, as shown in Fig. 13 b. The inbound and outbound traffic between the UoE and UoB is routed according to the OF rules installed by the ODL controller.
The UoE vision is to establish multiple e2e network services, enabling a mesh NFV/VNF-based network that supports traffic flows routing among the different partners. When a network service is configured between the UoE and any partner in the INITIATE project, a virtual local area network (VLAN) number in the range, [3,20], is assigned to the created network service, as shown in Fig. 13 a. A media server network function has been converted to a VNF which is deployed as a network service that provides multimedia traffic to the end-users associated with LiFi APs at the UoE and UoB, as shown in Fig. 13 b. The OF instance control plane, which runs on the VLAN 21 in the switch, sets the appropriate OF rules on the switches to run the deployed video server (video) streaming NS, as shown in Fig. 13 b.
Traffic measurements have been conducted to measure the throughput and latency of video stream packets transmitted between the UoE and UoB. A single LiFi user device associated with a LiFi AP configured in the high performance networks lab at UoB receives around 25 Mbps, as shown in Fig. 14 a. The video traffic packets experience a delay (latency) around 13.3 ms, as shown in Fig. 14 b. These results demonstrate that the 5GUKex can support high throughput and real-time traffic transport between the different partners in this project. This is the first time two LiFi attocellular networks have been connected through a high speed optical network. This enables experimenters to develop novel VNFs to provide new services to end-users or networking algorithms to further enhance the performance of wide LiFi attocellular networks.

D. HIGH-AVAILABILITY PROXY
This generic use case, developed by Lancaster University, demonstrates the basic interaction of components over the INITIATE test network. Several instances of an Apache web cache or server reside at one site, while a high-availability proxy (HAProxy) load balancer operates at the other, scanning for the cache instances on their shared L2 wide-area link. HTTP requests received by the HAProxy are distributed across the caches, and additional caches are automatically detected by the load balancer to deal with higher loads. A scenario of this use-case has been realized by deploying a single Apache server at one island and a HAProxy at another: see Fig. 15. The system was tested using an Apache Bench, with 1 GB RAM and one CPU; the results are illustrated in Table 2.   A new VM is required per 10 clients approximately, when the CPU utilization approaches 100%.

E. REMOTE HEALTHCARE AND TELEOPERATION
A use case on remote healthcare and ultra-reliable lowlatency communication (URLLC) has been implemented by using the 5GUK infrastructure, which interconnects paramedics and ambulance staff attending patients with medical experts located in the hospital. A haptic communication system is deployed across the network to enable medical experts to guide the paramedic using haptic interfaces with kinesthetic and tactile feedback, as shown in Fig. 16. This use case is designed for ultrasound diagnosis in ambulance, in partnership with some industrial partners [39], which has been implemented and trialled by KCL in a connected ambulance in the UK [40]. The performance of haptic communication system is affected in terms of QoE and task completion VOLUME 8, 2020 times, when latency exceeds 100 ms, due to the desynchronisation between video and haptic data streams [41].
This use case has been expanded in the INITIATE testbed, which runs an experiment between three islands at KCL, DCAT offices and UoB using two Geomagic Touch devices and two video cameras. A haptic interface and a camera are installed at KCL, which are used by an operator representing the medical expert while the other haptic interface along with a camera are installed in the DCAT offices to act as a tele-operator used by the paramedic in an ambulance. This enables the medical expert to observe and guide the paramedic, and a medical specialist at the UoB to supervise the whole procedure. Despite the stringent URLLC requirements, the network is able to provide adequate QoS for the robotic controllers to operate in a stable state. The RTT latency between the DCAT offices and KCL is measured to be 4.5 ms (ICMP ping test) and the robotic controllers were tuned to keep the system stable under such conditions. Generally, bilateral teleoperation systems, as the one used in the aforementioned experiment, are able to compensate for the high amounts of delay in order to achieve stability but, as a trade-off, the quality of telepresence is reduced [42].

V. CONCLUSION
This paper has explained the laboratory capabilities and test-bed architecture that each partner has contributed to build the INITIATE large federated experimental networking test-bed across the U.K. Concise details have been provided on the deployment of SDN, NFV orchestration and capabilities within each partner's test-bed and its interactions with the others through the 5GUKex [35]. This SDN exchange enables researchers to remotely request access to specific test-beds and carry out complex experiments involving any combination of their desired resources across any combination of the disparate test-beds. The INITIATE 5GUKex services provide a scalable plug-and-play framework for connecting future test-beds and commercial networks, as well as inter-connection with other established test networks, such as NDFF and GEANT. A data plane routing mechanism was developed to support concurrent experimental access to the test-beds running on the different islands. The solution provided by the data plane broker is already integrated with the 5GUKex to simultaneously route traffic among the different sites and hence enable users to support simultaneous access to the multi-site test-beds. By making its test-bed accessible to 3rd parties, INITIATE provides an opportunity for a wide range of the UK ICT and digital economy academic communities, industry/small and medium-sized enterprises (SMEs) to enhance the current features of the INITIATE test-bed. Innovative SMEs are encouraged to explore the diverse capabilities for Internet research offered by the different test-beds available under the INITIATE 5GUKex. INITIATE provides experimenters with resources at each island to support realtime, low-latency, high-bandwidth, wide-area experiments at multiple sites with resource isolation. As well as providing a distributed test-bed for SMEs and experimenters across the U.K., the platform is continuously evolving along with the individual test-beds. For instance, the adoption of new standards in ETSI NFV Release 3 into the 5GUKex is in progress and will help further integration of the various deployed NFV MANO systems including ONAP, 7 OpenBaton, 8  DRITAN KALESHI is the Head of Technology-5G with Digital Catapult. He has 25 years' of experience as a Researcher and Technologist in communication networks, distributed systems, and the IoT interoperability. Prior to this, he was with the University of Bristol, where he led a 15-Strong Research Team, strategically combining research with industrial collaboration, making a wide range of contributions and practical implementations in networking, interoperability, smart energy, cities, and digital health, including large scale testbeds and trials. He has published over 75 papers, holds three patents, and has edited two international standards on interoperability, and is often invited to speak on advanced digital infrastructure. He has represented the U.K. in international standardization bodies and served on/chaired international conferences and technical committees.
DAVID HUTCHISON is a Professor of Computing with Lancaster University and the Founding Director of InfoLab21. He has served on the TPC of top conferences such as ACM SIGCOMM and IEEE INFOCOM. He has helped to build a strong research group in computer networks, which is well known internationally for contributions in a range of areas, including the quality of service architecture and mechanisms, active and programmable networking, multimedia systems, content distribution networks, networking testbeds, and Internet science. He is currently focusing on resilient and secure networking, with interests in future Internet and also the protection of critical infrastructures, including industrial control systems. He served on editorial boards of Springer Verlag's Lecture Notes in Computer Science, the Computer Networks journal, and IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT (TNSM), and was an Editor of the Wiley Book Series on Computer Networks and Distributed Systems. Her research is focusing on the fields of high performance networks, programmable networks, wireless-optical convergence, 5G/B5G, and smart city infrastructures. She is increasingly working with social sciences on the topics of digital transformation for society and businesses. She has been the Technical Architect and the CTO of the smart city project Bristol Is Open. She is currently leading the Bristol City/Region 5G urban pilots. She has authored or coauthored over 600 publications, numerous patents, and several major contributions to standards. She has been the co-founder of two spin-out companies, the latest being the University of Bristol VC funded spin-out Zeetta Networks (http://www.zeetta.com), delivering SDN solutions for enterprise and emergency networks. She is a Fellow of the Royal Academy of Engineering and a Royal Society Wolfson Scholar. VOLUME 8, 2020