A Multi-Site NFV Testbed for Experimentation With SUAV-Based 5G Vertical Services

With the advent of 5G technologies, vertical markets have been placed at the forefront, as fundamental drivers and adopters of technical developments and new business models. Small Unmanned Aerial Vehicles (SUAVs) are gaining traction in multiple vertical sectors, as key assets to generate, process, and distribute relevant information for the provision of value-added services. However, the enormous potential of SUAVs to support a flexible, rapid, and cost-effective deployment of vertical applications is still to be exploited. In this paper, we leverage our prior work on Network Functions Virtualization (NFV) and SUAVs to design and build a multi-site experimentation testbed based on open-source technologies. The goal of this testbed is to explore synergies among NFV, SUAVs, and vertical services, following a practical approach primarily governed by experimentation. To verify our testbed design, we realized a reference use case where a number of SUAVs, cloud infrastructures, and communication protocols are used to provide a multi-site vertical service. Our experimentation results suggest the potential of NFV and SUAVs to flexibly support vertical services. The lessons learned have served to identify missing elements in our NFV platform, as well as challenging aspects for potential improvement. These include the development of specific mechanisms to limit processing load and delays of service deployment operations.


I. INTRODUCTION
The combination of 5G communications and Small Unmanned Aerial Vehicles (SUAVs) has recently turned out to be more than just a trendy idea. Separately, they have an unquestionable impact in our society, and an increasing number of new related applications are continuously appearing. However, it has only been recently that the amalgamation of these fields has received attention from the research community, and their natural synergies are beginning to be explored considering aerial vehicles as enablers of 5G communications [1], [2]. On the one hand, the development of 5G standards and technologies does not only aim at providing performant The associate editor coordinating the review of this manuscript and approving it for publication was Yang Tang . communications to end users. Whereas this was a primary driver in the previous generations of mobile networks, 5G considers vertical sectors as key adopters, and aspires to create a novel ecosystem where technical innovations and business models are also driven by vertical-specific use cases. On the other hand, SUAVs are currently gaining prominence as key assets in different vertical markets, such as precision agriculture, city management and public safety.
In these contexts, SUAVs have traditionally been considered as appropriate platforms to generate, process and/or transport relevant information (e.g., video and other sensed data). However, with the recent advancements on the miniaturization of electronic devices, SUAVs can onboard lightweight hardware platforms (e.g., battery-powered single board computers [3]), providing compute, storage, and network resources. These platforms open new and exciting opportunities to support a cost-effective and flexible provision of vertical applications. As an example, a smart farming provider could use a number of SUAVs to rapidly build an aerial network infrastructure over an extension of farmland. These SUAVs could support the collection, generation, processing, and dissemination of relevant information to offer smart farming operations, e.g., surveying and evaluating the status of a crop field, or detecting objects of interest to perform closer inspection or aid tasks such as precision spraying. In a different environment, an emergency service provider could rapidly deploy a fleet of SUAVs to support voice and data communications to a team of firefighters, working on a fire extinction activity along a large forest area, where existing infrastructures may be insufficient or even unavailable.
We want to emphasize that the practical realization of this view presents a first and fundamental challenge, i.e., the lack of flexibility exposed by existing SUAV products. These products are commonly equipped for specific purpose missions with a fixed set of software and hardware appliances, and base their operation on proprietary mechanisms and protocols. It is only when a programmable logic is embedded into the aerial vehicles that the flexibility of multi-mission SUAVs can be freely expressed [3], [4]. On the one hand, a programmable logic might allow the provision and configuration of different functions and services over adaptable SUAV units (e.g., to act as an aerial relay for voice and data communications). On the other hand, such a logic could enable the flexible interconnection of SUAVs to build programmable aerial networks, which could be rapidly deployed over delimited geographic areas. Based on these observations, in a prior work [3] we explored the utilization of Network Functions Virtualization (NFV), a key technology in 5G networking, to provide such programmable logic. In that work, we gave the first steps towards validating the practical feasibility of this approach, showcasing the deployment of a functional IP telephony service over a set of NFV-enabled SUAVs.
This paper continues the abovementioned research line, studying the feasibility of utilizing programmable SUAVs to support heterogenous vertical services. To this purpose, we followed a practical approach. We designed and built a multi-site NFV experimentation testbed, including an infrastructure of server computers, SUAVs and other resourceconstrained devices. This testbed was set up with the main goal of facilitating experimentation with vertical services, with SUAVs allowing for the prototyping and validation of these services in a realistic multi-site environment.
The remainder of this paper is structured as follows. Section II presents the background and the related technical literature, positioning our work with respect to the existing initiatives. In section III, we describe the specificities of our multi-site NFV testbed. We validate the testbed design in section IV, with the realization of a smart farming vertical service over this testbed and the analysis of the obtained results, particularly regarding service deployment delay. Section V concludes the paper with a discussion of the lessons learned, highlighting technical aspects for further development.

II. BACKGROUND AND RELATED WORK A. BACKGROUND ON ETSI NFV
It is unquestionable that end-user preferences, as well as their utilization of the information technology systems, have undergone a major change in the last decade. Social media applications like WhatsApp or Twitter, and multimedia services such as YouTube, Netflix or Spotify have been acquiring a notable prominence within all the Internet traffic. This tendency has led to an exponential increase in such traffic over the networks worldwide. Additionally, some of the most relevant forecasts, such as [5], point to an even more significant boost in this traffic due to the massive connection of different forthcoming devices (e.g., wearables, smartphones, sensors, etc.), and to the new emerging services brought about the new generation of communications (i.e., 5G).
These services (e.g., virtual reality, augmented reality, healthcare, autonomous driving, etc.) are not only expected to cause an enormous growth in the amount of traffic, but also to demand a high performance from the network. The latter is the responsible for providing higher data rates, ultrareliable and low latency communications, reduced and optimized energy consumption, and ubiquitous network access that enable the proper operation of the mentioned services. For this reason, and also with the objective of paving the way, stakeholders (i.e., network operators, vendors, etc.) are exhaustively working to make possible a complete change in the networking paradigm. One of the key technologies to enable this major shift is Network Functions Virtualization (NFV). NFV was initially defined with the main purpose to mitigate the restrictions entailed by the use of proprietary hardware in the provision of network functionalities by telecommunication operators. To this end, it emphasizes the importance of decoupling the network functionalities from the specific hardware, providing them as softwarization units that can be executed in commodity servers. In addition, the softwarization of network functions enables a reduction in investment and maintenance costs (i.e., related with operating and upgrading the network facilities), allows telecommunication operators to agilely integrate new network functionalities, reduces the network service creation time cycles, and enables the automatization of the network management processes, preventing human error prone operations.
The European Telecommunication Standard Institute (ETSI) is currently leading the standardization of the NFV technology, bringing together a wide range of participants including network operators, manufacturers, different industry stakeholders, and research entities and universities. ETSI published the design principles of the NFV architectural framework in [6], which describes the elements comprising the NFV architectural framework, and defines appropriate interfaces to enable interoperability among different vendor implementations. Following the ETSI view, the Virtual Network Function (VNF) provides the traditional hardwarebased network functionalities (e.g., firewall, DHCP server, IP router, residential gateway, etc.), implementing them in software and executing them through the use of virtualization technologies. VNFs are interconnected to provide network services, which are defined as a composition of VNFs. The NFV infrastructure (NFVI) comprises a set of server computers and other equipment that, through an abstraction layer, provides the virtual computing, networking, and storage resources that are needed to execute the VNFs. In the ETSI NFV reference architecture, the allocation of NFVI resources to the VNFs is handled by a Virtual Infrastructure Manager (VIM). This building block, along with the VNF Manager (VNFM) and the NFV Orchestrator (NFVO), comprise the NFV Management and Orchestration (MANO) platform. Regarding the VNFM, it is in charge of managing and monitoring the lifecycle of the VNFs, including their instantiation, configuration, scaling and deletion. The NFVO coordinates and manages the composition of functional network services across the NFV environment, interacting with VNFM and VIM entities.

B. ORCHESTRATION OF NFV SERVICES OVER CONSTRAINED DEVICES
Virtual infrastructure management can be nowadays considered a mature and consolidated technology, with multiple well-known open source and proprietary solutions in the field of cloud computing services. Examples of successful, widely adopted solutions are OpenStack [7], OpenVIM [8], VMWare vCloud Director [9], and Amazon Web Services Elastic Compute Cloud [10]. These solutions are typically used by cloud-computing service providers to support the execution of virtual functions through hypervisor-based virtualization technologies, supported by datacenters with high-profile server computers interconnected by high-speed local networks. As an alternative to traditional hypervisorbased approaches, container virtualization aims at providing a more lightweight solution, by sharing the operating system kernel of a compute node by all the containerized functions running on top of it. This approach enables to extend the application scope of the VIM towards more resourceconstrained environments, where traditional hypervisorbased virtualization may impose inconvenient overheads in terms of computing and storage. Well-known examples of container-based cloud technologies are LXC/LXD [11], Docker [12], or Kubernetes [13]. However, despite their reduced footprint in terms of computing, storage and networking requirements, these solutions still present limited applicability in highly resource-constrained environments, especially in those that may be available beyond the edge of telecommunication operator infrastructures. This is the reason why new alternatives are appearing to fit those requirements, such as K3s [14], a lightweight version of Kubernetes for Internet of Things (IoT) and edge environments; and Fog05 [15], a completely distributed end-to-end virtualization solution for provisioning and managing NFVI resources of IoT environments. The latter is a recent open source initiative, although preliminary tests in the scope of virtual and augmented reality [16] showcase its potential for supporting distributed resource-constrained NFV environments.
Regarding NFV orchestration and VNF management, there are several well-known and widely used open source implementations aligned with the ETSI NFV architectural framework. These include Open Source MANO (OSM) [17], ONAP [18], Cloudify [19], and Tacker [20]. Table 1 collects a summary of different software projects and initiatives that provide NFV orchestration and virtual infrastructure management, with an indication on the capacity of the latter to support resource-constrained devices.
The utilization of devices with limited computing resources, but with outstanding and inherent capacities in terms of mobility and flexibility, has been considered in the technical literature to enable ubiquitous network and service access, (even in rural areas with low population densities). For instance, the authors in [1] propose an architecture for 5G and the upcoming generations of mobile communications that integrates Unmanned Aerial Vehicles (UAVs), aiming at assisting the cellular networks to accommodate occasional flash crowds in specific areas. Works in [2], [21] address the extension of network coverage in future wireless networks, distributing UAVs around a macro base station to aid relying and sharing of user information, or even to let the UAVs to act as base stations to support end-to-end communications among multiple end-users. Following a similar research line, the work presented in [22] analyzes the next generation of wireless communications, and identifies UAVs as key devices to achieve the expected performance indicators in 5G networks, as enabling platforms to complement the cellular network resources and assist network communications in underserved areas.
Following an interesting and complementary direction, the research community is giving the first steps towards realizing the softwarization of network functions over mobile adhoc networks. Whereas these networks have been intensively studied by the research community in diverse application fields, their prospect of realizing the view of building a potentially unlimited pool of NFV infrastructure resources beyond the network edge, opens new research directions in this field that need to be explored with careful detail. In [23], authors use Software-Defined Networking (SDN) techniques to manage a heterogeneous vehicular network. The work in [24] proposes a model for a future infrastructure of Internet-ofvehicles using both SDN and NFV technologies. In [25], authors use SDN techniques to manage a flying ad-hoc network built by Unmanned Aerial Vehicles (UAVs). In [4], the authors present a video-surveillance system for large underserved areas using UAVs. NFV is used as the platform to transmit the video signal through a network of several VNFs running on the aircrafts. Considering the limitations on the resources that can be onboarded on the aircraft, the authors propose using para-virtualization, i.e., a solution where virtual machines directly share the hardware with each other, simplifying the interaction between the hypervisor and the operating system. This helps saving valuable execution time and allows a better management of the available resources.
In our prior work [3], we addressed the design and validation of an NFV system capable of deploying moderately complex network services over a wireless ad-hoc network of single board computers. We relied on container virtualization (LXC/LXD) to support the network functions, and on mobile ad-hoc routing protocols to facilitate multi-hop endto-end communications across the single board computers.
The management and orchestration platform was based on Open Source MANO (OSM) and OpenStack, given their potential to build a functional MANO platform aligned with the ETSI NFV architectural framework (OSM is an ETSIhosted project), as well as their maturity, their strong community support and endorsement, and their availability under open source licenses. The system design targets SUAV networks, although it is generic to be considered for other resource-constrained environments. This work was further developed in [26], showcasing the capacity of our solution to accommodate multi-site network services, making use of SUAVs and cloud infrastructures (our work was accepted by ETSI as a proof-of-concept for Open Source MANO [27]).
In this paper, we present the design and implementation of a distributed NFV testbed spanning three different (remote) sites. This testbed has been created following the methodology and the lessons learned from our work on MANO operations on SUAV networks [3], [26], and on NFV experimentation platforms [28], with the main goal to aid experimentation activities with SUAVs and constrained device applications on a multi-site NFV environment. The testbed design has been validated with the realization of a multi-site vertical service.

III. DESCRIPTION OF THE DISTRIBUTED NFV TESTBED
The distributed NFV testbed has been built taking advantage of the joint efforts of different Spanish universities, i.e., Universidad Carlos III de Madrid (UC3M), Universidad Politécnica de Cataluña (UPC), and Universidad del País Vasco (UPV-EHU), together with the 5G Telefonica Open Network Innovation Centre (5TONIC) laboratory [29], based in Madrid. In the following, we describe the infrastructure and services at each experimentation site, as well as the mechanisms that support their interconnection and joint operation. An overview of the whole NFV ecosystem is depicted in Fig. 1.
A. THE 5TONIC/UC3M SITE As a member of 5TONIC, UC3M has created an NFV experimentation infrastructure that can be flexibly interconnected and operate in coordination with other sites holding NFV capable equipment. This experimentation infrastructure is currently available at the 5TONIC premises. It includes a MANO system based on two open source and widely adopted software components: an Open Source MANO (OSM) software stack, which provides the functionalities of an NFV orchestrator and a VNF manager, and an OpenStack controller, acting as a virtual infrastructure manager (VIM). Both components run in a single server computer, each in an independent virtual machine. This approach facilitates the independent evolution of each component, as the software base of OSM and OpenStack evolves, as well as their vertical scaling in terms of allocated resources. The MANO system controls a set of compute, storage and network resources provided by a cloud of three interconnected server computers, each equipped with four Gb/s Ethernet ports. The server computers account for a total of 24 vCPUs, 96 GB of RAM, and 6 TB of storage.
In addition, the experimentation site at 5TONIC includes a resource-constrained NFV infrastructure. This infrastructure encompasses a set of ten battery-powered Single Board Computers (SBCs), Raspberry Pi 3 Model B+, each with a 100 Mb/s Ethernet interface and several Wi-Fi adapters. Due to their characteristics in terms of size and weight, these SBCs represent a hardware platform that can be conveniently used in SUAV and constrained device applications. The SBCs may be flexibly interconnected to form different multi-hop wireless network topologies, according to any specific experiment requirements. In addition, we have integrated the SBCs into our NFV ecosystem, following the approach specified in our prior work [3]. This way, the experimentation resources provided by the SBCs are accessible to the OSM stack of 5TONIC via a separate OpenStack VIM, and VNFs can be automatically executed onto the SBCs using container virtualization technologies (LXC/LXD). We want to mention that, during the realization of this work, ETSI published OSM Release SEVEN, which also supports Kubernetes. The software base of our MANO platform was updated to this OSM version, which was used to conduct the experimentation activities in this paper. A more detailed analysis is still required to evaluate the OSM support of Kubernetes and its suitability to enable experimentation activities with SUAVs. This analysis will be conducted as part of our future work.
The 5TONIC/UC3M site also provides two modular computer platforms conforming to the PC/104 embedded computer standard, each with one 8-port Gb/s Ethernet switch, enabling the interconnection of local equipment and applications at a UAV unit, and two Gb/s Ethernet ports, to support the exchange of information with external entities (e.g., with a ground control station in a realistic scenario). These platforms were designed to support IP communications in a tactical surveillance UAV from the Spanish Ministry of Defense [30]. In addition, a testbed with ten mini-ITX computers with Linux Operating system serves several purposes. For example, to complement the resources of the cloud computing infrastructure offered by the SBCs, and support the execution of VNFs that require more powerful equipment. Or to provide the functionalities of wireless routers and/or access points, e.g., with OpenFlow support.
The physical disposition of components in the UC3M site allows the flexible and on-demand interconnection of the aforementioned equipment, according to the experimentation needs. This enables experimentation with diverse and heterogeneous VNFs (e.g., lightweight VNFs) and physical network functions (e.g., mini-ITX or PC/104 computers), which can exchange data through the network infrastructure available at UC3M.
Finally, the 5TONIC/UC3M site provides a portable UAV-specific NFV infrastructure, which can be used to support different experiments. This infrastructure consists of four small-size UAVs (Parrot Bebop 2 and DJI Phantom 3). Each UAV onboards an SBC with two Wi-Fi interfaces (a Raspberry Pi 3 Model B+ with an additional Wi-Fi USB adapter). The infrastructure can be complemented as needed with the battery-powered SBCs that are available in the site, which could represent ground units for the purposes of experimentation (e.g., SUAVs perched on the ground or other terrestrial vehicles). All the SBCs involved in an experiment can be attached as compute nodes to a portable OpenStack VIM that runs on a laptop. This VIM offers an interface to the OSM stack to handle the deployment of VNFs over the whole infrastructure of SBCs. Finally, we want to mention that, for regulatory reasons, tests with real SUAV equipment at 5TONIC can only be carried out indoor, using specific locations that have been made available for this purpose.

B. THE UPV/EHU SITE
The Smart Networks at Industry (SN4I) infrastructure at UPV/EHU includes three OpenStack nodes comprising each one a controller node and three compute nodes, accounting for a total of 256 vCPUs of processing, 502 GB of RAM, and 19.6 TB of storage, with 10 Gb/s Ethernet connectivity between the nodes.
This OpenStack deployment is integrated in an NFV/SDN experimental facility that interconnects three main premises of the UPV/EHU: the Faculty of Engineering in Bilbao, the UPV/EHU rectorate in Leioa, where the connection to the National Research and Education Network points of presence are located, and the Aeronautics Advanced Manufacturing Center in Zamudio. All three premises are about 10-20 Km from each other, being interconnected at 10 or 1Gb/s with fiber-optics data-links by means of five OpenFlow-enabled switches in a double ring topology. For this aim, two independent networks are used: one provided by the UPV/EHU and the other by I2Basque. Each of the premises has an OpenStack deployment with similar characteristics as the one in the Faculty of Engineering of Bilbao, described above.
Additionally, the infrastructure of the UPV/EHU includes a reconfigurable node based on a Raspberry Pi 3 Model B+ integrating a high-resolution video camera and Bluetooth and Wi-Fi interfaces, onboarded in an SkyWalker X7 SUAV. This node hosts the connectivity system to allow the communication between the SUAV and the infrastructure. In order to deploy different services over the reconfigurable node or remove them, a Kubernetes-based control module has been deployed in the OpenStack node of the Faculty of Engineering of Bilbao. This module is also in charge of powering down, or reducing power consumption of devices connected to the system.
One of the services available to run in the reconfigurable node is the management of a Digital Mobile Radio (DMR) repeater, for emergency situations. The repeater has also LTE-support in order to connect to an emergency mission command center. Another service allows taking pictures in visible or infrared spectrum, that can either be processed on board (i.e. fire detection), stored or even have transmitted in real time.

C. THE UPC SITE
There are three OpenStack private clouds currently deployed at UPC with a total compute power of 262 vCPUs, 864 GB of RAM, 21 TB of storage, and 10 Gb/s Ethernet interconnection. The infrastructure comprises five controllers and eight compute nodes. All three sites are physically located at the UPC campus at Castelldefels in logically isolated silos within the campus internal network. Such ecosystem provides a distributed cloud-based platform for the deployment of multi-domain, multi-site testbeds for vertical services and use cases.
Within this infrastructure, one of these OpenStack private clouds has been allocated for the distributed NFV testbed described in this section. The private cloud includes two compute nodes and one controller, providing storage, processing, and networking capabilities to handle VNF provisioning. This enables cloud computing services in realistic use cases, where SUAVs and other limited-capacity devices, operating over distant geographic areas, interact with vertical-specific serving functions deployed over a dedicated cloud computing infrastructure. The UPC cloud provides both full VM and container-based deployments, supporting flexible VNF-enabled architectures and fast resource provisioning/orchestration under the control of the 5TONIC MANO system. The remaining two sites at UPC support other research activities on advanced network automation, as well as joint NFV/SDN experimentation activities with other UPC partners.

D. INTER-SITE COMMUNICATIONS AND JOINT OPERATION
To integrate the three experimentation sites into a single NFV ecosystem, and support their joint operation, an overlay network has been created to support inter-site communications. This is supported through a VPN service hosted at 5TONIC. This service enables the configuration of secure point-to-point virtual links between 5TONIC and every other site, each of them with a gateway function that runs a VPN client. Data transmitted over these virtual links are physically disseminated towards their proper destinations over the highspeed backbone network of RedIRIS. The VPN server at 5TONIC behaves as a network router, relaying data traffic across the point-to-point virtual links, which guarantees endto-end network connectivity among the different sites.
The coordination of orchestration actions among the three sites is guaranteed with the utilization of a single NFV orchestrator, which is provided by the OSM stack installed at 5TONIC. The OSM software provides a northbound interface that can be used to support common NFV operations, including the onboarding of VNF and service descriptors, and the commission and termination of multi/single-site network services composed by different VNFs. The latter is achieved through MANO-specific communications, which take place between the OSM stack at 5TONIC and the VIMs available at the three experimentation sites.
In the distributed NFV testbed, the VPN-based overlay network plays a fundamental role to support orchestration actions, enabling the distribution of the following types of information: 1) control communications between the OSM stack and the remote VIMs, to support the management of the compute, storage and network resources available at each site; 2) management communications to support the configuration of VNFs after their deployment; and inter-site data communications, to support the exchange of information among VNFs running at different sites.

IV. EXPERIMENTAL VALIDATION A. DEFINITION OF THE VERTICAL USE CASE
SUAVs have lately gained attention as enabling platforms to support the concept of smart farming. An SUAV can carry different payloads, such as high-resolution daylight video cameras, thermal imaging cameras, Global Positioning Systems (GPS), and a myriad of other low-cost sensor and electronic devices. These include wireless transceivers (e.g., Wi-Fi, lineof-sight radio, or 3G/4G), to support network communications towards ground units and other aerial vehicles. This way, this new type of devices emerges as a suitable platform to facilitate observation and remote sensing operations over crop fields.
On the one hand, the design space of smart farming applications has considered the utilization of standalone SUAV units. Based on information that can be acquired by an SUAV, by means of the sensors and the electronic devices that it can carry as payload, aerial vehicles may serve multiple purposes including, among others, surveying of the agricultural field [31], the evaluation of water status in a crop field [32], and the detection of objects of interest in agricultural environments through computer vision, e.g., to perform closer inspection or precision spraying [33]. But other research studies also contemplate the deployment of SUAV swarms to enhance the situational awareness on the farmland via cooperative and/or simultaneous operation, e.g., to support the collaborative generation of relevant images [34], or to increase the precision of monitoring operations over delimited areas [35].
To verify the capacity of our distributed NFV testbed to support experimentation activities with smart farming services, we considered the reference use case depicted in Fig. 2. In this use case, a smart farming service provider deploys a number of SUAVs over an extension of farmland. These SUAVs are configured to provide functions specific to smart farming, such as collecting data from sensors deployed over the crop field, and recording high-resolution daylight/thermal video/images, relaying all this information towards a remote equipment owned by the service provider. These functions are implemented in software and deployed on these SUAVs on demand, as Virtualized Network Functions (VNFs), using a MANO system under the control of the service provider. The collected information may serve different purposes, like the early identification of diseases, the estimation of the production volumes, etc.
In our reference use case, a Flight Control VNF could be instantiated at each aerial vehicle, being configured after its deployment with information on fixed trajectories in the form of waypoints, to be autonomously followed by the vehicle. Real-time control of an SUAV by a human operator could also be supported by the Flight Control VNF, for instance to enable a more detailed examination of a given zone of the farmland due to the detection of a potential disease. The information produced by the electronic devices onboarded on the SUAVs (e.g., live video/images and other sensed data), as well as information collected from sensors on the ground, would be delivered to a gateway function in a ground control station (a facility close to the farmland in charge of the maintenance of the SUAVs). In our reference use case, we consider that SUAVs are flying over a remote area, with limited access to communication infrastructures. Therefore, the collected information is disseminated towards the ground control station through a multi-hop wireless network conformed by the SUAVs themselves (flying or perched on land, for instance in specific-purpose ground structures). To this purpose, SUAVs execute the functions of wireless access points and/or routers as VNFs. Other devices in the farmland (e.g., harvesters, tractors, sprayers, etc.) might also be opportunistically used to provide similar functions. The information received by the gateway function would be relayed towards the service provider operating the deployment, via a fixed or a cellular (3GPP or non-3GPP) access network. In a realistic scenario, this would be supported by a telecommunication operator, who could offer 5G core network services under the control of a MANO platform. To facilitate the realization of the use case and provide an experimental setup, we will consider a single MANO platform, as it is indicated in section V-C.
In the reference scenario, a larger UAV equipped with a thermal imaging camera flies at a higher altitude, recording images that may serve to inspect and monitor the crop field and detect potential diseases. These images would be transmitted to the service provider through a second ground control station, which maintains a line-of-sight radio link with the UAV.
At the service provider location, relevant information (e.g., sensed data, video, images, etc.) would be processed and stored in specific-purpose servers, which would also be provisioned as VNFs. This information could also be inspected in real-time by a human operator.

B. EXPERIMENTAL SETUP
To evaluate the feasibility of utilizing virtualization technologies and resource-constrained platforms to support a smart farming use case, we have used our distributed NFV testbed to build the experimental setup shown in Fig. 3, which is aligned with the use case presented in the previous section.
In the 5TONIC/UC3M site, we used three SUAVs Parrot Bebop 2, each onboarding a compute node (i.e., a Raspberry Pi 3 model B+). A Wi-Fi access point, deployed at one of these compute nodes as a VNF, provides network access connectivity to a ground equipment with a sensor that measures humidity, pressure, and temperature. The information read by this sensor is sent to a gateway function deployed as a VNF on a ground compute node (i.e., a mini-ITX computer). The gateway function retrieves data from the sensor using the Message Queuing Telemetry Transport (MQTT) protocol [36], being the information delivered through the Wi-Fi access point VNF and a router VNF (deployed on another SUAV). To emulate the generation of information by electronic devices onboarded onto an SUAV, the experimental setup includes two additional VNFs: an MQTT Publisher VNF, which generates random temperature values (integers between 30 and 35 • C) and transmits them to the Gateway VNF; and a Video Producer VNF, which emulates the generation of a real-time video as if it were generated by a video camera at the SUAV. This VNF has been implemented using an open source traffic scheduler, Trafic [37]. We want to highlight that our experimental setup has been created within a laboratory environment with indoor controlled flight conditions, with a main focus on the network and verticalspecific functions that are needed to support the collection, dissemination, processing, and storage of relevant data in a smart farming use case. The implementation of specific functions for autonomous flight control, which might be needed in realistic scenarios, is out of the scope of this paper.
In our experimental setup, a second video source is provided at the UPV/EHU site, where a reconfigurable node is instructed to transmit a real-time video produced by a highresolution daylight video camera. This serves to represent a video recorded by an SUAV flying at a higher altitude. The video produced by this camera is disseminated using the Real-Time Transport Protocol (RTP) [38]. Alternatively, the video generated by this SUAV might be emulated by a Video Producer VNF, which is also available at the UPV/EHU site. This enables test procedures involving the transmission of real-time video from the SUAV at different qualities. In the experiments presented below, we utilize the high-resolution daylight video camera.
Real-time video flows, originated at the 5TONIC/UC3M and the UPV/EHU sites, are delivered to a Video Consumer VNF at the UPC site. This VNF, which has been built using Trafic and the VLC media player, handles the received video information to obtain performance metrics for validation purposes. In addition, the UPC site holds an Internet-of-Things (IoT) server VNF, which uses the MQTT protocol to retrieve all the data collected by the MQTT Gateway VNF, making these data available to authorized parties through a web interface. The IoT server VNF has been developed using an open source IoT platform, MainFlux [39].
Finally, in our experimental setup we emulate that the information collected and/or relayed by UAVs is sent towards the smart farming service provider through a non-3GPP cellular access network. To this purpose an Access Router VNF is deployed at the 5TONIC/UC3M site and the UPV/EHU site. This VNF supports the user-plane protocol stack defined by 3GPP for untrusted non-3GPP accesses [40], which is based on the GRE [41] and IPsec [42] protocols. The Access Router VNFs encapsulate the information transmitted from each site over a GRE/IPsec tunnel, sending it towards a 5G Core VNF running at UPC. The 5G Core VNF has been developed to support the same protocol stack as the access routers.
Hence, it decapsulates the information received from the GRE/IPsec tunnel, forwarding it to the IoT server VNF or the Video Consumer VNF as appropriate.

C. FUNCTIONAL VALIDATION
We used the MANO system at the 5TONIC/UC3M site to automatically deploy and configure the vertical service shown in Fig. 3 (all the NFV descriptors and images have been made available as open source software [43]). To support the automated configuration of all the VNFs, we developed specific Ansible playbooks, which are supported by the OSM software stack. Configuration aspects include, among other things, the definition of appropriate IP addresses and network routes in the case of access points and routing functions, and the activation of the services that might be needed to provision functional VNFs (e.g., starting the MainFlux service in the IoT server VNF).
Once the vertical service is successfully deployed and configured, all the sensed data (either real or emulated) starts to be disseminated towards the IoT Server VNF. Analogously, the Video Consumer VNFs starts receiving the video from the 5TONIC/UC3M and the UPV/EHU sites. The SUAV executing the Router VNF was preconfigured to fly (indoor) hovering over a static position for the whole duration of the experiment. This has served to verify that the SUAVs can fly while carrying a single-board computer as payload. Fig. 4 presents the average throughput of both videos, measured at the Video Consumer VNF, during a time interval of 180 seconds. They were received uninterruptedly, with an average rate of approximately 2 Mb/s and 9 Mb/s (their corresponding average sending rates). Additionally, the small frames in the top part of the figure represent a 100 seconds interval of temperature, humidity and pressure readings from the real sensor that is available in our experimental setup, as well as random temperature readings provided by the MQTT Publisher VNF in the same time interval. Readings are provided by the real sensor at a low rate, with a temperature, humidity and pressure value transmitted towards the IoT Server VNF every 5 seconds (random temperature values are also provided at the same pace). No sensed data was lost during the experiment. These results validate the functional behavior of the multi-site smart farming service depicted in Fig. 3, and suggest the potential of using virtualization technologies and SUAV platforms to support vertical-specific applications.
As previously commented, a very relevant advantage of programmable SUAV platforms is their ability to support the rapid deployment of different vertical services. In this respect, the service creation time cycle is recognized as a paramount performance parameter in the context of 5G networking. In particular, according to the 5G Infrastructure Public Private Partnership (5G-PPP) view, it must be reduced to an average of 90 minutes [44]. To verify the service creation times that can be achieved by our NFV solution for moderately-complex multi-site vertical services, we have done 30 consecutive deployments of the vertical service defined in Fig. 3. The deployment delays for this multi-site service are represented in the boxplot of Fig. 5. The figure also presents the time delays that are needed to deploy the vertical service at each site (5TONIC/UC3M, UPV/EHU, and UPC). To estimate these time delays, we created an NFV vertical service descriptor per site, including only the VNFs that are to be deployed at that site (i.e., 6 VNFs at 5TONIC/UC3M, 2 VNFs at UPV/EHU, and 3 VNFs at UPC). Each one of the three vertical service segments was deployed 30 times at its corresponding site, leading to the delay values presented in the boxplot.
The average service deployment delay for the multi-site vertical service was 1,052.2 s (less than 18 minutes). This represents an adequate performance figure in line with the 5G-PPP view. On the one hand, it enables the deployment of a moderately complex vertical service over several cloud and portable resource-constrained NFV infrastructures. On the other hand, it leaves sufficient time to carry out the flight procedures that may be needed in real scenarios, to position SUAV units at their corresponding locations. The average deployment times for the vertical service segments at 5TONIC/UC3M, UPV/EHU, and UPC where 505.3 s, 277.7 s, and 315.8 s, respectively. From these values, we observe that service deployment time increases with the number of VNFs, as it was expected. In addition, the average time required to deploy the multi-site vertical service (1,052.2 s) is lower than the aggregation of the deployment delays of the three vertical service segments (accounting for a total of 1,098.8 s). This suggests that OSM provides a certain degree of concurrency in the instantiation of NFV services.
We want to highlight that the deployment time for the multi-site vertical service under consideration includes two main components: 1) the time needed to instantiate and interconnect the required virtual machines and virtualization containers; and 2) the delay incurred in performing their configuration through Ansible playbooks to provide functional VNFs. To gain a better understanding on the impact of each of these components on the service deployment times, we have performed an additional experiment. We have created new versions of the vertical services of Fig. 5, which do not execute VNF configuration procedures (i.e., the deployment of these services only involves the instantiation and interconnection of their corresponding virtual machines and virtualization containers). Fig. 6 shows the average deployment times of such services, comparing them with the average delays that were previously obtained when VNF configuration procedures were executed. From the figure, we can observe that the deployment time of the services significantly increases when their corresponding VNFs are subject to configuration.
The reason for this lies in the mechanism of common use in OSM to carry out the configuration of VNFs. OSM supports a limited form of Juju charms [45], referred to as VNF configuration charms or proxy charms. Once the OSM software stack requests the VIM to instantiate the virtual machines and virtualization containers needed by the VNFs, and while these instantiations are in process, a Juju agent deploys a proxy charm for each of the VNFs that require configuration. A proxy charm is basically a collection of scripts and software that are specified as part of the VNF descriptor. In particular, the proxy charms implemented for our smart farming use case support the remote access to the VNFs via SSH, and perform their configuration through Ansible playbooks (these can also be specified as part of the VNF descriptors). The Juju agent installs each proxy charm on a Linux container, and provides an interface to the OSM software stack to handle the execution of the scripts bundled within the charm. This allows the automated configuration of a VNF. The OSM software stack monitors the outcome of the VNF configuration tasks, through status information reported by the Juju agent. This is an effective mechanism to support the configuration of VNFs, which also provides a high degree of flexibility to VNF developers in defining configuration actions. Still, our experimental results indicate that the complexity of the configuration processes, required to provision the Linux containers and to synchronize the execution of the configuration primitives, results in a significant increase of service deployment times and processing load. The latter is illustrated in Fig. 7, which shows the complementary cumulative distribution function (CCDF) of the average CPU utilization during one deployment of the multi-site vertical service, measured in 5 second intervals at the virtual machine that hosts the OSM stack. From the average CPU utilization samples, we observe that the configuration process of the VNFs imposed a significant load to the OSM virtual machine, with 79.63% of the samples taken showing an average CPU utilization higher than 90%. During 56.94% of the deployment time, the average CPU utilization exceeded 99%. This suggests that the efficiency of VNF configuration based on proxy charms still has room for improvement.
To observe the dependency between the CPU utilization and the number of VNFs of the vertical service, the figure represents the CCDF of the average CPU utilization for a single deployment of each vertical service segment, measured at 5 second intervals. It also shows a complementary cumulative distribution for a reference scenario, in which no NFV deployments were conducted by the OSM stack during a period of 1 hour. Table 2 summarizes some of the relevant values collected during these four deployments.
As it was expected, the average CPU utilization declines with the number of deployed VNFs. Moreover, we can observe that the distribution of CPU utilization exhibits a trend towards the baseline scenario as the number of deployed VNFs decreases. In the case of the 5TONIC/UC3M deployment, there was still a noticeable degree of overload, with the average CPU utilization exceeding 99% for approximately one third of the deployment time. This behavior is related with the VNF configuration mechanism detailed before, which increases the requested amount of CPU utilization with the number of VNFs to be configured. In the case of UPC and UPV/EHU, the average CPU utilization was lower, with reduced periods over 90% (20% of the deployment time for UPC, and 3.33% for UPV/EHU). Even in that cases, with lower CPU utilization values, the execution of VNF configuration primitives imposes a substantial delay to service deployment, as observed in Fig. 6.

V. LESSONS LEARNED AND FUTURE DIRECTIONS
In this paper, we studied the amalgamation of NFV and SUAV technologies to support 5G vertical services. To this purpose, we built a functional NFV testbed, capable of supporting prototyping, validation, and experimentation activities with such technologies in realistic multi-site scenarios. We validated the testbed design with the realization of a smart farming use case. Our results suggest the potential of NFV-enabled SUAVs to support the flexible execution of moderately complex vertical services. In the following, we identify the main lessons learned from this paper, as well as the future directions for our work.

A. SOFTWARE AND HARDWARE PLATFORMS
A challenging aspect of this work was to identify the software and hardware platforms that could support effective NFV operations on Single Board Computers (SBCs), while at the same time providing appropriate performance to vertical services. This was a complex task that required a careful analysis of diverse SBC models, operating systems, and OpenStack releases. More concretely, the fundamental problem was to identify SBC models and operating systems that supported wireless ad-hoc network setups, while at the same time could be integrated as compute nodes into a functional NFV infrastructure. This was the main driver for the selection of Raspberry Pi 3 Model B+ and Ubuntu 18.04 LTS (Bionic) to build the SBC/SUAV NFVIs at 5TONIC, and OpenStack Queens as the virtual infrastructure manager. In any case, these compatibility aspects need careful consideration by stakeholders aiming at reproducing our work.
Regarding the NFV orchestration platform, our experimental results suggest that OSM provides a mature and stable software base to support the MANO functionalities, with the commitment of a strong community to incorporate new and appealing features in subsequent software releases. Still, some OSM features are yet to be explored for SUAV-based vertical services, such as the capacity to scale VNF resources to satisfy elastic service demands, according to different service and resource-utilization metrics.
Further development of our multi-site experimentation testbed will follow a continuous integration approach, allowing a seamless evolution of our operational environment while its hardware and software base also evolves (e.g., with new OSM and OpenStack versions). As part of our testbed development, we will study the feasibility of utilizing Kubernetes clusters to support NFV services over SUAVs, taking advantage of their recent support from OSM Release SEVEN.

B. VNF CONFIGURATION PROCESSING LOAD AND DELAY
Our experiments with the smart farming use case represent a first approximation to evaluate the service deployment times that can be achieved in SUAV-based vertical use cases, using existing open-source NFV technologies. The obtained results are aligned with the requirements indicated by the 5G-PPP for appropriate service creation time cycles in the context of 5G networking.
We want to highlight that the virtual machine where we have installed the OSM software stack was allocated 4 vCPUs, 12 GB of RAM, and 100 GB of storage. This comfortably covers the recommended footprint for the proper operation of the OSM software (2 CPUs, 8 GB of RAM, and 40GB of hard drive, according to the OSM documentation). Still, our experiments indicate that the execution of VNF configuration primitives may impose a significant CPU workload to the OSM machine. This could be a symptom of problematic situations in production NFV environments, hindering the capacity of the MANO platform to perform concurrent operations, such as managing the lifecycle of multiple moderately complex vertical services.
As an alternative to proxy charms, the OSM community has recently introduced the support of native charms in the latest version of the OSM software (release SEVEN). With these, Juju scripts and software may run at the VNFs themselves, not needing an independent container environment at the OSM machine. However, whereas native charms could represent a promising direction to reduce the operational workload of VNF configuration primitives, this mechanism does not support Ansible-based VNF configuration, which may be a preferable option for multiple VNF developers and stakeholders.
Our short-term work will consider these aspects, addressing the design and implementation of lightweight mechanisms for Ansible-based VNF configuration, aiming at: 1) distributing the configuration burden among sites where VNFs are actually deployed; and 2) reducing the execution delays of configuration primitives. The latter will enable to better support certain SUAV scenarios, where the deployment of services must be satisfied with very stringent time constraints (e.g., in emergency use cases).

C. RESEARCH ON VIM AND NFVI SOLUTIONS
One aspect that was raised when building the infrastructure was the complexity of the mechanisms that were needed to integrate new resource-constrained platforms, such as single board computers, into a functional NFV infrastructure. Whereas the deployment of services over the NFVI infrastructure can be automatized by the OSM stack, the composition of the NFVI, as a prior step to accomplishing service deployments, is still a rather manual process requiring complex and time-consuming configurations. This may be appropriate for a static experimentation testbed, but we expect that real scenarios will require a much higher degree of automation and interoperation, when building an NFV infrastructure of SUAVs and other resource-constrained devices.
On the other hand, our preliminary experimentation stressed the challenges related with the limited operational time of battery powered SUAVs. In fact, this is the reason why SBCs in our testbed are powered by external batteries that provide extended autonomy. While we may expect longer operational times for larger SUAVs in real scenarios, there is a clear motivation to consider battery status of SUAV units to be as relevant as the status of the compute, storage and network resources. Monitoring battery status by a VIM solution may enable enhanced operations, particularly: supporting energy-aware policies for VNF placement; applying optimized replacement mechanisms for SUAVs with the goal of providing resilient services over delimited geographic areas with a limited number of SUAVs; and aiding the coordination of the overall NFV environment, by exposing information regarding the remaining lifetime of devices to the NFV orchestrator (e.g., to take adequate decisions regarding VFN migration, possibly across multiple VIMs).
From a general perspective, all these aspects call for enhanced VIM solutions, capable of deploying virtual functions over novel NFV infrastructures that can be dynamically conformed by resource-constrained devices. The development of such solutions will be supported by the distributed NFV testbed presented in this article, as part of our future work.

D. EMULATION OF COMMUNICATION CHANNELS
Last but not least, we understand that laboratory experiments may not capture all the specificities of real situations, particularly regarding the conditions of the wireless medium and the geographic positioning of SUAVs. As a long-term work, we consider the incorporation of an emulation system [46] into the NFV testbed. This will allow to increase the portfolio of experimentation enablers at 5TONIC, supporting experimentation activities with NFV services and SUAV platforms, while using realistic communication channels for the SUAVs. VOLUME 8, 2020 IVAN VIDAL received the Ph.D. degree in telematics engineering from the University Carlos III of Madrid, in 2008. He is currently a Visiting Professor with the University Carlos III of Madrid. He has been involved in several international and national research projects, including H2020 5GINFIRE and 5G City. He has published more than 50 scientific papers in several conferences and international journals. His research interests include unmanned aerial vehicles (UAVs), 5G networks, and multimedia networking.
BORJA NOGALES is currently pursuing the Ph.D. degree with the University Carlos III of Madrid. He was involved in the European Research Project 5GINFIRE and the National Project 5G City. His research interests include network functions virtualization (NFV), 5G networking, and unmanned aerial vehicles (UAVs). CRISTINA CERVELLÓ-PASTOR received the M.Sc. and Ph.D. degrees in telecommunication engineering from the Barcelona School of Telecommunications Engineering (ETSETB), Universitat Politècnica de Catalunya (UPC), Barcelona, Spain. She is currently an Associate Professor and the Head of the Department of Network Engineering, UPC. Being part of BAMPLA Research Group, she has been responsible and actively participated in diverse national and European competitive projects, such as NOVI, FEDERICA, ATDMA, A@DAN, Euro-NGI, Euro-FGI, EURO-NF, and private funding research and development projects. She has published diverse papers in national and international journals and conferences. She has been supervising several thesis in the field of management, optimal resource allocation, topology discovery, and routing in SDN/NFV and 5G. VOLUME 8, 2020