Dynamic Load-Balancing Vertical Control for Large-Scale Software-Defined Internet of Things

As the global Internet of things increasingly is popular with consumers and business environment, network flow management has become an important topic to optimize the performance on Internet of Things. The rigid existing Internet of things (IoT) architecture blocks current traffic management technology to provide a real differentiated service for large-scale IoT. Software-defined Internet of Things (SD-IoT) is a new computing paradigm that separates control plane and data plane, and enables centralized logic control. In this paper, we first present a general framework for SD-IoT, which consists of two main components: SD-IoT controllers and SD-IoT switches. The controllers of SD-IoT uses resource pooling technology, and the pool is responsible for the centralized control of the entire network. The switches of SD-IoT integrate with the gateway functions, which is responsible for data access and forwarding. The SD-IoT controller pool is designed as a vertical control architecture, which includes the main control layer and the base control layer. The controller (main controller) of the main control layer interacts upward with the application layer, interacts with the base control layer downwards, and the controller (base controller) of the basic control layer interacts with the data forwarding layer. We propose a dynamic balancing algorithm of the main controller based on election mechanism and a dynamic load balancing algorithm of the basic controller based on the balanced delay, respectively. The experimental results show that the dynamic balancing algorithm based on the election mechanism can ensure the consistency of the messages between the main controllers, and the dynamic load balancing algorithm based on the balanced delay can balance between these different workloads in the basic controllers.


I. INTRODUCTION
IoT is a kind of the Internet which is connected to the things. IoT contains a large number of interconnected devices, including household appliance, public facilities, wearable equipments, residential office buildings, industrial processes, medical equipments, law enforcement equipments, military installations, unmanned aerial vehicles, interconnected cars and other network applications that may be almost impossible to imagine at present [1]. IoT  Traffic management is an important theme to optimize the performance in Internet of Things [3], [4]. Through the dynamic analysis, forecasting and adjustment of transmission data, the traffic management technology to optimize the performance has been widely used. Existing traffic management technologies rely on closed, rigid existing network architecture designs. The control plane and data plane of this network architecture are tightly coupled, integrated, which hinders current traffic management techniques to provide true differentiated services for largescale IoT to accommodate fast, growing, uneven, high-speed variable flow mode.
SD-IoT is a new paradigm [5], [6], which introduces the software defined framework into the IoT architecture. Under the software defined framework, users can get a dynamic, automated network, which is a bit different than in the past and provides full-virtualization for application requirements. The architecture of the SD-IoT separates the control plane from data plane, which has the following obvious characteristics: visibility, programmability, openness, virtualization, and new flow patterns and characteristics. More importantly, SD-IoT provides a unified global network view that paves the way for inherently flexible, adaptive, customizable traffic control and management technologies of large-scale IoT.
The architecture of SD-IoT consists of two main components: SD-IoT controllers and SD-IoT switches. When an SD-IoT switch receives a new flow, the first packet of the flow is forwarded to a corresponding SD-IoT controller. The SD-IoT controller to compute a forward path for the flow. All SD-IoT switches located on this path install a new forwarding rule. The SD-IoT switch sends the first packet of the new flow to the SD-IoT controller that can cause transmission delays, and the SD-IoT controller calculates the forward path of the flow to generate a processing delay. Even more frightening is that it takes a lot of time to install new forwarding rules on the forward-path switches and is prone to delay peaks. When a large number of new flows are injected into the SD-IoT switch, the control plane and data plane of the SD-IoT architecture all generate significant computing and communication costs. Beacon [7], which is the most advanced multi-threaded controller, has a maximum throughput of 12.8 million flow requests per second (Mfrps). The global M2M traffic reached 32407.4 billion bytes per second at the end of 2016 [2]. Obviously, a single SD-IoT controller has been unable to meet the needs of big data flow for large-scale networks. In order to solve the above-mentioned problems of large-scale IoT, our preliminary work [8], [9] studied the problem of load balancing of data plane in large-scale software-defined networks. In this paper, we will further study the dynamic load balancing in the control plane of the large-scale SD-IoT. The main contributions of this paper are as follows.
• A general framework for the large-scale SD-IoT is presented, and it includes two main components: SD-IoT controllers and SD-IoT switches. The SD-IoT controllers use resource pooling technology, which are responsible for the centralized control of network resources.
The SD-IoT switches integrate IoT gateway function, and are responsible for data access and forwarding.
• A vertical control structure for the control plane of SD-IoT is proposed, and it includes the controllers (main controllers) of the main control layer and the controllers (basic controllers) of the basic control layer. The main controllers are responsible for resource management and coordination of the basic control layer, and also provide a northbound interface for the upper application. The basic controllers are responsible for the interaction between the control layer and data forwarding layer.
• A dynamic load balancing algorithm for the main controllers based on the electoral mecha-nism is proposed, which selects a main controller called a Leader from the main controllers to coordinate the message consistency between the main controllers to ensure that the main controllers can get the same, real-time global network view.
• A dynamic load balancing model based on balanced delay is deduced, and then a dynamic load balancing algorithm of the basic controller is proposed to reduce the network delay and avoid the traffic load imbalance.
The remainder of this paper is organized as follows. Section II summarizes the relevant work and current research trend of the load balancing schemes of the software-defined networking.
Section III presents the general framework of the SD-IoT and the vertical control structure of the control plane. Section IV proposes dynamic load balancing algorithms for main controllers and basic controllers. Section V presents the experiment settings and evaluates the performance.
Finally, a conclusion is given in Section VI.

II. RELATED WORK
When an SD-IoT switch receives a new flow, it forwards the first packet of the flow to the corresponding SD-IoT controller. The SD-IoT controller calculates and determines the path to forward the flow. The forwarding rules are installed in all SD-IoT switches on the flow path.
Obviously, when a large number of new flows are injected into the IoT, the SD-IoT controller will frequently receive and forward the new flow request information and calculate the flow path. At the same time, the installation of the forwarding rule process in the SD-IoT switch will produces a delay peak, easy to cause the Internet traffic load imbalance. Thus, a single centralized controller is easy to become a performance bottleneck. At present, the feasible solutions of the performance bottlenecks can be divided into two categories [3], [4]: controller load balancing and switch load balancing.

A. Controller Load Balancing
Controller load balancing refers to the ability to enhance network data processing by using Typical controllers with level structure are HyperFlow [10], DIFANE [11], Onix [12],and BalanceFlow [13]. HyperFlow uses a publishing and subscribing method based on a distributed file system, but it adds additional overhead for subscription management and maintenance.
DIFANE uses a core switch instead of a controller, but the core switch increases resource consumption. Both HyperFlow and DIFANE have a logical centralized, physically distributed control plane, and all controllers (or core switches) share a global network view. Onix uses a publishing and subscribing method based on the data structure of the NIB (network information base). Each local controller has an NIB data structure, and the controllers share a copy of the network state with each other, but the controller adds additional subscription management and maintenance overhead. BalanceFlow uses a dedicated controller for load balancing of all controllers and periodic reporting of flow request information. Each controller maintains its own flow request information, but it increases the overhead of the control plane. Onix and BalanceFlow divide a large-scale network into many small networks, and each small network is managed by a local controller.
Typical vertical control structures are Kandoo [14], Orion [15], SOX and DSOX [16], Hy-bridFlow [17]. Kandoo uses the root controller to control all local controllers, and each local controller manages one or more switches and has a global network view. Orion is similar to Kandoo. SOX, DSOX, and HybridFlow all use logical centralized control plane but physically distributed controller cluster architecture, and each controller cluster shares NIB.
Controller load balancing improves the latency between the switch and the controller in a single controller environment. The level controller architecture provides better recovery capability, but is difficult to manage. The vertical control structure is easier to manage through the upper controller, but there are single points of failure and complete consistency.
However, the load balancing strategy of the control plane has not been exploited to some extent. It needs to solve a series of fundamental problems, which are aimed at finding the optimal number of controllers, deployment location, workload distribution, forwarding path of control messages, and coordinates an optimal balance between the delay performance of control messages and the control costs, and has the statistical characters of network traffic and the diversity of network topology.
Heller et al. [18] first studied the static deployment of controllers, by calculating the minimum propagation delay to determine the number and location of controllers. Dixit et al. [19] proposed a distributed controller pool architecture that dynamically adjusts the work status of controllers and to balances the real-time workloads of the controllers based on traffic conditions. Bari et al. [20] studied the dynamic supply of controllers in a large-scale local area network, and dynamically changed the controller deployment through the number of real-time flows in the network. Yao et al. [21] studied load balancing based on the capacitated K-center problem, minimizing the maximum delay between the switch and the controller. Jimenez et al. [22] uses Ma et al. [28] proposed a load balancing mechanism based on a hierarchical control structure, which meta controllers analyze the resources and utilization in local control planes and optimize the processing performance. However, these efforts focus on finding quantitative or even heuristic results, the lack of deep research to the load balancing mechanism of the control plane in largescale SD-IoT.

B. Switch Load Balancing
ECMP [29] is a load balancing strategy that uses a flow-based hashing method to optimal flow allocation, but two or more long flows are prone to conflict on their hash and share the same output port, resulting in network bottlenecks. Hedera [30] is an extensible dynamic flow scheduling system that collects statistics information of flows on edge switches. If the flows increases beyond a given threshold rate, it will dynamically compute a suitable path and install the path in the switch. This allows for a balance between the high utilization and minimal scheduling overhead of networks. Mahout [31] monitors and detects large flows on the terminal host through a mezzanine of the operating system. When the mezzanine detects that the socket's buffer exceeds a given threshold, it will mark the subsequent packet of the flow. The switch forwards these marked packets to the Mahout controller. The Mahout controller calculates the best path for the large flow and installs a specific flow entity in the switch. MicroTE [32] is similar to Mahout. The above three methods use the central controller to calculate the appropriate flow path, and ECMP routing is used in the switch for a small flow. However, Hedera increases the processing overhead of controllers and switches and the bandwidth overhead of switches.
Mahout and MicroTE increase the processing overhead of switches and hosts and the bandwidth overhead of the host.
In order to reduce the number of interactions between controllers and switches, the traffic management of SDNs implements the wildcard rule in OpenFlows witches. Switches can handle the local route of microflows. Controllers only process directional macroflows, especially the quality of service of macroflows is more and more important, such as DevoFlow [33]. Another approach is using a core switch to complete all the packet processing, such as DIFANE [11].
However, this approach don't require the controller to participate in the process at all, and reduces the load on the control plane, but increases the burden on the core switch.
However, load balancing policies based on hash ECMP and wildcard rules are static and difficult to adapt to flow dynamics. In [34], a decision strategy based on switch migration is proposed, which senses load imbalance through switch migration triggering metrics, establishes a corresponding migration efficiency model, and weighs between migration costs and load balancing rates. Paper [35] proposed an alternative to the Beacon controller, which collects statistical information of OpenFlow devices in real time. The Beacon controller reroutes the flows according to the queues in the switch to ensure queue balancing in the switch. A load balancing algorithm in SDNs is proposed by collecting the traffic statistics of the switch subset in [36].
Obviously, the traffic management in SD-IoT needs a dynamic load balancing mechanism, which can dynamically adapt to time-varying network status and fine-grained traffic characteristics, such as traffic burst and arrival time interval.
Recently, paper [37] proposed a dynamic load balancing method for SDN-based cloud centers, which offers improved flexibility to the task scheduling using of the SDN technology, and completes the real-time monitoring of service node flows and load conditions based on the OpenFlow protocol. The controller can deploy global network resources once imbalance occurs.
A constrained optimization particle swarm algorithm based on SDN is proposed, which can effectively reduce the network delay and improve the quality of service of cloud and fog networks in [38].
To sum up, in order to avoid the bottleneck of a single centralized controller, traffic manage-

A. SD-IoT Framework
In this subsection, we describe a generic framework for SD-IoT, as shown in Fig. 1

B. Vertical Control Structure
In the above proposed framework of SD-IoT, the controller pool is designed as a vertical control structure, as shown in Fig. 2. The vertical control structure includes the main control

A. Dynamic load balancing algorithm for the main control layer
Existing vertical control structures are generally used with a controller or super controller [14], [17]. However, they are easy to prone to a single point of failure. Multiple super controllers can solve the problem of single point of failure [16], but there is the inconsistency of messages between the main controllers. In order to solve this problem, it is necessary to construct a set of effective message consistency mechanism in the multi-controller structure. PAXOS [39] is a consistency algorithm based on the information transfer model, which is considered to be the most effective of similar algorithms. In this section, we proposed a dynamic load balancing algorithm for the main control layer based on PAXOS algorithm.

2) Algorithm design: Assume n controllers in a main control layer are sent to their respective
Proposer a message and claim to be a Leader, the Proposer sends a proposal with their own <K, V> message to all Acceptors. According to K, all Acceptors will complete the update of K and promise to ensure that the proposal is rejected less than the updated K value, at the same time pass the proposal <K, V>. When no more than half of the feedback was received by the Proposer, it will update itself of K, and continue to give proposal messages from all Acceptors. Finally, <K N , Controller N > is the proposal to pass the resolution, and K N is the largest number and Controller N is the largest numbered controller. At this point, the Learner perceives the adoption of the <K N , Controller N > proposal and learns the proposal. At the end of the process, Controller N is elected as a Leader. Once the Leader fails, it will return to zero to launch a new election.

3) Algorithm implementation: The dynamic load balancing algorithm proposed in this section
is deployed in the main controllers, so that when multiple main controllers interact with each other, the Leader manages the execution and replacement information to pass to other main controllers. The main implementation process of the dynamic load balancing algorithm in the main control layer is as follows in Algorithm 1. if any of Proposers receives the number Acceptors returned more than half then 8: this Proposer will update its own K 9: the new K is used to send election massages to other Acceptor where q ij = 0 indicates that the jth switch S j is not controlled by the ith base controller C i ; q ij = 1 indicates that the jth switch S j is controlled by the ith base controller C i .
Assume that the number of the Packet in packets sent by the jth switch S j to the ith base controller C i at time t is p ij , we have where f ij (t) is the rate at which the jth switch S j sends the packet.
Obviously, during the interval [0, T ], the total amount of Packet in packets sent by the jth switch S j to the ith base controller C i is P ij , and we have So, the total number of Packet in packets processed by the ith base controller C ij from m switches: S 1 , S 2 , ..., S m at time t is expressed by During the interval [0, T ], the total number of Packet in packets processed by the ith base controller C ij from m switches: S 1 , S 2 , ..., S m is expressed by Thus, the total number of Packet in packets processed by the n base controllers in the basic control layer from the m switches in the data plane layer can be expressed by (q ij f ij (t)))dt (6) In vertical control structure of the large-scale SD-IoT, once the edge switch receives a new flow, the edge switch will forward the first packet of the flow to the corresponding base controller.
The base controller calculates and determines the path to forward the flow. All switches on the flow path will install forwarding rules. Obviously, when a large number of new flows into the IoT, the basic controller will frequently receive and forward the new flow request information and calculate the flow path. The switches on the path will frequently install forwarding rules. It will produce a peak of the network delay, and cause the load imbalance of the IoT.
Network delay is mainly composed of processing delay, queuing delay, transmission delay and sending delay. In general, the processing delay and sending delay of the controller and switch can be considered constant. Therefore, the network delay depends mainly on the queuing delay and transmission delay. From the queuing model M/M/1, it can be concluded that the queuing delay T j,w of a Packet in packet sent by the jth switch S j in the ith base controller C i can be expressed as where λ i = P i T is the arrival rate of packets, that is, the average value of Packet in packets arriving at the ith base controller C i in unit time; µ i = P i T d is the service rate of controllers, that is, the average rate of the ith base controller C i precessing Packet in packets.
Assuming that the transmission delay of packets transmitted by the jth switch S j of the data forwarding layer to the ith base controller C i of the control layer is T i,c , which can be obtained by calculating the maximum delay of all effective shortest paths between the switch S j and the base controller C i [42], we have As a result, the total time L i of the jth switch receiving a new flow, the ith controller calculating the path and the switches on the path installing forwarding rules is expressed by where T s is the processing delay and T d is the transmission delay.
Thus, the sum of the delay between the base control layer and the data forwarding layer is In the SD-IoT vertical control structure, the workload from basic controllers can be approximated as the amount of Packet in packet requests [41]. Therefore, the load balancing model of basic controllers can be expressed as where Q nm , L and P are given by Equ. (1), (6) and (10), respectively.
In fact, if you can reduce the network delay, it is easy to achieve load balancing. That is, the load balancing model can be converted to the lowest delay model, this is When the switch's requests for the base controller are too large or the base controller fails, the switch is mapped to a different base controller for balancing delay [41]. It can be seen from Equ.(10) and Equ.(12) that the network congestion caused by the excessive load of basic controllers can be alleviated by reducing the queuing delay and transmission delay.

2) Dynamic load balancing algorithm for basic controllers:
In this section, we present a dynamic load balancing algorithm for basic controllers based on balanced delay. The proposed algorithm obtains the network topology G(S, C) of switches in the data forwarding layer through basic controllers. Assuming that the threshold for the Packet in packet processed by a base controller in unit time is P th , which can also be referred to as the load peak of base controllers.
When the total number of the Packet in packets which the ith base controller C i processes from the m switches is greater than or equal to the load peak P th of the ith base controller C i , and the request amount of Packet in packets which the jth switch S j sends to the ith base controller Algorithm 2 : Dynamic load balancing algorithm for the base control layer. Input: The initial value of Q nm , the rate p ij of Packet in packets.
Output: The value of Q nm .
1: for S j ∈ C i do 2: if C i goes down then if p ij > P th m and T s + T w (p ij − P th m ) > T c (p ij − P th m ) + T d then 8: Add and control the controller with the smallest distance in the topology for the switch.

10:
The packet-in message exceeding the threshold set by the controller is processed to the new controller. C i is p i , and p i > P th m . At the same time, the total value of the queuing delay and transmission delay exceeding the initial value is greater than the total value of the transmission delay and processing delay used to retransmit the Packet in packet to the unoccupied basic controllers at the shortest distance, and the corresponding basic controller changes Slave into Master, that is, q = 1. While the portion of the Packet in packets beyond the original base controller is handed over to the idle base controller which has an minimum distance. Otherwise, the packets continue to wait for the base controller to handle until the iteration is complete. When the base controller fails, that is, q = 0, the switch through the G(S, C) will make the idle Slave with a shortest distance as the Master, that is, q = 1, and it repeats the above steps. The specific implementation code of the proposed algorithm is as shown in Algorithm 2.
The dynamic load balancing algorithm proposed in this section transforms the load balancing problem into a network latency problem, which avoids load imbalance by reducing the queuing delay and transmission delay.

A. Experimental environment
In order to evaluate the effectiveness of the proposed SD-IoT framework and the performance of the proposed dynamic load balancing algorithms for vertical control structure of the SD-  Fig.3 shows a scenario of the SD-IoT. There are nine controllers, including three main controllers and six basic controllers. Each of main controllers connect to six basic controllers, and each of base controllers manages five switches in the local region and connects with other regional switches. We deploy Algorithm 1 presented in this paper in the main controllers, and deploy Algorithm 2 presented in this paper in the base controllers. Assuming that the average delay of the switch sending requests to the base controller responding to requests is T , the standard deviation is Therefore, in the basic control layer, the response time of the controllers, the standard deviation of the response time of the controllers and the CPU utilization rate of the controllers will be used as the evaluation index of the dynamic load balancing algorithm of basic controllers.
2) Results analysis: In order to evaluate the performance of the dynamic load balancing algorithm proposed in this paper. We deploy the dynamic load balancing algorithm of the main controllers in a variety of different scenarios. The results are obtained by duplicating the experiment multiple times.
The impact of test number on the time spent on electing a Leader from three main controllers is given in Fig.4. As can be seen from Fig.4 In order to evaluate the performance of the dynamic load balancing algorithm for basic controllers proposed in this paper. We deploy Algorithm 1 in three main controllers, and deploy Algorithm 2 in six base controllers.
The impact of average request response time of base controllers on the rate at which a switch sends Packet in packets. As can be seen from Fig.6, the greater the rate at which the switch sends Packet in packets, the more data packets received by the base controllers, and the average response time of the base controllers is gradually increased. The performance of the dynamic load balancing algorithm for basic controllers proposed in this paper is far superior to that of the original data without deploying load balancing algorithm. With the increase of Packet in packet rate, the advantages of the proposed algorithm are becoming more and more obvious.
The impact of standard deviation of average request response time of base controllers on the rate at which a switch sends Packet in packets. It can be seen from Fig.7 that the standard deviation of the controller response time increases rapidly as the Packet in packet rate increases without the deployment of the load balancing algorithm. In the case of deploying the load balancing algorithm for basic controllers, the standard deviation of the controller response time is almost unaffected by the Packet in packet rate of the switches. The dynamic load balancing algorithm for basic controllers balances the request response time for the switches to send Packet in packets to base controllers.    The impact of CPU utilization for configuring six base controllers without deploying the dynamic load balancing algorithm presented in this paper (case 1) is given in Fig.8. As can be seen from Fig.8, in the case of the CPU utilization of basic controllers changes dramatically over time fluctuations. The impact of CPU utilization for configuring six base controllers with deploying the dynamic load balancing algorithm presented in this paper (case 2) is given in Fig.9.
As can be seen from Fig.9, the CPU utilization of basic controllers fluctuates with slighter rate. The average (blue bar) and standard deviation (red bar) of the CPU utilization of basic controllers in both cases are given in Fig.10. In both cases, the average of the CPU utilization of the six base controllers is close. In case 1, the average CPU utilization of the six base controllers is 55.15%. In case 2, the average CPU utilization of the six base controllers is 54.18%. However, the standard deviation of the CPU utilization of the base controllers in case 2 is much less than that in case 2. The average standard deviation of the CPU utilization of the six basic controllers is about 15.44% in case 1, but it is about 1.55% in case 2. Obviously, the CPU utilization in case 1 has better stability than that in case 2, that is, the load is more balanced. The load balancing algorithm of basic controllers is based on the balance delay to select a controller, and achieves the purpose of load balancing.
To sum up, we can see that the dynamic load balancing algorithm for main controllers proposed in this paper can make main controllers to quickly synchronize the information of the global network view. The load balancing algorithm for basic controllers proposed in this paper can ensure that basic controller resources are distributed evenly to achieve the effect of load balancing.

VI. CONCLUSIONS
This paper describes a general framework for SD-IoT. The control plane of the framework uses a vertical control architecture, which is divided into the main control layer and the basic control layer. The main controllers in the main control layer coordinate and manage the basic control layer, and the basic controllers in the basic controller layer manage and control the data forwarding layer of switches. In the main control layer, we design an algorithm for electing a controller as a Leader, which is used to coordinate and manage the main controllers, in order to achieve the dynamic load balancing of main controllers. In the basic control layer, we design a dynamic load balancing algorithm based on balanced delay, which is used to process Packet in