Dynamic Bandwidth Scheduling of Software Defined Networked Collaborative Control System

From the perspective of collaborative design of control and scheduling, a software-defined networked collaborative control system (SD-NCCS) is established. On this basis, the system performance of SD-NCCS is analyzed and the average sensitivities of reference are used in utility function to evaluate the effects of network-induced delays. Moreover, the network pricing mechanism and game theory are introduced and a dynamic bandwidth resource allocation model of the overall control system for optimal performance is obtained. Thus, the problem of the network resource allocation of the SD-NCCS has been converted to be the problem of solving the Nash equilibrium point under the non-cooperative game model. Furthermore, the Nash equilibrium solution under this frame is obtained using the estimation distributed algorithm (EDA). Finally, a simple example is included to illustrate the performance of our scheme.


I. INTRODUCTION
The networked control system (NCS) is a distributed control system where the control loops are closed via a communication network. Compared with the traditional point-topoint control system, the most significant feature of NCS is that the sensors, controllers, and actuators within are interconnected through some communication network instead of direct point-to-point connections [1]. As a result, using NCS may increase system flexibility, reduce system wiring, break the space limitation, which can realize remote monitoring and control in complex environments. Software-Defined Network (SDN) is recognized as the next-generation network paradigm [2], which decouples the network control plane from the data forwarding plane to a logically centralized controller, enabling rapid network technology innovation for multiple users. Under the SDN architecture, it is feasible to provide flexible and efficient service slicing solutions for different QoS requirements of nodes at the application layer by utilizing the characteristics of OpenFlow (OF) switches and Network Virtualization (NV).
The associate editor coordinating the review of this manuscript and approving it for publication was Qiong Wu.
Due to the limited communication bandwidth of the network itself, many problems such as delay, packet loss and jitter of data transmission in the control system will affect the control performance of NCS [3]- [5]. Without additional network resources, how to allocate and utilize bandwidth reasonably in limited network resources, so that the entire control system has an optimal control performance is a problem worth studying.
Notably, most of the aforementioned research is focused on the control strategy to reduce the impact of delay, packet loss, and packet error on control performance. However, studying NCS from the perspective of the quality of control (QoC) ignores the complex dynamic behavior of the network and is difficult to apply to complex real systems. The integrated design strategy of both the QoC and the quality of service (QoS) has attracted a growing number of researchers' attention. The collaborative design of control and scheduling can be divided into two categories. One is a real-time calculation method using real-time feedback theory, see [6]- [7]. In [8] a collaborative design method combining adaptive controller and feedback scheduling strategy is proposed, which overcomes some limitations of application and execution platform. The other is to maximize the performance of the control system in collaborative design and see [9]- [11] for a partial list of references. For example, in [12] joint optimization of information transmission and state estimation developed to reduce the influence of network inducing factors on the convergence and accuracy of estimation. However, a key research issue in the NCS over SDN is the lack of dynamic scheduling mechanisms for network resources [13].
Motivated by the above analyses and under the SDN framework, from the perspective of collaborative design of control and scheduling, a software-defined networked collaborative control system is proposed. Furthermore, under the framework of SD-NCCS, the non-cooperative game model and network pricing mechanism are introduced, and a dynamic bandwidth scheduling strategy suitable for SD-NCCS is proposed. The goal is to find a flexible bandwidth allocation strategy based on Nash equilibrium and network pricing. Thus, the dynamic performance requirements of individuals and the whole are simultaneously satisfied. The network pricing mechanism is implemented in a centralized control mode in the network controller, and the overall performance of the control system is evaluated by the cumulative amount of the output error of each control loop. Also, considering the complexity of the calculation of the exact solution of the network bandwidth scheduling problem in practical operation, this paper gives an EDA-based Nash equilibrium point solution method.

II. PROBLEM FORMULATION A. SYSTEM STRUCTURE
Envisioning the advantages of a software-defined network, a general architecture of SD-NCCS is proposed as shown in Figure 1. First, the data plane and the control plane are decoupled. The data plane is mainly composed of a set of switches that support the OpenFlow protocol (referred to as the OF Switch). In addition to the packet matching and forwarding tasks according to the flow table, the OF switch provides a secure interface to the control plane according to the OpenFlow protocol for convenient control. The control plane queries and manages the flow table in the switch. Second, the control plane consists of a controller that supporting the OpenFlow protocol (referred to as the OF Controller) or a network operating system and a set of applications on top of it. Through the OpenFlow channel, the control plane can acquire the state of the data plane and program the data plane, and thereby achieving optimal management of network resources from a global perspective. The part responsible for bandwidth scheduling and allocation above the OF Controller in the control plane is called the policy center (one kind of control APPs with a bandwidth allocation algorithm). Also, the network control subsystem is no longer a closed-loop control of the straight-through structure, but an open-loop control using a hierarchical structure. Each control loop issues a control command by the main controller, and the execution agent receives the command and controls the output.
It is assumed that the SD-NCSS has M hierarchical network control loop subsystems, and each control loop subsystem includes a main controller and an execution agent. The execution agent includes a local controller, a sensor, and an actuator. The main controller sends a control command to the execution agent through the network, and the local controller in the execution agent converts the control device into a control signal according to the received command, and the sensor feeds the output action state information to the local controller. Both the main controller and the execution agent are time-driven and have the same sampling period h. The main controller sends bandwidth requirements to the policy center through the OF switch and the OF controller according to the task information, and the policy center integrates the bandwidth requirements of each subsystem to perform bandwidth allocation. A complete bandwidth scheduling operation cycle is called T O , as shown in Figure 2. First, during the bandwidth polling period T P , each control loop controller submits a bandwidth requirement to the center. Then, in the allocation period T A , the center calculates and allocates the bandwidth according to the collected information, that is, updates the Flow Table in the corresponding OF Switch. Finally, in the remaining data transmission period T T , each control loop alternately transmits data according to the allocated time slice until the beginning of a new cycle. Through the state information feedback and bandwidth resource reallocation of each new cycle, it can adapt to the changes in data service requirements of each control ring. Therefore, in this layered software-defined network collaborative control system, flexible scheduling strategies can be conveniently implemented.

B. PERFORMANCE ANALYSIS
Under the premise of ensuring the stability of each control loop, finding an optimal bandwidth allocation scheme is the problem to be solved in this paper. Therefore, it is necessary to further analyze the impact of time delays on control performance under the assumption that there is no packet loss and disorder.
First, assume that during a data transmission period T T , the main controller of the control loop i will periodically send a reference control signal r (t) to the corresponding execution VOLUME 8, 2020  agent with h as the sampling period, as shown in Figure 3. The total number of reference transmissions during T T is , the control reference signal is being sent to the corresponding execution agent on the main controller side, and the latest reference control signal at this time is recorded as r (n) t j k . However, at the same time, the reference control signal actually received by the actuator agent side is a previous reference control signal, denoted as Based on the above analysis, the average sensitivity of the control output to the reference control signal due to network delay is defined as follows [14]: In practical engineering applications, the average sensitivity of the reference control signal can be calculated in advance or determined by simulation experiments according to the control requirements and then obtained by table lookup.
Finally, the designed utility function must be an increasing function of the bandwidth x, which is also affected by the average sensitivity of the reference control signal. In addition, to satisfy the existence of the Nash equilibrium, it must also be quasi-concave and continuously differentiable. Based on the above conditions, the utility function of the control loop i designed in this paper during a data transmission period t ∈ t 0 k , t 0 k+1 is as follows: is the average sensitivity of the reference control signal change of the control loop i.
The optimization goal is to maximize the total utility of all control loops in the SD-NCCS as follows: where ω R ∈ (0, 1) is a retention scale factor, which was introduced to retain part of the bandwidth for unexpected overload conditions. However, simply using J as the optimization objective does not consider that each control loop should pay a corresponding price for bandwidth resources. Therefore, such bandwidth resource allocation without price mechanism constraints cannot prevent individual control loops from maximizing their private utility due to private interests. All bandwidth resources may be only assigned to one control loop and cause the overall system to crash.

III. MAIN RESULT A. GAME MODELING BASED ON NETWORK PRICING
The game model based on network pricing mechanism is discussed in this section. Applying economic model and game theory to the field of computer networks becomes of high research. Utilizing a game-theoretical framework with the pricing mechanism is an effective way of resource allocation of arbitration. Especially in the field of wireless access networks. For example, in [15] a utility-based power control via convex pricing scheme is proposed to obtain an efficient power allocation in the uplink of CDMA wireless networks.
In [16] a cooperative distributed interference pricing algorithm is proposed for power control in heterogeneous wireless networks. In [17] a Stackelberg game is formulated to study the price-based bandwidth allocation in multi-user Bluetooth low energy (BLE)-backscatter communication. In [18] a hierarchical auction-based mechanism is designed to reduce both global communication and unnecessary repeated computation in ad hoc networks. The pricing strategy mainly includes two modes: Non-usage-Based Pricing (NBP) and Usage-Based Pricing (UBP) [19]. Although NBP is simple in pricing and convenient to operate, it cannot use price leverage to adjust user resource allocation, which affects resource utilization and easily leads to network congestion. However, the unit bandwidth price of UBP can be tracked and changed according to the consumption of network resources. This dynamic pricing mechanism aims at maximizing the overall satisfaction of users and can achieve high utilization and fairness of network resources. Every control loop wants to maximize its final revenue under the framework of the SD-NCCS. In general, the revenue of a control loop depends on the pricing strategy of other control loops. Given the pricing strategy of other control loops, if no control loop has the motivation to choose other pricing strategies (the utility of this strategy is already the max), then no control loop will break this equilibrium. This equilibrium is called the Nash equilibrium (NE) [20]. On the one hand, it needs to provide services for each control loop to improve its QoC. On the other hand, it needs to balance the bandwidth requirements of each control loop so that the QoC of each control loop can be guaranteed. Therefore, for the SD-NCCS with limited bandwidth resources, the optimal resource scheduling strategy that takes into account the best performance of individuals and the whole is to allocate bandwidth at the NE.
The OF controller plays the role of price maker and each control loop mentioned above is referred to as a user. Each user does not know the existence of other users and their current status. Each one submits its bidding strategy according to its QoC need, and then pays the corresponding amount for the actual bandwidth allocated. The entire bidding process is as follows: Step 1: Each user has the same amount of funds g in the initial state of each round of bidding.
Step 2: Each user submits a bidding y i (y i ∈ 0, g ).
Step 3: The price maker will develop a bandwidth allocation scheme based on the bidding and allocates bandwidth resources to each user.
Step 4: Each user pays the amount of bandwidth allocated. The amount y i that user i needs to pay can be calculated by the following formula.
where λ is the unit bandwidth price, and the utility function of each user is as follows: It can be seen that the benefit of a user is determined by the corresponding QoC obtained and the payment under the given QoS. Each user independently makes its bidding strategy based on its earnings. Each user will not offer the high price arbitrarily, because too much high bidding will only lower its final income. It should be noted that the amount, the user ultimately pays is based on the bandwidth they have obtained, which may not match the initial offer.
The Nash equilibrium based on and non-cooperative game model can be described as follows: A non-cooperative game model with M users based on the network pricing can be described as i is the optimal strategy for user i when other users choose is the policy space of user i, and S i (·) (shown in equation (6)) is the utility function set of the user i. Then, the bidding strategy x * = x * 1 , . . . , x * i , . . . , x * M is a Nash equilibrium based on the network pricing and non-cooperative game model. However, each time a user submits status information, it is not practically a submission of a bidding, but the average sensitivityP i n,o (t k ) for a new task. The price maker runs the bandwidth allocation program according to theP i n,o (t k ) combined with the game model to calculate the Nash equilibrium point, and performs actual scheduling on each control loop.
The following theorem will be used to prove the existence and uniqueness of Nash equilibrium for MG.
Theorem 1: A Nash equilibrium exists in game NG = , P j , u j (·) , if for all j ∈ : (1) The policy space P j is a non-empty compact convex subset of the Euclidean space R n ; (2) Utility function u j (p) is continuous on p and concave on p j ; Proof: Please refer to [21] for details of the certificate. Theorem 2: A Nash equilibrium exists in SD-NCCS, Proof: This theorem can be proved by verifying that MG satisfies the two conditions of Theorem 1. Assuming that x min i ≤ x max i is always true, it is clear that the first condition is met. It remains to prove that the utility function S i (x) is quasi-concave on x i for all i.
The first-order differential of utility function S i (x) is: The second-order differential of utility function S i (x) is: Obviously, the utility function S i (x) is continuous and differentiable on x and concave on x i . A concave function S i (x) is also quasi concave, so the second condition is also satisfied, and the proof is completed.
Theorem 3: The SD-NCCS has a unique Nash equilibrium. Proof: The proof of the uniqueness of Nash equilibrium is similar to Theorem 2. The specific proof process can be found in the reference [22]. To ensure the simplicity of the text, it will not be repeated here.  So far, we have converted the SD-NCCS network bandwidth allocation problem into a Nash equilibrium calculation problem based on the network pricing non-cooperative game model.

B. EDA ALGORITHM
Envisioning the Nash equilibrium point under the network pricing non-cooperative game model is a nonlinear optimization problem with multiple constraints. The traditional numerical calculation method is difficult to solve the Nash equilibrium, so this paper adopts the EDA-based optimization algorithm for calculation. Furthermore, the implementation steps are as follows [23]: Step 1: Initialization: randomly generate a valid initial population B 0 meeting the bandwidth constraint.
Step 2: Select: select the first half of the best gain candidates from the parents.
Step 3: Update: reconstruct a probability model based on Gaussian distribution from the obtained dominant group.
Step 4: Sampling: a new generation of populations is sampled by a new probability model. Go to Step 2 until the termination criterion is met.

IV. EXPERIMENT AND RESULT ANALYSIS A. SIMULATION SETUP
In this section, a simple example is given to illustrate the effectiveness of the proposed method. The topology of the SD-NCCS test platform is shown in Figure 4, including a controller as a policy center, a switch supporting the Open-Flow protocol, and three control loops. Each control loop consists of the main controller and a simulation execution agent. The simulation test platform includes three networked dc-motor subsystems with the same control parameters. The state-space description of the subsystem can be expressed by (10) [24], and the corresponding motor parameters are shown in Table 1.   and The reference control signal inputs of the three control loops during three transmission periods are as shown in Figure 5, and the corresponding sensitivity of the control signals is as shown in Table 2. Substituting each group of data into equation (3) gives the utility function of the corresponding control loop at each stage. The remaining parameters are set as follows: ω R = 0.1, T O = 0.5 s, T P = 0.03 s, and the remaining time is T T . Among them, the bandwidth allocation period T A is very small relative to the polling period T P and can be ignored. The evaluation of SD-NCCS system performance can be jointly described by all control loop subsystems. The performance of each control loop subsystem can be measured by the integrated absolute error (IAE) [25]. The IAE for the control loop i can be expressed as It can also be directly expressed by controlling the output instantaneous error, and (12) is corrected to: where t d = d t, t d ∈ t j , t j+1 is the sampling time of the output signal measurement, d is the index of the sampling point, and t is the sampling interval. D represents the total number of sampling points and can be set to D = 100. y nom,j (t d ) is the output of the control loop i measured at t d without time delay. y act,j (t d ) is the output of the control loop i measured at the t d after the scheduling policy is adopted in the case of time delay.

B. RESULT ANALYSIS
The initial population of the Network Pricing-based Bandwidth Allocation (NPBA) method is 100. After 10 generations of evolution, the overall fitness is quickly converged and stabilized, and reaches the Nash equilibrium.  Figure 6. The detailed numerical results are shown in Table 3 and IV.
As shown in Figure 6, generally speaking for independent subsystems, the less bandwidth allocation, the worse the control performance will be. Since the EBA uses the average bandwidth method, each control loop maintains the same bandwidth allocation for different transmission periods, so there is a case of excess bandwidth in a certain period. Loop 2 during period 2 for example, the average sensitivity of the loop 2 is 1.00, and only a small amount of bandwidth is required to maintain the stable output of this subsystem. However, the result of a large amount of bandwidth wasted VOLUME 8, 2020 by this control loop is the lack of bandwidth of other control loops, which ultimately leads to a decline in overall control performance. In contrast, although the individual subsystems using NPBA are inferior to the corresponding subsystems using EBA in a certain transmission period, such as the first transmission period, the IAE of the control loop 1 and 2 using NPBA are higher than EBA. However, the overall IAE of EBA is higher than that of NPBA, indicating that the SD-NCCS using the proposed strategy provides satisfactory performance.

V. CONCLUSION
This paper presents a software-defined networked collaborative control system from the perspective of collaborative design of control and scheduling. Under the framework of the SD-NCCS, a dynamic bandwidth scheduling strategy based on the game theory and network pricing mechanism is proposed, which transforms the network resource allocation problem into the Nash equilibrium solving problem. The biggest difference from the existing research results is that the proposed method forces the control subsystems to work at the Nash equilibrium dynamically, and the overall performance of the SD-NCCS with limited resources is improved. The simulation results demonstrate the effectiveness and availability of the proposed method. In addition, as the control plane and data plane are separated in SD-NCCS, a centralized algorithm for reaching the NE can be handed over to the SD-NCCS controller, which can be realized by an application on it. Therefore, in terms of complexity, comparing to the decentralized methods aforementioned, it is simpler and more operational. As it comes to the cost, more spending may be saved supposedly. At last, further study and implementation of this research work in a realistic software-defined networking environment is practical importance and is part of our current and future work. He is currently a Chief Technology Officer with Zhejiang Yi Duan Machinery Company. His research interests include intelligent CNC metal forming machine, metal forming, and wheel equipment. VOLUME 8, 2020