QROUTE: An Efficient Quality of Service (QoS) Routing Scheme for Software-Defined Overlay Networks

Many computer network applications impose constraints for multiple quality of service (QoS) metrics, e.g., delay, packet loss, bandwidth, and jitter. These QoS constraints cannot be guaranteed by the Internet due to its best-effort service model. Overlay networks have been an effective technique at the application layer to support multiple QoS constraints of networking applications. In software-defined overlay networks, software-defined networking (SDN) paradigm is introduced in the overlay networks to enable centralized and efficient routing of traffic in the overlay networks, thus, enabling better QoS. One of the main challenges in software-defined overlay networks is the fast-changing overlay link QoS characteristics. However, the existing routing algorithms for satisfying multiple QoS constraints in software-defined overlay networks involve high route computation time and thus these routing algorithms cannot adapt to the fast-changing overlay link QoS characteristics. Moreover, as we scale the size of overlay networks, the size of forwarding tables increases exponentially. This is because the existing routing schemes for ensuring multiple QoS constraints use both the source and the destination address for data-plane forwarding. This leads to pushing a huge amount of forwarding table entries by the controller through the network and thus limiting the size of the overlay network. We propose an efficient routing scheme, QROUTE, for satisfying multiple QoS constraints in software-defined overlay networks. QROUTE consists of a control plane routing algorithm which has significantly low route computation time because of employing a novel directed-acyclic-graph (DAG) based approach. QROUTE also reduces the forwarding entries in the data plane by using a QoS-metric-based forwarding scheme. We extensively evaluate QROUTE using traces from a global overlay service provider. We also examine QROUTE on a testbed of P4-BMv2 switches controlled by the ONOS controller using P4Runtime protocol. We find that QROUTE outperforms other state-of-the-art QoS routing schemes in route computation time, size of the forwarding tables and meeting the QoS requirements of various applications.


I. INTRODUCTION
Many computer network applications such as video conferencing, interactive gaming, VoIP, virtual reality, telepresence, video-on-demand and live video streaming impose constraints for multiple quality of service (QoS) metrics, e.g., delay, packet loss, bandwidth, and jitter. This is usually The associate editor coordinating the review of this manuscript and approving it for publication was Muhammad Imran . expressed by a list of minimum/maximum bounds for each QoS metric and is commonly referred to as a QoS policy. For example, in [2] the QoS policy used for a 384-kbps video conferencing session is (150ms, 30ms, 460 kbps, 1%). The tuple is in the format (maximum delay bound, maximum jitter bound, minimum bandwidth requirement, maximum packet loss). These QoS constraints cannot be guaranteed by the Internet due to its best-effort service model. Overlay networks have been an effective technique at the application layer to VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ support multiple QoS constraints of networking applications [3]- [7]. An overlay network is a virtual network over an existing physical network and is composed of software routers running on commodity servers connected using tunneling protocols like Virtual Extensible LAN (VXLAN) and Generic Routing Encapsulation (GRE). In software-defined overlay networks, software-defined networking (SDN) paradigm is introduced in the overlay networks to separate the control from the forwarding. This enables centralized and efficient routing of traffic in the overlay networks and thus helps achieve better QoS [8]- [20].
One of the main challenges in software-defined overlay networks is the fast-changing overlay link QoS characteristics because of various reasons like the change in the corresponding path in the physical network or the change in non-overlay back-ground Internet traffic [14]. However, the existing routing algorithms for satisfying multiple QoS constraints in software-defined overlay networks involve high route computation time and thus these routing algorithms cannot adapt to the fast-changing overlay link QoS characteristics [8]- [20]. Moreover, as we scale the size of overlay networks, the size of forwarding tables increases exponentially. This is because the existing routing schemes for ensuring multiple QoS constraints use both the source and the destination address for data-plane forwarding [8]- [20]. This leads to pushing a huge amount of forwarding table entries by the controller through the network and thus limiting the size of the overlay network [15].
The problem of finding the optimal routes that meet multiple QoS constraints is known to be NP-hard. Therefore, several polynomial time heuristics algorithms [8]- [30] have been proposed in the literature to find approximate solutions. However, none of them can efficiently reduce both the route computation time and number of routing entries while ensuring additive, multiplicative and concave QoS constraints.
To address the above limitations, we propose an efficient routing scheme, QROUTE, for satisfying multiple QoS constraints in software-defined overlay networks. We make the following contributions in the paper.
• DAG-based primary routing algorithm: QROUTE consists of a control plane routing algorithm which has significantly low route computation time because of employing a novel directed-acyclic-graph (DAG) based approach.
• QoS-metric-based forwarding: QROUTE reduces the forwarding entries in the data plane by using a QoSmetric-based forwarding scheme.
• Backup DAG generating algorithm: To improve resiliency, we generate a backup DAG using a topological sorting based algorithm in our QROUTE routing scheme. During failures, we use the backup DAG along with the original DAG to achieve a considerable fraction of QoS while ensuring loop-free routing.
• Framework to evaluate path's conformance to QoS policy: QROUTE also includes a framework to evaluate a path's conformance to a multi-constrained QoS policy. In the absence of paths that completely satisfy a given QoS policy, this framework enables the network operator to find paths that slightly deviate from the QoS policy.
• Trace-based evaluation: We extensively evaluate QROUTE using traces from a global overlay service provider.
• Evaluation on P4 switches: We also examine QROUTE on a testbed of P4-BMv2 switches controlled by the ONOS controller using the P4Runtime protocol. We summarize our contributions in table 1 along with their respective benefits. We find that QROUTE outperforms other state-of-the-art QoS routing schemes in route computation time, size of the forwarding tables and meeting the QoS requirements of various applications.
In [1], we introduce our DAG-based primary routing algorithm and QoS metric-based forwarding and present its trace-based evaluation. In this paper, we also include a backup DAG generating algorithm and a framework to evaluate a path's conformance to QoS policy. We also include rigorous mathematical proofs to support our theory in this paper. This paper also includes the evaluation of QROUTE on a testbed of P4-BMv2 switches controlled by the ONOS controller using the P4Runtime protocol. We also present our evaluation of QROUTE on the topologies from the Internet Topology Zoo dataset [31] in this paper.
We organize the paper as follows. In section II, we give an overview of our system framework and our QROUTE routing scheme and mathematically formulate our QoS routing problem. Section III describes the control-plane of our QROUTE routing scheme. We explain our QROUTE data-plane in section IV. Section V provides the details of our framework for evaluating a path's performance. We present the results from a trace-based and a real test-bed evaluation of QROUTE in Section VI. Section VII summarizes the related work. The paper ends with a conclusion in section VIII.

II. PROBLEM FORMULATION
This section explains our system framework, an overview of QROUTE and the mathematical formulation of our QoS routing problem.
A. SYSTEM FRAMEWORK Our system framework consists of an overlay network connecting end-users to content servers and other end users as shown in figure 1. The overlay network is created using software routers deployed across multiple data centers all over the world. The routers are connected using an overlay tunneling protocol. A DNS system chooses the best ingress software router for an end-user. A hierarchical design multi-controller architecture [32] is used to handle more flows and reduce control plane latency in large networks. The entire overlay network is divided into domains where routing within each domain is managed completely by a domain controller. The root controller is required to synchronize between the domain controllers for setting routes for flows that originate and terminate at different domains. Various applications demand different kinds of QoS from the overlay network. A QoS requirement of an application is represented by a QoS policy, which is defined by a list of permissible ranges for the end to end delay, jitter, packet loss, and bandwidth. The bandwidth requirement for a QoS policy is determined based on the application and also the number of users using that application averaged over time. The link-wise delay, jitter, packet loss, and available bandwidth are monitored and reported to the domain controller using traffic measurement tools. With this information, the routing algorithm at each domain controller generates routing Directed Acyclic Graphs (DAGs) for all the QoS policies and destination routers in its domain. A DAG stitching algorithm stitches together the routing DAGs in different domains to create a global routing DAG. For simplicity, we omit the DAG stitching algorithm from this paper and consider only a single controller for the entire network.

B. OVERVIEW OF QROUTE
In this section, we illustrate the control-plane and data-pane of the QROUTE routing scheme using an example.
We first provide an example of a primary routing DAG generated using our QROUTE control-plane routing algorithm. Consider the network graph in figure 2 where each edge between the nodes is bi-directed and the pair of value on each link represents the delay (ms) and jitter (ms) of the overlay link. For simplicity, we are considering here only two metrics but this can be extended to other metrics as explained in later sections. We do not depict the cost of the links in the figure. Given a QoS policy, we generate a routing DAG for every destination router instead of generating routes for every source-destination pair. In this example, we consider a QoS policy (40ms, 10ms) where the pair of values are maximum bounds for the delay (ms) and the jitter (ms) respectively. The dashed arrows in figure 2 depict the routing DAG generated using our QROUTE algorithm for the QoS policy (40ms, 10ms) and destination R8. From any node in the graph, the DAG contains at-least one path to the destination R8 that satisfies the QoS policy (40ms, 10 ms). Using breadth-firstsearch, we reduce the total number of route computations for a given destination router and QoS policy from O(n) to O(l) The primary routing DAG generated by QROUTE for destination R8 and policy (40ms, 10ms) (represented by dashed arrows). All the links between network nodes are bi-directed. Each pair of values represents the delay (ms) and jitter (ms) of the corresponding link. The cost of the link is not indicated. (b) The paths generated from every source to destination R8 using the A*Prune algorithm. The number of route computations in path-based routing is 7 while QROUTE only performs 3 route computations. VOLUME 8, 2020 where n is the number of nodes in the network and l is the number of leaves in a breadth-first-search traversal from the destination router. For m QoS policies and all destinations, the number of route computations reduces from O(m × n 2 ) to O(m × n × l). In figure 2b, we depict the minimum cost paths generated by the A*Prune algorithm from every source to destination R8 that satisfy the QoS policy (40ms, 10ms). We can observe that for the given policy (40ms, 10ms) and destination router R8, the number of route computations using A*Prune is 7 while the number of route computations using QROUTE is only 3. Any path-based QoS routing algorithm like H_MCOP and MH_MCOP will also lead to the same number of route computations as that of A*Prune. QROUTE is thus useful for quickly computing routes in an overlay network with fast-changing traffic conditions.
To reduce the number of forwarding entries and to ensure that the packets for an application are forwarded along a route on the DAG that preserves the QoS policy of the application, we introduce a QoS-metrics-based forwarding scheme instead of forwarding based on source and destination IP address. We illustrate our QoS-metrics-based forwarding through the same example in figure 2. Table 2 shows the forwarding entries in router R4 for QoS policy (40ms, 10ms) and destination router R8. The policy (40ms, 10ms) is assigned an ID 1. We replace the source IP in the forwarding entries with the maximum amount of delay and jitter a packet can go through to be eligible to be forwarded to the corresponding outgoing link. These routing entries capture the QoS information of paths down the outgoing links. The packets carry the elapsed value of delay and jitter in their packet headers which are updated by each router by adding the delay and jitter of the outgoing link to the respective header field. These values are matched against the corresponding keys to determine where the packets should be forwarded to. For example, we can observe in the first entry in table 2 that a packet with QoS policy (40ms, 10ms) and destined to R8 can be forwarded to node R5 if the elapsed delay and the elapsed jitter of the packet is less than 4 ms and 8 ms respectively. We significantly reduce the forwarding entries by grouping them if they have the same next hop. Thus, we reduce the number of forwarding entries from O(m × n 2 ) to O(m × n × deg) where deg is the maximum degree of the network graph. In the given example, we reduce the number of forwarding entries from 16 to 9 for the QoS policy (40ms, 10ms) and destination router R8. Thus, QROUTE alleviates the problem of the proliferation of routing entries in large overlay networks.

C. MATHEMATICAL FORMULATION
In this section, we provide a mathematical formulation of our QoS routing problem. Our QoS routing problem entails finding a directed acyclic graph (DAG) for every QoS policy and destination router such that from any node in the graph, the DAG contains at-least one path to the destination router that satisfies the QoS policy.
Network representation: A network graph G = (V , E), where V is a set of overlay routers and E is the set of overlay tunnels.
Cost of edges: The column-vector of ''costs'' of the edges is denoted by c, c ∈ R |E| + . The ''cost'' denotes the price for using a link to deliver traffic and is determined by business relationships.
QoS policy: Let d, j, b and l, {d, j, b, l} ∈ R + , denote the bounds for delay, jitter, bandwidth, and packet loss probability respectively. QoS routing ''main problem'': Our QoS routing ''main problem'' involves finding a least-cost DAG for a destination node r and a QoS policy Q = (d, j, b, l) such that the DAG has a path from every node s in the graph to the node r which satisfies all the constraints given in the QoS policy Q. We generate this routing DAG by composing the routes returned from the ''sub-problem'' as explained in section III-A2. We generate such routing DAGs for all destination nodes and QoS policies.
''Sub-problem'': We define the ''sub-problem'' as finding a path of minimum cost from a source vertex s ∈ V to a destination vertex r ∈ V which satisfies all the QoS constraints.
Vertex-edge incidence matrix: Let H denote the |V |×|E| vertex-edge incidence matrix such that for all e = (u, v) ∈ E where u, v ∈ V , H ue = 1 and H ve = −1. Additionally, H we = 0 for any w = u, v. Let K , K ∈ R |V | + , be a vector such that K s = 1, K r = −1 and K v = 0 for all v ∈ V \{s, r}.
Decision variables: Let y be a column-vector (y 1 , y 2 . . . y |E| ), y i ∈ {0, 1} of decision variables where y i = 1 if the edge i belongs to the final routing path and 0 otherwise. The column vector y represents a path returned by the QROUTE routing algorithm.
We describe the ''sub-problem'' using the following optimization formulation.
The constraint (1a) restricts the value of y to a particular directed path from s to r. Constraints (1b) and (1d) ensure the additive metrics (delay and jitter) of the paths to be below their respective maximum bounds. Constraint (1c) checks if the product of success probability of the links of the paths is greater than the minimum bound on the success probability. Finally, the constraint (1e) limits the bottleneck bandwidth of the path to be greater than the minimum bound on the bandwidth. The ''sub-problem'' in equation 1 represents the formulation for an online QoS routing problem which adds the route for a single flow to a network such that the QoS requirements of the new and the existing flows are maintained. We use the solution from this ''sub-problem'' to find the solution for our QoS routing ''main problem'' of generating routing DAGs for all destination routers and QoS policies.
To reduce the route computation time by availing the fast algorithms available for solving online QoS routing problems, we do not model the problem as a large and complex optimization problem that considers all the traffic flows to comprehensively determine the routes.

III. QROUTE CONTROL PLANE
In this section, we explain our QROUTE control-plane routing algorithm that generates the primary routing DAG for every destination router and QoS policy. We also explain our control-plane algorithm which is used to generate a backup DAG for each primary DAG.

A. PRIMARY ROUTING DAG
Our QoS routing problem entails finding a primary routing DAG for every QoS policy and destination router such that from any node in the graph, the DAG contains at-least one path to the destination router that satisfies the QoS policy.
To compute a primary routing DAG per destination that satisfies all the constraints in a QoS policy, we decompose this problem to finding routes from a subset of nodes in the graph to the destination that satisfies all the QoS constraints. The task of finding every such route is referred to in this paper as the ''sub-problem''. We optimize this ''sub-problem'' to a polynomial-time algorithm using a Lagrange-relaxation-based technique as described in section III-A1. In section III-A2, we explain how we generate a routing DAG by composing the routes returned from the ''sub-problem''. In section III-A3, we describe how the bandwidth requirements of different QoS policies are ensured in our routing algorithm.

1) OPTIMIZING THE ''SUB-PROBLEM''
To find a near-to-optimal solution for a multi-constrained shortest path (MCSP) problem with only additive constraints, the Lagrange relaxation-based aggregated cost (LARAC) algorithm [33] is found to achieve one of the best performances [24]. To take advantage of the LARAC algorithm, we reduce our ''sub-problem'' described in equation 1 into a linear integer programming problem by converting the additive, multiplicative and concave constraints to only additive constraints. The concave constraint bandwidth was eliminated from the network graph by removing links that do not meet the bandwidth constraint of the QoS policy. For more details about how we meet bandwidth requirements, refer to section III-A3. We accomplish this pruning by setting delay and jitter on those links to infinity and packet loss probability to 1. The multiplicative constraint, packet success probability, is simplified into an additive constraint through the negative logarithm of packet success probability of each link. The routing ''sub-problem'' then reduces to the following: We relax the inequality constraints 2b-2d, by inserting the degree of violation of these constraints and their corresponding Lagrange variables into the objective function. Let λ 1 , λ 2 , λ 3 ∈ R + be the Lagrange multipliers for constraints 2b,2c, and 2d, respectively. Thus, the following Lagrange dual problem reduces from the above integer programming problem.
where LR(λ 1 , λ 2 , λ 3 ) is the Lagrangian dual function which is optimized subject to the non-dualized constraint. (Equation 4) The Lagrangian dual function 4 is reordered into the form: Since λ 1 d + λ 2 l + λ 3 j is a constant, the value of y corresponding to the optimal solution of the Lagrangian dual function 5 is equal to that of Lagrangian dual function 6, which is equivalent to the shortest path between s and r with the cost of the links given by the vector c+λ 1 D+λ 2 L +λ 3 J .
To find the shortest path for given node pairs, we can use single-source shortest path algorithms such as the Fibonacci based Dijkstra (with time complexity of O(E + VlogV )) or the faster A* algorithm.
Using a sub-gradient descent algorithm [34], we explore the dual problem's solution space. To start, we compute a solution path for the Lagrange dual problem (Line 9, Algorithm 1). If this solution is feasible, the cost of this path is set as the upper bound for the sub-gradient descent algorithm (Line 15, Algorithm 1). If this solution is non-feasible, we set the upper bound to the value stated in Theorem 1. During iteration, if a feasible path is discovered, the upper bound is updated using the cost of the found feasible path. The feasibility of a path is determined in Line 11, Algorithm 1. Theorem 2 is used to test if the solution to the Lagrangian dual function problem is optimal for the original problem or not (Lines 12, Algorithm 1).
Our scalar step size θ is given by The difference of the current upper bound (UB) and the current lower bound (LB) and the scaling factor (Dy sr −d) 2 + (L y sr − l ) 2 + (Jy sr − j) 2 informs the step size. Lines 23-25 of Algorithm 1 update the Lagrange multipliers and the dual function is then re-solved using the new set of multipliers. If a predefined number of iterations or the optimality condition (Theorem 2) is met, the algorithm will terminate.   (a) y sr is feasible for the original problem (b) cy sr = [cy sr +λ 1 (Dy sr −d)+λ 2 (L y sr −l )+λ 3 Proof: This follows trivially from the Lagrangian sufficiency Theorem for inequality constraints.

2) GENERATING A ROUTING DAG USING THE ''SUB-PROBLEMS''
In this section, we elaborate on our approach for generating a directed-acyclic-graph (DAG) for a specific destination router and QoS policy by combining the solutions of ''sub-problems''.
For additive QoS metrics, we found that the feasible routes between an ingress router and an egress router also contain feasible routes between intermediary routers and the same egress router. Thus, if we compute feasible routes for only a subset of nodes, we can achieve full graph coverage.
In our approach we are trying to reduce the subset of nodes for which we need to compute a feasible path. We start by reversing the links in the original graph and then perform a breadth-first search (BFS) from the destination router (Lines 1-2 of Algorithm 1) while storing the nodes in decreasing order of distance (in hops) from the destination router. Then using this order, we find a feasible path between each router to the destination router. The further a router is from the destination, the more intermediary routers it will contain. We terminate the algorithm once all the nodes in the graph are covered (line 5&27, Algorithm 1). This approach significantly reduces the route computation time for the QROUTE algorithm. The BFS for all the destination routers can be done offline since it does not change with the changing traffic conditions. Only when the topology changes, we will need to re-perform BFS.
We illustrate routing DAG generation of QROUTE using the same example mentioned in section II-B with more detail. As explained in section II-B, the dashed arrows in figure 3a depicts the routing DAG generated using algorithm 1 for the QoS policy (40ms, 10ms) and destination R8. In this example, we perform breadth-first-search (BFS) traversal from the destination router R8 and arrange the nodes in the decreasing order of their distance from router R8. We then use algorithm 1 to compute feasible routes between routers R3-R8, R1-R8 and R2-R8 in the given order. By combining these routes, we create a routing DAG for the entire graph. Note that route generated for one pair of routers can be used for other pairs of routers if the QoS constraints are not violated. To ensure proper forwarding in data plane that ensures QoS constraints, we introduce QoS-metrics-based forwarding as explained in section IV-A.
Using breadth-first-search, we reduce the total number of route computations for a given destination router and QoS policy from O(n) to O(l) where n is the number of nodes in the network and l is the number of leaves in a breadth-first-search traversal from the destination router. For m QoS policies and all destinations, the number of route computations reduces from O(m × n 2 ) to O(m × n × l). QROUTE is thus useful for quickly computing routes in an overlay network with fast-changing traffic conditions.

3) MEETING THE BANDWIDTH REQUIREMENTS
In this section, we explain how we meet the bandwidth requirements for different applications. For example, the QoS requirement for a single video streaming application used in our experimentation is (2000ms, 80ms, 0.5Mbps, 5%). We multiply the bandwidth requirement of an application by the number of flow requests corresponding to that application in a 30 second time interval. This time interval corresponds to the time between the last two route updates in the data plane. For example, if the number of flow requests corresponding to a video streaming application during that time interval is 500, we prune the links which have bandwidth less than 0.5 × 500 = 250 Mbps before computing DAGs for the video-streaming applications. The number of flows of a particular application in a given time interval is calculated by the controller using overlay header information of the flows sent by the ingress router to the controller when the first packet of a flow arrives in the network.
Once the routing DAG for a particular application and all destinations is calculated using algorithm 1, we subtract the bandwidth requirements of this application from the available bandwidth of the network. For example, we subtract 250 Mbps from the available bandwidth of the links used in the DAGs for the video-streaming applications. We, then, find the routing DAGs for other applications. This ensures that the bandwidth requirements of all the applications are met.

B. BACKUP ROUTING DAG
In this section, we describe the algorithm for generating a backup DAG which we use along with the primary DAG to achieve a considerable fraction of QoS while ensuring loop-free routing.
Whenever an outgoing link is down or having congestion, the router should be able to forward packets to a backup link instead of incurring the delay of contacting the controller and waiting for the controller to install new rules. Installing backup paths for all pairs of source, destination and policy will again lead to the proliferation of routing entries and thus we need to generate a backup DAG for every destination router and policy that can be used during failures. The primary DAG should be as disjoint as possible to the backup DAG so that the congested or failed links of primary DAG do not affect the backup DAG. Moreover, using backup DAG, we should be able to provide sufficient QoS even during failures.
We use algorithm 2 to generate back-up DAG for the primary DAG generated by Algorithm 1. Algorithm 2 takes as input the primary DAG (P_DAG) and the adjacency list of the network graph (Adj_G) and generates a backup DAG (B_DAG) using the topologically sorted list of the P_DAG (t_sort) and the mechanism described below. We use an example to illustrate Algorithm 2. For the primary DAG in Fig. 3a, the backup DAG is shown in Fig. 3b. Algorithm 2 first sorts the nodes of the primary DAG using topological sort (line 2 of algorithm 2). A topological sort for a primary DAG in Fig. 3a is shown in Fig. 4. For a node, we do not use those neighboring nodes which lie before in the topologically sorted order (marked with a minus sign in Figure 4) as the next hop in the backup DAG. This was done so that the packets do not enter a forwarding loop while traversing links of both the primary and backup DAGs.For example, node R1 and R4 cannot serve as the end if 20: end for end procedure next hop for node R5 in the backup DAG. We also avoid those nodes as next hops in backup DAG which are chosen as the next hops in the primary DAG (marked with a tilde sign in Figure 4). This was done to make the backup DAG as disjoint as possible to the primary DAG. This constraint is relaxed if there is no candidate for the next hop. For example, node R8 is avoided as the next hop for node R5 in the backup DAG. We then choose one of the remaining neighbors as the next hop in the backup DAG (marked with an asterisk sign in Figure 4). In the example, nodes R6 is used as the next hop for node R5 in the backup DAG. The backup DAG for destination router R8 is shown in Fig. 3b.

IV. QROUTE DATA PLANE
In section IV-A, we explain our QoS-metrics-based forwarding which reduces the forwarding entries in the data plane while ensuring multiple QoS constraints. In section IV-B, we describe our data plane forwarding algorithm which uses the primary DAG entries along with the backup DAG entries to ensure QoS.

A. REDUCING FORWARDING ENTRIES IN THE DATAPLANE
We reduce the forwarding entries in the data-plane by using QoS-metric-based forwarding. We replace the source address in the forwarding entries of the switches with the maximum amount of QoS metrics a packet can spend in the network to be eligible to be forwarded to the corresponding outgoing link. We illustrate the QoS-metric-based forwarding using an example. Consider the network graph provided in Fig. 3. For simplicity each pair of values on the network links only represents delay and jitter for that bi-directed link measured in milliseconds. The following approach will hold for any number of additive and multiplicative constraints. Figure 4 shows the DAG for a destination router R8, with a QoS policy of (40ms, 10ms) where 40ms is the maximum bound for delay and 10ms is the maximum bound for jitter.
Packets from nodes R1, R2, and R3, arriving at R4 should be forwarded to R5, R6, and R7 respectively to maintain QoS requirements for each flow. For router R4 to route properly, one would need to utilize a source IP address, destination IP address, and policy ID as the key in forwarding tables or use a different key that satisfies the above requirement. However, using the source IP address leads to a proliferation of routing entries in the forwarding tables.
We replace the source IP in the routing entries of the forwarding table with the maximum amount of delay and jitter a packet can go through to be eligible to be forwarded to the corresponding outgoing link. The packets carry the elapsed value of delay and jitter in their packet headers which are updated by each router by adding the delay and jitter of the outgoing link to the respective header field. These values are matched against the corresponding keys to determine where the packets should be forwarded to.
For example, Table 3 depicts the forwarding table of router R4 with the forwarding entries corresponding to destination router R8 and QoS policy (40ms, 10ms). The delay and jitter along the path R4 − → R5 − → R8 is 36ms and 2ms respectively. Thus, if the maximum permissible delay and jitter that a packet has gone through before reaching R4 are 4ms (40-36) and 8ms(10-2) respectively then the packet can be forwarded to the router R5. This corresponds to the first entry in the forwarding table (Table 3). For the packets going from ingress router R1 to R4 directly, the elapsed delay and jitter will be 2ms and 7ms respectively. We can see that only the packet which is coming form R1 to R4 directly is eligible for forwarding to R5. Similarly we generate other forwarding entries for other feasible routes generated by algorithm 1 and passing through router R4. If any incoming packets headed for R8 do not match any routing entry or there is a failure, they will be routed through the default entry for policy 1 (represented by the 4th entry in Table 3). These default entries are created using the backup DAGs as described in Section III-A-D.
We significantly reduce the forwarding entries by grouping them if they have the same next hop. Given a QoS policy and destination, we have only one entry corresponding to an outgoing link which is used in a feasible route generated by Algorithm 1 and passing through that router. Thus, the maximum number of forwarding entries a router can have for a given QoS policy and destination is its degree in the network graph. In contrast, a source-destination based forwarding can have a forwarding entry for every source router. Thus, for a network of n nodes and m QoS policies, we reduce the number of forwarding entries from O(m × n 2 ) to O(m × n × deg) where deg is the maximum degree of the network graph. Thus, QROUTE alleviates the problem of the proliferation of routing entries in large overlay networks.

B. DATA-PLANE FORWARDING ALGORITHM
Our data plane forwarding algorithm uses the primary DAG entries along with the backup DAG entries to ensure QoS and provide resiliency. Algorithm 3 presents the forwarding algorithm of the switches. The switches first try to find a rule in the forwarding table corresponding to the primary DAG which matches all the QoS constraints. If it finds such VOLUME 8, 2020 Algorithm 3 Data Plane Forwarding Algorithm procedure FORWARDING(Adj_G, Q_DAG) 1: for rule in Primary_DAG do 2: if elapsed_delay ≤ rule max_elapsed_delay && elapsed_jitter ≤ rule max_elapsed_jitter && elapsed_pkt_loss ≤ rule max_elapsed_pkt_loss then 3: if rule output_port is up then 4: add_delay(rule output_link_delay ) 5: add_jitter(rule output_link_jitter ) 6: add_pkt_loss(rule output_link_pkt_loss ) 7: send_packet(rule output_port ) 8: end if 9: end if 10: end for 11: send_packet(Backup_DAG_output_port) {If no rule matches in Primary_DAG or the output ports for the matched rules are down} end procedure entry and the corresponding output port is up, the switch adds the QoS values of the forwarding link to the respective QoS values in the packet header and forwards the packet on the corresponding output port. If the switch does not find any entry or the output ports for the matched rules are down, it forwards the packet on the outgoing link along the backup DAG if it is up. If the outgoing link along the backup DAG is down, the switch drops the packet. Fig. 3c shows the path followed by a packet in case of failures of two links. We can see from the figure that the packet traverses links of both the primary and the backup DAGs. The QoS is maximized by trying to re-route the packets to the primary DAG even if the packet has arrived using the backup DAG link.

V. FRAMEWORK FOR EVALUATING PATH PERFORMANCE
Some applications are not stringent about meeting their QoS requirements and thus the paths which satisfy their QoS policies to a reasonable fraction can be accepted. Moreover, in scenarios of network congestion or failures, the routing algorithm might not be able to find a path that satisfies the QoS policy for a particular application and thus the routing algorithm has to return a path that partially satisfies the QoS constraints. In such cases, the network manager needs to decide the degree to which the QoS should be satisfied for a path. We propose a framework to evaluate a path's performance in terms of meeting a multi-constrained QoS policy. Under the absence of paths that completely satisfy a given QoS policy, this framework enables the network operator to use the paths which slightly deviate from the QoS policy.
We represent each QoS constraint by a function which takes a value between 0 and 1. If the value of a particular QoS metric for a path returned by the routing algorithm is within its bound, then the function takes a value of 1. If it is outside its bound, then it takes a value less than 1. Beyond a certain threshold, the function takes a value of 0 which denotes that the QoS metric deviates a lot from its bound.
As an example, the QoS requirement used for video streaming in our experiments is (2000ms, 80ms, 0.5Mbps, 5%). The tuple is in the format (maximum delay bound, maximum jitter bound, minimum bandwidth requirement, maximum packet loss). Their respective functions are defined in Fig. 5. We can observe from Fig. 5a that the function for delay takes the value of 1 when the delay of a path is less than 2000ms. After that the function gradually drops to a value of 0. This function indicates the degree to which the delay metric is satisfied. Figure 5b to 5d can be interpreted along similar lines. A steeper slope for a QoS metric function indicates that the application is more stringent about meeting that QoS requirement. This slope is decided by the network operator and requires domain knowledge.
The vertical lines in the graphs denote the value of the corresponding QoS metrics for a path returned by the routing algorithm. The degree to which a path satisfies the QoS policy is equal to the smallest function value amongst all the QoS constraints. For example, in Fig. 5 the minimum function value is min(.78, 0.58, 0.38, 1) = .38. Thus this path satisfies the QoS policy by 38%. If the threshold for accepting a path for routing is 60%, this path will be rejected.

VI. EVALUATION
In this section, we describe the evaluation of the QROUTE routing scheme using traces from a global communications technology solutions provider's real overlay networks. We also explain the evaluation of QROUTE carried out in a test-bed of P4-BMv2 [35], [36] switches controlled with ONOS SDN controller [37] using P4Runtime [38]. All experiments were carried out in servers with Intel Xeon E5-1630 v4 @3.7GHz processors.

A. TRACE-DRIVEN EVALUATION
We determine the scalability and optimality of the QROUTE routing algorithm using trace-driven evaluation. We had access to 24-hour traces with measurements after every 30 seconds from a global communications technology solutions provider's real overlay networks. These traces consist of delay, jitter, packet loss, and bandwidth measurements of the links between the routers. The network topology the traces measured was comprised of routers spread across multiple data centers worldwide. The traces provided covered different sizes of overlay networks with different topologies (partial mesh, star, ring, and tree). We also perform an evaluation of QROUTE on the real topologies from the Internet Topology Zoo dataset [31] using synthetically generated traces. The delay between the network nodes in the topologies from the Internet Topology Zoo dataset was estimated based on their geographical locations. The bandwidth, jitter and packet loss values of the links in the synthetic generated traces were derived from the above traces. We wrote the QROUTE algorithm in Python using fast libraries like Dijikstar and used a CSV file of the traces for input.  For our evaluations, we select 10 different kinds of userapplications' QoS requirements as mentioned in Table 4.
The resulting plots are informed by the average value of 28,800 (24×60×2×10) instances of data points. Figure 6 shows the distribution of QoS metric values of the links in the traces using a box plot.
Using these traces we compared the scalability and optimality of the QROUTE algorithm to the current state-of-the-art multi-constrained shortest path (MCSP) algorithms, like A* Prune [21], H_MCOP [22], and MH_MCOP [23].
We measured four performance characteristics: • Percentage Reduction in Forwarding Entries: The percentage reduction in forwarding entries compared to those generated by source-destination-based routing algorithms such as A*Prune, H_MCOP, and MH_MCOP.
• Percentage Cost Deviation: Indicates the percentage deviation of the cost of the approximate solution with that of the solution from A*Prune which is the optimal case.
• Route Computation Time: This includes generating the routes for all the destination routers and policies and the forwarding tables for the routers. Route computation time is measured in seconds.
• Relative percentage difference:The relative difference between the maximum QoS bound and the QoS of the paths found. The formula is: (QoS_bound − path_QoS)/QoS_bound × 100.
We obtain Fig. 6b, 6c, 6d and 6e with partial mesh topologies and Fig. 6f with networks of size 1000. From Fig. 6b we can observe that as the network size grows, QROUTE achieves significant reduction in forwarding table sizes compared to other source-destination based routing algorithms. As depicted in Fig. 7b, we can observe the similar reduction in forwarding entries by QROUTE on the topologies from the Internet Topology Zoo dataset. This demonstrates the impact of replacing the source IP address with maximum elapsed QoS constraint values.
From Fig. 6c and Fig. 7a, we observe that QROUTE achieves a much lower route computation time than the other algorithms, like A*Prune, H_MCOP, and MH_MCOP. This is because of generating DAGs as described in Section III-A2. This reduction in time complexity comes at a cost of solution optimality, and as a consequence, H_MCOP and MH_MCOP have much more optimal solution outcomes than QROUTE as depicted in Fig. 6d and Fig. 7c. However, unlike H_MCOP and MH_MCOP, QROUTE will always return a feasible route if one exists. If overlay providers do not consider a small increase in the cost of overlay links as a significant issue and desire a quick adaptive route computation algorithm for dynamic traffic, QROUTE is the ideal choice. Fig. 6e shows the comparison of QROUTE and A*Prune in terms of relative percentage difference of delay QoS metric. For this comparison we did not include H_MCOP and MH_MCOP as they do not always generate a feasible path. We see that the relative percentage difference of delay of QROUTE increases quicker than that of A*Prune as the network size increases. This is because the feasible paths from some nodes to a destination node are contained in already computed routes.
QROUTE is well suited to ring, tree and mesh topologies and has fast route computations compared to that in the star topology (Fig. 6f). This is because the star topology does not benefit from our heuristic of a feasible route between two routers covering the feasible routes of many intermediary routers.

B. EVALUATION ON P4 SWITCHES
In this section, we describe our evaluation using a test-bed of P4-BMv2 [35], [36] switches controlled with ONOS SDN controller [37] using P4Runtime [38]. We evaluate the overhead incurred due to the addition and the range operation in the switches on the end-to-end delay. We also evaluate the degree to which the QoS requirements of the flows were satisfied as we increase the traffic in the network. We also measure the impact of failures on the QoS.

1) TEST-BED DESCRIPTION
P4 language [35] allows us to define custom forwarding behavior of a switch. Any forwarding behavior is composed of matches on header fields and based on that taking an action like changing header fields, forwarding on an output port, packet drop, etc. P4 allows us to define custom header fields and custom actions that are not supported on legacy or Openflow switches. The P4 codes can be compiled to many targets like FPGAs, ASICS, software switches, etc. We use the behavioral model (BMv2) [36], a software switch simulation which supports P4 language, in our experiments. We use P4 Runtime [38] API to install routing entries in the forwarding tables generated using the P4 code. We use the ONOS SDN controller [37] to control the network comprising of P4 BMv2 switches using P4 Runtime. To implement the QROUTE algorithm in a real overlay network, the routers should support the range and the addition operations on packet header fields along with the capabilities of OpenFlow switches [39], [40]. These operations are supported in P4, BMv2 switches and ONOS controller.

2) P4 PSEUDO-CODE FOR THE QROUTE OVERLAY HEADER AND ROUTERS
Algorithm 4 shows the P4 pseudo-code for our overlay header QROUTE which is inserted between the Ethernet header and the IPv4 header. The QROUTE header contains the unique identifier for the QoS policy called QoS_policy_ID, the unique identifier for the egress router called egress_router_id, and the elapsed value of the QoS constraints called elapsed_delay, elapsed_jitter and elapsed_pkt_loss. header IPv4 We describe the P4 pseudo-code for the forwarding table used in the ingress routers in algorithm 5. The ingress router matches the IPv4 address of the destination host of the incoming packets and encapsulates it with the QROUTE tunnel header. The encapsulation is performed using the encapsu-late_QROUTE_header action.  A range_match match checks if a match field is between a lower and upper bound. For example, for the first entry in table 3, we use the range match to check if 0 ≤ elpased_delay ≤ 4 and 0 ≤ elpased_jitter ≤ 8. The intermediary routers match on all the 5 match fields mentioned in Algorithm 6 and accordingly forward the packet to an output port using forward_QROUTE_packet action described in Algorithm 7. The forward_QROUTE_packet action also adds to the respective headers of the packets the delay, jitter and packet loss of the link to which it is supposed to forward the packet. The delay, jitter and packet loss of the outgoing links are populated in the P4 switches by the ONOS controller. The action decapsulate_QROUTE_header mentioned in algorithm 6 is used by the egress router to remove the QROUTE tunnel header from the packet and forward the packet to its original destination host. A detailed description of our experimentation is given in Appendix A. We also provide an approach to measure available bandwidth in overlay networks having background traffic in Appendix B. add link_delay to header_elapsed_delay 3: add link_jitter to header_elapsed_jitter 4: add link_pkt_loss to header_elapsed_pkt_loss 5: forward pkt on out_port

3) RESULTS
All the figures in this section are generated on a mesh network of 1000 P4-BMv2 switches. We compare the average end-to-end delay observed by end-users with and without the overhead of range and addition operation on overlay headers (Fig. 8a). The paths taken by a packet between a pair of source and destination for both cases is the same. We observe that overhead imposed by QROUTE addition and range operation is not significant and as we increase the number of switches between end hosts, the overhead increases negligibly.
In Fig. 8b, we depict the percentage of flows that satisfy their QoS requirements using QROUTE in the absence of failures. We use the QoS policies for 10 different applications mentioned in Table 4 in this experiment. We can observe from Fig. 8b that QROUTE supports the QoS requirements for 90-99% of flows across different applications.
We also measure the percentage of flows that satisfy their QoS requirements using QROUTE after failures. This measurement is done until the SDN control responds to the change in the topology and installs new routes in the data plane. The QoS policies mentioned in Table 4 are used for this experiment also. As the number of failed links increases (Fig. 8c), we observe that the majority of the flows are still able to meet their QoS requirements. This resiliency is because we use the backup DAG along with the primary DAG and the forwarding scheme which maximizes QoS.

VII. RELATED WORK A. LAGRANGE-RELAXATION BASED ROUTING SCHEMES
There are several works [12], [13], [21]- [24] which solve the multi-constrained shortest path problem using Lagrange relaxation but they generate routes for every source-destination pair and QoS policy instead of generating a routing directed-acyclic graph. Thus, their approaches lead to an exponential increase in route computation time and the proliferation of routing entries. A*Prune [21] is an optimal approach to solve the MCSP problem by assuming that there is a guess function available for the constraints and costs and the algorithm is made faster by pruning certain paths based on their projected constrained values. However, its run-time increases much faster with network size compared to heuristics based algorithms [21]- [23]. A heuristic and multi-constrained version of the LARAC algorithm, H_MCOP, proposed in [22], searches in the direction of all constraints and cost simultaneously, but this approach does not always return a feasible solution if it exists [23]. MH_MCOP [23] finds a closer solution to the optimal as compared to H_MCOP but also does not always find a feasible solution if it exists [24]. In [12], the authors model the QoS routing problem as a multi-commodity flow problem which is decomposed to simpler constrained shortest path problems. They use the GEN-LARAC algorithm [41] to solve the constrained shortest path problem that satisfies all QoS constraints. The authors in [13] model the overlay routing problem as a maximization of the Quality of Experience (QoE) instead of QoS and performed Lagrange decomposition of their original problem. The subproblems are solved using Lagrange relaxation and sub-gradient optimization. They used a k-shortest path algorithm to find the shortest paths in their sub-problems. All these approaches generate routes for every source-destination pair and QoS policy and thus leads to a significant route computation time and size of forwarding tables.

B. DIRECTED-ACYCLIC-GRAPH (DAG) BASED ROUTING SCHEMES
Directed-acyclic-graphs (DAGs) has been used in multiconstrained QoS routing [42]- [44]. However, none of them can efficiently reduce both the route computation time and the number of routing entries while ensuring additive, multiplicative and concave constraints. In [42], the authors use DAGs for routing in low-power and lossy networks. They consider the delay, jitter and packet loss while creating a destination-oriented DAG but they omit the bandwidth requirement (concave constraint) which is a very crucial network resource. They are using a distributed approach of exchanging messages between the nodes to construct a DAG rather than using a centralized routing algorithm. Moreover, they do not address the problem of the proliferation of routing entries in large networks. The authors in [43] consider all types of QoS constraints, that is, multiplicative, additive and concave. They prune the original network to a DAG that contains paths between a source and destination which satisfy all the constraints. They find the optimal path using this DAG. However, they do not use DAG for routing but only as an intermediary step to generate feasible paths. Thus, their approach leads to high route computation time and a large number of routing entries since they are generating paths for every source-destination pair. In [44], the authors use a DAG-based approach to provide resiliency against single arbitrary link failure. They also introduce delay and bandwidth constraints while generating DAGs. However, they do not consider jitter and packet loss constraints. Moreover, their approach leads to high route computation time and the proliferation of routing entries because of generating a DAG for every source-destination pair.

C. OTHER APPROACHES FOR ROUTING IN SOFTWARE-DEFINED OVERLAY NETWORKS
In [8], the authors propose a time-slot-based routing algorithm for finding multiple paths for a particular flow for ensuring QoS of multimedia applications in software-defined overlay networks. They model the cost of the overlay links in terms of its availability and consider only bandwidth constraints. In [11], the authors use a routing engine based on random neural networks with reinforcement learning for their software-defined overlay networks. They only incorporate the latency of the paths while making their routing decisions. In [14], the authors propose a multipath routing algorithm that finds a path of least cost that satisfies the delay and bandwidth constraint. However, these approaches have high route computation time and a large number of forwarding entries because of generating routes for every source-destination pair and QoS policy.

VIII. CONCLUSION AND FUTURE WORK
We propose an efficient routing scheme, QROUTE, for satisfying multiple QoS constraints in software-defined overlay networks. QROUTE consists of a control plane routing algorithm which has significantly low route computation time because of employing a novel directed-acyclic-graph (DAG) based approach. QROUTE also reduces the forwarding entries in the data plane by using a QoS-metric-based forwarding scheme. QROUTE uses backup DAG combined with the primary DAG to provide sufficient QoS even during failures. We also provide a framework to evaluate a path's conformance to a multi-constrained QoS policy which enables the network operator to find paths slightly deviating from the QoS policy. Our experimental results demonstrate that the proposed QROUTE routing scheme not only significantly reduces the route computation time, but also decreases the forwarding table size considerably. Evaluations also show that the addition and range operations performed in the data plane do not incur significant overhead. Results demonstrate that QROUTE achieves QoS to a considerable degree with and without failures. QROUTE can be either implemented in software routers (e.g., BMv2 P4 switches) deployed in the cloud, or in hardware switches which support P4 (e.g., NetFPGA [45] and Barefoot Tofino [46]).
We are working on extending QROUTE to the hierarchical design multi-controller architecture. This involves designing a DAG stitching algorithm which stitches together the routing DAGs generated by QROUTE to create a global routing DAG.
According to the Cisco Annual Internet Report [47], the number of devices connected to Internet will be 29.3 billion by 2023. The majority of devices and the traffic that will dominate the Internet will be machine-to-machine communications, Internet of Things and enhanced 5G mobile broadband. These applications have significantly diverse QoS requirements [48]. To meet such diverse QoS requirements of huge network traffic requires more efficient QoS routing schemes.

APPENDIX A DETAILED DISCUSSION OF OUR P4 SWITCHES TEST-BED
In this section, we provide a detailed description of our experimentation on test-bed of P4-BMv2 [35], [36] switches controlled with ONOS SDN controller [37] using P4Runtime [38].

A. INGRESS ROUTER OPERATION
Listings 1, 2, 3 and 4 show some important portions of our P4 code that implement the QROUTE algorithm. Listing 1 shows the overlay header qroute_tunnel_t which is inserted between the ethernet header and the ipv4 header and is used to carry the protocol ID of the next layer of header called proto_id, the QoS policy ID called policyID, the unique identifier of the egress router called egress_router_id, and the elapsed value of the QoS constraints called elapsed_delay, elapsed_jitter and elapsed_pkt_loss. Listing 2 describes the table used in the ingress router which matches the IPv4 address of the destination host for a packet using the longest prefix match (lpm) and then encapsulates the packet with the qroute_tunnel_t tunnel header using the qroute_tunnel_ingress action.

B. INTERMEDIARY ROUTER OPERATION
To implement comparison on the elapsed value of delay, jitter and packet loss, we use the match type range supported by P4, P4Runtime and BMv2 switches. A range match checks if a specified header is between a low and a high value inclusively. ONOS supports a function matchRange (PiMatchFieldId fieldId, byte[ ] low, byte[ ] high) of type PiCriterion.Builder which adds a range field match for the given P4 header field ID fieldId, low value and high value of range match. In our scenario, we define three range matches for delay, jitter and packet loss (Listing 3) in our P4 code. We set the low limit to 0 and the high limit to the maximum permissible amount of delay, jitter or packet loss respectively in our matchRange() function call. Table t_qroute_tunnel_fwd (Listing 3) uses action qroute_tunnel_transit in the intermediate nodes and qroute_tunnel_egress in the egress nodes. The qroute_tunnel_transit action(Listing 4) is used in the intermediary nodes to forward the encapsulated packet on a output port based on the egress_router_id, policyID and the elapsed values of the QoS metrics. The BMv2 switch adds to the respective headers of the packets the delay, jitter and packet loss of the link to which it is supposed to forward the packet and then sets the output port of the packet. The delay, jitter and packet loss of the outgoing links are populated in the P4 switches by the ONOS controller.

C. EGRESS ROUTER OPERATION
The qroute_tunnel_egress action is used in the egress nodes to remove the qroute_tunnel_t header before forwarding the packet to the output port.

D. MEASURING QoS METRICS FOR THE LINKS
We use P4-BMv2 switches in integration with Mininet. Mininet provides us the options to specify the performance parameters of the links like delay, jitter, packet loss and maximum bandwidth. We need to start the mininet topology using -link tc command to set these performance parameters. This uses the TCLink class which is a wrapper around the Link class of mininet and allows us to specify the performance parameters like delay, jitter, etc. To create such a link in the custom topology python file, we need to use the command self .addLink(switch1, switch2, bw = 10, delay = 10ms, jitter = 8ms, loss = 1). This creates a link with the maximum bandwidth of 10 Mbps, a delay of 10 ms, a jitter of 8 ms and a packet loss percentage of 1%. We have hard-coded the delay, jitter and packet loss percentage of the links in our test-bed. To measure the available bandwidth of the links, we create counters at every switch using the counter() function supported by P4. The switch counters count the bytes sent by the ports which is polled by an ONOS application at regular intervals and is stored in a CSV file. We compute the available bandwidth in the links using the bytes sent by the ports and the elapsed time using the approach mentioned in [49]. Using this telemetry information, the central controller computes new routes after every 30 seconds and installs the new rules in the forwarding tables of P4-BMv2 switches. A detailed discussion on measuring the QoS metrics in a real-overlay network is given in appendix B.

E. SIMULATING LINK FAILURES
Mininet provides a command link s1 s2 up and link s1 s2 down which activates and deactivates respectively the link between switch s1 and s2 of the network on the fly. We use this command to fail a link for testing our routing scheme during failures. Since there is no support in P4 switches to determine the link status, we store the status of the links in another table in the switches which are updated using the P4 switch's API used for accessing tables. When the outgoing link in the primary DAG is down, the packet is forwarded to the outgoing link in the backup DAG if it is up. If the outgoing link in the backup DAG is down, we drop the packet. Meanwhile, the ONOS controller detects that a link is down and the routing application calculates the new paths based on the new topology and installs new rules in the forwarding tables.

APPENDIX B MEASURING AVAILABLE BANDWIDTH IN OVERLAY NETWORKS HAVING BACKGROUND TRAFFIC
We also provide an approach to measure available bandwidth in overlay networks having background traffic in this section. In a network with background traffic and unknown link capacity, available bandwidth can be computed using the values of delay and packet loss as described below. Since the throughput-intensive applications use either TCP or TCP like congestion control, the authors in [7] use the following equation for modeling throughput: where p is the packet loss probability and rtt is the round trip time. However, they have only considered last re-transmit and not timeout as the indicator for packet loss. Moreover, they have assumed that a received ACK is acknowledging at most 1 packet. They have also not considered that the congestion window size can be restricted. Thus, the below equation for throughput [50] will give a much accurate measurement of TCP throughput.
In equation (9),W max is the maximum congestion window size, b is the number of packets that are acknowledged by a received ACK, and T 0 is the initial time out. Current Windows and Linux operating systems often set T 0 = 3 sec, W max = 64 kb and b = 2 and putting them into Equation 9 will give a better estimate of TCP throughput in current networks. The delay, jitter and packet loss can be calculated by sending packet probes between the routers with timestamps.