LGCC: A Novel High-Throughput and Low Delay Paradigm Shift in Multi-Hop Congestion Control

Technological advancements have provided wireless links with very high data rate capacity for 5G/6G mobile networks and WiFi 6, which will be widely deployed by 2025. However, the capacity can have substantial fluctuations, violating the assumption at the transport layer that the capacity is (almost) steady. In this paper, we present a general and efficient, yet deployable solution to this problem through a novel design empowered with a rich theory, allowing a significantly improved experience in using new technologies, especially mobile cellular services. We employ the well-known theory of food-chain models in biology, where a bottleneck link can be modeled as prey, while flows are predators. We extend this model to a chain of predators and preys to form a multi-hop congestion controller, called LGCC. Through simulation evaluation with real-life 5G traces we show the effectiveness of LGCC, compared with the state-of-the-art ABC (Accel-Brake Control). Our results show an order of magnitude bottleneck queuing delay decrease, with only a small decrease in throughput because LGCC tries to never exceed link capacities. LGCC’s design can additionally open a new paradigm in stable multi-hop congestion control and flow aggregation.

LGCC: A Novel High-Throughput and Low Delay Paradigm Shift in Multi-Hop Congestion Control Peyman Teymoori , Member, IEEE, Michael Welzl , Member, IEEE, and David A. Hayes , Senior Member, IEEE Abstract-Technological advancements have provided wireless links with very high data rate capacity for 5G/6G mobile networks and WiFi 6, which will be widely deployed by 2025.However, the capacity can have substantial fluctuations, violating the assumption at the transport layer that the capacity is (almost) steady.In this paper, we present a general and efficient, yet deployable solution to this problem through a novel design empowered with a rich theory, allowing a significantly improved experience in using new technologies, especially mobile cellular services.We employ the well-known theory of food-chain models in biology, where a bottleneck link can be modeled as prey, while flows are predators.We extend this model to a chain of predators and preys to form a multi-hop congestion controller, called LGCC.Through simulation evaluation with real-life 5G traces we show the effectiveness of LGCC, compared with the state-of-the-art ABC (Accel-Brake Control).Our results show an order of magnitude bottleneck queuing delay decrease, with only a small decrease in throughput because LGCC tries to never exceed link capacities.LGCC's design can additionally open a new paradigm in stable multi-hop congestion control and flow aggregation.

I. INTRODUCTION
I N THE Internet design, congestion control has been deployed at the transport layer or above, and it is supposed to operate in an "end-to-end" manner, where "end" represents the application host [1].The commonly used transport protocol, TCP, reacts to packet drops as the main congestion signal from the network.This design greatly impedes the consideration of link-specific properties.Today, wireless links are prevalent, and the capacities that they expose to TCP are often time-varying, especially with new link layer technologies.
Notable technological advancements in communication have made it possible to exploit millimeter waves (mmWaves) [2] to benefit from the very high data rates it can provide.The mmWave spectrum has also been considered for next generation wireless networking such as cellular networks (e.g., 5G/6G).mmWave communication is at odds with TCP's assumption of a relatively stable end-to-end path with a static capacity at the bottleneck.It promises extremely low latency, yet it will often expose a drastically fluctuating network capacity.TCP should be able to react to these fluctuations-but the reaction speed of the end-to-end control loop is inevitably in the order of end-to-end round-trip times (RTTs).Such a control loop cannot ensure low latency and good utilization in the face of capacity changes that occur at sub-RTT timescales [3].
Even with today's 4G networks, it is already common to shorten the control loop by installing Performance Enhancing Proxies (PEPs)-devices which interfere with TCP connections, e.g., by dividing them in two (acting as a receiver towards a sender and acting as a sender towards a receiver) [4].Splitting an end-to-end connection like this (or with an application-layer proxy, which is currently under discussion in the IETF for the QUIC protocol [5]) has shown benefits, and it may even be the only feasible way to attain efficient communication over mmWave links [6], [7].Most importantly perhaps, connection splitting eliminates the need to operate "normal" TCP congestion control throughout the path, allowing the deployment of a more suitable congestion control mechanism on the wireless segment only.
What, then, should a mechanism for the wireless segment look like?Rather than suggesting a quick fix for one specific link layer technology, we argue that it is necessary to design a general congestion control mechanism which: • can swiftly adapt to changing physical capacity.
• will stably operate when multiple instances of the mechanism are used in sequence (e.g., when transferring data from one wireless segment to another), as a chain of control loops.• has a low deployment barrier, e.g., by not requiring to update every router along the path.
To the best of our knowledge, this paper is the first attempt at devising a congestion control mechanism that satisfies all of these requirements.Chains of congestion control loops have been shown as a key feature in new network architectural patters such as the Recursive InterNetwork Architecture (RINA) [1].In our previous work [8], we investigated various forms of feedback for a recursive congestion control as in RINA; the main finding was that faster more direct feedback is better, suggesting that chained congestion control should relay congestion signals back to the source without delaying them; this constitutes our main design principle in this paper.Similar to earlier explicit feedback methods such as XCP or RCP [9], [10], we opt for an approach that does not only rely on physical queue growth for feedback.These mechanisms were found not to be deployable in practice as they require to change all routers on an endto-end path.Our situation is more favorable, as we aim to apply our mechanism on path segments only.By basing the feedback method on the Datacenter TCP (DCTCP) [11] style of ECN usage (interpreting ECN marks from multiple packets as a multi-bit congestion signal), we made it even easier to deploy.
Specifically, the method that we propose, called "Logistic Growth Chain Control" (LGCC): • can be deployed by only updating end systems, and introducing some devices that support it at loop intersection points (a key difference from explicit rate feedback schemes like XCP or RCP); within each control loop, commodity hardware can be used, provided that this hardware supports Random Early Detection (RED) AQM marking with Explicit Congestion Notification (ECN).
• is general enough to be adopted in different use cases, and can be easily enhanced by feeding it more information from the network.For example, it is possible to further accelerate convergence if there are more bits available in the packet header than ECN.Because we use an additive signal akin to cost as defined in the Network Utility Maximisation (NUM) framework [12], it would also be possible for routers to mark packets before a physical queue even grows (as proposed in [13] and [14], for instance).
• is globally asymptotically stable.
• is fair, and can attain different types of fairness.
• is fast in terms of convergence due to the use of the LG function.Since we shorten control loops, flows can converge faster in case of capacity decrease/increase.• responds quickly to capacity changes since the capacity can be measured more directly than by inferring it based on queue-related metrics such as ECN and delay.
LGCC uses the concept of carrying capacity, the maximum available resource, denoted by K.In a 5G case, K is time varying, so we add a time parameter, i.e.K(t).
There are already useful signals at lower layers that can feed into K(t)-e.g., various MAC solutions calculate a physical rate using rate adaptation techniques, making the current physical rate a promising candidate as input to K(t).K(t) is then promptly and explicitly sent back to the sender inside the next ACK packet. 1 This enables the sender to know the current bottleneck capacity in a duration much shorter than the RTT.Consequently, sources do not need to rely on large, delay increasing, buffers to avoid fluctuations.By not relying solely on buffers and contrasting with methods like ABC [15], which necessitates a minimal persistent queue for its stable operation, an insignificant reduction in throughput can occur.
LGCC is based upon an early version of our mechanism, the biology-inspired Logistic Growth Control (LGC) [16], combined with our additive ECN feedback theory [17] that enables us to take advantage of higher marking probabilities.In contrast to prior work, we now assume multiple consecutive control loops between each source and destination, one per administrative domain.This allows us to react much faster to congestion than with an end-to-end loop, but it is a more complex design approach which inevitably raises stability concerns.A stability analysis is, thus, a key contribution of this paper.
In Section II, we survey related work.Section III discusses our motivations behind this paper, and Section IV introduces the theory behind LGCC.We establish the theory of LGCC in Section VI, and in Section VIII, we analyze stability and fairness properties of LGCC.Section IX presents how LGCC is implemented, and its simulation results are presented in Section X.Finally, Section XI concludes the paper.

II. RELATED WORK
The advent of high capacity wireless links (e.g.mmWave) has raised serious challenges for traditional congestion control [18].In a series of work [3], [7], [19], [20], [21], [22], it was shown that the full potential of mmWave cannot be achieved at higher layers, especially at the transport layer.
One of the main solutions of the above problem has been to introduce some intermediate boxes helping the end-toend congestion controller, e.g. by breaking it into several shorter loops.For example, in [6], a TCP proxy architecture was presented without any modification at the sender side.The proxy is installed in the Radio Access Network (RAN), and by collecting statistics from the 5G base station (gNB), it estimates the downlink Bandwidth-Delay Product (BDP) and performs flow window management using the advertised window field in ACKs.Another method, called X-TCP [23], was proposed to adjust the congestion window of TCP at the mobile equipment side.It estimates the current physical layer rate using the Signal to Interference plus Noise Ratio (SINR), thus keeping the buffer small.
Explicit feedback control protocols (e.g.XCP [9] and RCP [10]) can overcome some of the above challenges.However, they have deployment limitations in the Internet because they need drastic changes to all packet headers, routers, and endpoints.
The ABC mechanism [15], [24] has targeted the above problems by providing a deployable method, compared with prior explicit schemes, since it utilizes the existing Explicit Congestion Notification (ECN) [25] infrastructure.This makes the mechanism the closest competitor of the one that we present in this paper.ABC uses a Multiplicative-and-Additive-Increase/Multiplicative-Decrease (MAIMD) scheme to ensure fairness among flows.ABC adjusts the sender's window upon receiving an ACK from the receiver: it is increased by one if the ACK is marked as accelerate, 2 and in case of a brake mark, it is decreased by one; this allows the sender to have a larger window size dynamic within one RTT to follow the link capacity dynamics faster.In the network, ABC routers mark packets with accelerate or brake proportional to the queue length.Additionally, ABC has a fall-back mechanism to co-exist with non-ABC traffic.
New network architectures such as the Recursive InterNetwork Architecture (RINA) [26] represent a different approach to many aspects of networking, including congestion control: here, network mechanisms are designed from scratch, without compromise. 3In RINA, it is natural to break an end-to-end path into several shorter control loops, each with its own customized congestion controller.In addition, a larger control loop can operate over several shorter ones; this is the recursive property of RINA, which can provide many benefits [28].What we propose in this paper can also be adopted by RINA.
Since the advent of hop-by-hop congestion controllers, as the extreme case of multi-hop controllers, their stability has been of paramount concern because some unstable behavior was observed in some situations [29], [30].This had led to the "fear of instability".However, as demystified by [30], the cause of instability was the unsuccessful use of nondiscriminatory on-off type controls.In addition, research work such as [29], [30], [31], [32], and [33] has shown that per-hop controllers accelerate the response to load changes in the path, and that certain conditions can be obtained under which the chain of control loops is stable.For example, in case of Split-TCP (where a middlebox spoofs IP addresses to act like a receiver towards the sender and act like a sender towards the receiver), the system can be stable under certain conditions [31], [34].
The assumption of having links with time-varying capacity also complicates stability conditions of the congestion controller.Analyses such as [35] show system stability with feedback delay can be achieved in the joint congestion control and scheduling for multi-hop wireless networks.In [36], a family of optimization-based distributed traffic control laws was developed to enable load balancing and/or rate adaptation in response to capacity variations in the network.Instead of proving system stability around a single equilibrium point, the authors in [37] propose a primal-dual congestion control algorithm which is proven to be trajectory stable when link capacities are time-varying; this guarantees the system is stable around a time varying reference trajectory.

III. MOTIVATION
In this section, we motivate the need for a deployable multi-hop congestion controller with preliminary simulation and evaluation; this shows why the controller needs to be fast, in both rate increase and decrease, and also in terms of convergence rate, with the direct knowledge of link capacity and its changes as well as employing ECN to increase deployability.
In previous sections, we discussed about limitations of endto-end congestion control and why shortening control loops can respond faster to congestion and capacity change.ABC is a very recent state-of-the-art mechanism, which, according to its authors, operates at the Pareto frontier when compared with other congestion controllers such as TCP Cubic, Vegas, XCP, BBR, and PCC [15].Although ABC is empowered by ABC routers on the path to signal a rate increase/decrease via ECN, it still operates in an end-to-end manner.This means that ECN marks first arrive at the destination, and then they are echoed back to the source.Considering the topology illustrated in Fig. 1, in Fig. 2 we show a preliminary simulation 4comparison of LGCC with ECN-based mechanisms: LGC (our previous end-to-end approach [16]), DCTCP [11], and ABC.We observe that DCTCP produces the largest queue because it cannot incorporate link capacity fluctuations, and because it is additive-increase, it is very slow in filling the newly increased capacity.LGC, ABC, and LGCC.The links have a capacity of 500 Mbps, and Source's end-to-end RTT is 6 ms.The link capacity between Router 2 and Destination is time-varying; it drops to one third in [2,3]s of the simulation.Other parameters are set such that 100% utilization with minimal delay is achieved for each method.
LGC can incorporate link capacity in its rate dynamics, but it cannot track capacity changes since it is end-to-end, and starts fluctuating when the capacity drops, leading to a larger queue.ABC only incorporates link capacity at ABC routers through queue drain rate, but still the ABC sender can track changes faster because the ABC router motivates the source in both increase and decrease through ECN marks.LGCC can, however, outperform all of the others in terms of rate fluctuation, convergence speed, and keeping the queue small because of a multi-hop design where Router 1 and Router 2 act as connection splitters, shortening the control loop, and incorporating the link capacity and its changes directly in their rate dynamic.
In terms of convergence rate, we need a fast method for flows to reach their target rate.Although ABC flows can follow the underlying link capacity quickly with the help of ABC routers, they do not get a fair share of the capacity promptly as they fill the capacity.The main problem is that when a new flow joins, to attain fairness among flows, ABC's MAIMD adds/subtracts less than 1 packet per RTT to/from the congestion window (cwnd) of the flows, and the increase/decrease becomes smaller as flows converge.
We evaluate ABC's convergence rate through a simple analysis: assume n flows with equal RTTs and congestion window w i , competing for a bottleneck capacity.ABC's congestion window of flow i increases by 1 + 1/w i per accelerate and decreases −1 + 1/w i per brake signal.Therefore, during an RTT where the proportion of accelerates is f , the window increases by f w i (1 + 1/w i ) = f (w i + 1) and decreases by . Adding these two changes to w i to obtain the new window size, and considering the bottleneck capacity is completely filled, we have As   are shorter in LGCC because of shorter control loops.Fig. 3 shows a simulation comparison on the convergence of ABC and LGCC in OMNeT++.The first flow fills the capacity, and then the second flow joins.We can observe that it takes a long time for the ABC flows to converge.However, this is shorter than half a second for LGCC.In the following sections, we elaborate on the design of LGCC and show how it incorporates link capacity, capacity changes, and ECN in its rate dynamic.

IV. LOGISTIC GROWTH (
LG) The base of LGCC is the Logistic Growth (LG) function.
LG is usually used to model the growth of a species population over time.This function is defined by the following differential equation where N is the population size, K the "carrying capacity" determining the maximum value of N , and γ is growth rate.By the dot notation, we mean time differentiation, i.e.Ṅ = dN dt .We summarize the notation in Table I.The idea of growth constrained by a capacity limit in LG has inspired the use of this function in network congestion control [16], [38], [39].
The LG function (and its dynamics (1)) is usually normalized by defining x(t) = N (t)/K, which yields with the solution where x(0) denotes the initial population.Clearly, lim t→∞ x(t) = 1.

A. Lotka-Volterra Model
The LG function specifies how one species population grows over time.The Lotka-Volterra model consists of a set of equations examining the effect of inter-species competition where two or more species compete for some limited resource.The growth rate of a species is then not only bounded by the carrying capacity, but also by the growth rate of a competing species.If a generalized Lotka-Volterra model for n interacting species has a non-trivial equilibrium, x * i > 0, then there can be found a Lyapunov function that is zero at the equilibrium and negative at other points [40].This implies that the model is globally stable.In Section VI, we will use this relationship to model the interconnected loops of a path like individual species, using this relationship to interconnect them.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

V. LOGISTIC GROWTH CONTROL (LGC)
LGC [16] was presented as a congestion controller using the Lotka-Volterra competition model [41].In this controller, the rate of source r is updated by or in the discrete form, where γ r is a constant value and pr is the (end-to-end) measured marking probability at source r.This yields a normalized rate which sources can multiply by their link capacity to obtain a send rate in bits per second.Since LGC needs the link capacity, it was proposed for data centers in which link capacities are known/fixed.If there is only one bottleneck on the path, it is proven that pr converges to S−1 S where S is the number of competing flows, and hence, x * r = 1 S .To illustrate how LGC works, we numerically evaluated (5): we assume that there were 5 flows competing for a bottleneck capacity, and excess traffic was marked proportional to the capacity.Formally speaking, where [.] 1 0 = max(min(.,1), 0).Fig. 4 shows how the 5 flows compete when they join one by one.The first flow starts at time step 0, and then after every 20 time steps, one new flow joins.We see that the normalized rate of flows converges to 0.2, and their observed marking probability converges to 0.8.At step 150, flows leave one by one every 20 steps until there is only one flow left.We can also see that the convergence speed of pure logistic growth decays as more flows enter the system-flow 5 is quite slow at obtaining its bandwidth share.We have fixed this by making γ adaptive in [16], and we apply the same method in this paper, in Section IX-A.

A. Deflated LGC
In [17], we obtained the utility function of LGC, and showed how LGC can be adopted by the NUM theory.This also helps change the equilibrium marking probability of LGC.In other words, we can deflate its marking probability, which can become very high when the number of competing flows is large.The send rate dynamic is where ϕ 1 and ϕ 2 are the base parameters that can be configured to control the marking probability.If ϕ 1 = ϕ 2 , then (7) reduces to (4).In case there is one bottleneck, the equilibrium marking probability can be obtained, which is given by where S is the number of competing flows.In this case, if ϕ 1 > ϕ 2 , the equilibrium marking probability becomes smaller.For example, if we set ϕ 1 = 10 and ϕ 2 = 2, then p * r = 0.1897 (this was 0.8 in Fig. 4(b)).However, the congestion controller behavior does not change-it exhibits the exact behavior shown in Fig. 4(a).

VI. LG CHAIN CONTROL (LGCC) MODEL A. Network Model
In this section, we model a chain of LGCs and discuss its deployment considerations.Our main aim is to have a deployable solution with minimal required changes to current commodity hardware.ECN is a widely-deployed mechanism in commodity hardware with broad utility.We utilize this feature and assume that routers in the network support packet marking via RED.However, some of the nodes on the path towards the destination need to support LGCC, which we call LGCC router; these nodes could be domain border routers.In other words, LGCC does not need to be implemented as a hop-by-hop solution although that is also possible.
Fig. 5(a) illustrates an example network topology of a source and a destination with m routers on the path.This can be mapped into the diagram in Fig. 5(b) in which only some of the routers deploy LGCC, meaning that an end-to-end congestion controller between the source and the destination is broken in several smaller congestion control loops.The other routers are assumed to have RED deployed, which ECN-marks packets randomly if the output queue is non-empty.Therefore, there is one LGC loop from the source to Router 2, another one from Router 2 to some other router, and finally, there is one from Router m to the destination.There is one LGC loop per source from sources 1 and 2 to Router 2, one loop from source 3 to Router 2, another one per source from Router 2 to some other router (continued in that way until Router m), and finally, there is one per source from Router m to the destination.This example already highlights the possibility of aggregating traffic to reduce unnecessary competition between flows, but we deem a full flow aggregation mechanism for heterogeneous traffic out of scope for the present paper.However, here we discuss the possibility of flow aggregation and its benefits with regards to LGCC, and provide a fairness analysis.
The number of LGCC routers and where they operate depends on the network design.For example, each loop can be formed over one domain, or if the medium is wireless, the loop can operate over only that medium specifically.We assume that it is a design issue and up to the network manager where to deploy LGCC routers; however, we will provide performance evaluations about different deployments in Section X.
Fig. 6 shows a high level internal structure of an LGCC router.It has several "Receivers" to control the congestion control loops ending at it, and several "Senders" to start new ones towards their destination/next LGCC router.Senders have their own send queues with the aim of holding excess incoming packets if their current send rate is lower than the incoming rate.The role of "Schedulers" is to prioritize taking packets from receivers and pass them to the correct next sender.If packets are accumulated at receivers waiting for schedulers to pass them, a virtual queue management scheme can be deployed to ECN-mark packets.These marks are sent back towards the previous senders via ACKs from the receivers.In addition to the ECN echo each ACK carries back to the sender of the loop, the current send rate of the next loop is also copied to the receiver window size field of the ACK.This ensures that previous senders as well as the original source are updated very quickly, i.e., in at most half an end-to-end RTT depending on the bottleneck location, with the latest rate changes of the next loops to reduce the risk of instability.The value of this field can be interpreted as bit (or Byte) per second at the previous sender side; this determines the carrying capacity of the previous loop ending at this LGCC router.The sender of the last loop receives the buffer size from the destination as normal since the last hop sender needs to ensure that it does not overflow the destination.However, it feeds back its current rate inside the receiver window size5 field of ACKs originating from it.In the next section, we will show how carrying capacity is used in rate calculations.
LGCC routers could be implemented as PEPs or a part of border routers.Here we assume that they explicitly break a connection, and hence, packets of one LGCC router are destined explicitly to another one, and so on.As such, it is not possible for one LGCC router to be somehow bypassed by changes in the traffic route.However, it is possible to have route changes between two LGCC routers.The goal of ECN is to capture traffic load changes on the path segment, even due to routing changes.
To elaborate the LGCC router design in Fig. 6, assume there are two loops arriving at an LGCC router from different links, and departing it from the same link.In case of per-flow chaining, which is our main focus in the paper, each incoming flow has its own dedicated receiver and sender in the router.Packets from the receivers are passed to their corresponding senders, which now compete for the same output queue; their packets might also be ECN-marked in the next loops as well.The senders accordingly adjust their send rate to the fair share of the capacity of the next link/loop, and their current send rate is echoed back to the senders of the previous loops using every ACK generated by the receivers.The previous senders will accordingly adjust their send rate in the same manner.In case of aggregated chaining, there is only one sender at the output link, and the scheduler passes packets from the receivers to the sender; with the assumption that there are only homogeneous greedy flows, sharing for aggregations is simply weighted with respect to the number of flows in each aggregated flow.If there are no other senders at this output link, the sender can send at the link capacity without queuing.However, the sender of the previous loops need to send at a lower rate, which can be done through ECN-marking packets at the receivers' virtual queue.We will demonstrate a simple policy to perform this kind of marking in Section X-E.
In the above architecture, each LGCC loop operates within an autonomous administrative domain, e.g., from an ingress router to an egress router, where these two can also act as LGCC routers.Incorrect ECN configuration might lead to either longer delays or competing flow instability within that domain.However, since LGCC uses the same ECN as DCTCP, it should be able to coexist harmoniously with traditional TCP in the L4S (Low Latency, Low Loss, and Scalable Throughput) architecture [42].A detailed evaluation is out of the scope of this paper.

B. Congestion Controller Model
Here, we introduce LGCC.First, we form a chain of (4) where the carrying capacity is not normalized to 1. Instead, it is determined by the send rate of next loop.In addition, we use notation pr,i to reflect the marking probability of only loop i that is measured by its sender.Assume a flow broken into n loops from a source to a destination.The rate dynamics at the sender of each loop are given by . . . where . . .
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
and x r,n+1 (t) and c l denote the destination's processing rate and link capacity, respectively.
LGC uses the notion of carrying capacity, and so does LGCC.Therefore, the carrying capacity K should be determined in each loop.This can be the minimum of the send rate of the next loop and the link capacity in the current loop, which is obtained by (10).p includes the price of the receiver's virtual queue to the next sender as well in case of aggregated chaining.Since we assume that the formation of each loop is under one authority/domain, link capacities may be known in that domain.In case a link capacity is changing over time, e.g. in a wireless medium, information about the current capacity and modulation is usually available at the output link of the LGCC router.This could be used as the current link/carrying capacity.A similar technique has been done by ABC [15].
In case of aggregated chaining and if packets of two or more receivers are forwarded to the same sender in the next loop, i.e., the loops are aggregated into one flow in the next loop, we can enable schedulers to prioritize packets of certain receivers while passing to their next sender.Here, we assign weights to receivers and employ a weighted fair queuing discipline in the scheduler.This is under the assumption that flows are homogeneous and greedy.Other cases will be left as future work.In this case, the rate dynamic of loop i is given by where ω r,i denotes the normalized weight assigned to the send rate of flow r in loop i.

C. Benefits of the LGCC Design
The notion of carrying capacity is an important feature of the LGCC design.It helps the controller to have an accurate estimate of the upper capacity limit, which is a direct signal sent back to previous controllers, relieving them from having to wait for indirect congestion/cost signals such as queue marks to adjust their rates.In other words, each loop uses the send rate of the next loop as its own K, which leads to faster rate adjustment to available capacity at next loops.
LGC's increase/decrease behavior is non-linear.As opposed to controllers with, e.g.additive increase, how fast to increase/decrease depends on the difference between x r,i and (1 − pr,i )k r,i .This means that in case of a sudden increase in the medium capacity (assume a wireless link that is recovered from a rate drop due to an intermittent noise), capturing the available capacity can be fast.

D. Utility Function
For LGCC, we apply the same method that we used to obtain the LGC utility function in [17].In equilibrium, we know that q * r,i = U ′ (x r,i ) where q * r,i = − log ϕ2 (1 − p * r,i ).Solving for p yields p * r,i = 1−ϕ q * r,i 2 .We also have 1− kr,2(t) − p * r,2 (t) = 0.These give which is a strictly concave, twice differentiable function in the range (0, k r,i ].These properties facilitate its stability and convergence. VII. CONGESTION CONTROL ALGORITHMS The LGCC dynamics (9) can be converted into a Network Utility Maximization (NUM) problem.This enables us to analyze LGCC better and find deployable solutions, especially on commodity routers.In [17], we obtained the utility function of LGC, and showed how LGC can be adopted by the NUM theory.Here, we use the utility function behind LGCC dynamics, i.e., (12), and then, we divide the optimization logic between sources and routers.Since sources and LGCC routers use the same module to send packets, by sender dynamics we mean the controller logic in both of them, and by router dynamics we mean commodity routers that do not break connections.Our assumption about these routers is that they can either implement our specific packet marking scheme or at least, they support RED, which is a widely-deployed mechanism.

A. Primal Algorithm
In this algorithm, the optimization logic is deployed at the sender side.The routers only return a network cost to senders, which is a direct mapping of the current total traffic of a link to a non-negative real value.It is given by where U (.) is the utility function of senders [17].Substituting the derivative of ( 12) in ( 13) yields Since our assumption is that routers support RED, the network cost can be sent back to senders via ECN marks.This is denoted in (13) by pr,i .In the next subsection, we introduce an ECN marking algorithm for this purpose.

B. Dual Algorithm
In the dual algorithm, the optimization logic is deployed in routers, and senders directly map the cost they get from the network into a send rate.First, we discuss a general router which could run this algorithm, and then discuss how routers with RED AQM can employ it.
We use the fact that at equilibrium, q * r,i = U ′ (x * r,i ) where q * r,i is the cost sender i gets.As we already showed in [17], in case of employing RED, q r,i = − log ϕ2 (1 − pr,i ).Combined, these two yield x r,i = k r,i e log(1− pr,i) log(ϕ1)/log(ϕ2) (15) meaning that the sender can directly calculate its optimal send rate using pr,i .However, routers need to iterate on their cost until they converge to the optimal value.The first dual algorithm, called Dual1 is given by where p l and y l denote the marking probability of link l (at the router's output queue to the link) and the total traffic sent over link l in the last interval.β l denotes the step size.At output queues, C = c l where c l is the link capacity, and at receivers' virtual input queues, C = x r,i+1 , i.e. the send rate of the next loop, and y l is equal to the rate of the traffic sent to this queue from the sender of loop i in the last interval.
As we showed in our previous work, we can use RED with a special configuration (see Appendix E of [17]) as the dual algorithm.This enables us to have a more deployable solution.The cost/marking probability dynamics at routers' output queues (links) in a discrete form is given by where b l , max th , and n denote the queue length, the maximum threshold parameter of RED over which all the packets are marked, and the interval number.Other RED parameters are min th = 1, w q = 1, and max p = 1.This algorithm implies that the current backlogged size can be divided by the maximum threshold to have a marking probability for the current interval.
One of the benefits of the dual algorithms of LGCC is that, due to the notion of carrying capacity, senders can sometimes jump to a send rate if, for example, the carrying capacity suddenly changes.In case the sender's loop is not shared with other loops, jumping to higher rates does not affect the network.Otherwise, if some links of a loop is shared with other loops, the primal algorithm can be used instead of (15); the use of a primal-dual algorithm can also handle case, which will be explained in the next subsection.

C. Primal-Dual Algorithm
This type of algorithm divides the optimization logic between senders and The combinations ( 14)-( 16) and ( 14)-( 17) form two sets of primal-dual algorithms.

D. Numerical Evaluation
In order to have a notion on how ( 9) is effective and can accelerate convergence of a chain of loops due to the usage of the carrying capacity, we perform a numerical evaluation here on the topology shown in Fig. 7.We consider two scenarios: 1) a chain of 5 loops where inside each loop, there is no other competing loop (i.e.Source 2 to Source 5 do not send any packets), and 2) a chain of 5 loops where inside each loop, there is another competing loop, which will be aggregated into one flow in the next loop (i.e.all sources send packets, which will be aggregated at the next router).Therefore, in the last loop, only two flows compete for the bottleneck capacity, and each one should get half of the capacity; in the previous loop, the two flows that are aggregated in the next loop using one of the flows should also get half of the send rate of that flow.
If we assume a normalized capacity of 1 in the last loop, then last-loop flows get 0.5, each flow in the previous loop gets 0.25, and the values are 0.125, 0.0625, and 0.03125, respectively for the flows in the previous loops.Apparently, in both scenarios, the rate of loop i, x r,i , is the carrying capacity of loop i − 1, but p * r,i is equal to 0 and 0.5 in scenarios 1 and 2, respectively.In the figures, we only report the rate of one flow in each loop labeled with "Loop i".At step 200, we cut the capacity of the last loop to 0.5 to observe how fast the chain of congestion controllers can converge.
We evaluated the Primal-Dual and Dual algorithms under the above scenarios.In addition, to see how a chain of controllers that do not employ the notion of carrying capacity behaves, we used a controller with a logarithmic utility function, i.e.U (x) = log(x).Hence, the rate dynamics in loop i are given by ẋr,i = γ r,i 1 pr,i .
The Dual algorithm ( 16) was used for the algorithms, and the controllers were also tuned for fastest convergence.Fig. 8 and Fig. 9 illustrate the results of scenario 1 and scenario 2, respectively, and Fig. 10 shows the results of the controller with a logarithmic utility function.From the figures we observe that LGCC can converge faster when the capacity drops to 0.5 in loop 5 at step 200, and also at step 400, when it changes back to 1.It might seem that a Dual method is superior to the other one.As mentioned before, if links in a loop is not shared by other flows, a Dual method can be used to jump to the rate routers imply; however, it is not true in a general topology.
VIII.ANALYTICAL EVALUATION In this section, we analytically evaluate properties of LGCC such as equilibrium, stability, and fairness.

A. Equilibrium
Depending on where the bottleneck in the network is, LGCC can converge to different values.For this analysis, we assume that the bottleneck is always in some loop n, and then, the min operator in (10) always kicks in, and for the sake of simplicity, we assume that schedulers in Fig. 6 employs a fair queuing discipline.Hence, assuming x r,i+1 (t) ≤ c l (t), ∀l ∈ L r,i , ∀t: and when the marking probability is deflated Employing a fair queuing discipline in the scheduler implies that x * r,i = x * s,i for competing flows r and s in loop i.Using the fact that r x * r, i = x * r,i+1 , we get where S is the number of competing flows.Solving for p * r,i yields p * r,i = 1 − e log(1/S) log(ϕ2)/log(ϕ1) .

B. Stability Analysis
Here, we discuss the stability of LGCC.The following theorem will prove its global stability.
Proof: See Appendix A-A.□ Although Theorem 8.1 proves that the system of equations ( 14)-( 16) is globally stable, we also analyze its local stability; Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.this provides us a relationship between the step size parameters γ and β, and we can use this as a guideline for setting those parameters.
Local stability refers to the analysis of the system behavior close to an equilibrium.We use the dual algorithm (16) as the cost generator at routers.This indeed includes a step size parameter, β, which can affect the system stability.
We use an aggregation tree topology that is illustrated in Fig. 11 with variable depth (the number of loops of the chains -denoted by δ) and breadth (the number of competing flows in each loop -denoted by ζ).The circles represent LGCC routers at which incoming flows compete.For the sake of simplicity, we assume that the flows in each loop only compete at the input queue of the next loop, and they are all aggregated into one flow in the next loop.
Theorem 8.2: The system of equations ( 14)-( 16) is locally stable if and only if Proof: See Appendix A-B.□ The above theorem implies that the stability in each loop i only depends on the number of competing flows in that loop; this means that the value of γ i and β i determine the stability, and as we get closer to the bottleneck link, i.e. i = 1 in this case, γ should be smaller than β.

C. Fairness Analysis
Fairness properties of a congestion controller determine how fairly the capacity is shared among a number of competing flows and how much each flow can expect to receive.Since LGCC chains controllers, we focus on two types of fairness: intra-loop and inter-loop.
1) Intra-Loop Fairness: Here, we focus on only one loop.We consider a number of flows competing for links capacity in an arbitrary topology.First, we define a new type of fairness.
Definition 1 (Logarithmic Proportional Fairness): Every rate change from the equilibrium proportional to the logarithm of the equilibrium is non-positive.Formally speaking, This is obtained from the fact that if the feasible set is convex and U (.) has a globally maximized at x * , then for any feasible allocation vector x, we have ∇(U )| x * (x − x * ) ≤ 0 meaning that the gradient of U , evaluated at x * , times the change is not positive.The above definition also implies that if there are n sequential links with the normalized capacity of 1, 1 n-hop flow with rate x 1 crossing all the links, and n 1-hop flows crossing each link with rate x 2 , then x * 1 is the multiplication of the equilibrium rate of other 1-hop flows in its path.In other words, 2) Inter-Loop Fairness: To focus on only inter-loop fairness, here, we assume that all routers support LGCC.This implies that loops are formed in a hop-by-hop fashion.In this case, schedulers determine the fairness type.We consider a number of flows in a loop arriving at an LGCC router and analyze two types of chaining: 1) the flows are aggregated into one flow in the next loop, and 2) a separate flow is created in the next loop for each flow.We employ a weighted fair queuing discipline in LGCC routers.LGCC is able to attain max-min fairness in type 1) if weights are set equal to the number of aggregated flows, and proportional fairness in type 2) if weights are set in a special manner.The following theorems prove these characteristics.
Theorem 8.3: In case of aggregated chaining, if weights are set equal to the number of aggregated flows, LGCC attains max-min fairness.
Proof: See Appendix B-A.□ Theorem 8.4: Let p u r,i denote the upstream marking probability of flow r, i.e. the marking probability at the output queues of LGCC routers 1 to i.In case of per-flow chaining, if weights are set inversely proportional to − log(1 − p u r,i ) at the scheduler in LGCC router i + 1 for all routers i on the path, then LGCC attains proportional fairness.
Proof: See Appendix B-B.□ Since in LGCC we assume that each loop operates with its own marking probability, accessing p u r,i at downstream routers would need a special bit in the LGCC header to convey the marks received so far at the output queues of LGCC routers.

IX. LGCC ALGORITHM AND IMPLEMENTATION
Algorithm 1 shows how each sender adjusts its rate.It can be either a Primal-Dual or a Dual algorithm.At startup, send rate x is set to the initial value x init ; this can be interpreted as the initial window size of TCP.However, the unit of x is bits per second.We estimate the loop marking probability, p, using an exponential smoothing average with parameter α.As a rate-based method, we define intervals to sample ECN-marks, and at the end of each interval, p is updated.The carrying capacity is also obtained: it can be the minimum link capacity the loop, or the last send rate of the next loop, which is echoed to the previous loop via the receiver window size field; the value of this field is interpreted as a rate in bits (or save space, Bytes) per second.Then, depending on the algorithm, and in case of Primal-Dual, ( 14) is run.Before the rate calculation, from line 11 to 18, the adaptive step size algorithm is run.The goal is to be more aggressive if the calculated rate from a Dual algorithm is far from the current send rate; this will be discussed in the next subsection.Then, the send rate is set, ensuring it is not lower than the initial send rate.In case of Dual, the send rate is directly calculated from the measured marking probability, p.Details of the parameter setting are presented in the next section.
In both of the algorithms, we assume that routers run (17), which can be realized through a special configuration of RED.At the end of the loop, the receiver module of the next LGCC router, echoes back ECN marks.x ← k e log(1− p) log(ϕ1)/log(ϕ2) x ← xγ log ϕ2 1 − p − log ϕ1 x k + x 20: x ← max(x, x init ) 21: else if Dual then 22: x ← k e log(1− p) log(ϕ1)/log(ϕ2)

23:
end if 24: end procedure The following subsections elaborate on some aspects of this algorithm that go beyond the previously discussed basic LGCC dynamics: adapting γ, obtaining the right capacity value to use (in line 9, we assume that the minimum of all link capacities is known), and how often to trigger "OnIntervalEnds".

A. Adaptive Step Size
To improve the performance of LGCC via faster convergence, we use the same technique that we used in LGC [16] to adjust the step size of the Primal-Dual algorithm, i.e. γ.In LGCC, p and accordingly, (15) provides an indication of what the target rate can be, and how far the current send rate is from that target.In the Primal-Dual algorithm, when the current rate is far from the target, we use a larger value of γ.When the rates are close, then we use a small value for γ to have a smoother behavior.The algorithm (line 11 to 18 in Algorithm 1) determine if the difference of x and x are larger than the threshold g 2 .If so, γ is set to a large value, i.e. γ init ; if the difference is smaller than the threshold g 1 , then γ conv is used; otherwise, it is set linearly based on the proportion of the difference.The range [γ conv , γ init ] is chosen such that it does not affect stability.

B. Carrying Capacity
As shown in (10), LGCC senders need to know the carrying capacity of their loop to obtain their rate; this could be either the minimum link capacity on their path in the loop, or the current send rate of the next sender in the next loop of the chain.In [16], we showed that if an LGC sender cannot get the carrying capacity, it still can operate (as we see in Fig. 2(b)).However, the queue length fluctuates more.When each loop is formed under one domain or over a specific Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.medium, e.g. a wireless link, obtaining the carrying capacity is sometimes straightforward.In wireless links with timevarying capacity, the current send rate (depending on the noise level and modulation) is the carrying capacity.This is how the current transmission rate is obtained in some work such as [23].ABC also uses a method based on the output queue drain rate to estimate the current link capacity.Hence, we assume that if there is a link with time-carrying capacity, an LGCC router is run on that node to start a new loop over that link; in case of wired links, LGCC routers can be run on interior/exterior routers of each authority domain, e.g.ISPs.

C. Rate Update Time Intervals
The rate is regularly updated by each sender per interval.The LGCC dynamics, (9), imply that the fair rate only depends on p in each loop, and as long as all sources see the same p, they converge to the same rate.This means that the steady state behavior of the controller is not sensitive to the update interval: updating it more or less often will naturally make the control converge faster or slower, but the point of convergence is not affected (but updating it extremely fast could lead to oscillations).There are several options for the update frequency such as using the smallest RTT value seen over a certain time interval (an estimate of the RTT without queuing delay).If a flow's smallest RTT is still a long time interval, especially for a large-RTT flow, we can use shorter intervals.Based on our observation in simulations (performed in [16], the performance is robust if a flow can receive at least 4 ACKs per interval.Hence, another option is to use small, equal-sized intervals that can fit receiving a few ACKs.

A. Simulation Setup
In this section, we present simulation results of LGCC over various topologies and scenarios, compared with ABC as the state-of-the-art.Simulations were performed in the INET framework 6 of OMNeT++.We set the general parameters as:

B. Flow Competition and Fairness Evaluation of LGCC
Here, we evaluate LGCC using when 5 sources (flows) competing for the bottleneck in a dumbbell topology, sending packets to 5 different destinations.Links have a capacity of 500 Mbps with 1 ms propagation delay.Router 1 and Router 2 that connect sources to destinations are LGCC routers, making 3 loops, and the capacity of the link between them changes every 1 second: it is 500 Mbps in the first simulation second, and drops to 400 Mbps at second 1 and to 300 Mbps at second 2.Then, it changes back to 400 Mbps and 500 Mbps.Fig. 12(a shows how the rate of the 5 flows in the middle loop changes over time, and the dashed, black line is the fair share of the capacity.We see that sources closely track the capacity change while competing with each other.Fig. 12(b) plots the marking probability of the flows in the second loop, i.e. flows starting from Router 1 to Router 2. These flows compete at the output queue of Router 1.We used RED as the Dual algorithm with max th = 75 KB, and ϕ 1 = 10 and ϕ 2 = 3.The Equilibrium Marking Probability (EMP) is 0.6.Referring to the figure, we observe that EMP only depends on the number of competing flows, not the current link capacity.There are two drops for a short period in MP because the carrying capacity suddenly increases, and it takes a short period for the flows to converge by adjusting their rate.

C. Comparison With ABC
We compared LGCC with ABC.As the first topology, we used a dumbbell topology with 5 sources, sending packets to 5 different destinations.Fig. 13 shows the result of the first scenario, in which the flows start together, and in [2,3]s, the capacity drops to 150 Mbps.The links have a capacity of 500 Mbps, and the link propagation delay is 1 ms.The queue length of the bottleneck router is also shown.Both ABC and LGCC have a special router at the bottleneck link.From the figures we see that LGCC operates faster and more smoothly in terms of convergence, which also results in a smaller queue at the bottleneck.
In the next scenario over the previous topology, flows join one-by-one every two seconds, and they last for 12 seconds.Fig. 14 illustrates how the two protocols operate; again, LGCC converges faster and behaves more smoothly than ABC with a smaller queue length.
We ran the above two scenarios 10 times, each with a different random seed and a random start of flows.Fig. 15 shows how LGCC and ABC operate.Short-term fairness was calculated over the average values of time windows of 24 ms.This also confirms that LGCC operates more smoothly, and flows converge fast.The queue length is accordingly smaller because of the faster reactions to changes in the number of flows.
In the next scenario, we ran ABC and LGCC over three real LTE link capacity traces of the AT&T, TMobile, and Verizon operators.These are the traces used by [15] to evaluate ABC.The topology includes a destination, connected through an LTE link whose capacity changes over time (the capacity  follows the pre-measured trace) and a source 3 hops away from the destination.Fig. 16 illustrates the difference between the send rate of Source and the rate in the trace file at any time; the smaller the difference, the faster the controller at convergence.In case of 2 flows, each flow should get half of the current capacity, and the difference is calculated with this.Referring to the figure, we observe that LGCC flows can track capacity changes faster, leading to a smaller queue at the LTE link.As the number of flows increases, the difference between ABC and LGCC becomes smaller because each flow should receive a smaller share of the capacity.The link utilization of the protocols are 93% for both ABC and LGCC.

D. 5G Use Case
In the next scenario, we used a 5G trace collected from a major Irish operator in [43], which is the first publicly available dataset containing throughput information for 5G networks.The dataset contains traces of two different mobility patterns: static and car.We used the traces of the static pattern because results of [43] have shown that 5G coverage in streets was not complete, so the car traces mostly included LTE/4G.The trace files had some dramatic capacity changes dropping to values of around 0.3% of the maximum achieved throughput.We did not consider cases with connection timeouts or with dis-connectivity since our focus was on devising a congestion control method.
We applied the traces in OMNeT++ to dictate link speeds over time, where the maximum capacity was 333 Mbps.The topology was the one in Fig. 5(a) with m = 5 except there was only one source, i.e.Source 1, and there were 5 routers, making 6 hops to Destination.Link capacities were 500 Mbps with a propagation delay of 5 ms, yielding an endto-end RTT 60 ms.There was one flow from Source 1 to Destination transferring a large file.We used this application as an example of various high-bandwidth applications 5G is envisioning such as holographic video call. 7We set ABC's δ parameter to 300 ms; ABC recommends setting this to at least two third of the base RTT to guarantee stability.The ABC's minimum queuing delay threshold parameter were set to 1 ms.In general, we tried to set the ABC's parameters such that it is more responsive to capacity changes, has a shorter queue, while stability is not affected.
We ran ABC and LGCC over this network with various link configurations.First, we assumed that only the link between Router 5 and Destination is 5G with capacities collected by [43]; the results are shown by "ABC-R5" and "LGCC-R5" in Fig. 17, and there is only one ABC/LGCC router deployed, which was on Router 5. Second, we assumed that the link between Source 1 and Router 1 is 5G, with results shown by "ABC-S" and "LGCC-S"; in this case, the ABC queue is installed on Source 1 while the LGCC router logic should be installed on Router 1. "LGCC6-R5" shows the results of 6 LGCC loops, operating hop-by-hop where the last hop was 5G.In the third scenario both of the above links were 5G, shown by "ABC-2B" and "LGCC-2B".We used different combinations of traces as capacities, combined all collected data, and measured queuing delay illustrated as box-whisker plot, excluding outliers, in Fig. 17 with a logarithmic y-axis.
The queuing delay measurements of LGCC were performed in two categories: 1) the bottleneck link queuing delay, measuring only the delay at the bottleneck, and 2) the end-to-end queuing delay, which measures the delay of every queue (including all the senders' queues at LGCC routers) along the path to destination.The aim of the latter measurement is to include effects of all congestion control loops since they might momentarily delay forwarding packets if their current send rate is smaller than the send rate of their previous loop.In case of ABC, the end-to-end queuing delay is equal to the bottleneck queuing delay.
From the figure, we observe that LGCC can reduce queuing delay significantly.The achievable utilization8 values of ABC and LGCC for the scenarios shown in Fig. 17 from left to right were 93.9%, 85.8%, 93.9%, 90.2%, 87.8%, 94%, and 90.3%, respectively. 9In total, LGCC's utilization was a bit smaller than that of ABC because LGCC tries not to exceed the carrying capacity, which is the current link bandwidth; however, ABC needs a small standing queue for the sake of stability, which helps the link in sending more packets when its capacity increases, but at the cost of an order of magnitude Fig. 17.Queuing delay represented by a Box-Whisker plot over a 5G use case.Figure 5(a) topology with: i) last hop is 5G-ABC-R5 and LGCC-R5; ii) first hop is 5G-ABC-S and LGCC-S (LGCC R1); iii) last hop is 5G-LGCC6-R5 (LGCC all routers); iv) both first and last hops are 5G: ABC-2B and LGCC-2B (LGCC R1 and R5).Boxes span from the 0.25 quantile to the 0.75 quantile, where outliers are not shown for the sake of visibility.increased delay.LGCC-R5 has the lowest utilization and larger delay values than the other LGCC configurations because it comprises a long loop from the sender to the last router, where the bottleneck is; this also implies the benefits of shortening control loops.The end-to-end queuing delay of LGCC-R5, in Fig. 17(b), shows larger variations compared with the delay of ABC-R5 because in LGCC-R5, the sender at Router 5 might delay forwarding packets due to a lower send rate than the incoming rate, which does not happen in ABC.However, the median value is smaller in LGCC.
Any capacity change takes time to affect the source.We see that if the first link is 5G, shown by LGCC-S, any change is reflected very fast at the source, resulting in the highest utilization (90.2%).LGCC6-R5 which has 6 loops, can compensate for this without increasing queuing delay.ABC's utilization is approximately equal in all the cases because it is end-toend.If both source's and destination's links were 5G, we see that LGCC and ABC achieve approximately equal utilization even though LGCC has only 3 loops; this is because the first loop, similar to LGCC-S, is very fast, compensating for the second loop that is longer, and the average utilization becomes comparable to that of ABC.
In order to evaluate the effect of increasing the number of loops, we also measured the utilization of two other cases where the last link is 5G: LGCC with 3 loops, where Router 1 and Router 5 are LGCC routers, and LGCC with 4 loops where Router 3 acts as an LGCC router as well.utilization values were 85.8%, 86.4%, 86.6%, and 87.8% respectively, for the 2 loop (LGCC-R5), 3 loop, 4 loop and 6 loop (LGCC6-R5) scenarios.All the queuing delays are very close.This shows that adding more loops can increase the utilization, without increasing queueing delay, due to shortening control loops and faster convergence.
As discussed previously, the trace files included some dramatic capacity changes.So, if future 6G (and beyond) links have similar characteristics, as expected with sub-THz and Visual Light Communication (VLC) technology, LGCC is expected to operate similarly well.However, it should be noted that LGCC is only a congestion control method, expecting other underlying network mechanisms, such as multipath forwarding and load balancing, to manage cases where a link is down for a long time.LGCC could also be extended to work as a multipath congestion control scheme operating similarly to MPTCP [44], but with the additional advantages that LGCC routers provide.

E. Flow Aggregation
One of the potentials of LGCC is flow aggregation.Although a detailed evaluation of this needs a complete aggregation protocol (including scheduling algorithms and some form of signaling for fair sharing), which is out of the scope of this paper, however, using a simple topology we demonstrate how it looks in practice.Fig. 18 shows the results of two flows aggregated at an LGCC router into one flow on a dumbbell topology.The link capacity is 500 Mbps, but the capacity at the second link drops to 150 Mbps in [2,3]s.Since there is only one flow operating on the second link, there is no competition there.The LGCC router only applies a fair queuing discipline on the flows to be aggregated.The general LGCC router design shown in Fig. 6 and the two types of LGCC algorithms (Primal in (14) and Dual in (15)) allows different ways of implementing this scenario.We adopted the Primal algorithm along with a marking policy at virtual queues of the LGCC router at the bottleneck.The marking policy operates as follows: it randomly marks every incoming packet with a fixed probability of 0.4 that is calculated by the scheduler using (22) to fairly divide the next sender's rate among these incoming loops.So, the flows receive half of the capacity, and in case of a drop, they follow the capacity change fast.The scheduling does not change the equilibrium marking probability seen by sources; the spikes in Fig. 18(b) only happen when on-the-fly packets arrive at the bottleneck link whose capacity is just dropped.This scenario, however, represents a simple case showing the possibility of aggregation, which can be of a huge potential in LGCC.For example, the use of a Dual algorithm at sources eliminates the waiting time for them to reach the maximum/equilibrium capacity, and newly-joined flows can instantly hop on the already established aggregated flow in the bottleneck.However, a more complicated scenario needs further design considerations, which are considered as future work.

XI. CONCLUSION
In this paper, we presented a novel multi-hop congestion controller using the well-known Logistic Growth (LG) function in a food chain style.The idea of carrying capacity in LG parallels with the bottleneck link capacity, which in case of link layer capacity measurements, can provide an excellent feedback to sources to adapt their send rate, especially for links with time-varying capacity.This combined with employing ECN enables us to deploy LGCC over path segments, and react faster to available capacity changes.
We analytically evaluated LGCC for stability and fairness; we proved that the system is globally stable, and provided conditions for the step size parameter setting.Through simulation study, we evaluated the performance of LGCC in terms of latency (queuing), utilization, and short-term fairness.We also used real-life traces of some LTE/4G/5G link capacities, and compared LGCC with ABC.Results confirm that LGCC can improve the delay and fairness performance significantly.
LGCC greatly reduces queuing delay on the bottleneck link, minimizing the delay impact on other flows.It does this by moving queuing from the router's shared output queue, to a separated LGCC sender queue.LGCC also keeps a low e2e queuing delay, while maintaining a high utilization.In this paper we focused on delay.The balance between delay and utilization can be adjusted to meet QoS requirements, and does not impact the QoS of other flows sharing the bottleneck.In addition, LGCC can provide flow aggregation, which is of great benefits for congestion control.
As future work we will look at the effect of delay on control loop stability.Preliminary analysis indicates a generally stable system but suggests the corner case of connecting loops with slightly different equilibrium points needs further investigation.In addition, we will perform a deeper analysis of flow aggregation, devise a flow aggregation protocol, and provide a scalable solution considering heterogeneous flows, which might not be greedy; this will include the case where a flow has a bottleneck after it is split from an aggregation.Performing experiments on real use cases is a key interest of ours.
which is inspired from the Lyapunov function in [17] where the marking is mapped to cost using the function − log ϕ (1 − p * l ).Fixing the cost dynamics to the Dual1 algorithm and taking the time derivative of V(x, p) yields The sum of (26a), (26b), and (26c) is equal to zero.(26d) and (26e) are less than or equal to zero: if, for example, y * l = c l , then (26d) is zero, and if y * l < c l , then p * l , which makes log ϕ (1 − p * l ) zero, and since − log ϕ (1 − p l ) ≥ 0, this term is non-positive.The same argument holds for (26e).In order for (26f) to be non-positive, if x r ≤ x * r , then we should have U ′ r (x r ) ≥ − log ϕ (1 − p * r ), or vice versa.This means that, for example, in case of x r ≤ x * r , x r,i+1 should be large enough to let the first term become non-negative.Since we connect each loop to the next one (i.e. the carrying capacity in each loop is only dependent on the rate of the next loop, not the previous one), loops converge from the last one, and for each loop, x r,i+1 will increase until it meets the above condition and stabilizes the system.In case of x r ≥ x * r , the first term should be non-positive.Since the value of x r,i+1 does not depend on x r,i , this means that x r,i+1 should be small enough for the term to become non-positive.Again, as the next loop converges first, and it does not depend on the previous one, this condition is met.This means that the Lyapunov function is negative, and it is zero at the equilibrium, making the system globally, asymptotically stable.□ B. Proof of Theorem 8.2 The following theorem establishes the necessary and sufficient conditions for local stability.
Theorem 1.1 (see [45]): The equilibrium x * of the system of equations ẋ = F(x) is locally stable if and only if all the eigenvalues of the Jacobian of the system of equations evaluated at x * have real parts less than zero.Now we prove Theorem 8.2: Proof: First, we write the system equations and then, we obtain the equilibrium.We use (9) to obtain system dynamics.Consider some LGCC router i at depth d of the aggregation tree whose flows are aggregated into flow x k,j in loop j at depth d + 1.The incoming flows to router i are governed by ẋ1,i = x 1,i (t)γ d 1 − x x y,i − x k,j .
In the loop at depth δ, x k,j = K.The equilibrium of the above system is given by x * .,i= (1 (this is obtained by considering that the aggregation tree is a perfect m-ary tree, where the number of nodes determines the number of rate dynamics, and the number of non-leaf nodes also determines the number of LGCC routers).Although obtaining eigenvalues of a matrix is not trivial, since the above matrix has a special form, we can calculate its eigenvalues for an arbitrary ζ and δ.Each element of the Jacobian matrix is given by ∂ χ ∂υ where χ and υ are any two system variables.For example, for 0 ∂ ẋb,i and the partial derivatives are zero for other variables.This matrix has a number of eigenvalues; some of them are negative, and some of them are in the form of Constraining this eigenvalue to be negative and solving for a relationship between γ and δ yields (23).□

APPENDIX B FAIRNESS ANALYSIS
A. Proof of Theorem 8.3 First, we provide a definition and theorem from [46] on max-min fairness.
Definition 2 (Bottleneck Link): Link l is a bottleneck for source r if and only if link l is saturated, and source r has the maximum rate among all sources crossing link l.
Theorem 2.1: A feasible rate allocation vector to all sources is max-min fair if and only if every source has a bottleneck link.Now we prove Theorem 8.3: Proof: At every LGCC router, there is a schedule per-output port.Taking into account that all the incoming flows that are forwarded towards the same output port are aggregated, and they are fairly scheduled, if this is LGCC router 1, then it is a bottleneck for source r; if the bottleneck is at some router i, then our weight assignment guarantees that again this is a bottleneck for source r, and hence, every source has a bottleneck, confirming that the policy attains max-min fairness.□ B. Proof of Theorem 8.4 Lemma 2.2: A proportionally-fair rate allocation vector assigns rates inversely proportional to the end-to-end cost the network charges each source r.
Proof: With a logarithmic utility function, which attains proportional fairness, x * r = 1 l∈Lr q l .□ Now we prove Theorem 8.3: Proof: According to the employed scheduling policy, as flow r crosses multiple routers, each router's cost is accumulated.Consider the last router with a non-zero cost: if it constrains x r to 1 l∈Lr q l , then every flow r gets a share of the bandwidth inversely proportional to its cost.Since the sum of costs is less constrained at previous routers, it is the role of the last router to enforce fairness; previous loops of flow r are backpressured due to this constraint.This is similar to limiting the carrying capacity at the last router, not to increase more than a proportionally fair share.In [17], we proved that the function − log(1 − p u r,i ) provides the cost using ECN.Therefore, setting weights inversely proportional to this value guarantees proportional fairness. □

Fig. 1 .
Fig. 1.A simple topology used for a preliminary comparison of DCTCP,LGC, ABC, and LGCC.The links have a capacity of 500 Mbps, and Source's end-to-end RTT is 6 ms.The link capacity between Router 2 and Destination is time-varying; it drops to one third in[2,3]s of the simulation.Other parameters are set such that 100% utilization with minimal delay is achieved for each method.

Fig. 2 .
Fig. 2. Performance comparison of 1 flow of DCTCP, LGC, ABC, and LGCC.The dotted line in the top row illustrates the link capacity, and the blue lines show the senders' rate using each of the aforementioned methods.

Fig. 5 .
Fig. 5.An example of how loops are formed.

Fig. 9 .
Fig. 9. Chaining 5 loops of where there are two competing flows in each loop, aggregated into one the next loop.(a) and (b): Primal-Dual, and (c) and (d): Dual.

Fig. 10 .
Fig. 10. 5 loops of the logarithmic controller where there are two competing flows in each loop, aggregated into one the next loop.(a) and (b): Primal-Dual, and (c) and (d): Dual.

Fig. 12 .
Fig. 12. Send rates and marking probability of 5 flows when the carrying capacity changes.

)Fig. 13 .
Fig. 13.Performance comparison of ABC and LGCC in a dumbbell topology: 5 flows start at the same time, and the capacity of the bottleneck link drops to 150 Mbps in [2, 3]s.

Fig. 14 .Fig. 15 .
Fig. 14.Performance comparison of ABC and LGCC in a dumbbell topology: 5 flows join one-by-one at every two seconds, and leave after 12 seconds of transmission.

Fig. 16 .
Fig. 16.Box-Whisker plot of rate difference and queuing delay of ABC and LGCC.

Fig. 18 .
Fig. 18.Send rate and marking probability of 2 aggregated flows when the carrying capacity changes in a Dumbbell topology.
We define a Lyapunov function based on the Primal-Dual algorithm in the form of V(x, p) =