A Novel QoS-Aware Multi-Connectivity Scheme for Wireless IIoT

Wireless technology is envisioned to be a major enabler of flexible industrial deployments, allowing agile installations and mobility of production elements. However, for industrial IoT (IIoT) use-cases, reliability and service availability remain as key concerns for wide adoption. To enhance wireless link reliability, we propose in this paper a Selective Duplication with QoS (SDQoS) technique, a cross-layer scheduling solution which leverages radio access technology metrics and transport layer metrics to schedule data transmission across one or more links. The framework is implemented as part of a multi-access gateway solution and its capability is demonstrated in a realistic two-hall industrial production environment for a multi-data flow autonomous mobile robot use-case, using Wi-Fi. The proposed QoS-aware solution shows a major improvement in preserving low latency and reliability for critical control data in the presence of large background traffic as compared to state-of-the-art solutions.


I. INTRODUCTION
The rapid development of the Industrial Internet of Things (IIoT) has led to a diverse set of new technological enablers and concepts that promise to improve flexibility, efficiency, and reliability of production setups in factories [1]. Among them, industrial mobile applications such as Autonomous Mobile Robots (AMRs), or cloud computing, which enables optimized control in a factory by migrating the 'intelligence' of Programmable Logic Controllers (PLCs) from the production shop floor to a centralized location, are becoming a reality [2]. The different industrial use cases and applications built on top of these new technological enablers, rely on the integration of communications, robotics, and edge-cloud; and are based on the exchange of data flows which, typically, exhibit a wide range of requirements in terms of e.g., latency, reliability, and throughput [3]. The The associate editor coordinating the review of this manuscript and approving it for publication was Tiankui Zhang . industrial evolution has also motivated the parallel development of the industrial communication ecosystem which, in general, has evolved from the use of field-buses to Ethernet as the main control communication technology [4]. In this respect, wireless communication is now recognized as a major enabler of flexible industrial deployments, as it allows for agile installations, cable replacement, and mobility of production elements [5]. Here Wi-Fi and 5G New Radio (NR) are identified as main candidates for deployment in industrial scenarios and, while both support Industrial Internet-of-Things (IIoT) applications, in particular, effective reliability and service availability remain as key concerns [6].
As a potential solution, multi-connectivity is identified as an excellent candidate technique to enable the support of reliable communication in case of variable radio channel conditions [7], e.g. due to a harsh radio environment (as it is the case in industrial environments), or due to mobility, while eventually preserving sufficiently high spectral efficiency. There are multiple approaches to multi-connectivity, which VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ can be classified into two groups depending on the layer of the protocol stack in which the functionality is implemented: 1) technology specific (lower layers), or 2) networkagnostic (higher layers). For 5G NR there is native multi-connectivity support at different layers. At the Physical Layer (PHY), NR supports non-coherent joint transmissions from Multiple Transmit and Receive Points (multi-TRP), while duplication is also possible at the Packet Data Convergence Protocol (PDCP) layer to utilize both a primary and secondary base station for transmission and reception [8]. Additionally, Dual Active Protocol Stack (DAPS) provides functionality to enable make-before-break handover [9]. A common characteristic of these approaches is that they require standard-compliant implementation of the multi-connectivity solution at both the User Equipment (UE) device side and network side [10]. Such features are generally not available in existing 5G NR networks as initial 5G NR deployments are mainly targeting traditional Mobile Broadband (MBB) applications. However, in future 5G NR industrial deployments, these specific multi-connectivity features could be enabled -provided that both network and user equipment (UE) vendors implement them.
For Wi-Fi there is no native multi-connectivity support, however there do exist proprietary technology-specific solutions which can provide make-before-break handover by utilizing multiple radio interfaces and Access Points (APs) in coordination, as well as improved reliability by employing a network controlled Time Domain Multiple Access (TDMA) scheduling scheme as a collision avoidance mechanism [11]. However, this requires specialized devices both at the network and device side and is, therefore, not well suited for environments requiring technology co-existence and multivendor interoperability.
Compared to the above techniques, which require network/device side specific features, network-agnostic multiconnectivity is performed at the network/transport layer. Therefore, Radio Access Technologies (RATs), such as 5G NR or Wi-Fi, are simply viewed as communication links which provide a data service with certain characteristics in terms of latency, bandwidth and reliability. This abstraction has multiple benefits, including fast time-to-market and the ability to combine multiple RATs easily. In this regard, this study elaborates on the network-agnostic multiconnectivity concept where multiple communication interfaces are exploited to enhance the communication quality in industrial environments, especially in mobility conditions. Previous works have highlighted the benefit of either duplicating packets or selecting the best interface for transmission [12]. However, quality of service (QoS) was disregarded, i.e. every packet was treated the same regardless of the criticality of the supported application. Multi-flow QoS differentiation on the transport layer has previously been investigated by the authors of [13] and our previous work in [14] for Quick User Datagram Protocol (QUIC) and multi path-QUIC (MPQUIC). Such work illustrates the impact of leveraging the transport layer statistics such as congestion window estimates to correctly allocate sufficient resources to different flows using the same underlying communication channels. However, published solutions do not consider metrics of the underline radio technology, possibly leading to sub-optimal design and performance. This is especially important in e.g., Wi-Fi where the mobility is handled at the device side, as shown in our previous work [15].
In this paper, we propose a novel adaptive multiconnectivity scheme which exploits specific radio metrics of the underlining RATs to improve QoS performance at run-time. Besides looking at suitable metrics to improve the performance of the cross-layer hybrid multi-connectivity solution, we validate its performance by experimentation in a realistic IIoT scenario based on a real industrial use case: AMR with two different data flows (control link and video link), with different QoS requirements. For the experimental validation, a specific implementation of the novel multi-connectivity scheme based on Wi-Fi was done. However, it is important to note that the proposed connectivity solution is fully agnostic to the specific underline RATs, and can also be applied for multi-connectivity over other RATs, such as 5G NR, or across multiple different RATs. Further, in the considered implementation, we leverage QUIC as the transport layer protocol, as it already possesses a mechanism that allows for easy differentiation of traffic flows as well as for providing transport layer statistics. However, the presented multi-connectivity scheme is flexible enough to allow for a different choice of a different transport protocol, if required. As such the contributions of the paper are: • Design of a novel two-step cross-layer link selection scheme, Selective Duplication with QoS (SDQoS), for network-agnostic multi-connectivity, aiming improving reliability of selected data links, based on specific QoS rules, especially in mobility scenarios.
• Implementation of the SDQoS a scheme into an industrial application-agnostic multi-access device with Ethernet traffic support, using Wi-Fi as the baseline RAT.
• Performance evaluation and validation of the SDQoS scheme operation in a real-world factory test environment considering a real industrial AMR use case and traffic flows.
• Comparison of the performance of SDQoS with respect to other state-of-the-art connectivity solutions, and discussion of the achieved gains.
The rest paper is structured as follows. Section II presents the algorithms for the proposed novel crosslayer multi-connectivity link selection scheme, and its implementation-specific details considering an IIoT Ethernetbased gateway and Wi-Fi 6 as wireless technology, which are used in the subsequent performance evaluations. Section III describes the experimental setup and operational industrial environment, as well as the baseline algorithm parameter calibration and optimization. In Section IV, the main performance results of our novel proposed QoS-aware multi-  K ← 2 4: else 5: K ← 1 6: end if 7: for v ← 1, K do 8: if EBW(I v ) > BW i then 9: transmit(I v , pkt) 10: end if 11: end for connectivity scheme are compared to those from other stateof-the art connectivity schemes, and the main findings are analyzed; followed by conclusions and future outlook discussion in Section V.

II. QoS-AWARE LINK SELECTION SCHEME
In this section, we introduce the Selective Duplication with QoS (SDQoS) scheme, a novel cross-layer multi-connectivity link selection scheme that is able to shape and steer traffic from distinct packet flows with separate QoS requirements over a shared set of network paths.
For its formal definition, we consider generally F packet flows (J = {J 1 , J 2 , · · · , J F }) mapped over N network paths (P = {P 1 , P 2 , · · · , P N }), where each of the flows has its own set of QoS policies. The operating principle of the SDQoS scheme, as shown in Figure 1, consists of two stages: 1) Radio-aware link selection: the full set of N network paths are evaluated based on radio quality which leads The procedure for the radio-aware link selection in stage 1 is presented in Algorithm 1. This algorithm sorts the network paths according to the value of an associated radio quality metric r i , i = 1, · · · , N , and selects the best M paths. The sorting happens accordingly to a RAT-dependent function f (·) of the quality metric of the paths and a threshold δ , which introduces a margin in the sorting process. In practice, a value v i is sorted to be larger than v j if and only if v i > v j + δ.
Then, the selected M paths undergo another selection to K interfaces by taking into account a set QoS parameters (q i , BW i ) associated with each flow. Here, BW i denotes an allocated bandwidth for the flow i.e. the priority and q i a parameter determining optional replication of packets across multiple network paths to further improve reliability. The details of the selection process in stage 2 are presented in Algorithm 2, where high priority flows are critical and served with duplication over K = 2 interfaces, while low priority flows are less critical and only served on a single interface. Then, the size of the estimated bandwidth EBW is compared to the parameter BW i of each flow to decide whether the packet is transmitted or not. The intended effect of BW i for flow a F i given a link capacity EBW is illustrated with an example in Figure 2. In this simplified example, there are three flows F 1 , F 2 and F 3 with associated parameters BW 1 , BW 2 and BW 3 . While the link capacity EBW is above both thresholds, packet transmissions occur normally. However, when EBW decreases and goes below a threshold BW i , the packet transmission for that specific flow will be stopped until EBW recovers and ensures enough capacity to re-establish the transmission of the flow. It should be noted that, when significant congestion is experienced (q i == high) and duplication is introduced (K ← 2), a short delay could potentially be added to fine-tune the connection and avoid a more serious congestion.
In case multiple groups of paths belonging to different RATs are available, Algorithm 1 would be used to select candidate paths within each technology group and afterwards, and Algorithm 2 would be used to perform link selection based on transport layer mechanisms in a fully-agnostic manner to the underline RAT. Note that the radio-aware 'prefiltering' of the communication interfaces simplifies the next QoS-aware scheduling procedure. This is meant as a first approach towards novel cross-layer schedulers. An optimal scheduler should eventually take into account both radio performance and QoS parameters when performing the scheduling decision. However, this may lead to a significantly larger complexity in the presence of several radio interfaces and technologies, and will be considered for future work.

A. QoS-AWARE MULTI-CONNECTIVITY SCHEME IMPLEMENTATION
In order to validate its design, the presented novel crosslayer QoS-aware multi-connectivity link selection scheme was implemented. Figure 3 details the functional block diagram of the implementation, which is based on the wireless multi-access gateway (MAGW) with layer-2 tunneling capabilities, previously presented in [16]. This device allows for transparent end-to-end integration of Ethernet-based systems. In transmission, this is done by taking Ethernet packets as an input, processing them, and encapsulating their information into configurable transport protocol packets that are later sent for wireless transmission on selected interfaces. In reception, the process is opposite. The received wireless packets are processed, and their transport protocol payload is decapsulated to re-generate the original Ethernet-based frame information, which is later forwarded as an output.
A number of modifications and inputs are to be provided in order to accommodate the proposed two-step scheduling design to the current MAGW implementation. First, flow awareness needs to map Ethernet (or IP) traffic to a specific QoS rule (sets of BW i , q i ). Second, the scheduling algorithm needs to receive an input of M interfaces, sorted according to the link quality (independently of the RAT). Finally, suitable transport layer statistics and flow parameters are needed to properly shape the outgoing traffic according to the specific BW i and q i constrains for the packet in question.
The specific implementation chosen for presentation in this paper is based on the following design choices: 1) For flow mapping, we leverage IEEE 802.1Q [17], which allows for traffic separation via the use of Virtual LAN (VLAN) IDs in the Ethernet header. Furthermore, this allows for full support of Ethernet and IP traffic flows. 2) For simplicity, we focus on a single RAT (IEEE 802.11, Wi-Fi). However, note that, while the study focuses only on a single RAT, we separate the radio-aware link selection part of from the rest of the tunnel implementation in order to maintain flexibility for future RAT additions (e.g., 5G). 3) Finally, we make use of QUIC [18] as transport layer protocol. This provides a simple way to obtain link capacity estimates in a similar manner to the stream-aware congestion algorithm from our previous work [14].
Under those considerations, the functional blocks of the updated MAGW with QoS-aware multi-connectivity capabilities can be described as follows: • Radio-aware plugin: provides a set of M sorted interfaces eligible for transmission according to Algorithm 1. As Wi-Fi is the selected RAT for the implementation show in study, the selected radio quality metric r i for sorting the available interfaces is the average beacon Received Signal Strength Indicator (RSSI) measured for a given service Access Point (AP). Therefore, the quality metric function f (·) in the implemented radio-aware link selection algorithm is simply defined as where θ is the value which triggers a handover to another AP. The radio-aware link selection procedure will then avoid including in the sorting procedure those paths who are about not to be eligible given the low signal quality. The implementation is based on wpa_supplicant [19] which also allows to perform the connection management. This includes initiating and updating network scans when the trigger level θ is hit, re-establishing connectivity in case of disconnections, maintaining mutual-exclusive connectivity for the modems, as well as polling the radio signal statistics.
• QoS rules: define the mapping of the VLAN IDs to each flow. This corresponds to the GetQoSOfPacket, resulting in the the q i flag and the BW i as per Algorithm 2.
• Peer lookup: contains the mapping of Medium Access Control (MAC) addresses behind a gateway (tunneled). The implemented look-up table relies on the Address Resolution Protocol (ARP) for automated network discovery, i.e., each endpoint MAC address gets assigned to a gateway ID.
• Ethernet packet handler (sender) is responsible for handling input Ethernet frames, extracting the 802.1Q header, and looking up the associated QoS rule.
• Transmission scheduler is responsible for scheduling an Ethernet frame according to Algorithm 2 based on the input received from the Ethernet packet handler (a QoS rule), the radio-aware plugin (sorted list of wireless interfaces allowed for current packet transmission), and protocol-specific statistics (depending on the configured transport protocol). Once a frame is scheduled, it is encapsulated and ready for transmission. This would correspond to the entire cross-layer multi-connectivity link selection scheme, integrating the radio-aware and the congestion-aware link selection stages.
• Wireless transport protocol payload insertion takes care of setting the encapsulated information as payload over a selected transport protocol (layer-4) for wireless transmission, as well as of handling broadcast and unicast packets. This toggles the use of Algorithm 2 based on the configured QUIC transport layer protocol (where quinn-rs [24] is used). Once the packets are ready, they are forwarded to the wireless interfaces for actual transmission. • Wireless transport protocol payload extraction performs reverse processing as compared to the wireless transport protocol payload insertion. It takes care of packet reception over the actual wireless interfaces, and decodes the transport layer payload to forward it to the next stage for decapsulation of the information.
• Decapsulation and peer discovery handles incoming data from the underlying wireless transport protocol and forwards the information for (re)constructing an Ethernet frame. It also updates the peer lookup table in case the source MAC has not been observed before, e.g., in case new devices are added to the network.
• Ethernet packet handler (receiver) is responsible for handling decapsulated received information and shaping it as Ethernet frames. More details about the specific hardware and software components used in the implementation of the MAGW terminals with QoS-aware multi-connectivity capabilities are given in Table 1.

III. EXPERIMENTAL VALIDATION
The experimental validation was performed at the AAU 5G Smart Production Lab, in Aalborg, Denmark. This lab, displayed in Figure 4, is an industrial research testing facility composed of two small operational factory halls, expanding over a total area of 1250 m 2 , resembling a real world factory, where researchers have access to a wide range of operational industrial-grade manufacturing and production equipment from different vendors including production line modules, robotic manipulators, AMRs, etc. [2].

A. TEST DESCRIPTION
A set of tests was conducted in order to evaluate the performance of the proposed QoS-aware multi-connectivity scheduling scheme for industrial applications. In our tests, we considered an AMR which needs to have two distinct traffic flows serviced via Wi-Fi network: 1) A critical control loop which handles the operations of the AMR from a centralized server location. This loops typically handles Priority Data (PD) such as robot status or mission information updates. 2) 2) An uplink Best Effort (BE) stream of background data (BD); which depending on the AMR specific use could be, for example, video data (in case of surveillance or industrial process inspection robot) or robotic K ← 2 4: else 5: K ← 1 6: end if 7: for v ← 1, K do 8: transmit(I v , pkt) 9: end for arm steering control information (in case of a mobile manipulator robot).
To capture the impact and performance gains of the proposed SDQoS scheme, it is benchmarked against the following 3 alternative state-of-the-art Wi-Fi based connectivity strategies: • Single Path (SP) Wi-Fi connectivity with optimized handover; as presented in [15].
• Best Path Scheduling (BPS); as presented in [12]. This corresponds to a generalized execution of Algorithm 1 for the case M = 2, K = 1, i.e. there are two paths and there is no duplication.
• Selective Duplication (SD), i.e. duplicating the critical traffic and steering the background traffic according to the BPS. The SD procedure is shown in Algorithm 3, which is a simplified version of Algorithm 2, where the congestion control based QoS mechanism is missing, meaning that infinite capacity is assumed. It should be noted that, as the SP, BPS, and SD connectivity schemes do not make use of QoS indicators, their implementation makes use of User Datagram Protocol (UDP) as transport layer protocol; this is different from the proposed SDQoS scheme whose implementation is QUIC-based.

B. SETUP AND MEASUREMENT PROCEDURES
Practically, testing was done with a MiR200 AMR [26] equipped with the MAGW with the implemented QoS-aware multi-connectivity scheme, which was set to navigate a predefined route with an approximated total length of 140 m in the factory test environment. Such route is marked as a black thick line in Figure 5, which displays the layout of the factory test environment. The route is designed such that there are at least two APs at each position with a RSSI level above the handover threshold θ. Multiple iterations are taken in the route, in order to collect a sufficiently large number of measurements. The MiR200 is able to navigate between predefined checkpoints, using Light Detection and Ranging (LiDAR) technology combined with other onboard built-in sensors such as 3D cameras or proximity sensors, with a +/-5 cm accuracy. The use of such an AMR allows for automated execution of the measurement route. This ensures repeatability of the mobility pattern across the industrial scenario in the different measurement campaigns performed in the study. The factory floor is equipped with 3 CISCO Meraki MR36 Wi-Fi 6 APs [25], which are deployed at the positions also highlighted in Figure 5. The APs are ceiling-mounted and provide full coverage to both factory halls. The test environment is fully radio-controlled, meaning that only devices and APs that are a part of the experiment generate traffic in the network, and that their spectral channel allocations can be fully organized. In this case, each of the APs are configured with 20 MHz non-overlapping channels in the 5.6 GHz region.
For the AMR data flows, the critical PD control loop is emulated via transmission of ping messages of 64 bytes between AMR and a centralized server location, transmitted in intervals of 50 ms. The BE BD stream is generated via iperf3 transmission started at the AMR at a rate of 10 Mbps in uplink. Each of the flows is continuously monitored by the MAGW software in order to collect relevant performance statistics in terms of Round-Trip Time (RTT) latency, throughput, packet loss, and number of Wi-Fi handovers (number of switches between APs). For the critical control loop analysis based on PD, at least 100.000 samples are collected, while for the BE stream of BD, the amount of collected data in the same runtime is limited to 50.000 samples due to practical limitations of the tool [27]. Such a difference in measurement samples available for analysis between the two types of IIoT traffic types is not significant due to their different nature. While the PD can be considered of ultra-reliable type, insight into high reliability percentiles of the statistical distributions of the results is needed to assess its performance. However, the BD is BE and thus, lower statistical significance would suffice to understand its behavior. In any case, with the collected number of samples per test for both types of traffic, it is possible to get a statistical significance with a 95% confidence level close to the 99.99% (10 −4 ) and 99.9% (10 −3 ) reliability levels for the PD and BD traffic, respectively [28].
The evaluation of the implemented novel SDQoS scheme and comparison with the SP, BPS, and SD strategies were done for both idle network and loaded network cases. In the idle network case, the only active device in the multi-AP Wi-Fi 6 network was the MAGW on the AMR with the two configured traffic flows. In the loaded network case, two additional active Interferer (INT) devices were deployed per AP. These were deployed co-located at the static positions denoted by the colored crosses in Figure 5, and were configured to generate symmetric traffic, one by generating uplink traffic and the other by generating downlink traffic. Different interference levels (5, 10 and 25 Mb/s) were tested in order to gain further insights on the performance of the connectivity schemes under different network load conditions. Although the experiments presented in this study were performed with a single active AMR, the effect of having other active AMRs is implicit in the test cases considering interference. Figure 6 illustrates the full multi-connectivity test setup including the AMR with the two configured emulated traffic flows handled by the MAGW (PD flow between AMR controller and edge cloud server, and BD video flow from a camera sensor to the edge cloud server), the 3 Wi-Fi APs, and the interfering devices. Utilization of the implemented connectivity solutions require one MAGW at each end of the communication link. In this centralized IIoT case, the two MAGW are deployed as follows: one at the AMR (GW1device side) and one at the edge cloud server (GW2 -network side). Note that Device-to-Device (D2D) decentralized communication will also be easily enabled and configured by using the MAGW, just by deploying the two MAGW at device sides on different machines. The figure also depicts the 2 active (multi-connectivity) connections to different APs, and how the PD and BD flows are mapped over these connections via MAGW using VLANs.

C. PARAMETER OPTIMIZATION
As the sorting of the paths in Algorithm 1 depends on the threshold δ and the handover trigger level θ, we first performed a set of measurements to determine the optimal threshold in our industrial lab environment with respect to the two flow types. Note that both traffic flows are simultaneously active in each experiment. These measurements were performed using BPS with the cases of no interference (0 Mb/s) and 10 Mb/s (in each direction) for each AP. For each level of interference, we studied the impact of different threshold values, i.e. [0, 3,9,12] dB. Figure 7 displays the obtained results in terms of 99.9%-ile RTT and 99%-ile throughput, as well as loss rate, and number of switches, for different values of switching threshold, considering both no load and loaded cases. The RTT results show clearly the benefit of using a small value of the switching threshold, especially in case of high load. This is because a high threshold leads to 'stickiness' to a certain AP, which delays the switching to another AP with more favorable radio conditions. The presence of the BE BD flow in interfering scenarios and degraded radio conditions leads to the usage of more conservative transmission modes, therefore occupying more resources and possibly delaying the transmission of critical PD messages. Though a 0 dB threshold should offer the best performance as it results in instantaneous switching, it leads to a small throughput degradation and a significant packet loss rate, especially in the presence of interference. As it can be deduced from the number of switches results, this can be due to the emergence of ping-pong effects in the selection of the serving AP, that in our implementation may cause 'race conditions' and packet drops prior transmission.
Based on these results and observations, we set the δ threshold to 3 dB to ensure optimal performance of the connectivity schemes in our industrial lab test scenario, with reduced occurrence of switching events (and packet losses), and limited throughput impact, while still preventing excessive 'stickiness' to a given AP. Using this threshold configuration together with a handover trigger level, θ, of -80 dBm, we experience the link selection behavior described by the trace displayed in Figure 8. In this trace, the MAGW is initially connected to two APs at a time (AP2 and AP3), perceiving different RSSI levels, with AP2 and AP3 acting as primary AP and secondary AP, respectively. When the weaker secondary link approaches the -80 dBm threshold, the MAGW triggers a handover operation, and AP1 becomes the secondary AP. As the GW is moving further from AP2, the RSSI drops and a switch of the primary AP role from AP2 to AP1 happens once the received RSSI from AP1 is higher than the RSSI from AP2 by a margin of 3 dB.

IV. RESULTS AND DISCUSSION
To compare the performance of the SDQoS scheme to the one from SP, BPS, and SD, results are presented in terms of complementary cumulative distribution function (CCDF) of RTT for the critical PD flow (Figure 9), and throughput for the BD BE flow (Figure 10), for all test configurations (idle network and load conditions).
As observed in Figure 9, for the PD flow, SP offers the worst performance of all of the schemes even at no load. This is due to the impact of the Wi-Fi handover as well as excessive 'stickiness' to primary APs. Also, performance worsens visibly as load increases. BPS shows a visible gain with respect to SP thanks to the radio-aware link selection mechanism. When comparing it to the more advanced scheduling mechanisms, we observe that BPS offers similar performance to SD and SDQoS at 0 Mb/s extra network load (idle network configuration) for the 99.99%-ile, but degrades quickly once additional network interference load is introduced. This is because BPS is not designed to leverage the additional network path for duplication.
SD offers better or similar performance to the SDQoS scheme up to the 5 Mb/s network load. This is to be expected since the overhead of QUIC as compared to pure UDP introduces additional delay in the system. However, at increased network load (10 Mb/s) we observe a performance degradation of 26 ms, indicating that the capacity of the network is being exceeded as we can no longer serve the data flows fully. This performance difference increases up to approximately 100 ms further once the load is increased to 25 Mb/s. In general, our novel QoS-aware multi-connectivity scheme, SDQoS, offers consistent performance of around 57 ms for low-medium load levels (0, 10 Mb/s) and increases slightly to 85 ms in the presence of 25 Mb/s interfering load, indicating that, although a capacity issue arises, its effect is not very severe.  While it is clear from the PD flow performance results that SDQoS offers better QoS (prioritization) when the network load is increased, this performance gain comes at a cost. Since the QoS mechanism deliberately drops incoming packets if available capacity is estimated to be lower than the reserved bandwidth for the PD flow, we expect there to be a clear difference in the BD data rate at high network load levels. Thus, Figure 10 illustrates a clear performance throttling of the BE BD whose trend is consistent for both 0 and 5 Mb/s where it is around 7.9 Mb/s at the 99%-ile, whereas for SD it is around 9 Mb/s even though the PD RTT is similar. This suggest that the throttling mechanism may be too aggressive with the setting used for the test at these load levels. When increased to 10 Mb/s, the throttling is however at a similar level but with a performance gain for the priority data. At the 25 Mb/s level, SDQoS throughput decreases to 6.3 Mb/s as compared to SD which is at 8.5 Mb/s, but the tail of the PD flow is significantly better contained.
Summary statistics for achievable priority data RTT and effect on the background data rates are collected in Table 2. The PD critical flow latency performance gains at the 99.99% reliability level achieved with a certain scheme with respect to standard SP operation are computed as SP RTT . For the novel SDQoS multi-connectivity, these gains range from 62.5% to 90% for idle network and loaded network with 25 Mb/s interference traffic, respectively. As previously addressed, these gains for the critical data flow come at the cost of a reduced performance for the secondary flow. SP TP indicates the throughput performance difference for the BD flow between the different connectivity schemes as compared to the one achieved in standard SP operation mode. With SDQoS, the BD flow is throttled by 22% in idle network conditions, while the impact is reduced to 12% in loaded network conditions in the presence of 25 Mb/s interference traffic, which clearly illustrates the benefits of using the proposed novel scheme as compared to less advanced schemes.
It is worth mentioning that the results related to the BPS, SD and SDQoS schemes are marginally affected by implementation-related issues caused by the Wi-Fi cards.
It has been observed throughout the testing that sporadic disconnections can occur; especially when multiple Wi-Fi cards are active simultaneously [29]. This small issue is not related to the implementation of the schedulers itself, and does not impact its decision-making; but rather to firmware/driver issues, which affect mainly the performance of the non-priority data flow. This impact is evident from the fact that the SP scheduler offers slightly better BD throughput than BPS and SD; where the expectation would be that these should be similar. Supplementary debugging could be done to address this issue and further optimize the implementation. This is, however, beyond the scope of the paper, as it is expected that the performance would only improve for all three schemes if this issue was not present, while keeping the observed relative performance and interference tolerance trends unchanged.

V. CONCLUSION AND FUTURE WORK
In this paper we have presented Selective Duplication with QoS (SDQoS), a novel network-agnostic cross-layer scheduling solution which leverages radio access technology metrics and transport layer metrics to schedule data transmission across one or more links. The presented two-step design first down-selects a number of eligible network paths and afterwards applies QoS differentiation on the flows that are to be served based on the estimated capacity of the chosen paths. We developed and integrated the scheme as a part of an operational industrial multi-access gateway (equipped with two Wi-Fi interfaces) and experimentally-validated its performance by running tests based on a realistic AMR industrial use case in a real-world production environment.
Experimental results show that for low load levels up to 5 Mb/s, the SDQoS scheme does not achieve any gain over simpler state-of-the-art multi-connectivity techniques when evaluated for priority data packet delay at the 99.99%ile. However, when load increased to 10 Mb/s we start to observe a clear distinction between SDQoS and SD with a ∼25 ms latency reduction, which increases up to ∼110 ms at 25 Mb/s to 113 ms, at the cost of slightly throttled background data flow. Therefore, the proposed solution shows a major improvement in preserving low packet latency of priority data in the presence of large background traffic as compared to state-of-the-art solutions.
Future work includes the evaluation of the presented QoSaware multi-connectivity scheduling scheme for scenarios with larger deployments and number of links, including more explicit testing with multiple active AMRs. We also aim at extending the presented solution to other (cellular) technologies such as 5G NR and investigating its impact on performance as compared to technology-specific multi-connectivity implementations. Along the same lines, we plan implementing and testing QoS-aware multi-RAT multi-connectivity solutions. Finally, we aim at designing more advanced schedulers which jointly combine the link selection and transport layer QoS, instead of having it separated as a two-step process.

ABBREVIATIONS
The following abbreviations are used in this manuscript: