Processing math: 0%
Timely Survey of Time-Sensitive Networking: Past and Future Directions | IEEE Journals & Magazine | IEEE Xplore

Timely Survey of Time-Sensitive Networking: Past and Future Directions


This illustrates an overview of TSN standardization history. In addition to the quantitative increase in TSN research, a steady effort has been made to standardize TSN. W...

Abstract:

Time-sensitive networking (TSN) is a next generation local area network technology for the coexistence of information and operation technology, targeted to industrial aut...Show More

Abstract:

Time-sensitive networking (TSN) is a next generation local area network technology for the coexistence of information and operation technology, targeted to industrial automation, in-vehicle networks, and avionic networks in the industrial Internet of Things. This paper presents an up-to-date survey of the research efforts in TSN over the past few years, including the latest efforts in standardization by the IEEE. We review more than 170 TSN-related academic research papers published by major academic publishers and categorize these studies according to the topic, purpose, and methodology, analyzing research trends in TSN. The TSN core function deals with the functional and internal perspective to achieve ultra-low latency/jitter and ultra-reliability. Network management includes work with macroscopic and external views, such as configuration, management, and maintenance of TSN-enabled networks. In particular, we analyze the characteristics of existing studies that have evaluated the TSN network and propose future directions to advance the limits of TSN for real-world deployment. We aim this study to be a cornerstone for fellow and new researchers.
This illustrates an overview of TSN standardization history. In addition to the quantitative increase in TSN research, a steady effort has been made to standardize TSN. W...
Published in: IEEE Access ( Volume: 9)
Page(s): 142506 - 142527
Date of Publication: 15 October 2021
Electronic ISSN: 2169-3536

Funding Agency:

Figures are not available for this document.

CCBY - IEEE is not the copyright holder of this material. Please follow the instructions via https://creativecommons.org/licenses/by/4.0/ to obtain full-text articles and stipulations in the API documentation.
SECTION I.

Introduction

The Ethernet [1] is a powerful technology capable of meeting the communication needs of vast number of general purpose applications, and has become commonplace in the Internet of Things (IoT). Traffic from IP-based applications such as web browsing, audio/video streaming, or sensing data from IoT devices are essentially best-effort, and supported on the Ethernet or Wi-Fi, which share the same service philosophy.

Ethernet technology has proved to be quite successful, although no real-time support for the IEEE standard Ethernet existed until recently. Real-time communication is an essential requirement in some industries; thus, it has been developed for specific tasks or domains through numerous proprietary protocols, such as EtherCAT, PROFINET, and Sercos III for industrial automation; controller area network (CAN) and FlexRay for automotive; and avionics full-duplex switched Ethernet (AFDX) for avionics. These protocols perform their designated tasks capably but have compatibility limitations for converging Ethernet-like networks and devices.

As each solution is tailored to a specific application, expanding of these solutions to other industries is also limited. Therefore, the IEEE Time-Sensitive Networking (TSN) [2] task group has been vigorously active since 2012, standardizing real-time capabilities in the Ethernet.

TSN describes several mechanisms for ensuring or improving the real-time transmission of Ethernet traffic and defines the first IEEE standard for time-triggered message delivery in switched Ethernet networks [3].1 The key feature of the TSN standard is to allow flows with various requirements, such as ultra-low latency, low-jitter, and zero-loss reliability, to be controlled on an integrated bridged network. To achieve deterministic real-time communication over the Ethernet for tightly demanding applications, TSN uses globally synchronous time and schedules that route messages across multiple network components. By defining a queue that transmits messages according to a time schedule, TSN guarantees deterministic bounded latency for time-critical reserved traffic, integrating all traffic classes and applications into a single standard-based Ethernet network. In addition to standardization’s documentation and administration perspective, the technological and practical perspectives of maximizing efficiency through flexible prioritization and fine-tailored transmission emphasize the importance of TSN at the core of information technology (IT) such as Industry 4.0, cyber-physical system, and industrial Internet of Things (IIoT).

This paper is an up-to-date statistical survey of research on TSN (as of the end of 2020), aiming to be an informative source to those who wish to conduct academic research on TSN. Instead of rewriting a tutorial [4]–​[6], we present information and pointers that we wished we had known when we first entered the field of TSN. Our work is motivated by the fact that many early studies on TSN are outdated, as TSN was officially standardized only recently (2018 [3]) and is still an ongoing and evolving effort. Some research has even been innocently incorrect because it was based on discussions and terms from prestandard drafts.

Specifically, when we research new technology, we ask the following questions:

  • What is it, and what are the key challenges?

  • What topics are researchers working on? What has been focused on so far, and what is expected in the future?

  • What methodologies are used for this research?

  • What applications, scenarios, and topologies are considered?

  • What kind of devices or simulators are used for experiments or simulations, and what is the scale?

The goal is to provide up-to-date answers to these questions to aid and promote new researchers in TSN.

The remainder of this article is structured as follows. First, in Section II, we clarify the scope, methodology, and taxonomy of this work. Section III provides a statistical analysis of the surveyed papers to discover the temporal and demographic research trends and investigate the methodology, setup, and scale used for TSN evaluations. Based on the classification, Sections IV, V, and VI briefly review the relevant studies in each subtopic, and Section VII summarizes what has been learned and the critical future challenges in TSN. Lastly, Section VIII concludes this study.

SECTION II.

Overview

We first define the scope of the survey and the taxonomy in this work. A note follows this discussion regarding the related tutorials for TSN.

A. Scope of the Survey

This paper limits the survey scope to TSN-based studies to the extent possible and does not consider studies on audio/video bridging (AVB) [7] or time-triggered Ethernet (TTEthernet) [8] for the following reasons. First, studies on AVB have sufficient results for time-sensitive traffic with predictable latency (looser than TSN, but tighter than the best-effort), and have been well covered as a predecessor to TSN. For those who want to learn more about AVB, Teener et al. provided details in [9]. Second, most studies on the TTEthernet have also been passed to TSN or conducted concurrently because time-triggered traffic scheduling is similar to gate control list (GCL) scheduling of TSN. Zhao et al. elaborately compared both in [10].

This decision excludes remarkable studies on extensions to Ethernet-based real-time communication protocols, such as AFDX or flexible time-triggered Ethernet (FTT-Ethernet), among others. However, we believe this decision provides a more accurate view of the work done on TSN because the above protocols are generally expected to be integrated into IEEE TSN as standardization progresses.

We searched academic research publications on IEEE Xplore,2 ACM Digital Library,3 and Google Scholar4 with the keywords “TSN”, “Time-Sensitive Network” and “Time-Sensitive Networking” from 2014 to 2020. Additionally, we went through the references, following papers that cited or were cited by the papers, but studies not found in these databases were excluded. Among these, we reviewed 174 research publications.

The numbers of publications collected for each year are plotted in Fig. 1. After the AVB Task Group changed its name to TSN Task Group, research papers related to the TSN have been published since 2014, and the work volume has steadily increased. As of 2017, 38 research papers were published. Then, the number has increased rapidly since 2018 when IEEE 802.1Q-2018 [3] with TSN key features was officially standardized reaching 59 publications in 2020. Fig. 2 illustrates an overview of TSN standardization history. In addition to the quantitative increase in TSN research, a steady effort has been made to standardize TSN. We believe that this growing trend will continue because the ultra-low latency/jitter or multiclass traffic converged LAN/MAN is expected to become a critical foundation technology. This technology will be required in various industries in the future, such as industrial automation and in-vehicle networks, which are the current main targets. In addition, from the perspective of 5G, which is one of the megatrends of the current technology development, TSN is of great importance as a 5G fronthaul to support traffic classes transmitted and multiplexed through the packet-based switching fabric. Further, as TSN still has technical challenges in standardization and implementation issues and fundamental challenges that must be addressed or evolved, more research publications are expected over the next few years.

FIGURE 1. - Number of surveyed publications per year.
FIGURE 1.

Number of surveyed publications per year.

FIGURE 2. - History of IEEE TSN standardization (with OPC-UA).
FIGURE 2.

History of IEEE TSN standardization (with OPC-UA).

B. Taxonomy

We believe that the most insightful way to classify previous work is to categorize it by purpose (i.e., What are the primary challenges behind the work?). Based on this, the taxonomy was divided into the following three main categories with 12 subtopics.

1) TSN Core Function

In this category, we report work in which the subject is TSN internal functions. TSN is a toolbox providing various functions, not a single technology or protocol. Several functions can be selected individually or unitively according to the requirements of the target application that need to enable TSN characteristics. Therefore, this category is divided again into several subtopics based on individual functionality.

  • Time-Aware Shaper (TAS): IEEE 802.1Qbv Enhancements to Traffic Scheduling [11] pursues the zero-jitter transmission of critical traffic through open/close scheduling of the GCL in each switch’s egress port. We present work that analyzes the performance of the TAS (primarily compared with other shapers) or enhances the TAS and provides insight on the scheduling synthesis approach (SSA). Although the TAS is a novel feature for ultra-low-latency transmission of time-critical traffic, it has a high implementation complexity and additional overhead due to the GCL schedule generation/deployment and time synchronization. Moreover, it is unsuitable for aperiodic traffic flows. Accordingly, TSN also provides other shapers suitable for the requirements of other traffic types.

  • Credit-Based Shaper (CBS): For more relaxed latency-bound traffic, IEEE 802.1Qav Forwarding and Queueing of Time-Sensitive Streams [12] specifies two traffic classes: AVB-A (less than 2 ms per seven hops) and AVB-B (less than 50 ms per seven hops), and defines the CBS that is relatively simple to implement compared to the TAS. The CBS allocates a logical bandwidth for each class by regulating a ‘credit’ to limit the bursts and prevent the starvation of lower priority classes. The characteristics of the CBS can be regarded as a token-bucket-based per-class shaper.

  • Asynchronous Traffic Shaper (ATS): IEEE 802.1Qcr [13] defines ATS, which achieves, as the name suggests, limited low-delay transmission without global time-synchronization. The ATS is flexible in handling mixed traffic, including aperiodic traffic, and does not require time synchronization unlike the TAS.

  • Frame Preemption (FP): The basic concept of IEEE 802.1Qbu Frame Preemption [14] is that express traffic can interrupt preemptable traffic. In detail, when express traffic is ready to be transmitted on a given switch egress port, even if a (preemptable) frame is in the state of being transmitted, the transmission halts immediately, and the transmission of express traffic starts. When the express traffic transmission is completed, the transmission of preempted traffic is resumed. Further, IEEE 802.3br Interspersing Express Traffic [15] abstracts from the PHY layer to provide preemptable MAC and express MAC service interfaces with additional MAC merge-sublayer in egress ports.

  • Stream Reliability: As networks that consider TSN primarily aim for deterministic transmission of critical traffic (e.g., safety-related), one of the key differences in comparing AVB with TSN is reliability. Several industrial reliability architectures exist, such as high availability and seamless redundancy (HSR) [16] and parallel redundancy protocol (PRP) [17]. Nevertheless, TSN published a stand-alone standard IEEE 802.1CB Frame Replication and Elimination for Reliability (FRER) [18], which facilitates reliability via redundancy by applying the stream concept.

2) Network Management

In this category, TSN research work from a macroscopic viewpoint, such as network component discovery, negotiation of management or configuration information (e.g., GCL schedule deployment), or user interaction, is organized. These studies range from the simplest single TSN domain to integrating TSN with legacy real-time or wireless communication systems.

  • Time Synchronization: Because most TSN standards require a global time reference, time synchronization between network components is one of the most critical issues in TSN. Moreover, IEEE 802.1AS Timing and Synchronization for Time Sensitive Applications [19] defines the generic precision time protocol (gPTP) as a new profile that adapts IEEE 1588-2008 precision time protocol (PTP) [20] for TSN environments. The precision and efficiency of time synchronization techniques directly affect TSN performance; thus, several studies have been conducted to improve the mechanisms in IEEE 802.1AS.

  • Framework/Modeling: The anticipated quality-of-service (QoS) can be achieved only when complex and sophisticated TSN functionalities work as intended. Thus, each network component must know the relevant configuration and information promptly. In this subtopic, with the background of IEEE 802.1Qcc Enhancements to Stream Reservation Protocol [21] standard and centralized management, we introduce studies that present network models to communicate configurations among network components efficiently. We also report studies on fancy frameworks for network designers that facilitate network analysis, configuration, or monitoring.

  • Case Study/Proof-of-Concept: As TSN is considered for applications in various target domains, designing a specialized network configuration for a specific purpose or requirement is common. Therefore, in this subtopic, studies that analyze how to configure TSN components efficiently and studies that serve as a cornerstone for replacing or integrating with the legacy network are covered.

  • Wireless Integration: The advantages of wireless, such as the convenience of physical deployment and mobility, are essential elements for industrial automation. Ultimately, extending representative wireless communication technologies, such as 5G or Wi-Fi, to TSN is essential to construct a wired/wireless integrated real-time industrial network. However, as TSN is primarily an Ethernet-based wired system, various additional studies (e.g., clock synchronization, protocol conversion, and radio link failure handling) are underway for incorporation into wireless environments.

3) For TSN Researchers

The previous two categories summarize studies for the internal function and overall use of TSN. In this category, we organize information for researchers beginning TSN research to grasp where to begin quickly, for example, which simulators and real devices support TSN features or which stage of research for these features is underway. We also introduce important but inevitably unclassified studies, such as introductory documents or reports on the industry status for TSN.

  • Simulator: A network simulator can significantly reduce costs when designing and validating new protocols or identifying new scenarios. We introduce TSN simulators based on proven simulator frameworks, such as OMNeT++ [22] and Riverbed [23], or studies that implement specific libraries or modules based on those frameworks.

  • Hardware: Representative commercial off-the-shelf (COTS) equipment, or even development or evaluation kits for academic research, are lacking in TSN. Nevertheless, some papers have investigated practical challenges of TSN support in COTS equipment, or proposed hardware designs to support various functions of TSN more harmoniously while directly implementing in application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or systems-on-a chip (SoC).

  • Miscellaneous: Through tutorials for beginners or reports on the future applicability of TSN technology, it is possible to obtain a rough understanding of why the technology is necessary, what features and characteristics it has, and how it operates internally or with the surroundings. Interested readers can gain preparatory knowledge through these documents before studying TSN in earnest.

In classifying TSN studies of various subjects, it is sometimes challenging to separate studies from each other in an entirely disjointed manner, especially those dealing with combined or converged subjects with ambiguous classification boundaries (e.g., a performance comparison between multiple shapers). However, as one of the significant contributions of this study is tracking changes in research trends, we aggregate the granular classification criteria to eliminate duplicate counting in the statistical analysis section. In contrast, the review section includes as many references as necessary even if they appear more than once, to reduce the misunderstanding of research results during this aggregation process.

C. Related Tutorials on TSN

To focus this paper on a survey of the current state-of-the-art TSN studies and their statistical analyses, we decided not to deal with tutorials on TSN in this paper. This decision was because we wanted to focus on providing concise pointers to interested readers and because a few fine tutorials have already been published [4]–​[6], [24]. We aim to provide information that is not in those tutorials.

For example, Nasrallah et al. [4] regarded IEEE TSN and the IETF deterministic network (DetNET) as L2 and L3, respectively, of an ultra-low-latency network, and provided a comprehensive survey and broad overview on both standards. However, the subjects of the papers they surveyed are weak in the distinction between the TSN and AVB (or TTEthernet) research heritage, and the time range of the papers they surveyed ends in 2018, after which TSN was standardized [3]. Therefore, it is necessary to update many novel studies (over 100) conducted since then.

Bello et al. [24] focused more on TSN and provided a relatively detailed overview of the standards and mechanisms related to the transmission scheme for time-critical traffic, which is the core of TSN. It also provides an analysis of two primary TSN use cases, industrial communication and automation system, as a reference for TSN integration or applicability in both domains. However, they discuss other domains or the general-purpose use of TSN less.

These tutorials are all good references for an overview of TSN and understanding the basic mechanisms of TSN core functions; therefore, we recommend them. However, to the best of our knowledge, our work is the most up-to-date (as of the end of 2020) and the first to include a statistical analysis of research trends and topic categorization in TSN. We believe this paper provides valuable information to readers who want to conduct academic research on TSN.

SECTION III.

Statistical Analysis

This section presents the statistical analysis of TSN studies gathered according to the method discussed in Section II-A. Research papers are summarized and analyzed based on the subtopic classification, evaluation methods, publication outlet, and demographics.

A. Subtopic Classification

Table 1 lists the subtopics of the three main categories and publication statistics for each subtopic by year. First, the TSN core function category was researched the most with 83 publications. As expected, TAS has been studied the most with 46 papers of the five subtopics of the TSN core function category.

TABLE I Number of Time-Sensitive Networking Related Research Papers in Each Subtopic, Tabulated by Publication Year
Table I- 
Number of Time-Sensitive Networking Related Research Papers in Each Subtopic, Tabulated by Publication Year

Early TAS research focused on implementing a solution for flow-based Qbv in a multihop full-switched TSN network while being inspired by the existing frame-based scheduling of TTEthernet. Based on this, more drastic and sophisticated SSAs were proposed to solve more complex problems, including various additional components (e.g., mixed-criticality or joint routing). In addition, studies for TAS enhancements and studies that analyze TAS performance in various environments are included.

Second is framework/modeling, covered by 29 papers, which ranked no.1 in 2019. Starting with the research on using TAS efficiently in 5G fronthaul [25] in 2017, research on wireless integration has increased rapidly, reaching 17 papers. Next, stream reliability and case study/proof-of-concept have 13 publications each.

Fig. 3 illustrates the relative publication ratio of three categories from 2016 to 2020. Most notably, despite the growing number of publications on the TSN core function, their overall share of TSN research has been decreasing. Instead, the sum of the other two categories has been steadily increasing in publication count. Especially in 2020, the network management category recorded its largest number of publications, with more than the TSN core function for the first time. This trend means the efforts on how to implement the basic functions of the TSN have been paying off, leading to focusing on how to use TSN well.

FIGURE 3. - Changes in research trends.
FIGURE 3.

Changes in research trends.

B. Evaluation Method

Proper evaluation is essential to make the tremendous effort devoted to standardizing and enhancing TSN productive. The two rightmost columns of Table 1 display the statistics of the evaluation methods: simulation or experiment.

Overall, 49 studies performed simulations, and 35 studies conducted experiments with real devices. Two studies used both methods (marked with *). Thus, a total of 82 studies evaluated their work using either simulation or experiments, which is less than half of all surveyed papers (47.12%). Fortunately, as TSN research matures, the work ratio evaluated through simulations or experiments has also increased. Using simulations or experiments, 15 and 8 studies (23 of 44; 52.27%) in 2019 and 18 and 21 studies (39 of 59; 66.10%) in 2020 evaluated TSN (or the proposed enhancement), respectively.

For the most popular subtopic, TAS, few papers have conducted simulations or experiments because most TAS studies (primarily related to SSA) have employed numerical analysis methods calculating the proposed objective functions and constraints using a solver (e.g., satisfiability modulo theories (SMT) or integer linear programming (ILP)) or network calculus. Another reason is that decent TSN simulators that support TAS (e.g., NeSTiNg [26]) have only emerged since 2018, and later to support on real devices. Only three of the TAS studies have conducted experiments with real devices, among which two of these papers have evaluated their proposed SSA only in simulations and merely deployed their presynthesized schedules to devices for experiments. Likewise, stream reliability had only one study conducting simulations. In contrast, CBS, ATS, and FP have relatively lower implementation complexity, and more papers have conducted simulations or experiments.

It is disappointing that cutting-edge proposals for TAS improvements have not performed simulation- or experiment-based evaluations due to the lack of an appropriate environment. However, this situation does not mean that a basic performance evaluation for TAS has not been conducted. Some studies on framework/modeling and case study/proof-of-concept have implemented a relatively simple network topology with a basic TAS, and evaluated the proposed architecture through simulations or experiments.

C. Network Scale

We counted the nodes and time-critical flows used in evaluations to demonstrate how closely TSN studies mimic the current real-world scenarios. We were motivated to do this for the following reasons. As many studies have already revealed, the time required for GCL scheduling is dependent on the network size and number of scheduled traffic (ST) flows [27]–​[31]. For TSN to be deployed in the real world, it must complete GCL scheduling within a reasonable time for a generally applicable network size and number of flows. This criterion is generous in that it does not consider the worst cases where network topology is dynamic.

The following guidelines have been established for a high-level understanding of the network scale used for simulations and experiments in vast number of papers with diverse evaluation setups and methods. First, all TSN-enabled switches (SWs) or bridges (including logical bridges) in the evaluation are counted as SWs, and all other network devices other than SWs (i.e., talker, listener, background traffic generator, traffic sink, etc.) are counted as end systems (ESs). Similarly, we count all scheduled flow/stream/traffic as ST flow. Although this may exclude the specificity of each paper, it allows us to understand the overall trend of the TSN evaluation scale.

Further, we targeted all scenarios in which the motivation for each experiment is distinguished, but in the case of a single experiment with multiple network topologies, only the largest-scale topology was selected to focus on the scalability. We believe that by selecting the most daring scenarios attempted by each paper, we demonstrate the limits of the network scale used by TSN evaluations.

Referring to the experimental settings of each paper with the guidelines above, we count the number of SWs, ESs and ST flows used in the evaluation of TSN studies conducted so far and obtained some basic statistics. Table 2 presents the number of scenarios used in simulation- and experiment-based studies and the median, average, and maximum SWs, ESs, and ST flows in those scenarios.

TABLE 2 Network Scale in Simulation/Experiment-Based Evaluations for Time-Sensitive Networking Research
Table 2- 
Network Scale in Simulation/Experiment-Based Evaluations for Time-Sensitive Networking Research

In total, 44 scenarios were used for simulations, and the network topology of each scenario had an average of 7.5 SWs and 12.91 ESs. Among them, 37 scenarios simulated the TAS with an average of 8.89 ST flows. The simulation scenario with the most SWs was by Gutiérrez et al. [32], who analyzed the time synchronization accuracy of a large network using 100 time-aware nodes with up to 100 hops to the grandmaster clock. Moreover, Nsaibi et al. [33] presented a simulation scenario involving the largest number of ESs to analyze the feasibility of integrating TSN into a legacy real-time Ethernet for automation systems. Further, Hellmanns et al. constructed a line-topology consisting of 50 SWs, 52 ESs, and 50 ST-flows. They compared the performance of stream-based scheduling, class-based scheduling, and FP [34]. This study used most SWs in a simulation-based evaluation that included a performance analysis of TAS scheduling.

For real-device experiments, fewer scenarios were used than simulation-based evaluations. The network scale was also smaller with an average of 2.63 SWs, 4.17s ES, and 2.52 ST flows. Nevertheless, Vlk et al. validated the applicability of GCL schedules with two TSN SWs (capability stack: 802.1AS, 802.1Qbv, 802.1Qcc, NETCONF, and OPC-UA Pub/Sub), 14 ESs (13 actuators and one PLC emulated by the IXIA traffic generator), and 13 ST flows [35]. The scenario using the largest number of SWs is a study comparing the performance of TAS and FP in a multi-hop TSN network (maximum of six hops) using seven SWs (capability stacks: 802.1AS, 802.1Qbv, and 802.1Qbu) [36].

To understand the distribution of the network scale used in simulation- and experiment-based evaluations, we plotted the CDF in Fig. 4. Of the studies involving simulation, 80% of the studies used only six or fewer SWs. The number of SWs used in the experiments was even smaller. More than half of the scenarios used only two SWs, and 83% of the scenarios included three or fewer SWs.

FIGURE 4. - Cumulative distribution function (CDF) of network scale (number of switches, end systems, and scheduled traffic flows) used to evaluate TSN research.
FIGURE 4.

Cumulative distribution function (CDF) of network scale (number of switches, end systems, and scheduled traffic flows) used to evaluate TSN research.

These numbers are disappointing. More evaluation must be done on larger topology to demonstrate the satisfiability of TSN’s target specifications, such as the worst-case delay of 100 \mu \text{s} over five hops for ST. In addition, we point out that, while some studies that evaluated the work using numerical analyses have actively used a case study from real-life scenarios, such as the Orion crew exploration vehicle (CEV) [37] or General Motors [38], they have not been simulated. The first study has 31 ESs, 15 SWs, and 100 ST flows, and the second has 20 ESs, 20 SWs, and 27 ST flows, so they are a meaningful experiment target scenario as they represent an appropriate network size for an actual use case.

D. Simulator Frameworks

We investigated the types of simulators used in 49 studies that conducted simulations. The use frequency of the simulators is presented in Fig. 5, referring to the dominant frameworks and their characteristics. This information can be helpful to TSN researchers aiming to include simulation-based evaluation in their work.

FIGURE 5. - Statistics for the simulator framework usage.
FIGURE 5.

Statistics for the simulator framework usage.

First, OMNeT++/OMNEST has been used in 26 papers (23 and three respectively), more than half of the studies that include simulation. Moreover, INET, which is often used together, was used in 21 publications.5 OMNeT++ [22] is an extensible, component-based C++ simulation framework for building a network simulator, and OMNEST is a commercial version that provides more features, and is compatible with OMNeT++. OMNeT++ is a simulation framework without models. Thus, it is primarily used in combination with external frameworks and libraries, such as INET [39], containing models for the Internet TCP/IP stack or wired/wireless link layer protocols. In addition to providing various features, both OMNeT++ and INET are open source based on the GPL license; allowing researchers to use thees when implementing a TSN simulator.

NeSTiNg, CoRE4INET, and TSimNet are representative extension frameworks that implement specialties for TSN and can be used appropriately as needed for TSN simulation. A novel simulation framework especially designed for TSN, NeSTiNg includes 802.1Qbv, Qbu, Qav, and VLAN tagging, and has been used in seven papers, which is the highest frequency among the three despite being the newest. Next, CoRE4INET, originally an extension for simulating real-time Ethernet such as TTEthernet, can simulate AVB and TSN using 802.1Qbv, Qav, and Qci extensions and has been used in three papers. Finally, TSimNet, which primarily implements non timed-based features, such as 802.1Qbu, Qci, and CB, was used in two papers.

The Riverbed simulator has gradually expanded its capabilities through various studies and has been used in seven papers so far. Similarly, TSN functional modules based on the Riverbed simulator have been implemented by several researchers. In addition, some studies use other simulators, such as ns-3, UPPAAL, ALEVIN, PAnSim, and Mininet, which we categorized as ‘etc.,’ along with seven papers that did not indicate what simulator was used.

E. Examples of Real-World TSN Devices

We cannot sufficiently emphasize the need for better equipment and a development environment for researchers to study fast-evolving TSN. From this perspective, we investigated various products for reference to establish TSN now or in the near future. Table 3 lists the names of products by major vendors and the TSN standards currently supported by each product. This list is only an example of TSN-supported devices, not a complete list. All the information is based on the official webpages or flyers that introduce each product. In addition, the rightmost column lists papers that explicitly indicate the use of a related product, and Fig. 6 shows the images of some example products. Several development kits, including SoC and the TSN solutions, offer relatively many features and a high degree of freedom in implementation. In contrast, off-the-shelf SWs allow the ease of configuration and use, although support for additional functionality may be relatively limited and slow. In addition, PCIe NIC can be used to support TSN functionality in ESs.

TABLE 3 Current State of Affairs of Real Devices and Supported Time-Sensitive Networking Standards
Table 3- 
Current State of Affairs of Real Devices and Supported Time-Sensitive Networking Standards
FIGURE 6. - Examples of TSN-supported real devices.
FIGURE 6.

Examples of TSN-supported real devices.

F. Publication Demographics and Outlets

Fig. 7(a) plots publication distribution of demographics based on the first author of each paper. Germany overwhelmingly ranked first in terms of the number of publications, at 60, more than a third of all publications, even more than Asia and America combined. As Germany launched Industry 4.0 as a national strategic initiative, these overwhelming statistics are believed to result from their enthusiastic research on ICT and IoT for the digitization of manufacturing and industrial practices. Denmark (19 papers), Spain (nine papers), Sweden (eight papers), and Austria (eight papers) contributed to a combined total of 73.56% of research from Europe. In Asia, China is ranked second overall with 20 papers, and the Republic of Korea (seven papers) and Japan, India, and Taiwan (one publication each) followed. North and South America published papers from the United States (10 papers), Canada (five papers), and Brazil (one paper). Out of the 24 countries in Fig. 7(a), 15 new participants are listed in TSN research after 2018, showing that countries are increasingly interested in TSN. We believe that this trend will continue in the coming years, leading to innovative research in more regions.

FIGURE 7. - Distribution of publication information.
FIGURE 7.

Distribution of publication information.

Fig. 7(b) plots the distribution of outlet types in TSN-related publications. Overall, 27% of the papers were published in journals, 55.8% were published in conference proceedings, 6.3% papers were published in symposiums or workshops, and 4.6% papers were published in magazines. In particular, 13 papers were published in the IEEE Access journal, and 26 papers were published by the IEEE Emerging Technologies and Factory Automation conference, the two most popular publishing outlets.

As illustrated in Fig. 7(c), 65.7% of all papers were published in IEEE-sponsored publishing outlets, 15.7% were in ACM-sponsored outlets, and 18.5% were published by other publishers, such as Elsevier, Springer, and MDPI, which is unsurprising considering that TSN is an IEEE standard.

SECTION IV.

TSN Core Functions

The most crucial aspect of a time-critical system is to ensure that timing requirements are always met. The TSN core functions are primarily significant enhancements to IEEE 802.1Q in the form of amendments, and most have already been quickly integrated into the IEEE 802.1Q-2018 standard [3]. According to the required QoS, various traffic types are classified by the name of each stream (or flow) and are given a definite priority. Through prescheduled queue gate control [11] or exclusive resource reservation [12], TSN ensures QoS requirements of high-priority streams. Furthermore, transmission integrity is also as crucial. For example, imagine having a necessary appointment. Finding the way to arrive there on time is just as important as scheduling time for it. In addition, it is also necessary to have a backup plan for unexpected catastrophes.

This section presents studies on TSN core functions for ensuring ultra-low latency and reliability for time-critical traffic in multihop switched networks. For readers who require a more detailed understanding of the introduced features, please refer to the representative tutorial papers recommended in each subsection.

A. Time-Aware Shaper

Frame-level scheduling is an NP-complete problem. The general formulation using SMT or ILP was carried over from TTEthernet research or even earlier. This carryover is because the method that enables the transmission of hard real-time traffic based on perfectly synchronized time is similar to the GCL scheduling in TAS that schedules the open-close times of the egress port [56]–​[60]. Although this is a somewhat valid approach, it has limitations [27]. Oliver et al. articulated these limitations and presented a gate window-level approach based on the first-order theory of arrays, which is close to the principle method of GCL scheduling. Reusch et al. [61] extended this window-level synthesis approach to improve flexibility by mitigating mutual isolation constraints between ST flows.

Early SSAs did not consider the possibility of additional routing paths (i.e., simply using the shortest-path bridging that is the most basic use of off-the-shelf SWs). This fixed-route scheduling is the most intuitive to select a fast route by minimizing the number of hops but has latent problems, such as load imbalance and frequent bottlenecks. These problems occur because many flows, including low-priority flows, select the same path. Moreover, because scheduling on a fixed route is formulated in a relatively small solution space, it is not sufficiently representative of the performance, and schedules with better performance that are feasible can be excluded. Altogether, the latent disadvantage of assigning multiple time-critical flows to a small number of specific paths becomes more pronounced as the overall network load increases.

Consequently, research is underway to determine optimized scheduling in the expanded solution space by considering various paths of flows (i.e., solve routing and scheduling problems jointly). However, it is exceedingly difficult to determine an optimal solution within a limited amount of time because routing constraints must be considered in addition to scheduling constraints that are already highly time-complex. Therefore, methods are proposed to obtain practically satisfactory solutions using meta-heuristics, such as the greedy randomized adaptive search procedure, genetic algorithm, or Tabu search [28], [29], [62]–​[73]. Using a completely different approach, such as source routing [74], is also proposed.

Although the primary purpose of SSA is generating a schedule for the deterministic transmission of time-critical traffic, due to the nature of TSN, where various traffic types coexist, the transmissions of lower priority traffic must also be considered. This consideration can be achieved with the proper configuration of scheduled queues, according to the IEEE 802.1Qbv standard, and with routing according to traffic class [75]–​[80].

In this mixed-criticality environment, a guard band (GB) can be a useful option. It can be used to ensure the exact start time of the ST transmission by preventing low priority traffic (e.g., best effort) from interfering with the ST transmission. However, GB is a time interval in which no traffic can be transmitted; thus, the bandwidth is wasted. Therefore, there are SSAs that reduce the size or number of GBs in GCL [81], [82].

Another option for increasing the efficiency of TAS is IEEE 802.1Qch Cyclic Queuing and Forwarding (CQF) [83]. This CQF allows the bridge to periodically synchronize the frame enqueue/dequeue operations with bounded latencies that depend only on the number of hops and cycle time. The goal is to reduce the dependency on the network topology and increases determinism. It can also be used in combination with ingress policing from IEEE 802.1Qci Per-Stream Filtering and Policy [84] to support more accurate guarantees of an allotted cycle time in TAS, especially for isochronous traffic with high priority [85], [86].

While proposals for novel SSAs with higher scalability and schedulability continue, various analysis methods have also been continuously proposed to assess the performance of these SSAs better [34], [38], [53], [87]–​[97]. In particular, Bello et al. [95] conducted a comprehensive simulation-based analysis that includes all of TSN’s major shaping features, such as TAS, CBS, FP, and GB, and the corresponding traffic types such as Class ST, Class A, and Class B.

B. Credit-Based Shaper

Many notable studies on IEEE 802.1Qav and CBS exist from the AVB era, inherited by TSN. Nevertheless, because TSN needs to define a separate traffic class for ST traffic so that they are guaranteed to be transmitted in isolation from other traffic types while the highest traffic class number has already been occupied by AVB,6 CBS enhancements considering this were continuously studied [98]–​[102]. Similarly, analysis of latency and backlog bounds of AVB traffic in TSN has actively been researched [103]–​[106].

C. Asynchronous Traffic Shaper

The ATS originated from the urgency-based scheduler (UBS) [107], which implements a per-flow interleaved regulator [108] based on rate-controlled service disciplines, providing deterministic latency with low implementation complexity. Specht and Samii [107] analyzed the key parameters of UBS and presented a UBS parameter synthesis approach based on the topology rank solver heuristic and SMT [109]. Through these works, the authors address the shortcomings of traditional asynchronous schedulers to provide deterministic low-latency. Le Boudec [108] also proved that interleaved traffic regulators, which are the key concept behind the ATS, do not increase the worst-case latency. The IEEE 802.1Qcr standard provides a paternoster scheduling (PS) solution.

Many studies have analyzed the performance of ATS using various methods, such as numerical analysis and simulations [108], [110]–​[114]. Zouh et al. provided process modeling for both solutions (UBS and PS) and analyzed the differences [115], [116]. In addition, the authors presented the first hardware implementation using FPGAs [117]. Prados-Grazon et al. used reinforcement learning to cope with the complexity of asynchronous based flow scheduling [118]. Nevertheless, there is no work until 2020 that includes a formal formulation of the flow scheduling/configuration of asynchronous TSN networks, as in synchronous ones. Samii and Specht [107] used STM to solve the problem, but their work does not include an explicit formulation to express the constraints of the problem. This could be a future research direction for asynchronous TSN.

D. Frame Preemption

Frame preemption allows selected express traffic to move faster with relatively low implementation and management complexity (no time synchronization required). When the requirements are less fastidious, it can be used solely by itself to guarantee the bounded worst-case end-to-end latency of express traffic [36]. Moreover, FP allows reducing the GB size when used with TAS [119], [120].

Zhao et al. used network calculus to analyze the theoretical worst-case delay upper bound of the influence on AVB traffic by time-triggered traffic with FP [105]. In addition, the influence of FP has been evaluated using various methods, such as numerical analyses [121], simulations [34], [95], [119], [120], or experiments [36], [121]. In particular, Gogolev and Bauer [36] implemented 802.1AS, TAS, and FP in SoC to compose a simple and practical network with six-hop daisy-chained bridges and nonsynchronized end devices for real experiments. Their analysis indicates that FP can demonstrate relatively reasonable performance in the transmission of high-priority traffic in networks where TAS is unsuitable, but it does not consider the effects of FP on low-priority traffic.

E. Stream Reliability

Kehrer et al. provided a FRER overview and compared the differences with other redundancy protocols [122]. The spatial redundancy concept of FRER is robust against both permanent defects, such as physical link failure or temporary defects, such as frame drops. Spatial redundancy is a method to ensure that there is no problem in data transmission, even if a failure occurs in a specific location, by sending duplicate packets through multiple paths. The TSN standard separately defined maximally redundant tree multipath algorithm for this purpose. However, this does not allow the user or main controller to set the route statically; thus, the algorithm for setting the static route has also been studied by several researchers.

Despite the intention of FRER to eliminate duplicate frames seamlessly, the network resource consumption for indiscriminately redundant transmissions is an obvious drawback. Therefore, appropriate mixing with temporal redundancy to minimize unnecessary transmissions in a so-called proactive manner has been studied [123], [124].

In terms of cost efficiency, critical traffic is the main target of guaranteeing reliability. Therefore, fault tolerance of ST can be considered in the GCL scheduling stage [30], [125]–​[130]. Atallah et al. proposed a degree-of-conflict aware iterative routing and scheduling approach [30], which is a greedy heuristic that creates a schedule to minimize the degree-of-conflict between generated graphs. The authors calculate the degree of conflict by analyzing conflict elements among streams and creating a graph by grouping streams with high interdependence. A disjointed routing set that shares the same source and destination is created using this process, providing a traffic multi-redundancy level. The authors also proposed a fault-resilient joint topology routing and schedule synthesis that provides tolerance levels of critical traffic by generating multiple disjoint paths like [30] in their study [127].

Well-scheduled transmissions can prevent errors, such as frame drop due to interference from other traffic (lower or same priority). However, streams with stringent requirements also consider the redundancy level when scheduled, as they must be robust against link failures.

It is also possible to determine the robustness of a schedule by detecting or diagnosing latent hazards in computation-completed schedules. Atallah et al. [131] detected possible defects by injecting a single event upset (SEU) error into the generated schedule. This injection determines the weak point of the schedule for gate-blocking or message-lagging defects. Although this paper is specified as an SEU error scenario, it sufficiently represents defects caused by comparing the robustness of single path and multiple paths and manipulating schedule information. Likewise, machine learning-based fault detection frameworks to accelerate the schedule viability have also been devised because the scheduling fault diagnosis process usually occurs in pre-deployment stages [132], [133].

SECTION V.

Network Management

Those managing a large industrial automation network with hundreds of SWs must ensure accurate real-time requirements. Network devices can be clustered based on physical location or have separate hierarchies according to their role groups and can be divided into various domains. Different domains may imply different protocols and may also include various wireless components. To operate this network seamlessly, we need a flexible and scalable method for efficient network configuration and management, preferably automated.

This section deals with studies on network management in TSN, such as discovery, configuration, policy, control, clock synchronization, and GCL deployment specific to TSN. The potential of TSN combined with next-generation network technologies, such as network virtualization or data modeling for IT and operation technology (OT) convergence, is also presented. Finally, we introduce several case studies for the application of TSN in various industrial fields.

A. Time Synchronization

Stanton [134] provided a tutorial on IEEE 802.1AS [19] time synchronization for TSN with a detailed background explanation including comparison of gPTP and PTP.

Beyond the standard, a few studies have proposed enhancements [135], [136] for more precise or reliable time synchronization with simulation-based performance comparison.

Zhu et al. [135] proposed a method for re-synchronization when the main clock is abnormal and a method for increasing synchronization accuracy by adjusting the synchronization interval and proposing a separate algorithm for sending synchronization messages. Furthermore, IEEE 802.1AS assumes symmetric communication links; therefore, Baniabdelghany et al. [136] proposed an extended synchronization protocol to compensate for asymmetric delays by measuring accurate delay values based on the neighbor rate ratio and cumulative scaled rate offset.

In addition to research for precision, another study proposes a software defined networking (SDN)-based solution that mitigates the complexity associated with the IEEE 802.1AS configuration and analyzes its performance using real-world devices [52]. For time synchronization in heterogeneous networks, the authors of [137] suggested a solution to integrate the clock synchronization methodology of EtherCAT and TSN with high precision and compatibility.

B. Framework/Modeling

From AVB, IEEE 802.1Qat Stream Reservation Protocol (SRP) [138] features stream registration, reservation, and deregistration [139] procedures for the transmission path from talker to listener. This distributed method is less agile to changes in stream or network configuration than the centralized approach despite other advantages. Moreover, it is unsuitable for TSN because GCL scheduling requires information from the entire network and streams in advance. These are the motivations of the research work, such as a feasibility analysis framework [140], a modeling study for network configurations aiming for the ease of computing and deploying GCL schedules, and studies to support autoconfiguration (plug-and-play) mechanisms [141]–​[144].

Accordingly, IEEE 802.1Qcc [21] defines centralized network configuration (CNC) and centralized user configuration (CUC) entities that support the NETCONF or RESTCONF protocols. The IEEE 802.1Qcp YANG Data Model [145] is proposed for enabling more flexible network configuration [146], [147]. As centralized control is regarded as a viable option in TSN, the use of SDN for TSN has also actively been studied [148]–​[150]. Several solutions using SDN to implement fully centralized models of 802.1Qcc have been presented [41], [48], [49], [51], [151], [152].

In addition, heuristics that enable runtime management are presented, demonstrating its applicability in the fog computing platform [153], [154]. Moreover, IEC 62541 OPC-UA [155] is an industrial protocol and modeling standard for IT-OT convergence. Additionally, OPC-UA is expected to integrate with TSN by providing reliable network interoperability at OSI 5–7 layers, and its standardization is currently in progress (Fig. 2). In particular, OPC-UA Pub/Sub expands the existing server/client structure to a flexible structure for many-to-many connections, making it more useful in industrial machine-type networks. In [50], [156], [157], the authors introduced the integration of OPC-UA and TSN and discussed open issues and obstacles to be addressed for IT-OT convergence. Open625417 [158], node-opcua,8 and python-opcua9 are in development as open-source projects to implement OPC-UA stack, and research demonstrates TSN with OPC-UA using these projects [40], [55], [159].

C. Case Study/Proof-of-Concept

The in-vehicle network is one of the critical target domains of TSN. Motivated by typical mixed-critical automotive applications, Farzaneh and Knoll [160] tested the performance of TAS via actual experiments using three SWs with FPGA-based 802.1Qbv ports. Brunner et al. [161] foresaw the direction of future in-vehicle network architecture as the evolving automotive electrical/electronic architecture demands additional requirements, and highlighted the appropriateness of TSN’s low latency, redundancy, and bandwidth restriction properties.

A larger portion of TSN work has been in many industrial domains that urge IT-OT integration convenience. Nsaibi et al. [33] composed a tree topology network that integrates TSN and Sercos and demonstrated the possibility and potential of integrating a legacy industrial Ethernet with TSN through simulations. Integration of TSN with next-generation technologies, such as IEEE 11073 service-oriented device connectivity [162] or openSAFETY [163] has also been researched [164], [165].

Sanchez-Garrido et al. [166] presented a use case of TSN, which implements a deterministic system in which all substations can communicate with each other through an Ethernet-based infrastructure. The authors installed the TSN system directly at a substation facility of Grupo Cuerva SL, one of Spain’s electricity providers, and demonstrated that the TSN system satisfies IEC 61850 [167] mixed-criticality traffic requirements well. Likewise, Docquier et al. [168] studied the use of the IEC 61850 protocol over TSN, and Muguira et al. [45] focused on the security of electrical sector digitalization through IEC 61850 over TSN.

Many case studies have been conducted to demonstrate the effectiveness of integrating OPC-UA and TSN. Li et al. [46] proposed a two-tier OPC-UA TSN communication architecture for a manufacturing system network and evaluated their prototype through experiments to demonstrate the feasibility of TSN for an industrial network.

Gogolev et al. demonstrated the basic concept of OPC-UA over TSN using COTS devices that are widely used in the industrial field, such as a laser level or gauge pressure transmitter. The authors revealed that when OPC-UA is used with TSN, communication becomes much faster and stabler even under high network congestion [43]. They also indicated that OPC-UA communication latency could be reduced even if there is no TSN support in end devices, instead only using traffic shaping (TAS and CBS) of TSN-enabled SWs, (i.e., without additional updates for end-devices) [44].

Golatowski et al. [169] used TSN for deterministic communication of system control tasks (crane-crane remote controller) using the OPC-UA framework and evaluated their proof-of-concept. In contrast, Agarwal et al. [170] investigated integrating a data distribution service [171], a distributed system information management technology similar to OPC-UA, with TSN.

D. Wireless Integration

Cavalcanti et al. described the requirements (e.g., time synchronization, reliability, scheduling, and network management) in detail when applying TSN to wireless systems. They analyzed the challenges that arose while performing TSN features through wireless networks (5G, Wi-Fi, ZigBee, etc.) [172].

Seijo et al. analyzed the potential problems in the PTP synchronization mechanism due to wireless characteristics, such as sampling and multipath propagation. They proposed an improved wireless timestamping method robust to these problems [173].

Shrestha et al. [174] also conducted a study to improve the performance of PTP in wireless by estimating the clock drift factor considering wireless characteristics. They included it in the clock offset estimation.

Romanov et al. [175] focused on Wi-Fi to enable time synchronization in a wired/wireless hybrid network where the TSN backbone and Wi-Fi cells are mixed by modifying the synchronization algorithm without re-implementing the MAC layer. A new QoS access category was introduced to improve the delay and jitter performance of Wi-Fi when integrating the Wi-Fi into a TSN network [176].

The 3rd Generation Partnership Project (3GPP) recently finalized 5G release 16, providing ultra-reliability and low-latency communication services for the industrial Internet of things (IIoT). This release was a step closer to core wireless technology for factory automation, such as motion control or control-to-control.

Neumann et al. presented various scenarios for combining 5G and the industrial Ethernet [177]. Further, Khoshnevisan et al. covered the overview of TSN and 5G integration for industrial factory automation, presenting a prototype system to validate the potential for using an integrated 5G New Radio system with PROFINET (although not TSN, but considering the possibility of substitution in the future) [178].

Furthermore, some researchers [179]–​[182] have analyzed the integration of 5G systems as a logical TSN bridge in a TSN network environment. Particularly, a system-level simulator was presented in [179] to analyze the effects of a 5G system supporting ST in TSN, emphasizing the necessity for more research for seamless integration. Moreover, 5G and TSN-integrated time synchronization mechanisms have been analyzed [42], [182].

In addition to the perspective of 5G as an industrial Ethernet link, the incorporation of TSN to enhance the 5G mobile network functionality has also been studied. A change in the structure of the existing fronthaul is required to handle the tremendous increase in traffic expected when switching to 5G. Therefore, conversion from the circuit-based common public radio interface to the packet-switching-based common public radio interface has also been studied, and TSNs are expected to play an essential role in the next fronthaul architecture.

The TSN Task Group standardized the IEEE 802.1CM Time-Sensitive Networking for Fronthaul [183]. It provides profiles for the time-sensitive fronthaul and specifies synchronization, scheduling, and preemption mechanisms.

Bhattacharjee et al. introduced the 802.1CM overview. They evaluated the performance of IEEE 802.1Qbv and Qbu with real fronthaul traffic [184].

Liu et al. proposed a simple and effective heuristic ‘higher rate flow scheduled later’ scheme for flow scheduling in fronthaul networks with detailed background knowledge of 5G fronthaul [185]. In addition, TAS enhancement [25] and asynchronous scheduling approaches [186] have been studied for deterministic transmission in 5G fronthaul networks.

Furthermore, Ansah et al. adopted the concept of network slicing of 5G. This concept was to implement the logical separation of various services with different requirements and demonstrate a simulation-based analysis for the worst-case end-to-end latency of each service [187].

SECTION VI.

For TSN Researchers

In this category, we cover a few subtopics that readers who wish to research TSN should know before addressing the cutting-edge of this field. We introduce simulators regarding which frameworks are widely used or which modules are available, depending on the specific features requiring simulation. Furthermore, as TSN demands more ‘real’ assessments of its proposed extensions and improvements, studies demonstrating implementation and evaluation of TSN capabilities on real hardware provide references concerning the currently available devices and limitations.

A. Simulator

In 2016, Heise et al. proposed TSimNet [188], a TSN simulator that extends INET to modularly design non-time-based TSN features, such as FP, per-stream policing, and FRER (802.1Qbu, Qci, and CB). This study was before the official standardization of TAS in 2018. Soon after, in 2018, Jiang et al. [47], [189] implemented time-based features, such as IEEE 802.1AS for time synchronization and the IEEE 802.1Qbv TAS simulation model, by extending CoRE4INET [190].

In 2019, Falk et al. developed NeSTiNg [26], a TSN framework based on INET, including IEEE 802.1Qbv, Qbu, Qav, and VLAN tagging. Next, Martenvormfelde et al. modeled 5G as a logical TSN bridge in addition to the NeSTiNg framework in 2020 [191].

Riverbed [23] (formerly known as OPNET) has also been widely used for developing TSN models since 2017. According to our survey, research on modules by function has continued: IEEE 802.1AS [192], 1Qbv [193], [194], 1Qci [194], and 1CB [195]. Riverbed is also a reliable simulator used to design complex communication protocols and large-scale networks. Another study analyzed TSN scheduling using UPPAAL [196], a well-defined timed model checker [197].

B. Hardware Design

Software-level timestamping has difficulty achieving the precision required by time-triggered Ethernet protocols; thus, support from designated hardware is required. Kyriakakis et al. [198] proposed an architecture that includes a PTP hardware-assist unit in an FPGA-based multicore platform to improve the precision of timestamps. They analyzed the worst-case execution time through experiments.

Groß et al. [199] proposed a flexible and scalable architecture that can run entirely in the hardware, entirely in software, or both. This selection is so that certain features can select an appropriate method according to their timing requirements.

Coleman et al. [200] demonstrated that the performance of TSN Ethernet could degrade depending on the synchronization accuracy between the network clock and CPU clock. For a more sophisticated synchronization, the authors proposed an improvement using PCIe. They evaluated the improvement by experimenting with COTS hardware.

Pruski et al. [201] revealed that the hardware for the existing switch queuing system has a fundamental problem, limiting the performance of TAS, CBS, and FP. They proposed a new SW architecture to compensate for this.

Li et al. [202] proposed an efficient and fault-tolerant memory architecture to facilitate complex TSN functions, even within limited SW memory. Lastly, Vlk et al. [35] proposed a method to detect and handle frames sent differently from the schedule in the SW, relaxing the schedule constraints and increasing schedulability.

C. Miscellaneous

Many concepts are confused among TSN and TTEthernet; thus, it is vital to understand their differences. Zhao et al. [10] presented a comparative analysis of these in various aspects, such as network architecture, clock synchronization mechanism, and redundancy approach. Steiner et al. [31] compared the transmission mechanism of traditional time-triggered switching and IEEE 802.1Qbv TAS. We encourage the reader to refer to the mentioned tutorials to understand TSN overall [4]–​[6], [24].

Network calculus is frequently used to compute the upper bounds on the latency and buffer size of various queuing and scheduling mechanisms in TSN. Maile et al. [203] surveyed studies that present network calculus models for TSN and analyzed their differences.

The future direction of TSN applications in each industry is also of great importance. Samii and Zinner [204] reviewed the suitability of TSN as an internal network for autonomous driving systems taking the five levels defined by SAE J3016 as a use case [205]. The authors introduced switched Ethernet technologies enhanced by several critical attributes of TSN and mapped the requirements for future automotive systems. Hallmans et al. [206] discussed the main challenges and issues of using TSN technology in long-life industrial distributed control systems, such as power plants and smart energy transmission systems, and proposed strategies for designing of these systems.

SECTION VII.

Future Challenges

This article describes literature reporting features making TSN different from the traditional Ethernet, namely the TSN core functions. We cover the studies on network management that enable these powerful features to be exquisite, flexible, and interoperable. Subsequently, we survey several reference studies for TSN Researchers, hoping that more diverse and novel TSN research will follow. In this section, based on the efforts made so far, we outline future challenges that TSN technology must address to become more mature for practical large-scale deployment and use.

A. Flexible Traffic Class

The TSN achieves low-latency and fault-tolerance through traffic prioritizing, streamifying, shaping, and policing. Traffic in TSN is classified according to the requirements and configuration known a priori, and bounded latency is guaranteed through suitable shapers assigned to each class or stream in SWs. All is well in a fully prepared state, but TSN is vulnerable to unexpected runtime changes in network conditions. To compensate for this vulnerability, TSN requires a method to recognize network status changes in real-time and seamlessly modify the traffic class configuration on-line without halting or pausing the system.

For example, due to an out-of-sync clock or unexpected change in the network topology, the schedule may become meaningless, or a new schedule may need to be synthesized (which takes a long time). If the class of ST flows can adaptively switch to (for example) Class A so that ATS or CBS regulates ST flows without regenerating and redeploying a new schedule, it may still be possible to guarantee a certain degree of QoS. Additionally, in preparation for this situation, the network designer can reserve a separate class subject to ST’s urgent downgrade with a lower priority lower than ST but higher than Class A. This method has no side effects on lower classes because there is no traffic of that class under normal circumstances, and the TSN group allows the AVB classes to be completely customized. Furthermore, data analytics and ML-assisted algorithms are becoming more important to realize online solutions that enable the TSN network to adapt to the changing traffic demands and network conditions.

B. Routing for Time-Aware Shaper Scheduling and Reliability

Routing is an indispensable and vital process for network systems, and TSN is no exception. Shortest-path bridging is the intuitive default for low end-to-end latency of critical traffic. Alternatively, equal-cost multipath is an exemplary routing strategy for load balance. Among TSN standards, IEEE 802.1Qca Path Control and Reservation [207] is an IS-IS protocol that provides a set of routing functions to manage bridged networks.

There are many potential disadvantages when multiple critical flows select the same path. Therefore, by considering the routing problem jointly in SSA for time-triggered transmission of ST traffic, it may be possible to obtain a more optimized schedule (e.g., minimizing flowspan and maximizing ST window utilization). Not just for SSA, the importance of routing in TSN research can also be emphasized from a reliability perspective, such as multipath routing for redundancy to address link or device failures.

Fig. 8 is a diagram for understanding the coupling of routing, scheduling, and reliability at a glance. We identify the shared focus areas for the main interests of relevant studies in each part and then discuss the most important challenges.

FIGURE 8. - Classification of routing, scheduling, and reliability related studies.
FIGURE 8.

Classification of routing, scheduling, and reliability related studies.

For example, studies that handle routing and scheduling simultaneously belong to Part A, and Part B includes papers addressing both reliability and routing issues by, for example, using spatial redundancy. Studies in Part C are mostly GCL schedule synthesis techniques considering ST with fault tolerance for reliability. Although studies in Part C do not consider routing, some use temporal redundancy in the scheduling stage or verify the schedule stability. Studies in Part D concurrently consider all three characteristics of routing, scheduling and reliability. In other words, studies in Part A or Part B focus on achieving each goal of scheduling and reliability, respectively, by combining appropriate routing information, whereas Part D includes studies dealing with both simultaneously. Two studies by Atallah et al. [30], [127] are good examples of Part D, which address both scheduling and reliability by considering proper routing, and we believe that there should be more studies going forward.

C. Integration With Other Protocols or Technologies

The accuracy of scheduled transmission depends on how sharp and agile each window in the schedule timeline responds to its offset. However, research on stability and recovery, such as the fault handling process of the clock system, is still insufficient. In addition, IEEE 802.1AS specifies a protocol that provides accurate synchronization time on the links for the Ethernet passive optical network (EPON) and coordinated shared network (CSN). However, no research has been conducted on these. All studies have only been subjected to Ethernet full-duplex point-to-point links (minimally for 802.11 wireless).

The problem of TSN research focusing on Ethernet is the same in time synchronization and areas that require interaction with external networks, such as framework/modeling or wireless integration. Some studies evaluating the integration of TSN with a powerful communication middleware called OPC-UA [155] to provide interoperability between systems have been preferentially conducted, but this is a very basic step. We emphasize the importance of evaluating interfaces for compatibility between heterogeneous network protocols, including OPC-UA. Appropriate north/southbound, east/westbound interfaces should be implemented, and a benchmark should be evaluated.

The need for such interoperability evaluation studies must aim particularly at even more complex wireless communication systems. Smart factories in the future will be wirelessly connect various last-hop devices, such as actuators and robots, for mobility and scalability through 5G NR, Wi-Fi, wirelessHART, ZigBee, Bluetooth, and others. Therefore, related research is ongoing in each field, as it must be rapidly and precisely controlled for industrial purposes. This interoperability evaluation is also necessary for external vehicle networks, such as dedicated short range communication (DSRC), wireless access in vehicular environment (WAVE), and cellular vehicle-to-everything (C-V2X).

D. Network Virtualization

These futuristic wireless networks can secure deterministic communication, flexibility, scalability, and convenience of configuration by decoupling through hardware virtualization. They aim to provide customized QoS by multiplexing virtualized independent logical networks in a single physical network infrastructure. The 5G network slicing and SDN network function virtualization are prevailing types of these virtualizations. Access networks can provide virtualized resources in their own way, and core networks slice, share, or isolate them and provide control as needed in software. From this viewpoint, the virtual LAN concept of TSN serves as the nucleus of the core network. Thus, more realistic research on contact points with various access networks is needed.

E. Fully Featured Simulator and Hardware

We learned the key differences between NeSTiNg and CoRE4INET, the two most recently and widely used TSN simulators, by using them in our research. We realized that more TSN features must be incorporated, and some practical implementation-specific restrictions limit what can be simulated. For example, NeSTiNg is only available in a Linux environment, but CoRE4INET is available in Windows and Linux. Both NeSTiNg and CoRE4INET use a configuration file to set the routes and GCL but in different formats. The NeSTiNg models queue loss by importing INET code, but CoRE4INET has its own queue loss code which requires customizing. Moreover, NeSTiNg follows the priority-to-traffic class mapping table in the IEEE 802.1Q standard (Class A: 2 and Class B: 3). However, CoRE4INET maps priority and traffic class numbers one-to-one (Class A: 5 and Class B: 6) Thus, TSN simulators with more TSN features and realistic data and environment models are required to evaluate TSN and its research more accurately.

Finally, fully featured COTS equipment (e.g. SW, CNC) and development/evaluation kits for academic research are lacking in TSN. Similar to the simulators, more TSN features must be incorporated, and some implementation-specific restrictions limit their capabilities. We have emphasized several times throughout the paper the need for better equipment and development environment for researchers to study TSN. Only then would we be able to take TSN to the next level for real industrial deployments.

SECTION VIII.

Conclusion

Tremendous efforts on open standards for the integrated real-time networking protocol, TSN, are underway in industry and academia. In addition to the functional strengths of ultra-low latency/jitter and reliability, the fact that it is based on the most widely used Ethernet, makes TSN economical in many ways let alone interoperability. With these advantages, TSN is expected to become the de facto next-generation LAN infrastructure that will be the foundation of near-future leading technologies, such as Industry 4.0, smart factories, intelligent transportation system, and 5G. Throughout this paper, we conducted a comprehensive statistical survey of 174 TSN publications, the first study of its kind to the best of our knowledge. We have examined in a timely manner what topics have been researched so far in TSN, their trends, and what is lacking and needed. We aim for this paper to serve as a pointer and directional guide to many fellow researchers who wish to study TSN in the future.

Appendix–List of Acronyms

Nomenclature

AbbreviationExpansion
3GPP

3rd Generation Partnership Project.

AFDX

Avionics Full-Duplex Switched Ethernet.

ASIC

Application-Specific Integrated Circuit.

ATS

Asynchronous Traffic Shaper.

AVB

Audio/Video Bridging.

CAN

Controller Area Network.

CBS

Credit-Based Shaper.

CDF

Cumulative Distribution Function.

CEV

Crew Exploration Vehicle.

CNC

Centralized Network Configuration.

COTS

Commercial off-the-shelf.

CQF

Cyclic Queuing and Forwarding.

CSN

Coordinated Shared Network.

CUC

Centralized User Configuration.

C-V2X

Cellular Vehicle-to-Everything.

DetNET

Deterministic Network.

DSRC

Dedicated Short Range Communication.

EPON

Ethernet Passive Optical Network.

ES

End System.

EtherCAT

Ethernet for Control Automation Technology.

FTT-Ethernet

Flexible Time-Triggered Ethernet.

FP

Frame Preemption.

FPGA

Field Programmable Gate Array.

FRER

Frame Replication and Elimination for Reliability.

GB

Guard Band.

GCL

Gate Control List.

gPTP

Generic Precision Time Protocol.

HSR

High Availability and Seamless Redundancy.

IIoT

Industrial Internet of Things.

ILP

Integer Linear Programming.

IoT

Internet of Things.

IT

Information Technology.

OPC-UA

Open Platform Communications Unified Architecture.

OT

Operation Technology.

PRP

Parallel Redundancy Protocol.

PS

Paternoster Scheduling.

PTP

Precision Time Protocol.

QoS

Quality-of-Service.

SAE

Society of Automotive Engineers.

SDN

Software Defined Networking.

SEU

Single Event Upset.

SMT

Satisfiability Modulo Theories.

SoC

Systems-on-a-Chip.

SRP

Stream Reservation Protocol.

SSA

Scheduling Synthesis Approach.

ST

Scheduled Traffic.

SW

Switch.

TAS

Time-Aware Shaper.

TSN

Time-Sensitive Networking.

TTEthernet

Time-Triggered Ethernet.

UBS

Urgency-Based Scheduler.

WAVE

Wireless Access inVehicular Environment.

Figures are not available for this document.

References

References is not available for this document.