By Topic

• Abstract

SECTION I

## INTRODUCTION

Teleoperation1 is the extension of a person's sensing, decision making, and manipulation capability to a remote location [1]. Today's teleoperation systems allow the interaction with environments at a distance and can also scale human force and motion to achieve stronger, bigger, or smaller action capabilities. Inaccessible, hazardous, or scaled environments become accessible to the human. Applications range from classical nuclear/toxic/explosive material handling and disposal [2], space and underwater exploration [3], [4], [5], rescue applications in dangerous environments [6], minimally invasive surgery [7], cell/molecule manipulation and telenanorobotics [8] to training, education [9], and entertainment applications; see [10] for an overview.

The user of a teleoperation system is able to interact with remote objects and/or humans and/or robots and to execute tasks without being there [1]. In order to make the user feel like being there, the teleoperation system supplies him/her with the multimodal sensor information of the remote environment. At the same time, he/she may multimodally interact with the remote object through a robotic system. Here, “multimodal” refers to the perceptual modalities of human beings, such as the visual, the auditory, and the haptic modality. Haptics, as an extension to classical bidirectional multimedia, includes the feeling of force, motion, and vibration, and is accordingly divided into the kinesthetic, the proprioreceptive, and the tactile submodality. In the following, we use the notion haptics to cover the kinesthetic and proprioreceptive submodalities. Haptic feedback provides the operator with more complete information and increases the sense of being present in the remote environment thereby improving the ability to perform complex tasks [11].

A typical multimodal teleoperation system architecture is depicted in Fig. 1. The human operator, equipped with a headmounted display and headphones, manipulates a human–system interface (HSI), a robot equipped with motion and force sensors, and thereby commands the remote executing robot, the teleoperator. While the teleoperator interacts with the remote environment, the multimodal sensor data are fed back and displayed to the operator. A communication network realizes the transmission of the command and sensor signals between the operator and the teleoperator side.2

Fig. 1. Multimodal teleoperation system.

#### Remark 1

Depending on its level of autonomy and assistance, teleoperation is classified into bilateral, shared, and semiautonomous schemes [1], [11], [13]. In bilateral teleoperation, the interaction between the human and the remote environment through the teleoperation system is exclusively based on low-level continuous-time signals and the human is strongly coupled with the environment. In shared control schemes, the teleoperation system anticipates human actions and assists in order to achieve a better task performance, for example, it reduces the natural tremor of the hand. In semiautonomous teleoperation, the human commands subtasks, e.g., “grasp object” through symbolic interaction. In this paper, we concentrate on bilateral teleoperation as a tool to facilitate human decision-making capabilities when solving manipulation tasks in remote environments.

Haptic teleoperation is a typical example of a human-in-the-loop system with the particular challenge of tight physical coupling between the human and the machine. Haptic interaction does not only provide the human operator with information about his/her environment, but also enables the human operator to manipulate the environment. This implies a bilateral exchange of energy between the human operator and the environment. Thereby, a global control loop is closed over the communication system with the human, the HSI, the teleoperator, and the remote environment being in it. Stability of this control loop is a fundamental requirement: an unstable system is inoperable, and furthermore, dangerous to the human operator and the environment. Accordingly, the first design goal for haptic teleoperation control is stability. Besides stability, transparency is the second principal goal in teleoperation system control design. Teleoperation is called transparent if the human does not distinguish between direct and tele-interaction with a remote environment; ideally he/she feels as if directly interacting with the remote environment [14]. Transparency is an important prerequisite to facilitate human decision making in the remote environment, e.g., in terms of object recognition from haptic properties.

The inclusion of the human has two major consequences for the control design: 1) the dynamics of the human operator and the teleoperation system are tightly coupled, i.e., stability of the overall system is affected by the human operator dynamics; and 2) the performance of the teleoperation system is subjectively evaluated by the human, which means that typical standard control performance metrics are not suitable. The original contribution by the authors is the systematic consideration of human perception and action aspects in the control design of haptic teleoperation systems. Until now, only individual aspects of human-oriented control have been treated in earlier works [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35]. The contribution of this paper is a concise treatment of the human-oriented analysis and design approach including the following individual innovations: 1) dissipativity-based modeling of the human closed-loop dynamics for improved and robust control performance; 2) introduction of the generalized wave/scattering variable transformation [36], [37], [38], [39], [40] based on the dissipative human dynamics model to teleoperation systems with communication unreliabilities in the feedback loop; and 3) the definition of a novel human-oriented evaluation criterion of perceived transparency. From the view of this novel concept earlier results of 4) analysis of the influence of communication parameters on the human-perceived performance and 5) exploitation of human haptic perception limits for haptic data compression are revisited. In summary, it is shown that the inclusion of improved human dynamical models leads to an improvement of the level of transparency, and on the other hand, that due to human haptic perception limits perfect transparency is not necessary thereby closing the gap between achievable and ideal performance.

The remainder of this paper is organized as follows. In Section II, an overview on current control approaches is given, in particular, the wave variable control architecture. In Section III, a novel dissipativity-based modeling paradigm for the human dynamics and other subsystems of the teleoperation system is introduced and a novel approach to guarantee stability of the closed-loop system with communication unreliabilities is presented improving the level of transparency and therewith human decision making and manipualtion capabilities. In Section IV, transparency metrics in a human-oriented fashion are introduced, and in Section V, human-oriented evaluation and control design aspects are discussed. Finally, in Section VI, a psychophysically motivated data compression scheme is presented.

SECTION II

## STABILITY IN BILATERAL TELEOPERATION

The communication between the HSI at the operator side and the teleoperator at the remote side typically takes place over a communication network as depicted in Fig. 1. Through the command and feedback signals energy is exchanged between the HSI and the teleoperator thereby closing a global control loop through the communication network. Naturally, the transmission of the command and sensor signals is afflicted with the time delay and potential loss of information. In terrestrial telerobotics application, the transmission time delay is typically in the range of some milliseconds up to several hundred milliseconds depending on distance and communication infrastructure; in inner space applications, the data transmission may easily take several seconds. The time delay in the haptic feedback loop represents one of the key challenges in control design with respect to stability and also transparency. Without appropriate control measures even small time delay can destabilize the teleoperation system [41] resulting in a severe hazard to the safety of the human and the remote environment.

#### Example 1

In intercontinental experiments as, for example, presented in [42] and [43] between Europe and Japan (see also Fig. 2 for the experimental setup), a round-trip time delay of $T=$ 278 ms has been measured. The variance of the time delay was in the range of 1 ms, i.e., negligible.

Fig. 2. Intercontinental teleoperation experiment between Germany and Japan with the 7DoF-HSI Vishard7 [44] and the humanoid HRP2 as the teleoperator; results are presented in [42] and [43].

The coupling control design for the HSI and the teleoperator over time-delayed communication channels has been extensively treated in the literature of the past two decades. One of the most prominent control approaches for the stabilization in the presence of communication unreliabilities is the passivity-based scattering/wave variable transformation [45], [46], which in its original version stabilizes the system for arbitrarily large constant time delays and can cope with nonlinear manipulator and human and environment dynamics. Its extensions concern other communication unreliabilities such as the time-varying delay and the packet loss, e.g., [15], [16], and [47], [48], [49], [50], [51], [52]. Related to the concept of scattering/wave variables is the geometric scattering approach [53], [54], [55], where a port-Hamiltonian representation of the teleoperation system is employed. Using PD-type control laws and passivity arguments in combination with Lyapunov–Krasowskii techniques, stability is ensured for known constant time delay [56] and time-varying delay [57]. In the time-domain passivity concept [58], [59], [60], the energy balance is evaluated online through a passivity observer and gains are adapted such that passivity is maintained and in consequence stability is guaranteed. In model-mediated telemanipulation [61], an improved performance with large time delays also beyond one second is achieved through an interaction with local regularly updated models of the remote side; see also [62] for a survey on related approaches where control is based on (online) estimated local models of the human, task, and environment. A conceptual framework ensuring passivity and transparency on different layers of a two-layer architecture is presented in [63]. Alternative techniques employ an $n$-port network with a hybrid matrix formulation teleoperation system model and design linear dynamic filters based on stability criteria from the linear time-invariant system theory under different assumptions on the transmitted signals (four-, three-, two-channel architectures) [64], [65], [66], [67], [68], [69]. In [69], it is shown that under zero time delay for the ideal transparency two channels are necessary and sufficient. Other approaches employ $H_{\infty}$-type control designs [70], [71], $\mu$-synthesis [72], and a Nyquist-type stability criterion [73]. For an overview on state-of-the-art control methods, see [10], [74], [75], and the (experimental) comparisons of different control schemes can be found in [76], [77], [78].

In the remainder of this paper, the focus is on passivity-based approaches in conjunction with the scattering/wave variable transformation. It should be noted though that the principle concept of human-oriented control performance evaluation and design is not restricted to this particular control architecture but can be extended to any other control approach. In the following, an introduction into the passivity-based control and the scattering/wave variable transformation is given.

### A. Passivity-Based Control

One of the major challenges for teleoperation control design is the largely unknown dynamics of the human and the remote environment, both being part of the global control loop. Passivity-based approaches address those challenges and in addition facilitate a modular design approach, where the overall stability certificate can be derived from subsystem properties.

A bilateral teleoperation system can be represented as cascaded feedback interconnection of its components, i.e., the human, the HSI, the teleoperator, and the environment. The human dynamics in haptic closed-loop control can only be approximated. Such approximative models can be classified into nonparametric models [79], according to which a trained human can be considered to interact passively with passive environments, and parametric models [80], [81]. The nonparametric passivity-based model [79] has been most successful in the context of teleoperation due to its stability robustness and the ability to also include nonlinear subsystem dynamics as the robotic manipulators (HSI, teleoperator) into the stability analysis. The HSI and the teleoperator are passive or can be made passive by an appropriate local control, and most environments can be considered to be passive as well.

Passivity is an energy-based concept characterizing a dynamical system by its input/output behavior. It provides a sufficient input/output stability condition, and assuming zero state detectability, it also provides the stability of the origin. The systems we consider are continuous time, nonlinear time-invariant dynamical systems described by TeX Source $$\Sigma\cases{\mathdot{x}=f(x,u)\cr y=h(x,u)}\eqno{\hbox{(1)}}$$ where $x\in\BBR^{n}$, $u,y\in\BBR^{p}$ are the state, input, and output vectors, respectively, and $f(0,0)=h(0,0)=0$. The function $f$ is assumed locally Lipschitz, and $h$ continuous. Thus, for each fixed initial state $x(0)$, (1) defines a causal mapping from the input signal $u(\cdot)$ to the output signal $y(\cdot)$.

#### Definition 1 (Passivity) [82]

System $\Sigma$ (1) is passive if there exists a positive-semidefinite function $V:\BBR^{n}\rightarrow\BBR_{+}$, called storage function, such that for each admissible $u$ and for each $t\geq 0$ TeX Source $$V\left(x(t)\right)-V\left(x(0)\right)\leq\int \limits_{0}^{t}u^{T}y\,d\tau.\eqno{\hbox{(2)}}$$ For ease of notation, the dependencies of $u,y$ from time $\tau$ are not explicitly written.

Passivity can be considered as an energy nongeneration condition: The energy entering the system, represented by the right-hand side integral over the scalar product of the input vector $u$ and the output vector $y$, is either stored in the internal states $x$ or dissipated. Note that the input energy does not necessarily correspond to any actual physical energy if the input/output variables are not chosen to be the colocated effort/flow variables, for example, force and velocity in a mechanical system.

An important feature of the passivity formalism is its closure property. This implies that the combination of two passive systems connected in either a feedback or a parallel configuration is again passive [82].

Such passivity arguments hold in the typical velocity–force and force–velocity teleoperation architectures. Exemplarily, the velocity–force architecture is depicted in Fig. 3; the blocks with the wave variable transformation are explained in Section III-C. In the velocity–force architecture, the combined system of the teleoperator and the environment is represented by a map from the (desired) velocity $\mathdot{x}_{t}^{d}$ to force $f_{e}$ (admittance causality), and the combined system of HSI and human arm by a map from the (desired) force $f_{h}^{d}$ to velocity $\mathdot{x}_{h}$ (impedance causality). Typically, there is a local velocity control loop at the teleoperator side and force control at the HSI side; vice versa for the force–velocity architecture. Through passivity of their subsystems, also the combined systems teleoperator/environment and HSI/human are passive with respect to their input/output variables $\{\mathdot{x}_{t}^{d},f_{e}\}$ and $\{f_{h}^{d},\mathdot{x}_{h}\}$, respectively. Both are connected in negative feedback over the forward and backward communication channels. In Fig. 3, the negative feedback is hidden in the HSI block; the blocks with the wave variable transformation are ignored for now. If the communication channel does not introduce any additional dynamics, then the overall system is passive from the human voluntary force input to the velocity output of the teleoperator/environment.

Fig. 3. Teleoperation velocity–force control architecture with wave/scattering variable transformation.

For communication with the time delay, the passivity argument is no longer valid [45]. In fact, the overall system becomes unstable even for a small time delay.

### B. Stability With Communication Unreliabilities

In order to achieve the stability with communication unreliabilities such as the time delay, the wave/scattering variable transformation [45], [46] is introduced. In its original version, it passifies the communication two-port for an arbitrarily large constant time delay. It is a linear transformation from the input and output signals of the teleoperator/environment and HSI/human subsystems to the so-called wave/scattering variables transmitted over the communication channel. For the velocity–force architecture, as depicted in Fig. 3, the wave/scattering variable transformation is given by TeX Source \eqalignno{u_{l}=&\,{1\over\sqrt{2b}}\left(f_{h}^{d}+b\mathdot{x}_{h}\right)\quad u_{r}={1\over\sqrt{2b}} \left(f_{e}+b\mathdot{x}_{t}^{d}\right)\cr v_{l}=&\,{1\over\sqrt{2b}}\left(f_{h}^{d}-b\mathdot{x}_{h}\right) \quad v_{r}={1\over\sqrt{2b}}\left(f_{e}-b\mathdot{x}_{t}^{d}\right)&\hbox{(3)}} with $u_{l},u_{r},v_{l},v_{r}$ representing the wave/scattering variables transmitted over the communication channel. The tuning parameter $b\in\BBR_{+}$ can be interpreted as the wave impedance of the communication line. As a result, the overall system is delay-independently stable. For details concerning the proof, refer to [45] and [46]. Note that this approach also guarantees the delay-independent stability for the force–velocity architecture with accordingly modified transformation (3).

However, the passivity-based methods are known to lead to a conservative control design. As a result, the environment properties are displayed in a distorted way. With the increasing time delay, hard objects are displayed softer, and in free-space motion, an increasing inertia is displayed [17], [18]; see also Section V. A high level of transparency, however, is required to enable similar decision making and manipulation capabilities by means of a teleoperation system as without it.

SECTION III

## IMPROVED TRANSPARENCY THROUGH DISSIPATIVE HUMAN MODELING

Including more model knowledge of the subsystem, for example, of the human closed-loop dynamics, into the control design, potentially improves transparency but may also reduce robustness to modeling uncertainties and errors. In order to improve transparency without giving up the robustness and modular design of passivity-based approaches, we propose to use approximate knowledge of the damping properties of the human arm, the controlled manipulators, and/or the environment for stabilizing control design of a teleoperation system with communication unreliabilities. In fact, we can show that these subsystems are dissipative with a quadratic supply rate, also called QSR-dissipative [83], [84], [85], [86], [87], [88], which can be exploited for better transparency. Applying the generalized scattering transformation (GST) [37], [38], [39], [89], we can ensure the stability with communication unreliabilities, in particular, finite gain ${\cal L}_{2}$-stability for arbitrarily small gain network operators, including the arbitrarily large constant time delay and the appropriately handled packet loss. The following derivations extend the results from our recent conference papers [19], [20].

### A. Background on Dissipativity and Finite Gain ${\cal L}_{2}$-Stability

Instead of the passive subsystems, we now consider the larger class (less conservative) of QSR-dissipative subsystems in a teleoperation system; its validity is discussed in Section III-B. In this section, the most relevant results for general dissipative systems are presented.

Consider a system $\Sigma$ described by (1) with the slightly relaxed assumption that input and output vectors may be of different dimension, i.e., $u\in\BBR^{p}$ and $y\in\BBR^{q}$.

#### Definition 2 (QSR-Dissipativity)

System $\Sigma$ (1) is said to be QSR-dissipative if there exists a positive-semidefinite function $V:\BBR^{n}\rightarrow\BBR_{+}$, called storage function, such that for each admissible $u$ and each $t\geq 0$ TeX Source $$V\left(x(t)\right)-V\left(x(0)\right)\leq\int\limits_{0}^{t}{u\brack y}^{T}P{u\brack y}\,d\tau \eqno{\hbox{(4)}}$$ with the dissipativity matrix TeX Source $$P=\left[\matrix{Q&S\cr S^{T}&R}\right]$$ where $Q\in\BBR^{p\times p}$, $R\in \BBR^{q\times q}$, and $S\in\BBR^{p\times q}$. For ease of notation, the dependencies of $u,y$ from time are not explicitly written.

First, we observe that the choice $Q=R={\bf 0}$, $S=(1/2)I$ recovers the notion of a passive system from Definition 1 showing that indeed QSR-dissipative systems include passive systems as a special case ($I$ is the identity and ${\bf 0}$ is the zero matrix of appropriate dimension). Another special case of QSR-dissipative systems, interesting for the application in teleoperation, is given by the class of input-feedforward/output-feedback passive (IF-OFP) systems with $u,y\in\BBR^{p}$, which are represented by the choice [90] $Q=-\delta I$, $R=-\epsilon I$, and $S=(1/2)I$ with parameters $\delta,\epsilon\in\BBR$. The system is called lossless if $\delta=\epsilon=0$, output-feedback strictly passive (OFP $(\epsilon)$) if $\delta=0$ and $\epsilon>0$, and input-feedforward strictly passive (IFP$(\delta)$) if $\delta>0$ and $\epsilon=0$. If one or both of the values $\delta,\epsilon$ are negative, then there is a shortage of passivity; otherwise, there is an excess of passivity.

The negative feedback interconnection of QSR-dissipative subsystems $\Sigma_{1}$ and $\Sigma_{2}$ as illustrated in Fig. 4 is still QSR-dissipative indicating the potential for modular design.

Fig. 4. Negative feedback interconnection of systems $\Sigma_{1}$ and $\Sigma_{2}$.

Subsequently, we purposely concentrate on the special class of IF-OFP systems because of their simplicity and intuitiveness. The IF-OFP description and the QSR-description are equivalent for systems with single input and single output. For systems with multiple input and multiple output, the IF-OFP formulation might be conservative. It should be noted, however, that all subsequent derivations can be straightforwardly derived for the general QSR-dissipative formulation [40].

#### Lemma 1 [90]

Assume that system $\Sigma_{1}$ is ${\hbox {OFP}}(\epsilon_{1})$, and system $\Sigma_{2}$ is ${\hbox {IFP}}(\delta_{2})$. Their negative feedback interconnection is ${\hbox {OFP}}(\epsilon_{1}+\delta_{2})$ from input $e_{1}$ to output $y_{1}$.

#### Lemma 2 [19]

Assume that system $\Sigma_{1}$ is ${\hbox {IFP}}(\delta_{1})$, and system $\Sigma_{2}$ is ${\hbox {OFP}}(\epsilon_{2})$ with $\delta_{1}\geq 0$ and $\delta_{1}+\epsilon_{2}\geq 0$. Their negative feedback interconnection is ${\hbox {IFP}}(\kappa)$ with $\kappa=\min(\delta_{1},\delta_{1}+\epsilon_{2})$ from input $e_{1}$ to output $y_{1}$.

For proofs, the reader is referred to [19] and [90].

From the IF-OFP properties of the two subsystems, finite gain ${\cal L}_{2}$-stability of their negative feedback interconnection can be concluded.

#### Definition 3 (Finite Gain ${\cal L}_{2}$-Stability) [90]

System $\Sigma$ (1) is called finite gain ${\cal L}_{2}$-stable if there exist a positive-semidefinite function $V:\BBR^{n}\rightarrow\BBR_{+}$ and a scalar constant $\gamma>0$ such that for each admissible $u$ and each $t\geq 0$ TeX Source $$V\left(x(t)\right)-V\left(x(0)\right)\leq \int\limits_{0}^{t}\gamma^{2}u^{T}u-y^{T}y\,d\tau.\eqno{\hbox{(5)}}$$ In particular TeX Source $${\Vert y_{t}\Vert}_{{\cal L}_{2}}\leq\gamma^{2}{\Vert u_{t}\Vert}_{{\cal L}_{2}}+\sqrt{V\left(x(0)\right)}\eqno{\hbox{(6)}}$$ holds, where ${\Vert y_{t}\Vert}_{{\cal L}_{2}}$ represents the ${\cal L}_{2}$-norm of the truncated signal $y$ TeX Source $${\Vert y_{t}\Vert}_{{\cal L}_{2}}=\sqrt{\int_{0}^{t}y(\tau)^{T}y(\tau)\,d\tau}.$$ The smallest possible value $\gamma$ satisfying (6) is called the ${\cal L}_{2}$-gain of system $\Sigma$. A system $\Sigma$ is called to have the small gain property if its ${\cal L}_{2}$-gain satisfies $\gamma\leq 1$.

The following proposition indicates that the passivity shortage in one subsystem can be compensated for by the passivity excess in another subsystem, which is the essential benefit from this modeling paradigm.

#### Proposition 1 [90, Theorem 6.2]

Consider two IF-OFP systems $\Sigma_{1}$ and $\Sigma_{2}$ with $\delta_{i}$, $\epsilon_{i}$, $i\in 1, 2$. The negative feedback interconnection of $\Sigma_{1}$ and $\Sigma_{2}$ is finite gain ${\cal L}_{2}$-stable if TeX Source $$\epsilon_{2}+\delta_{1}>0\ {\hbox {and}}\ \epsilon_{1}+ \delta_{2}>0.$$

### B. Dissipativity-Based Modeling of Teleoperation Systems

In the following, we derive the IF-OFP properties of the subsystems of a teleoperation system. The human arm is intrinsically damped [91] and the environment may contain velocity-dependent damping as well. Also, the controlled as well as the uncontrolled dynamics of the manipulators follow the Euler–Lagrangian equations with a velocity-dependent dissipation term that accounts for natural friction in the uncontrolled case or purposely introduced damping in the manipulator impedance control to achieve a certain compliant behavior and avoid contact instability; see, e.g., [92]. The behavior of this type of a mechanical system can be described by QSR-dissipativity (4).

#### Example 2

To illustrate the relationship between damping terms and the IF-OFP parameters we exemplarily investigate a simple mechanical impedance represented by a second-order differential equation in Cartesian space TeX Source $$M\mathddot{x}(t)+D(x,\mathdot{x})+Kx(t)=f(t)\eqno{\hbox{(7)}}$$ where $x\in\BBR^{n}$ is the position vector in Cartesian space, $f\in\BBR^{n}$ is the Cartesian force, $M\in\BBR^{n\times n}$ and $K\in\BBR^{n\times n}$ are the positive-definite inertia and stiffness matrices, respectively, and $D(x,\mathdot{x})={\rm diag}\{D_{i}(x,\mathdot{x})\}$ accounts for viscous damping. The components $D_{i}(x,\mathdot{x})$ are assumed to be continuous, potentially nonlinear but unknown functions, for which $D(x,\mathdot{x})\geq d_{\min}\mathdot{x}$ with $d_{\min}\geq 0$ holds componentwise and for all $x$ and $D(x,0)=0$; see Fig. 5 for a visualization in the 1-degree-of-freedom (1DoF) case. Consider now the force $f$ as the input, the velocity $\mathdot{x}$ as the output, and the following storage function: TeX Source $$V(x)={1\over 2}\mathdot{x}^{T}M \mathdot{x}+{1\over 2}x^{T}Kx\eqno{\hbox{(8)}}$$ representing the kinetic and potential energy. Taking its derivative and integrating, it follows: TeX Source \eqalign{V \left(x(t)\right)-V\left(x(0)\right)=&\,\int\limits_{0}^{t}(M\mathddot{x}+Kx)^{T}\mathdot{x}\,d\tau\cr=&\, \int\limits_{0}^{t}(f-D(x,\mathdot{x})^{T}\mathdot{x}\,d\tau\cr\leq&\,\int\limits_{0}^{t}f^{T}\mathdot{x}-d_{\min}\mathdot{x}^{T}\mathdot{x}\,d\tau.} The system is, hence, shown to be ${\hbox {OFP}}(d_{\min})$ with the input/output force/velocity pair. Using the same storage function, the system can be shown to be ${\hbox {IFP}}(d_{\min})$ with input $\mathdot{x}$ and output $f$.

Fig. 5. Nonlinear damping with lower sector bound $d_{\min}$.

The IF-OFP parameters of a system can be derived through the experimental identification of the input/output behavior or alternatively by considering a certain structure of the model such as the second-order structure in (7). Note that only the lower bound of the damping needs to be known; the specific values of parameters $M$, $B$, and $D(\cdot,\cdot)$ are not required. Here, we shortly sketch how the IF-OFP parameters for the subsystems based on such a model structure are derived.

The human arm endpoint characteristics are approximately second order as recent work suggests [81], [93]. The damping depends on the neuromuscular activity, which in turn depends on the task and the human action intention. For example, in a 1DoF human–robot cooperation task, the human arm damping is identified to be 16 Ns/m [91]. From the lower bound of the damping represented by $d_{\min}^{h}\geq 0$, the subsystem is ${\hbox {IFP}}(d_{\min}^{h})$ if velocity is considered as the input to the human and force is considered as the output (admittance causality). With force as the input and velocity as the output the human dynamics would be characterized by ${\hbox {OFP}}(d_{\min}^{h})$. Analogously, the lower bound of the damping term can be retrieved for the locally controlled robotic manipulators of the HSI and the teleoperator. In particular, impedance/admittance-type control laws typically result in a second-order dynamics similar to (7). The lower damping bound is then given by $d_{\min}=\lambda_{\min}(D)$, where $\lambda_{\min}\geq 0$ represents the smallest eigenvalue of the nonnegative-definite matrix $D$. Accordingly, the parameters for the HSI and the teleoperator depend on the implemented damping matrices $D^{\rm HSI}$ and $D^{\rm TO}$, and are given by $d_{\min}^{\rm HSI}$ and $d_{\min}^{\rm TO}$, respectively. Also many environments induce dissipation, and the lower bound of the damping is represented by $d_{\min}^{e}\geq 0$. A concise discussion of the IF-OFP properties of all potential local control architectures is beyond the scope of this paper, see [19] for more details.

In our considered velocity–force architecture (see Fig. 6 for illustration), the human is represented by an admittance, i.e., it is ${\hbox {IFP}}(d_{\min}^{h})$, and the HSI has impedance causality, i.e., it is ${\hbox {OFP}}(d_{\min}^{\rm HSI})$. The negative feedback interconnection of the two, represented by the right-hand side subsystem $\Sigma_{l}$, has overall impedance characteristics and is ${\hbox {OFP}}(d_{\min}^{h}+d_{\min}^{\rm HSI})$ according to Lemma 1. Analogously, the input/output behavior of the environment is represented by an impedance of the teleoperator by an admittance. The negative feedback interconnection of the corresponding ${\hbox {IFP}}(d_{\min}^{e})$- and ${\hbox {OFP}}(d_{\min}^{\rm TO})$-systems results in an overall right-hand side subsystem $\Sigma_{r}$. As the teleoperator damping is nonnegative $d_{\min}^{\rm TO}\geq 0$, the overall system $\Sigma_{r}$ is ${\hbox {IFP}}(d_{\min}^{e})$ according to Lemma 2. The blocks denoted by $\Xi$ and $\Xi^{-1}$ in Fig. 6 are explained in Section III-C.

Fig. 6. Teleoperation system with the time delay and the generalized wave/scattering variable transformation.

#### Remark 2

In case where the dissipation term in (4), e.g., described by the lower bound of the damping term in (7), is unknown, the analysis is reduced to the case of simply passive systems as performed in [45].

#### Remark 3

The main difference between the force–velocity architecture and the velocity–force architecture is that the input/output causalities are switched for all subsystems. The IF-OFP properties can be derived straightforwardly.

Clearly, without communication unreliabilities, the stability is guaranteed by the IF-OFP properties of the subsystems according to Proposition 1. In our case, for the velocity–force architecture, $\epsilon_{1}=d_{\min}^{h}+d_{\min}^{\rm HSI}$, $\delta_{2}=d_{\min}^{e}$, and $\delta_{1}=\epsilon_{2}=0$ hold. We observe that the human damping properties (as well as the damping of the HSI) enlarges the stability margin. It also indicates that we may potentially trade this added stability margin against improved transparency. This is shown in the following, where we consider a communication channel with unreliabilities. Similar to the passivity-based approach, it can be shown that the time delay may destabilize the overall system, i.e., the feedback interconnection of the left-hand side and right-hand side subsystems $\Sigma_{l}$ and $\Sigma_{r}$ via the communication channel. In order to cope with communication unreliabilities, we extend the wave/scattering variable transformation in a way such that it uses the benefits of the introduced dissipativity-based modeling paradigm, thereby improving the transparency compared to the standard wave/scattering variable transformation.

### C. Generalized Wave/Scattering Variable Transformation

The generalized scattering transformation is a linear input/output transformation represented by the matrix $\Xi$ in Fig. 6. Using the same notation as in (3), we denote by $u_{l}$ the generalized wave/scattering variable transmitted over the forward channel instead of the velocity $\mathdot{x}_{h}$. The relationship between the power variables at the HSI side $\mathdot{x}_{h},f_{h}^{d}$ and the left-hand side generalized wave/scattering variables $u_{l},v_{l}$ is given by the transformation TeX Source $${u_{l}\brack\upsilon_{l}}=\Xi{\mathdot{x}_{h}\brack f_{h}^{d}}.$$ Analogously, $\upsilon_{r}$ is transmitted instead of the environment force $f_{e}$, where TeX Source $${u_{r}\brack \upsilon_{r}}=\Xi{\mathdot{x}_{t}^{d}\brack f_{e}}.$$

#### Remark 4

Note that we can also rewrite the standard wave/scattering variable transformation (3) in a similar form with the realization of the scattering matrix TeX Source \Xi_{scatt}=\left[\matrix{\displaystyle{b\over \sqrt{2b}}&{\displaystyle{1\over\sqrt{2b}}}\cr\noalign{\vskip6pt}{\displaystyle{-b\over\sqrt{2b}}}& \displaystyle{1\over\sqrt{2b}}}\right].\eqno{\hbox{(9)}}

The variables denoted by $u_{r}$ and $\upsilon_{l}$ are the output of the network operator for the forward and backward channels, respectively. We assume for now that both the forward channel and the backward channel are represented by unknown constant time delays $T_{1},T_{2}$, respectively, as depicted in Fig. 6 TeX Source $$u_{r}(t)=u_{l}(t-T_{1})\quad \upsilon_{l}(t)=\upsilon_{r}(t-T_{2}).\eqno{\hbox{(10)}}$$

The generalized wave/scattering variable transformation is parametrized using a rotation matrix $R_{\theta}$ and a scaling matrix $B$ TeX Source $$\Xi=R_{\theta}B=\left[\matrix{\cos\theta I&\sin\theta I\cr-\sin\theta I&\cos\theta I} \right]\left[\matrix{b_{11}I&0\cr 0&b_{22}I}\right]\eqno{\hbox{(11)}}$$ where $I$ represents the $n\times n$ identity matrix, $\det B\neq 0$, and $\theta\in[-(\pi/2),(\pi/2)]$. The choice of the transformation angle $\theta$ is based on the IFP and OFP properties of the subsystems $\Sigma_{r}$ and $\Sigma_{l}$; $b_{11},b_{22}>0$ represent free tuning parameters.

#### Proposition 2 [89]

Assume a system consisting of the networked negative feedback interconnection of an ${\hbox {OFP}}(\epsilon_{l})$-system $\Sigma_{l}$ and an ${\hbox {IFP}}(\delta_{r})$-system $\Sigma_{r}$ with $\epsilon_{l},\delta_{r}>0$, the bidirectional delayed communication channel, and the input/output transformation (11). Delay-independent finite gain ${\cal L}_{2}$-stability is ensured if and only if for each admissible scaling matrix $B$ the rotation matrix parameter $\theta\in[\theta_{l},\theta_{r}]$. Here $\theta_{l}$ and $\theta_{r}$ are one of the two solutions of TeX Source $$\cot 2\theta_{i}={\epsilon_{B_{i}}-\delta_{B_{i}} \over 2\eta_{B_{i}}},\qquad i\in\{l,r\}\eqno{\hbox{(12)}}$$ which simultaneously satisfies TeX Source $$a(\theta_{i})=2\eta_{B_{i}}\sin\theta_{i}\cos \theta_{i}-\delta_{B_{i}}\cos^{2}\theta_{i}-\epsilon_{B_{i}}\sin^{2}\theta_{i}\geq 0\eqno{\hbox{(13)}}$$ and $\epsilon_{B_{i}}$ and $\delta_{B_{i}}$ are given by the matrix $P_{B_{i}}$ TeX Source \eqalignno{P_{B_{i}}=&\,B^{-T}P_{i}B^{-1}= \left[\matrix{-\delta_{B_{i}}I&\eta_{B_{i}}I\cr\eta_{B_{i}}I&-\epsilon_{B_{i}}I}\right]\cr=&\,\left[ \matrix{-\displaystyle{\delta_{i}\over b_{22}^{2}}I&\displaystyle{1\over 2b_{11}b_{22}}I\cr\displaystyle{1\over 2b_{11}b_{22}}I&-\displaystyle{\epsilon_{i}\over b_{11}^{2}}I}\right],\qquad i\in\{l,r\}.\qquad & \hbox{(14)}}

For the proof refer to [89].

The conceptual idea of the generalized wave/scattering variable transformation is based on the equivalence of the dissipativity and small gain property of a system via the transformation of its input and output space. It is well known that the scattering operator of a system has an ${\cal L}_{2}$-gain $\gamma=1$ if and only if it is passive (lossless) [94]. Here, we extend this equivalence to QSR-dissipative systems and use it for the stability conditions; see also [40]. In fact, we can show that the choice $\theta=\theta_{r}$ results in the transformed system $\Sigma_{r}^{\prime}$ mapping the input $u_{r}$ to the output $v_{r}$, which has an ${\cal L}_{2}$-gain $\gamma_{r}^{\prime}=1$. With this choice, the left-hand side transformed system $\Sigma_{l}^{\prime}$, mapping the input $v_{l}$ to the output $u_{l}$, has an ${\cal L}_{2}$-gain $\gamma_{l}^{\prime}\ <\ 1$. As a result, the open-loop gain regarding the transformed variables including the constant unknown delay operator with $\gamma_{T}=1$ is $\gamma=\gamma_{r}^{\prime}\gamma_{l}^{\prime} \gamma_{T}\ <\ 1$. In consequence, the overall system is finite gain ${\cal L}_{2}$-stable for an arbitrarily large constant time delay. The exact proof can be found in [37] and [89], and goes beyond the scope of this paper.

#### Remark 5

For passive subsystems $\Sigma_{l}$ and $\Sigma_{r}$, i.e., with $\delta_{l}=\delta_{r}=\epsilon_{l}=\epsilon_{r}=0$ in Proposition 2, the proposed input/output transformation with $\theta=\pi/4$ and the elements of $B$, given by $b_{11}=\sqrt{b}$, $b_{22}=1/\sqrt{b}$, $b>0$, is equivalent to the standard wave/scattering variable transformation [45], [46]; see also (9).

#### Remark 6

Finite gain ${\cal L}_{2}$-stability of the overall system is not only guaranteed for an arbitrarily large constant time delay but also for any small gain operator in the communication loop. This includes an arbitrarily large constant time delay, which has an ${\cal L}_{2}$-gain $\gamma_{T}=1$, but also appropriately handled time-varying delay as, e.g., in [15], [47], [48], and [55], sampling [53], [95], and the packet loss [16], [49], [50], [51], [52], [54], as the passivity-preserving algorithms presented there have an ${\cal L}_{2}$-gain $\gamma_{P}\leq 1$. Applying those presented algorithms we can still ensure stability with the generalized scattering/wave variable transformation in the presence of the time-varying delay and the packet loss.

#### Remark 7

It is well known that due to the velocity encoding in the scattering/wave variables there might be an unrecoverable position drift between the HSI and the teleoperator if the wave/scattering variable is distorted through the packet loss or other disturbances [15], [47], [49], [96], [97]. In order to compensate for that, different approaches employing a potentially passivity violating position feedforward are proposed, which mainly differ in the way passivity or the small gain property of the corresponding operator is ensured. Most of the approaches employ an online energy monitoring and modulation strategy [47], [49], [97], which can straightforwardly be applied to the generalized wave/scattering variable transformation in order to avoid position drift.

#### Remark 8

More general conditions than Proposition 2 on finite gain ${\cal L}_{2}$-stability for any small gain network operator for QSR-dissipative systems $\Sigma_{l}$ and $\Sigma_{r}$ are provided in [37].

### D. Transparency Improvement and Discussion

In the following, we investigate the improvement in the transparency level due to the inclusion of the human (and other subsystems) damping knowledge in the generalized wave/scattering transformation approach. A high level of transparency is indispensable for facilitating human decision making based on haptic environment properties. For example, tissue differentiation and recognition based on haptic properties has a strong impact on the success of interactive robotic surgery systems.

For the sake of brevity, we illustrate the transparency improvement in a simulation study. An analytical derivation of the transparency properties as well as an experiments with a 3DoF-teleoperation system demonstrating the superiority of the generalized wave/scattering transformation can be found in [19] and [20].

For the simulation, we consider a linear time-invariant 1DoF spring-damper environment characterized by the transfer function $Z_{e}=(300/s)+30$. We assume that the dynamics of the controlled HSI and the teleoperator are negligible. The simplified control architecture for this simulation study is presented in Fig. 7, where the block for the wave/scattering variable transformation is replaced by the generalized wave/scattering transformation. Accordingly, the right-hand side system is ${\hbox {IFP}}(\delta_{r})$ with $\delta_{r}=\delta_{e}=30$. The left-hand side system is assumed to be ${\hbox {OFP}}(\epsilon_{l})$ with $\epsilon_{l}=10$ resulting from the human dynamics' minimum damping. A characteristic impedance $b=1$ is chosen for the standard wave/scattering variable transformation, and for fairness reasons, the generalized wave/scattering variable transformation is tuned with scaling components $b_{11}=\sqrt{b}=1$ and $b_{22}=2b_{11}b^{-1}\sin\theta\cos\theta$, such that in free space both methods have the same characteristics, i.e., display the same inertia. A more detailed explanation of the displayed mechanical properties follows in Section IV. The time delay is $T_{1}=T_{2}=$ 50 ms and the resulting system is delay-independently stable for all $\theta\in$ [3°, 87 °] according to Proposition 2. Note that for the standard wave/scattering variable transformation, $\theta={\hbox {45}}^{\circ}$, i.e., our proposed method, leaves more freedom for parameter tuning.

Fig. 7. Simplified control architecture based on the wave/scattering variable transformation for transparency analysis.

Let us compare the impedance $Z_{h}$ displayed to the human, as marked in Fig. 7, with the environment impedance $Z_{e}$. Ideally, both should be equal. In Fig. 8, we observe that the generalized wave/scattering variable transformation outperforms the standard wave/scattering variable transformation. For example, setting $\theta=\{{\hbox {10}}^{\circ},{\hbox {80}}^{\circ}\}$ a stiffness $k_{h}=\{{\hbox {166.3}}\}\ {\hbox {N/ms}}$ is displayed. This is by far closer to the real environment stiffness of $k_{e}=$ 300 N/m than with the standard wave/scattering variable transformation, which displays only 35.5 N/m; see Fig. 8. The structural difference between the two symmetric choices $\theta=(\pi/4)\pm\Delta$ and $\Delta\in[0,(\pi/4)]$ is currently under investigation.

Fig. 8. Comparison of the generalized and standard wave/scattering variable transformation: Frequency characteristic for the environment impedance $Z_{e}=(300/s)+30$ and the displayed impedance depending on the transformation angle $\theta$.

In summary, the inclusion of human (and environment) dynamics knowledge improves control performance in terms of transparency. The proposed methods of dissipative modeling in conjunction with the generalized wave/scattering variable transformation clearly show superior and robust performance when compared to many other methods with less information about the human. Future work should further improve human models, e.g., by identification and/or learning methods. Another challenge lies in unifying modeling paradigms for multiscale human haptic interaction behavior in closed loop with environments, other humans, and autonomous robots.

SECTION IV

## HUMAN PERCEPTION AND CONTROL PERFORMANCE

In Section III, we have demonstrated that the inclusion of refined knowledge of the human closed-loop behavior and manipulator control in terms of their dissipativity properties may improve the control performance. When evaluating the control performance in human-in-the-loop systems, the human perception of “performance” is essential, i.e., standard control performance measures such as squared error integrals may no longer provide valid evaluation metrics. Therefore, this section is concerned with the evaluation of the control performance on human terms and the consequences for the control design. Therefore, in this section, we review known facts from human studies about human perception capabilities and conclude that from a control perspective these facts need to be generalized towards a dynamical model of human perception in order to further improve control performance in haptic teleoperation systems.

Performance in a teleoperation system is determined using the notion of transparency. A teleoperation system is called transparent if the human may not distinguish between direct and teleinteraction with a remote environment; ideally he/she feels directly interacting with the remote environment [14]. Different transparency criteria have been proposed, for example, requiring that the mechanical properties of the remote environment described by its mechanical impedance are exactly displayed to the human operator based on mechanical impedances [64] or requiring the equality of the motion and force signals at the HSI and the teleoperator [98]. We concentrate here on the impedance formulation due to its convenient frequency domain representation.

#### Definition 4 (Transparency)

In the traditional sense, a teleoperation system is called transparent if the mechanical impedance displayed to the human operator $Z_{h}$ is equal to the environment impedance $Z_{e}$ [64] TeX Source $$Z_{h}=Z_{e}.\eqno{\hbox{(15)}}$$

#### Remark 9

Naturally, there is a tradeoff between transparency and robust stability in all control schemes, i.e., ideal transparency is not achievable in real systems [64], [99], [100] due to unmodeled dynamics. Transparency further deteriorates in teleoperation systems with the time delay as the bandwidth of the closed-loop system has to be severely reduced in order to achieve stability [17], [101]. Even with the improvements through dissipative modeling the achievable transparency depends on the time delays and might be far from the ideal transparency for large time delays. In fact, if the time delay between the operator action and the corresponding haptic feedback is too large, the telerobotic system becomes inoperable. Nevertheless, direct haptic feedback is still beneficial for task completion even with a significant time delay of up to one second as experimentally validated in [5] and [102].

### A. Novel Concept of Perceived Transparency

One of the major questions is: To what extent may transparency differ from the ideal transparency in the sense of definition (15) such that the human haptic decision making abilities are not deteriorated? Therefore, we propose to take human perception resolution limitations into account and refine the technical transparency definition (15) to a novel definition of perceived transparency that takes explicitly the limited human haptic perception resolution into account as follows.

#### Definition 5 (Perceived Transparency)

A teleoperation system is called perceived transparent if the displayed impedance lies within the nondistinguishable difference range of the environment impedance TeX Source $$Z_{h}(j\omega)\in\left[Z_{e}(j\omega)-\Delta(j\omega),Z_{e}(j\omega)+\Delta(j\omega)\right] \eqno{\hbox{(16)}}$$ where $\Delta(\cdot)$ represents the nondistinguishable difference range.

Concerning the perceptual resolution, empirically derived models exist in psychophysics, a branch of psychology concerned with the quantitative relation between physical stimuli and the sensations and perceptions evoked by these stimuli. The following paragraph gives an introduction to related models.

Table 1 Perceptual Discrimination Thresholds (JNDs) for Haptic Related Properties

### B. Just Noticeable Difference (JND)

The smallest difference in a sensory input that is perceivable by a human being is called the just noticeable difference (JND). For many sensory dimensions, the JND is an increasing function of the base level of the input. For most force-related physical properties, the ratio of the two is roughly constant over a large range [102] and can therefore be represented by Weber's law [103] TeX Source $${\Delta I\over I}={\hbox {JND}}_{I}\eqno{\hbox{(17)}}$$ where $I$ is the original intensity of stimulation, $\Delta I$ is the addition to it required for the difference to be perceived, and ${\hbox {JND}}_{I}$ is the Weber fraction. In this paper, we follow most of the literature (see, e.g., [104], [105], [106], [107], [108], [109], [110], [111]), where the JNDs for mechanical parameters such as stiffness, inertia, and viscosity and for quantities, such as position, velocity, and force, are given as the percental change with respect to the original intensity of the stimulation. Some of the most relevant JNDs for haptics are summarized in Table 1.

#### Remark 10

It is well known that the experimental conditions have a significant influence on the results gained in psychophysical experiments. This explains in some cases the wide variation for JNDs. Additionally, some parameters, as, e.g., inertia, are suspected of not following Weber's law [105] given by (17). As the results are empirically obtained, they generally represent a statistical quantity, i.e., individual differences exist.

### C. Perceived Transparency and JND

We observe that the term $\Delta(j\omega)$ in (16) is related to the JND from (17). However, existing empirically derived JNDs have only limited applicability in the control context for the following reasons: existing JNDs are determined for simple impedance models only. So far it is unclear how JNDs for more complex impedances such as mass-spring-damper systems can be derived. Moreover, known approaches usually only consider the gain characteristics neglecting, for example, the phase lag for higher frequencies, which are usually caused by hardware/control bandwidth limitations (low-pass) and/or the time delay. In fact, we have shown [26], [27] that for the same amplitude characteristics discrimination thresholds increase with increasing phase difference, i.e., there is a nonnegligible dependency of the JND on the phase difference. This fact strongly supports our request for dynamic JNDs in the sense of transfer functions $\Delta(j\omega)$ according to Definition 5. The difference between the existing empirical results and the required ones is exemplarily illustrated in the following example.

#### Example 3

Consider a stiff remote environment given by $Z_{e}(j \omega)=k_{e}(1/j\omega)$. Using the existing psychophysical results on stiffness resolution, we can state that the nondistinguishable difference range $\Delta_{k_{e}}(j\omega)=k_{e}{\hbox {JND}}_{k}(1/j\omega)$, where $k_{e}$ is the reference stiffness and JNDk is the stiffness JND as given in Table 1. Here only the difference to another purely stiff environment is considered, i.e., all three transfer functions $Z_{e}(j\omega)$, $Z_{e}(j\omega)+\Delta_{k_{e}}(j\omega)$, and $Z_{e}(j\omega)-\Delta_{k_{e}}(j\omega)$ have the same phase characteristics as illustrated in Fig. 9. Consider now a stiff environment which is displayed delayed by the time delay $T$, i.e., having a transfer function $Z_{h}(j \omega)=k_{e}(1/j\omega)e^{-j\omega T}$. While the amplitude characteristics are the same as for the original environment impedance $Z_{e}(j\omega)$, the phase characteristics are modified. To the best of our knowledge, current results, except our preliminary ones [26], [27], cannot capture the differences in the phase characteristics.

Fig. 9. Impedance (pure stiffness) and nondistinguishable difference $4\Delta(j\omega)$ as available today (the same phase characteristics).

The challenge for researchers in the area to develop full dynamic models of human haptic perception remains an open field. System identification is difficult because the percept (system output) can hardly be measured but can only be derived indirectly from discrimination and detection thresholds. New system identification methods from controls are needed to make significant progress in this area. Nonlinearities due to the arm/hand/finger pose need to be modeled as well.

SECTION V

## PERCEPTION-ORIENTED TRANSPARENCY ANALYSIS AND CONTROL DESIGN

Using the definition of perceived transparency (16) and the existing empirical results for the JND, we may evaluate the influence of communication parameters on the human perception of the remote environment. In order to achieve similar haptic decision making capabilities as without a teleoperation system, the difference between the displayed and the environment impedance should be smaller than the JND. As an example, we perform the perception-oriented transparency analysis and discuss the consequences for the control design for the standard wave/scattering variable control architecture.

### A. Transparency Analysis Approach

The displayed impedance $Z_{h}$ can be derived as a function of the environment impedance $Z_{e}$, the constant time delays $T_{1}$, $T_{2}$ in the forward and backward communication channels, and the wave impedance $b$. For the transparency analysis, we assume the ideal controlled behavior of the HSI and the teleoperator; the simplified overall system is illustrated in Fig. 7. Assuming a linear time-invariant environment impedance represented by $Z_{e}(s)$ in the Laplace domain and combining it with (3) and (10) yields for the displayed impedance TeX Source $$Z_{h}(s)=b{1+R(s)e^{-sT}\over 1-R(s)e^{-sT}}\quad {\hbox {with}}\quad R(s)={Z_{e}(s)-b\over Z_{e}(s)+b}\eqno{\hbox{(18)}}$$ where $T=T_{1}+T_{2}$ is the round-trip time delay. Note that only for zero time delay $T=0$ the displayed impedance is equal to the environment impedance, meaning ideal transparency in the sense of (15). For the nonzero time delay, the displayed and environment impedances differ in general. Ideally, we would like to ask whether this difference between the environment and displayed impedances is within the nondistinguishable difference range $\Delta(j\omega)$ according to the perceived transparency Definition 5. However, current psychophysical results only capture magnitude differences but not phase differences, i.e., they are not applicable. In order to still derive at least some approximative results we use the following interesting property: for the lower frequency range, we can show that the displayed impedance has similar phase characteristics as the environment impedance; see also Figs. 10 and 11 for an illustration of this fact in free-space motion and contact with a stiff wall cases, respectively. In the next paragraph, we derive a low-frequency approximation of the displayed impedance. This is performed in two steps. First, the time-delay element in (18) is replaced by a Padé approximation. Second, the resulting transfer function is decomposed into low- and high-frequency components, with the first one being the desired low-frequency approximation.

Fig. 10. Amplitude/frequency characteristics of the exact and approximated displayed impedances in the free-space motion case.
Fig. 11. Amplitude/frequency characteristics of the exact and approximated displayed impedances in the contact with a stiff wall case.

The approximation of the displayed impedance transfer function is derived employing the commonly used Padé series of the finite order to approximate the delay transfer functions $e^{-s T}$ in (18). The order of the displayed impedance approximation depends on the order $N$ of the Padé approximation. A Padé approximation of order $N$ is valid for frequencies $\omega\ <\ N/(3T)$. In order to simplify the analysis, the time-delay element is approximated here by a first order, i.e., $N=1$, Padé series TeX Source $$e^{-sT}\approx{1-\displaystyle{T\over 2}s\over 1+ \displaystyle{T\over 2}s}.\eqno{\hbox{(19)}}$$ This comes at the cost that for a large round-trip time delay the approximation validity range does not fully cover the frequency range of human proprioreceptive and kinesthetic perception (up to approximately 60 Hz). Inserting (19) in (18) yields the approximated displayed impedance TeX Source $$Z_{h}(s)\approx Z_{h}^{\ast}(s)=b{2Z_{e}(s)+bTs \over 2b+TsZ_{e}(s)}.\eqno{\hbox{(20)}}$$ In accordance to the limited frequency range of the approximation validity, for further analysis, this transfer function is split into a low-frequency component $Z_{h,lf}^{\ast}$ and a high-frequency component $F_{hf}$ TeX Source $$Z_{h}^{\ast}(s)=Z_{h,lf}^{\ast}(s)F_{hf}(s)\eqno{\hbox{(21)}}$$ with the high-frequency component having approximately the unity gain at lower frequencies. The component $Z_{h,lf}^{\ast}$ represents a good approximation of the low-frequency behavior of the displayed impedance. The mechanical parameters of the approximated displayed impedance $Z_{h,lf}^{\ast}$ can be derived analytically as a function of the round-trip time delay $T$, the wave impedance $b$, and the environment impedance $Z_{e}$, which is exemplarily carried out in detail for the prototypical cases: free-space motion and contact with a stiff wall.

#### Example 4

In the free-space motion case, no environment force is exerted on the teleoperator $f_{e}=0$, i.e., the environment impedance is $Z_{e}=0$. The exact displayed impedance (18) is TeX Source $$Z_{h}(s)=b{1-e^{-sT}\over 1+e^{-sT}}.$$ Inserting the environment impedance into (20) gives the approximation of the displayed impedance valid for low frequencies TeX Source $$Z_{h}^{\ast}(s)=m_{h}s{1\over 1+ \displaystyle{T\over 2}s}\eqno{\hbox{(22)}}$$ with TeX Source $$m_{h}={bT\over 2}.\eqno{\hbox{(23)}}$$ The left-hand side $m_{h}s$ in (22) represents the dominant low-frequency component $Z_{h,lf}^{\ast}$ from (21). The right-hand side factor is the high-frequency component $F_{hf}$ satisfying $\vert F_{hf}(0)\vert=1$. The similarity of the exact and approximated displayed impedances for low frequencies can also be observed from their frequency responses for a simulated example; see Fig. 10. The displayed impedance is an inertia with the mass $m_{h}$ [see (23)].

#### Example 5

In the contact with a stiff wall case, a force proportional to the wall penetration depth with the stiffness $k_{e}$ acts on the teleoperator; the environment impedance is described by the transfer function $Z_{e}=k_{e}/s$. The exact displayed impedance (18) is TeX Source $$Z_{h}(s)=b{k_{e}+bs+(k_{e}-bs)e^{-sT}\over k_{e}+bs-(k_{e}-bs)e^{-sT}}.$$ The approximation (20) of the displayed impedance for low frequency is analogously computed to the free-space motion case TeX Source $$Z_{h}^{\ast}(s)={k_{t}\over s}\left(1+{bT\over 2k_{e}}s^{2}\right)\eqno{\hbox{(24)}}$$ with TeX Source $${1\over k_{h}}={1\over k_{e}}+{T\over 2b}.\eqno{\hbox{(25)}}$$ The left-hand side factor $Z_{h,lf}^{\ast}=k_{h}/s$ is the low-frequency component from (21). The right-hand side factor in (24) exhibits high-pass behavior satisfying $\vert F_{hf}(0)\vert=1$. A simulation example in Fig. 11 shows the frequency responses for the exact and approximated displayed impedances, which are similar at low frequencies. The displayed impedance in the contact with a stiff wall case exhibits a springlike behavior at low frequencies, however, with a lower stiffness $k_{h}$ than the environment stiffness $k_{e}$; see also Fig. 11. As is observable from (25), the communication subsystem including the wave/scattering variable transformation can be interpreted as a rod with a stiffness coefficient $2b/T$ in the mechanical series connection with the environment.

#### Remark 11

The proposed approximation method extends the steady state consideration on time-delay effects from [96], [115], and [116] to a low-frequency approximation.

### B. Perception-Oriented Time-Delay Transparency Analysis

Based on the above transparency analysis and taking into account human haptic perceptual limits, the influence of the time delay on the perceived transparency along with design guidelines for bilateral telerobotic systems are provided in the following. The major goal for the design is to achieve a perceived transparent system. The rationale for that is the assumption of similar human haptic decision-making capabilities with and without a teleoperation system if the difference between the displayed and environment impedances is smaller than the JND.

#### 1) Communication-Induced Inertia Perception

In the free-space motion case, an inertia is displayed to the human operator even though no inertia is contained in the environment. The inertia characteristics are induced by the wave/scattering variable transformation and the communication delay. With increasing the round-trip time delay $T$ and the wave impedance $b$, the displayed inertia $m_{h}$ proportionally grows (23) as shown in the simulation example in Fig. 12. Given a time delay $T>0$, free-space motion is transparent in the sense of (15), i.e., $m_{h}=0$, only if $b=0$, which is unfeasible in terms of the tuning requirement $b>0$. Considering human perception, an inertia is not perceivable if it is below the absolute human perception threshold $\Delta m$ for inertia.

Fig. 12. Displayed inertia $m_{h}$ in the free-space motion case $Z_{e}(s)=0$ depending on the round-trip time delay $T$ and the wave impedance $b$.

#### Example 6

Let us assume free-space motion of the teleoperator, a communication round-trip delay $T=$ 200 ms, typical for the communication over the Internet, and the wave impedance tuned to $b=$ 1 Ns/m. Then, the operator feels an inertia $m_{h}=$ 0.1 kg. If the wave impedance is chosen to be $b=$ 1000 Ns/m, then the displayed mass is already increased to $m_{h}=$ 100 kg.

#### 3) Communication-Induced Stiffness Reduction

If the environment exhibits spring characteristics, a reduced stiffness is displayed to the human. The environment feels softer than it really is. The displayed stiffness coefficient (25) nonlinearly depends on the communication time delay as shown in Fig. 13 for different environment stiffness values. Ideal transparency in the sense of (15), i.e., $k_{h}=k_{e}$, is not achievable for the nonzero time delay. Considering the human haptic perception limits, however, a transparency degradation should not be perceivable if the displayed stiffness is within the JND range of the environment stiffness $k_{h}>(1-{\hbox {JND}}_{k})k_{e}$ with $0\ <\ {\hbox {JND}}_{k}\ <\ 1$, the stiffness JND. Accordingly, a stiff environment appears transparent to the human even for the nonzero round-trip time delay as long as it satisfies TeX Source $$T\ <\ {{\hbox {JND}}_{k}\over 1-{\hbox {JND}}_{k}}{2b\over k_{e}}\eqno{\hbox{(26)}}$$ which follows from inserting (25) in the previous equation. Note that an increase in the environment stiffness reduces the allowable time-delay margin. In contrast, a high value of the wave impedance $b$ increases the delay margin and reduces the impact of the time delay on the stiffness reduction as observable from (25), i.e., it increases the transparency of stiff environments. This, however, contradicts the design rule for free-space motion. Due to the time delay, good transparency in free-space motion and for arbitrarily stiff environments is not achievable at the same time.

Fig. 13. Displayed stiffness $k_{h}$ in the contact with a stiff wall case depending on the round-trip time delay $T$ and the environment stiffness $k_{e}$.

#### Example 7

Consider a stiff wall with $k_{e}=$ 30 000 N/m, and the wave impedance tuned to $b=$ 1 Ns/m. Assuming a JND of 23% for stiffness (see Table 1), any round-trip time delay $T>$ 0.02 ms would already be perceivable by the human. A very small round-trip delay of $T=$ 1 ms substantially decreases the displayed stiffness to $k_{h}=$ 1875 N/m, a reduction by 94%. At a delay of $T=$ 200 ms, the operator perceives only a stiffness of $k_{h}=$ 10 N/m, hence 0.03% of the environment stiffness. Contacting a soft environment with $k_{e}=$ 10 N/m (see Fig. 13), the displayed stiffness at $T=$ 1 ms is still $k_{h}=$ 9.95 N/m, and at $T=$ 200 ms, it is still $k_{h}=$ 5 N/m. If the round-trip time delay $T\ <\$ 57 ms, then a stiffness difference is perceivable (with stiffness JND of 23%). Increasing the wave impedance for the hard wall to $b=$ 1000 Ns/m would allow for a round-trip time delay $T\ <\$ 19 ms for a stiffness difference still to be perceivable. Note the increased inertia in free space from the previous example with these values. Human user studies validating these hypotheses are presented in [28].

#### 5) Communication-Induced Stiffness Bound

The displayed stiffness (25) cannot exceed TeX Source $$k_{h,\max}=\lim_{k_{e}\rightarrow\infty}k_{h}={2b \over T}.\eqno{\hbox{(27)}}$$ This result is also indicated by the asymptotic behavior of the displayed stiffness for increasing environment stiffness shown for a simulation example in Fig. 14. Considering the psychophysical fact that the human feels a wall to be rigid for $k_{h}\geq$ 24 200 N/m [117], it becomes clear that only for a very small time delay and a very large wave impedance $b$ a rigid wall can be realistically displayed with this control architecture. For a large time delay, the stiffness, especially in the case of hard walls, is not transparent. Appropriate tuning (high values) of the wave impedance $b$ increases the transparency in terms of the maximum displayable stiffness.

Fig. 14. Displayed stiffness $k_{h}$ depending on the environment stiffness $k_{e}$ and the wave impedance $b$.

#### Example 8

Assuming a communication delay $T=$ 200 ms with a wave impedance tuned to $b=$ 1 Ns/m, the maximum displayable stiffness is only $k_{h,\max}=$ 10 N/m. A stiff environment feels very soft.

#### 7) Bounded Displayable Stiffness Difference

In some tasks, not only the absolute value of the displayed stiffness is important but also the possibility to distinguish between various stiff environments. This is especially important for, e.g., telesurgery applications, where different tissue characteristics have to be distinguished. As indicated by the asymptotic behavior of the displayed stiffness in Fig. 14 at higher values of the environment stiffness, a stiffness difference in the environment results in a smaller difference in the displayed stiffness. However, a difference between a reference value $k_{e}^{0}$ and a value $k_{e}$ of the environment stiffness is perceivable by the human only if the corresponding percental difference in the displayed stiffness TeX Source $$\delta k_{h}=\left\vert k_{h}-k_{h}^{0}\right \vert/k_{h}^{0}\eqno{\hbox{(28)}}$$ is larger than the stiffness JND TeX Source $$\delta k_{h}={2b\delta k_{e}\over 2b+Tk_{e}}\geq{\hbox {JND}}_{k}\eqno{\hbox{(29)}}$$ with the percental difference in the environment stiffness $\delta k_{e}$ defined analogously to (28) and the displayed reference stiffness $k_{h}^{0}=k_{h}(k_{e}^{0})$ according to (25). The percental difference $\delta k_{h}$ of the displayed stiffness and the environment stiffness $\delta k_{e}$ is equal only for the marginal cases of the zero delay $T=0$ or the infinite wave impedance $b\rightarrow\infty$. At the high delay and the high environment stiffness, a large difference in the environment stiffness may result in a nonperceivable difference of the displayed stiffness. According to (29), the appropriate tuning (high values) of the wave impedance $b$ increases the transparency in terms of the range of environment stiffness where a difference is perceivable by the human.

#### Example 9

Let us assume a communication delay of $T=$ 200 ms and the wave impedance tuned to $b=$ 1 Ns/m. If the environment stiffness coefficient is $k_{e}>$ 40 N/m, then a difference to any larger environment stiffness is not perceivable under the 23% JND assumption; see Table 1.

#### 9) JND for the Time Delay

So far the distortion induced by the absolute value of the time delay has been investigated. This section discusses when a relative increase of the time delay can be perceived by the human operator and possibly influence his/her decision making. Here we again can only approximative the results as the phase differences are not yet captured in JND formulations. For now we assume that the delay difference is haptically perceived only by the difference in the mechanical properties of the displayed impedance, specifically its magnitude difference. Then, the JND for the time delay can be derived from the results from Section V-A and the well-known JND for mechanical properties. This result is interesting with respect to the design of control architectures for telerobotic systems over the Internet coping with time-varying delay, where data buffering strategies, as, e.g., in [52], introduce an additional delay. If the additional delay results in a distortion below the human perception threshold, then no change in transparency should be perceived.

The inertia $m_{h}^{0}=m_{h}(T^{0})$ [see (23)] and the stiffness $k_{h}^{0}=k_{h}(T^{0})$ [see (25)] represent the displayed mechanical properties at the reference time delay $T^{0}$. An additional time delay $\Delta T=T-T^{0}>0$ results in the further increased displayed inertia in the free-space motion case and the further reduced displayed stiffness in the contact with a stiff wall case. The further distortion due to the time-delay difference is just noticeable by the human if the corresponding percental difference of the displayed mechanical property is equal to the JND TeX Source \eqalignno{\delta m_{h}(T^{0},\Delta T)=&\,{\hbox {JND}}_{m}&\hbox{(30a)}\cr\delta k_{h}(T^{0},\Delta T)=&\,{\hbox {JND}}_{k}&\hbox{(30b)}} where $\delta m_{h}$ denotes the percental difference of the displayed inertia defined similarly to $\delta k_{h}$ in (28), and ${\hbox {JND}}_{m}$ is the inertia JND. The just noticeable time-delay difference is computed straightforwardly using (23) and (25) in (30a) and (30b), respectively. In the free-space motion case, a time-delay difference is expected to be just perceivable by the human if TeX Source $${\Delta T\over T^{0}}={\hbox {JND}}_{m}.\eqno{\hbox{(31)}}$$ Obviously, the just noticeable time-delay difference in the free-space motion case follows a linear law similar to Weber's law [see (17)]. In the contact with a stiff wall case, a time-delay difference is expected to be just perceivable by the human if TeX Source $${\Delta T\over T^{0}}={{\hbox {JND}}_{k}\over{\hbox {JND}}_{k}+1}\left({2b\over k_{e}T^{0}}+1\right).\eqno{\hbox{(32)}}$$ In contrast to the free-space motion case, the just noticeable time delay depends on the environment stiffness $k_{e}$, the wave impedance $b$, and the reference time delay $T^{0}$. In both cases, the absolute JND $\Delta T$ increases with the reference time delay. Accordingly, any additional time delay should be avoided in the haptic telerobotic system, especially if the reference time delay is small. At the high-reference time delay, an additional delay may not further perceivably degrade the transparency. In consequence, if the buffering strategies in the telerobotic systems with the time-varying delay induce an additional time delay below the JND, then human perceived transparency is not further degraded. For further results on the influence of communication effects on the transparency, refer to [28].

#### Remark 12

For the transparency analysis, in this section, the environment is assumed to be constant. Dynamic transitions between different environments, e.g., from the free-space motion case to the contact with a stiff wall case, also have an influence on the perceived transparency as indicated in [118]. The analysis requires different techniques and is beyond the scope of this paper.

#### Remark 13

The displayed impedance parameters in (23) and (25) are derived for the standard wave/scattering variable approach. Consequently, all results in (27), (29), (31), and (32) are valid only for this specific control architecture. However, the proposed perception-oriented transparency analysis is straightforward to apply in other control architectures by using the corresponding expressions for the parameters of the displayed impedance, e.g., from [20] for the generalized wave/scattering variable approach, and from [76] for other teleoperation control architectures.

#### Remark 14

Transparency with the packet loss is far more challenging and so far closed-form solutions for the displayed impedance are not available. In [28], a Monte-Carlo-simulation-based analysis of different data reconstruction methods for the packet loss provides an insight on the mean distortion of the displayed impedance due to the packet loss including a perception-oriented analysis. The influence of the packet loss on the perceived transparency, however, is not yet fully understood and poses an interesting research question for the future.

### C. Perception-Oriented Design Aspects With Time Delay

The tuning of the wave impedance $b$ has a high impact on the transparency of the communication subsystem as observed in Section V-B. The transparency criterion (15) requires for the free-space motion $b\rightarrow 0$ as observable from (23). In contact with a stiff environment, the time delay has no influence on transparency if $b\rightarrow\infty$; see (25), (27), and (29). These are contradicting design rules that can be relaxed by considering human haptic perception and a real telerobotic system. Therefore, in this section, we no longer neglect the HSI and teleoperator dynamics. In real telerobotic systems with limited control input and robustly designed controllers, the dynamics of the HSI and the teleoperator is generally not negligible. As a result, even without the time delay, transparency in the sense of (15) is not achievable. In the following, the HSI and teleoperator dynamics refers to the locally controlled device dynamics. We assume that the HSI and the teleoperator are impedance/admittance controlled as investigated in Section II, and their dynamics is represented by (7). As there are currently hardly any results for multidimensional JNDs available in the literature, we restrict ourself here to the 1DoF case, i.e., HSI inertia and teleoperator stiffness are assumed to be given by $m_{\rm HSI}\in\BBR_{+}$ and $k_{\rm TO}\in\BBR_{+}$, respectively.

In free-space motion, at least the inertia $m_{\rm HSI}$ induced by the HSI dynamics is displayed to the human. If the wave impedance is chosen TeX Source $$b\ <\ {2\over T}{\hbox {JND}}_{m}m_{\rm HSI} \eqno{\hbox{(33)}}$$ then the displayed overall inertia, the sum of the HSI and the communication induced inertia, is within the JND range of the HSI inertia $m_{h} \ <\ (1+{\hbox {JND}}_{m})m_{\rm HSI}$ as straightforwardly derivable from (23). No additional communication-induced transparency degradation should be perceivable by the human then. The original design requirement $b \rightarrow 0$ is relaxed.

In order to avoid the contact instability or oscillations, the teleoperator is typically compliance controlled. The resulting stiffness $k_{{\rm TO}/e}$ of the teleoperator together with the environment computes from the environment stiffness $k_{e}$ and the stiffness $k_{\rm TO}$ of the compliance controlled teleoperator according to the serial connection of springs $k_{{\rm TO}/e}^{-1}=k_{e}^{-1}+k_{\rm TO}^{-1}$. If the wave impedance is chosen to be TeX Source $$b>{T\over 2}\left({\hbox {JND}}_{k}^{-1}-1 \right)k_{{\rm TO}/e}\eqno{\hbox{(34)}}$$ then the communication-induced reduction is within the JND range of the combined teleoperator/environment stiffness $k_{h}>(1-{\hbox {JND}}_{k})k_{{\rm TO}/e}$. The upper bound of the teleoperator/environment stiffness $\sup_{k_{e}}k_{{\rm TO}/e}=\lim_{k_{e}\rightarrow\infty}k_{{\rm TO}/e}=k_{\rm TO}$, i.e., the teleoperator compliance, determines the lower bound of a transparently designed wave impedance $b$ [see (34)]. Clearly, the original transparency requirements for the communication subsystem design $b\rightarrow\infty$ are relaxed by (34).

#### Example 10

The haptic input device ViSHaRD10 [118] displays an inertia of at least $m_{\rm HSI}=$ 8 kg without time delay. With the Internet-realistic round-trip time delay of $T=$ 160 ms and an inertia JND assumption of ${\hbox {JND}}_{m}=$ 21% [110] for the communication transparent design in the free-space motion case, the wave impedance should be $b\ <\$ 21 Ns/m [see (33)]. In the contact with a stiff wall case, assuming a teleoperator compliance of $k_{\rm TO}=$ 900 N/m as in [120] and a stiffness JND of 23% [108], the wave impedance should be $b>$ 241 Ns/m [see (34)]. The gap between the design requirements for the wave impedance $b$ for free-space motion and contact with a stiff wall cases derived from the strict transparency criterion (15) becomes smaller by considering human perception aspects and the real telerobotic system. For round-trip time delays smaller than $T=$ 4.7 ms, this gap is closed with the wave impedance tuned to $b=$ 71.1 Ns/m as straightforwardly computable from (33) and (34). In general, the time delay has to be sufficiently small in order to guarantee perceived transparency simultaneously for the free-space motion case and for the contact with the same tuning of the wave impedance.

### D. Discussion

The major contribution of this section is the introduction of the notion of perceived transparency taking into account human haptic perceptual limits. Using the concept of perceived transparency, the effect of teleoperation control architectures and communication parameters on the human perception of the remote environment is analyzed. A key innovation is the consideration of human perception in the performance evaluation and control design. It is shown that the inclusion of human haptic perceptual limits decreases the requirements for a desirable level of transparency. Thereby, the gap between the achievable level of transparency and the desired level becomes smaller. From the control perspective, we conclude that existing human perception models are not yet sufficient. The modeling of human haptic perception based on dynamical models and their identification poses a major challenge for the future research of psychophysics and system theory. Further, major challenges in this area are the understanding of higher level human haptic signal-based decision making in the presence of disturbances, the appropriate adaptation of communication parameters in critical situations, and the augmentation of human haptic perception and decision making, e.g., by sensitivity-based design.

SECTION VI

## PERCEPTION-ORIENTED HAPTIC DATA COMPRESSION

While the design questions in Sections III and V are mainly concerned with the treatment of communication unreliabilities such as the time delay, also limited communication resources represent a major challenge in networked teleoperation systems. Severe communication constraints are imposed by the communication technology and infrastructure in space and underwater teleoperation applications; see, e.g., [121] and the references therein. Generally, in mobile (wireless) telepresence applications, higher network traffic is directly related to higher power consumption potentially reducing the lifetime of intermediate communication hops and the mobile agent. In common purpose communication networks, such as the Internet, the limited communication resources are shared by multiple network applications. High network traffic may lead to network congestion and hence large transmission time delays and the packet loss. In standard teleoperation systems, the coupling control is sampled at the same rate as local controls, i.e., around 1 kHz. In order to avoid the additional time delay, block coding cannot be performed. Instead, each sample is instantaneously sent over the communication channel, i.e., the packet rate equals the local sampling rate at the HSI and the teleoperator. From this point of view, it is of high interest to reduce the network traffic. Data compression methods as well as transmission protocols for digital video and audio are well developed. Perceptual models are successfully employed for the efficient compression of video and audio data; codecs such as MPEG-4 (video) and MP3 (audio) and multimedia transmission protocols such as H.323 are standardized and commercialized. In contrast to that, haptic data reduction techniques are rather poorly treated in the known literature. Only few researchers consider the compression of haptic data, mostly by applying quantization schemes [2], [122], [123], [124]. Differential pulse code modulation (DPCM) is proposed in [2] and [124]; an adaptive DPCM is considered in [122] and [123].

A major achievement in the area of haptic data compression is the psychophysically motivated deadband control approach originally proposed in [125] and further analyzed and extended in [21], [22], [23], [24], [25], [29] by the authors and coworkers. The deadband approach is a lossy perceptual coding approach for haptic signals that exploits human haptic perception limits using Weber's law of JNDs. It reduces the packet rate by removing data that are considered to be unperceivable by the human. A compression ratio of up to 90% on a two-channel velocity–force teleoperation system is achieved without significantly impairing human immersiveness as empirically shown in human user studies [23], [24], [25]. In this section, we introduce the basic principle and related results. For the sake of simplicity, the focus is on teleoperation systems without additional communication unreliabilities allowing for a simple velocity–force architecture. A concise treatment of the case with the time delay and the wave/scattering variable control architecture can be found in [24].

With deadband control, measurements are sent over the communication network only if the difference between the current measurement and the most recently sent value exceeds a certain threshold: the deadband width. As a result, data packets are no longer transmitted in equidistant time intervals. A similar approach is proposed in [126] to reduce the network traffic in the related networked control systems (NCSs). This is visualized in Fig. 15 where the instants of packet transmission for the example signal when using our deadband approach are represented by triangles. Deadband control obviously leads to a reduction of the number of transmitted packets. If, for example, the teleoperator moves in free-space motion (zero force), then no force data packets at all are transmitted. Accordingly, if the HSI does not move or moves with a constant velocity, no velocity packet is transmitted either.

Fig. 15. Principle of the human-perception-motivated deadband approach.

The deadband approach to reduce network traffic can be interpreted as a lossy compression algorithm exploiting the fact that the human is not able to discriminate against arbitrarily small differences in haptic stimuli; see Section IV. With the deadband approach, data are sent over the communication channel only if the difference between the most recently sent sample $x(t^{\prime})$ and the current value $x(t)$, where $t>t^{\prime}$, exceeds a threshold $\Delta_{x(t^{\prime})}$ TeX Source $$\left\vert x(t^{\prime})-x(t)\right\vert>\Delta_{x(t^{\prime})}.$$ In the velocity–force architecture without the wave/scattering variable transformation, value $x(t^{\prime})$ represents either the HSI velocity or the teleoperator force. Using the insights of Weber's law [see (17)] for motion and force data (see also Table 1), this threshold is chosen to grow proportionally with the magnitude of the signal $x^{\prime}$ TeX Source $$\Delta_{x(t^{\prime})}=\varepsilon\left\vert x(t^{\prime})\right\vert>0\eqno{\hbox{(35)}}$$ where $\varepsilon\in(0,1)$ is a factor that influences the size of the deadband. We intentionally use a different notation for the deadband parameter as for the JND, because the proposed compression scheme is inspired by Weber's law but not destined to identify JNDs. Therefore, a meaningful choice, from the perceived transparency (see Definition 5), would be $\varepsilon\leq JND$. If the haptic signal exceeds the perception threshold, then a signal update event is triggered and the packet is transmitted; the subsequent deadband is redefined based on the update sample magnitude according to (35).

### B. Stability Through Passive Reconstruction

The transmission of fewer data packets itself does not have an effect on the stability of the haptic feedback system. However, at each receiver side, the local control loops still operate at the original constant sampling rate. Accordingly, updates of the current velocity/force measurement are required in each sampling instant. Due to deadband control, however, these measurements are not directly available at each sampling instant, but only if a data packet is transmitted. The missing measurements, namely the desired teleoperator velocity $\mathdot{x}_{t}^{d}$ and the desired HSI force $f_{h}^{d}$, have to be reconstructed. As this takes place in the closed control loop, the data reconstruction has to be performed such that stability is preserved as it is well known that passivity of a continuous system is not an invariant property under sampling [127]. Therefore, we propose the data reconstruction algorithm TeX Source \eqalignno{f_{h}^{d}(t)=&\,f_{e}(t^{\prime})+{\rm sign}\left\{\mathdot{x}_{h}(t)\right\}\Delta_{f_{e}(t^{\prime})}\cr\mathdot{x}_{t}^{d}(t)=&\, \mathdot{x}_{h}(t^{\prime})-{\rm sign}\left\{f_{e}(t)\right\}\Delta_{\mathdot{x}_{h}(t^{\prime})}&\hbox{(36)}} with $f_{e}$ the environment force, $\mathdot{x}_{h}$ the HSI velocity, and the signum function TeX Source $${\rm sign}\{x\}=\cases{-1,& if x\ <\ 0 \cr 1,&otherwise.}$$ It preserves passivity of the human/HSI and teleoperator/environment subsystems, and thereby guarantees stability of the closed-loop system. The first line of (36) represents the reconstruction algorithm using the forward path; the second line represents the reconstruction algorithm using the backward path. The first term corresponds to a hold-last-sample algorithm, which is widely used in sampled data systems. It is modified by the latter term such that no energy is generated. For further details including the stability proof, refer to [21] and [28]. The overall system architecture with deadband control and data reconstruction is presented in Fig. 16.

Fig. 16. Velocity–force control architecture with the deadband approach (without the time delay).

### C. Perceived Transparent Design

The level of transparency with the deadband approach is determined by the size of the deadband parameter $\varepsilon$ in (35). A low value results in low distortion of the velocity and force signals. On the other hand, a significant packet rate reduction is achieved only for appropriate values of the deadband parameter. However, considering the limits of human haptic perception, the haptic data reduction may still be transparent with slightly distorted signals; see also Definition 5. Thus, the design goal is to minimize the network traffic by maximizing the deadband parameter while maintaining perceived transparency. Therefore, the deadband parameter $\varepsilon$ is determined in psychophysical studies.

#### Remark 15

The distortion of the velocity signal does not only affect the perception of the velocity itself, but also the velocity error between the HSI and the teleoperator may lead to a position drift between the HSI and the teleoperator. The position drift does not only deteriorate the transparency, but also it may drive the system to inoperability if the HSI or the teleoperator reaches the limit of its workspace. In [15], the velocity–force architecture is extended by a passive position feedforward to compensate for the eventual position drift. In order to improve the position tracking, this architecture is also applied here. A HSI position update is transmitted together with the HSI velocity and used for the teleoperator control. More details on the full control architecture can be found in [21] and [28].

### D. Experimental Results From a Psychophysical Study

Here we present a selection of experimental results from our earlier studies [23]. In order to achieve the design goal of a minimal network traffic while maintaining perceived transparency we determine the maximum value of the deadband parameter $\varepsilon$ where the system still appears transparent to the human operator.

The experimental setup consists of two identical 1DoF haptic devices connected to a PC and a stiff wall as the environment as shown in Fig. 17. The teleoperator can be moved in free space and a stiff wall (wooden board) can be touched. The angle of the devices is measured by an incremental encoder: force by a strain gauge. The sensor data are processed in the PC where all control algorithms (HSI force control, teleoperator velocity control) including the deadband control given by (35) and the data reconstruction given by (36) are implemented. The control loops operate at a sampling rate of 1000 Hz representing the standard packet rate without deadband control. The deadband control and the data reconstruction strategy are equally applied with the same deadband parameter value for the velocity and force signal, i.e., in the forward and backward paths. The lower bound (see Section VI-A) is set to a small value of $\Delta_{\mathdot{x},\min}=$ 0.001 rad/s for the velocity and to $\Delta_{f,\min}=$ 0.02 N for the force, chosen such that the measurement noise has no influence.

Fig. 17. Experimental setup based on a 1DoF haptic teleoperation system used for validation of the deadband approach in [23].

In the psychophysical study, 14 subjects were tested for their detection threshold of the deadband parameter $\varepsilon$. The deadband parameter detection thresholds were determined using a three-interval-forced-choice (3IFC) paradigm. The subjects were presented with three consecutive 20-s intervals in which they should operate the system. Only in one of the three intervals, which was randomly determined, the deadband algorithm with a certain value $\varepsilon$ was applied. The other two were without deadband control. Every three intervals the subject had to tell which of the intervals felt different than the other two. No feedback was given about the correctness of the answer. The experiment started with a deadband parameter $\varepsilon=$ 2.5% and was increased by 2.5% after every incorrect answer up to a maximum of 25%. When an answer was correct, the same value was used again. The pass ended when three consecutive right answers were given. A typical example from the experiments for the position and force trajectories at the HSI and the teleoperator with and without deadband is shown in Fig. 18(a). Except for the slightly deteriorated position tracking, no effect of deadband control is obvious. Only a closer look on the corresponding values for the desired teleoperator velocity $\mathdot{x}_{t}^{d}$ and the desired HSI force $f_{h}^{d}$ reveals the effects of deadband control and the data reconstruction algorithm.

Fig. 18. (a) Position and force trajectories from the experiments with and without deadband and (b) the effect of deadband/data reconstruction on the desired teleoperator velocity $x_{t}^{d}$ and the desired HSI force $f_{h}^{d}$, taken from [23].

The mean detection threshold over all passes for the deadband value is $\bar{\varepsilon}=$ 12.1% with a standard deviation of 3.6%. We observe that the detection threshold is in the range of the JNDs for velocity and force perception; see Table 1.

In order to investigate the effect of the deadband control, the induced network traffic was recorded during the experimental user study. The mean percentage of transmitted packets as a function of the deadband parameter $\varepsilon$ is shown in Fig. 19; 100% represent the standard approach with 1000 packets/s on the forward and backward paths, respectively. As expected, higher deadband parameter leads to higher traffic reduction. The traffic volume induced by velocity packets is already at 25% at a deadband size of $\varepsilon=$ 10% and keeps falling with an increasing deadband size. The impact on the number of transmitted force packets is even higher. Already at $\varepsilon=$ 2.5% the network traffic volume in the backward path is less than 10% of the standard approach. At $\bar{\varepsilon}=$12.1%, only 13.2% of the original number of packets are transmitted. Note that within the standard deviation range of $\bar{\varepsilon}$ the number of transmitted packets changes only slightly from 15.2% to 11.4%. Accordingly, the network traffic reduction is in the range of 84.8%–88.6%.

Fig. 19. Network traffic reduction by deadband control: Average number of transmitted packets depending on the deadband parameter $\varepsilon$, taken from [23].

In summary, psychophysically motivated deadband control is a very successful approach with respect to transparent network traffic reduction in haptic teleoperation systems.

#### Remark 16

Note that the reported network traffic reduction is a mean value. Strict communication rate guarantees can be given only in terms of the sampling rate. The maximum packet rate of up to the sampling rate may occur at very sudden changes in velocity and/or force, which in the experiments was observed, e.g., at the moment of impact with the environment.

#### Remark 17

Our work [24] extends the human-perception-motivated deadband approach to teleoperation systems with the arbitrarily large time delay. The wave/scattering variable approach is considered there. Compression rates of 96% are achieved at a round-trip time delay of $T=$ 100 ms. Stability is guaranteed by an appropriate data reconstruction that renders the deadband/reconstruction parameter small gain; see also Section II. It is interesting to note that in this coupling control architecture an absolute deadband value, i.e., not changing its size depending on the signal amplitude, achieved a better result than the proportional deadband, as presented in (35).

#### Remark 18

Human haptic perception is not isotropic, i.e., the perceptual resolution also depends on the force–velocity direction [128]. An efficient multidimensional deadband approach considering this anisotropy is introduced in our recent work [30], where also an optimal passivity-preserving reconstruction algorithm was introduced. The experimental setup used in this work is shown in Fig. 20.

Fig. 20. Experimental setup based on the 4DoF haptic teleoperation system used for validation of the multidimensional deadband approach in [29].

#### Remark 19

The deadband approach so far has been mainly discussed in the context of haptic data reduction in teleoperation systems. However, further straightforward domains of the application are in haptic virtual reality systems, where the computational load should be minimized without sacrificing perceived transparency, and in the storage of haptic data, e.g., in “haptic photography” and “haptic video.”

SECTION VII

## CONCLUSION AND FUTURE RESEARCH DIRECTIONS

In this paper, we presented recent results on the human-oriented control design for haptic teleoperation systems facilitating human decision making and action in inaccessible environments. The tight coupling and inclusion of the human into the control loop poses major challenges with respect to stability and performance evaluation. The goal of our work is to improve teleoperation system quality such that the human decision-making capabilities are preserved as well as possible during teleoperation while guaranteeing robust stability of the overall system. Therefore, we explicitly consider human factors such as human closed-loop dynamics and human perception for control design and performance evaluation. The major conclusion from our investigations is that human haptic closed-loop behavior and perception models—both still missing in the desirable depth—significantly improve the control design.

Haptic teleoperation systems typically are on the lower signal-based levels in the control hierarchy. The human directs the technical (teleoperation) system, provides all reference set points and trajectories, and also ultimately closes the loop from environment sensor measurements to actions within the environment. Recently, semiautonomous and/or assistive teleoperation systems have been investigated, where (part of) the goal-oriented behavior was also generated by the system itself. This led to higher level interactive systems, where not only a tight coupling on the signal level but also a coupling on the higher symbolic levels of human task performance was to be implemented. It is one of the grand challenges to investigate rigorous control methodologies for such new generation human-in-the-loop interactive systems performing joint actions on several levels of the chosen modeling and control hierarchy.■

### Acknowledgment

The authors would like to thank E. Steinbach, J. Kammerl, and P. Hinterseer for the fruitful collaboration in the area of haptic data compression; and T. Matiakis, I. Vittorias, M. Rank, and A. Peer for their theoretical and experimental contributions in the area of haptic teleoperation control design and human perception models. They would also like to thank W. Jaschik, J. Gradl, H. Kubick, R. Geng, D. Weilbach, T. Stöber, and T. Lowitz for technical support.

## Footnotes

This work was supported in part by the German Research Foundation (DFG) within the Collaborative Research Center SFB453 on “High-Fidelity Telepresence and Teleaction,” the Priority Program SPP 1305 “Control Theory of Digitally Networked Dynamical Systems,” and Technische Universität München.

The authors are with the Department of Electrical Engineering and Information Technology, Technische Universität München, Munich 80290, Germany (e-mail: s.hirche@ieee.org; m.buss@ieee.org).

1The terms of teleoperation, telemanipulation, telepresence/teleaction, and telerobotics are generally used in an interchangeable manner in the engineering community. In this paper, we refer to it as teleoperation.

2The technology of virtual reality (VR) systems is closely related to teleoperation; there the teleoperator and remote environment are replaced by a software implementation of the VR. In fact, many of the control mechanisms and results for haptic teleoperation carry over to haptic VR. The enhancement of VR is considered as one of the Grand Challenges of Engineering published by the National Academy of Engineering [12].

## References

No Data Available

## Cited By

No Data Available

None

## Multimedia

No Data Available
This paper appears in:
No Data Available
Issue Date:
No Data Available
On page(s):
No Data Available
ISSN:
None
INSPEC Accession Number:
None
Digital Object Identifier:
None
Date of Current Version:
No Data Available
Date of Original Publication:
No Data Available