Robot Multimodal Anomaly Diagnosis by Learning Time-lagged Complex Dynamics

Robots are prone to making anomalies when performing manipulation tasks in unstructured environments, it is often desirable that robots can detect, diagnose them, and then effectively provides prior experiences for rapidly adapting the robot anomalous behaviors. The traditional methods on anomaly diagnosis are to focus on univariate time series and ignore the modality correlation and time-dependent, which can't be applied in a Human-robot collaborative (HRC) system that is integrated with multiple sensors for improving autonomy and safety. The variability and temporal consistency of multimodal sensory data exacerbate anomaly diagnosis still an open problem in robotics. Here we propose a novel method of multimodal anomaly diagnosis by learning the time-lagged dynamics of anomalies detected during an HRC task. Specifically, a time-lagged variational auto-encoder model (
$tlVAE$
) is first proposed to compress complex multivariate dynamics into simpler manifolds, and the manifolds are used to fitting a dynamic time warping-based K-nearest neighbors model for anomaly diagnosis in a multi-classes classification scheme. A real-robot anomaly dataset is presented to evaluate the significance and effectiveness in extracting underlying time-dependent patterns, results indicate that the efficiency and precision of diagnosis can be improved by introducing a sparse representation of the multivariate sample. Meanwhile, we compare the accuracy of anomaly diagnosis with several commonly used sparse representation methods, including Principal Component Analysis (PCA), Time-lagged Independent Component Analysis (TICA), Autoencoder (AE) as well as Variational Autoencoder (VAE). The resulting anomaly diagnosis accuracy of 97.6% across 7 kinds of anomalies, which outperformed all the baselines.