Adaptive Tokenization Transformer: Enhancing Irregularly Sampled Multivariate Time-Series Analysis

Analyzing irregularly sampled multivariate time series (ISMTS) data poses significant challenges, with such irregularities frequently occurring in contexts like the Industrial Internet of Things (IIoT). However, most existing methods are designed for regularly sampled data, limiting their effectiveness in handling such complexities. These traditional approaches struggle with misalignments across time and variate dimensions, often requiring extensive preprocessing that can result in information loss and the introduction of noise. Furthermore, they may incorrectly utilize processing units, such as variate or temporal tokens, leading to suboptimal performance. To tackle these challenges, we present the Adaptive Tokenization Transformer (ATFormer), an innovative model designed to improve the analysis of ISMTS data. ATFormer employs an adaptive mechanism to select appropriate tokens (temporal or variate) based on the unique characteristics of the time series data. By capturing each observation at a finer granularity, the model enhances token representation. A masked attention mechanism aggregates observations, creating more comprehensive tokens and embedding information consistently, thereby mitigating incomplete embeddings and noise. Additionally, ATFormer facilitates the formation of fine-grained tokens and performs coarse-grained self-attention operations, enhancing the model’s utilization of tokens through the interaction of information at different granularities. This multilevel processing allows the model to effectively capture detailed information while integrating broader features, ultimately improving overall performance. Our evaluations on two healthcare datasets and one human activity dataset demonstrate that ATFormer outperforms existing methods in analyzing ISMTS.