Feature Extraction Method Using Lag Operation for Sub-Grouped Multidimensional Time Series Data

Systems in the real world often consist of multiple subsystems interacting with each other, for example, the musculoskeletal system or human-human interaction. The measurement of temporal changes in these systems involves a multidimensional time series. This study introduces a novel framework for extracting features from multidimensional time series data with a group structure using self-supervised learning techniques. Specifically, we use a “lag operation,” which is a temporal shifting operation applied to the features of a certain group. We propose a self-supervised learning method for a neural network model that uses the data automatically generated by the lag operation and its corresponding operation labels to capture and quantify the interaction between groups. Upon completion of the training process, the representation space is obtained with the expectation that it will capture timing-dependent features within its boundaries. We define and calculate the interaction score, R-score, on the obtained space. To validate our approach, we apply the proposed methodology to an artificial oscillator and approximately 4 hours of conversational data to evaluate the R-score properties. From the results of the artificial data, the R-score increases when the connection between the groups is large. From the high R-score region of the representation space of the conversation data, we extract the data that contain social behaviors such as “eye contact,” “turn-taking,” and “smiling,” which are related to the interaction between the participants. The experimental results suggest that the proposed method can obtain a representation space for time series data with a group structure.