A Multimodal Behavior Recognition Network with Interconnected Architectures

The feature extraction part of the behavior recognition network plays a crucial role in the results of recognition. Different feature extraction networks may lead to varying accuracies, and for higher efficiency, networks usually select only the optimal feature extraction network. In response to this, we propose a network architecture that combines the advantages of different feature networks, which is referred to as the connecting feature network (CFN). The CFN framework involves a two-stage method: in the first stage, we use ResNet as the feature extraction network; in the second stage, we utilize a behavior-aware network based on the vision transformer for feature extraction. We hope that the phased training will ensure the complete preservation of the advantages of different feature extraction networks. Importantly, CFN can be flexibly applied to various tasks involving multiple network architectures, thereby achieving the integration of diversified feature extraction capabilities. By strategically integrating these components, we aim to enhance the overall performance of behavior recognition systems across different domains. Finally, the effectiveness of CFN was validated in the Animal Kingdom dataset.