Transformer-Based Multi-Target Object Detection and Tracking Framework for Robust Spatio-Temporal Memory in Dynamic Environments

Multiple object tracking (MOT) is a critical task in computer vision, applied in areas such as autonomous driving, sports analytics, and human activity recognition. Traditional online MOT approaches, which rely on separate detection and re-identification (re-ID) stages, often suffer from high computational costs and suboptimal accuracy. To address these challenges, we propose MOTTNet, a unified framework that integrates detection and tracklet mapping using transformer-based architectures. MOTTNet leverages a spatiotemporal memory network to store past observations of tracked objects, allowing for more robust identity preservation and trajectory prediction over time. The Target Proposal Module generates object candidates using a transformer encoder-decoder structure, and the memory decoder integrates both proposal and track vectors to predict object locations and identities in real-time, minimizing post-processing. The framework incorporates a multi-scale attention pyramid to handle variations in object scale and utilizes a deformable transformer for generating accurate object proposals. A memory encoding-decoding aggregates object features and associates them across frames, allowing robust detection and identity preservation over time. MOTTNet achieves IDF1 scores of 79.8 and 80.4, MOTA scores of 79.3 and 80.1, and HOTA scores of 73.2 and 72.7, respectively, on on the MOT17 and MOT20 datasets. Additionally, MOTTNet exhibits lower identity switch rates, significantly outperforming existing transformer-based and traditional tracking models.