Mitigating the Bias in the Model for Continual Test-Time Adaptation

Modern machine-learning models often encounter streams of new, unlabeled data whose appearance gradually changes over time after deployment—for example, vision models on autonomous vehicles facing day-to-night transitions, surveillance cameras experiencing seasonal lighting shifts, or medical scanners with hardware ageing effects. Continual Test-Time Adaptation (CTA) refers to methods that, once a model is deployed, adapt it online during test time without any labeled examples or prior notice of distribution shifts to maintain accuracy in changing environments. However, as the model adapts, it can become over-confident in its mistakes and steadily biased toward recent but unreliable patterns. In this work, we introduce a simple prototype-based correction mechanism that stabilizes adaptation and reduces prediction bias. We maintain a running, class-wise exponential moving average (EMA) of target-domain feature centroids (prototypes) only from samples on which the model is confident. During adaptation, we softly re-cluster incoming features around these target prototypes and at the same time align them with the source-domain prototypes learned offline. This dual clustering alignment step can be seamlessly integrated into existing CTA methods with almost no extra computation. We validate our approach on standard corruption benchmarks, ImageNet-C and CIFAR100-C. When integrated into a strong CTA baseline, our method raises classification accuracy by 1.51% points on ImageNet-C and by 1.47% points on CIFAR100-C in the most challenging corruption streams, demonstrating that prototype-guided clustering significantly curbs drift without slowing down adaptation.