Persistent Test-time Adaptation in Recurring Testing Scenarios

Summary

The paper proposes a novel method called Persistent Test-time Adaptation (PeTTA), aimed at addressing the gradual performance degradation of models when used for long-term test-time adaptation (TTA). Traditional TTA methods adapt to continuously changing environments but fail to account for the cumulative errors and performance drops that can occur when these environments reoccur over time. The study demonstrates, through the simulation of a simple Gaussian Mixture Model classifier, how TTA methods might gradually fail in such recurring environments. PeTTA achieves a balance between adaptation and preventing model collapse by monitoring the model's tendency to crash and adjusting the adaptation strategy accordingly, significantly improving the model's stability and performance in long-term testing.

Strengths

(1) The study introduces a new test scenario—recurring TTA—which realistically simulates conditions that might be encountered in real-world applications. This setup is more practical than traditional static test scenarios and helps reveal potential long-term issues with TTA methods. (2) By using simulations with Gaussian Mixture Model classifiers, the study thoroughly analyzes the reasons for performance degradation in TTA methods and proposes a theoretically supported solution. PeTTA's stable performance across multiple benchmarks validates its effectiveness.

Weaknesses

(1) Although the recurring test scenario has theoretical significance, in practical applications, environmental changes involve more than just lighting conditions; factors such as shooting angles and weather conditions also impact the captured data. Therefore, a simple Gaussian Mixture Model classifier may not fully simulate the variations in real complex scenarios. (2) While virtual data experiments provide theoretical support, whether their results fully apply to real scenarios requires further validation. Using real images for experiments in theoretical analysis would better validate the practical applicability of the PeTTA method.

Questions

Please see the weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper investigates the risk behind long-term test time adaptation. To achieve this, the authors simulate a long-term test data stream called Recurring Test-Time Adaptation by repeating a single period continual TTA setting for 20 times, and propose a Persistent TTA (PeTTA). Within the proposed algorithm, the authors calculate the Mahalanobis distance between the feature distributions at the current time slot and before test-time adaptation to weight the regularization term and model updated momentum. Moreover, the authors also leverage an Anchor Loss in the probability space to further avoid over-adapting process. To validate the proposed method, the authors conduct the experiments on several TTA benchmark datasets, e.g. CIFAR10-C and ImageNet-C. The results demonstrate the effectiveness of the proposed method which provides stable adapting process in long-term test time adaptation scenarios.

Strengths

1. This paper investigates a realistic and valuable TTA setting, whereas the test data stream is enough long and most of the existing TTA methods based on self-training self-supervised objectives produce poor results. 2. This paper is well-written and sounds interesting. 3. The authors attempted to theorize the causes of long-term TTA failures.

Weaknesses

1. The novelty is incremental. Methodologically, the Anchor Loss proposed in this paper is similar to that used in [A] (compare eq.8 in the manuscript with eq. 5 in [A]), only replacing L2 distance between two probabilities to Cross Entropy. In the ablation study, I observe most of the promotion is provided by this Anchor Loss module on CIFAR100, DN and IN-C datasets. Based on these, I suggest the authors to provide some experimental comparisons on these two TTA method and more discussions on the differences between them. 2. The danger brought by self-training based TTA is well-known [B], while there is no denying that the author may be correct in making the theoretical analysis in the manuscript. There are many TTA methods proposed to alleviate this issue, for example, Balanced BatchNorm proposed by [A] alleviates the impact from the biased class to batchnorm, class diversity weighting utilized in [C] avoid the accumulation of prediction bias during test time. Unfortunately, they are not discussed and compared in the experiments of the manuscript. [A] Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalization, AAAI 2024. [B] On Pitfalls of Test-time Adaptation, ICLR 2023. [C] Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correction, WACV 2024.

Questions

See the weakness section above. By the way, how about the performance of feature alignment based TTA method e.g. [D] in the recurring TTA scenario. [D] Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering Regularized Self-Training. TPAMI 2024.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

There is not nearly enough technical novelty, and there is a lack of discussion and comparison of some of the competing methods that should be compared.

Summary

The paper provides theoretical and empirical analyses on the error accumulation and model collapse in continuous TTA scenarios. From the analyses, the authors discover the risk of using constant key hyperparameters ($\alpha$ and $\lambda$ in RoTTA) and periodic reset of model parameter (in RDumb). They propose Persistent Test-time Adaptation (PeTTA) with adaptive $\alpha_t$ and $\lambda_t$ depending on distribution shifts and anchor loss $\cal{L}_{AL}$. The proposed method shows lower and stabler error over the continuous distribution shifts.

Strengths

- The paper considers a practical scenario of TTA, continuous TTA. - The theoretical analysis explains an interesting outcome of error accumulation in TTA, called model collapse: a collapsed model is prone to misclassifying several classes as a few classes. This is also verified in an experiment. - Based on the findings of model collapse, the authors propose a mechanism of detecting divergence of $\theta_t$ and adaptively selecting $\alpha_t$ and $\lambda_t$, which is well justified and verified. - The authors provide an extensive set of experiments demonstrating their findings and the superiority of the proposed method, compared to the state of the arts: RoTTA and RDumb.

Weaknesses

- The paper focuses solely on recurring TTA scenarios, potentially overlooking other types of domain shifts such as non-cyclic domain shifts and label distribution shifts. The proposed method appears too specific to recurring TTA scenarios and may be prone to failures, particularly under label distribution shifts. - The sensitivity of hyperparameter choices, particularly $\alpha_0$ and $\lambda_0$, is not studied. Additionally, there is insufficient justification for the hyperparameter choices of other algorithms. This raises concerns about the empirical study's claim of the proposed method's superiority. - The proposed method requires access to the source dataset, as indicated in Equation 6.

Questions

The following questions include the concerns in the weakness. - Can PeTTA work under scenarios of (i) non-cyclic domain shift; and (ii) label distribution shift? - Can you provide further results on longer time horizon than 20? I want check if the error accumulation is fully resolved by PeTTA. - Is the performance of PeTTA sensitive to the choice of $\alpha_0$ and $\lambda_0$? If so, how can we select them in practice? - How did you choose the hyperparameters of PeTTA for the experiment result? Were the hyperparameters of other algorithms tuned in the same way of PeTTA? - Why there is no balancing hyperparameter in front of $\cal{L}_{AL}$? - Is PeTTA runnable without the access of source dataset?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- In the current manuscript, the limitation of this work is not described in detail. It would be great if you can provide more specific description on the direction of further improvement.

Summary

The authors proposed a practical TTA scenario called recurring TTA and, within this scenario, suggested the best-performing TTA methodology, which they named persistent TTA (PeTTA), measured by various benchmark performances.

Strengths

1. The proposed recurring TTA scenario reflects the challenging and practical situation well. 2. PeTTA is a simple yet effective solution. Especially, PeTTA excels in collapse prevention.

Weaknesses

1. I think Corollary 1 is not a rigorous condition for model collapse. Of course it guarantees the decrease of distance ($d_t^{0 \rightarrow 1}$), the distance could converge to another value because $\lim_{t \rightarrow \tau}\epsilon_t$ is not always $p_1$. Just saying 'The model collapse happens when this condition holds for a sufficiently long period' is not proper. 2. I think the role of $\mathcal{L}\_\mathrm{AL}$ is similar to $\mathcal{R}(\theta')$, but $\mathcal{L}\_\mathrm{AL}$ is not controlled by $\lambda$. However, in Table 3, adding $\mathcal{L}\_\mathrm{AL}$ leads to significant performance improvement in several benchmarks. The authors should provide the difference between $\mathcal{L}\_\mathrm{AL}$ and $\mathcal{R}(\theta')$, and explain why they chose the current design.

Questions

1. The authors said EATA is the baseline. However, I couldn't find any results of EATA. 2. It seems that the images on the left and right in Figure 4(c) are switched. 3. How about applying only $\mathcal{L}\_\mathrm{AL}$ without $\lambda$ in Table 3?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Response to Authors' Rebuttal

Thanks for the additions to the TRIBE and ROID experiments, especially the TRIBE experiments, which allayed my concerns about the anchor loss module. The discussion of similar existing work here is necessary, and I suggest that the authors include these discussions and experiments in their camera-ready revision. Up to this point, all my concerns have been addressed, and overall this is a good paper for exploring the robustness of long time TTA methods. I would like to increase my score to Weak Accept to suggest acceptance.