CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions

Summary

This paper introduces a novel continual multi-source adaptation method to tackle a new Test-Time Adaptation (TTA) task involving dynamic distributions. The method integrates multiple source models to adapt continuously to the evolving test data distribution. It efficiently computes the optimal combination weights for merging the source models and identifies which source model parameters require updating. Additionally, the authors present a thorough theoretical analysis of the optimization convergence and test risk bound, supported by extensive experiments demonstrating the effectiveness of the proposed method.

Strengths

1. The authors propose a novel continual multi-source adaptation method to address the TTA task with dynamic distributions, which is a challenging and practical problem. 1. The theoretical analysis of the optimization convergence and test risk bound is thorough and well-supported.

Weaknesses

1. The proposed method is not adequately supported by the experimental results. The authors claim that the method can adapt to dynamic test data distributions, i.e. sunshine interspersed with rain, modeled as a linear combination of source distributions. However, the authors did not construct a corresponding test dataset reflecting these conditions, but instead used a continual manner for the experiments. 1. As a plug-in method, the proposed approach lacks experiments based on recent TTA methods. The experiments only included three methods, Tent, CoTTA and EaTA. The authors should incorporate more recent TTA methods in realistic scenarios, such as RoTTA$^{[1]}$, TRIBE$^{[2]}$, and ROID$^{[3]}$. - [1] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, 2023. - [2] Yongyi Su, Xun Xu, and Kui Jia. Towards real-world test-time adaptation: Tri-net self-training with balanced normalization. In AAAI, 2024. - [3] Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. In WACV, 2024.

Questions

I have some questions and suggestions for the authors: 1. I suggest that the authors provide a toy experiment by constructing a dynamic test data distribution and conducting experiments using ground truth weights to validate the effectiveness of the proposed method. 1. According to the current test settings, the authors train four source models (vanilla, fog, snow, and frost) and test them in a continual manner. For the results in the fog, snow, and frost domains, it is expected that directly using the corresponding source model would yield the best results, i.e., X-best would be the best. However, this is not the case in the experimental results. Could the authors provide an explanation?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper introduces a new task to consider continual mutli-source test-time adaptation to dynamamically evolving distributions. A framework is proposed consisting of two key step: (1) learning the combination weights and (2) identify the most correlated source model to update. In order to speed up the optimization, this paper further designs a good initialization strategy and selects the optimal step size. Extensive experiments are conducted to validate the effectiveness of the proposed model.

Strengths

1. Writing quality sounds good. The paper is well-structured. Theoretical insights are also provided. 2. SOTA performance. The proposed framework can be integrated with various single source TTA models. It can improve the performance of single source TTA models by combining with the proposed method. 3. Ablations. Ablation experiments are provided to evaluate the impact of the proposed modules.

Weaknesses

1. Some parts of the writing are unclear. a) In Line 182, clarify what the t-th test batch refers to and how it is calculated. Why not compute the distance for each source model independently? b) In Line 270, unclear descriptions cause confusion for the reader. c) Algorithm 1 in the Appendix contains some typos. 2. The proposed framework largely follows strategies from existing multi-source domain adaptation methods. For example, the approach to learning combination weights and the use of weighted pseudo-labeling strategies are similar to those outlined in [18]. 3. The differences between challenges in the new task and multi-source DA and CTTA are not clearly described. The proposed task appears similar to methods addressing multiple source model adaptation without accessing the source data. The CTTA setting typically deals with catastrophic forgetting in single source adaptation. However, multi-source adaptation inherently mitigates forgetting, as noted in Line 110. It remains unclear how the challenges of the multi-source CTTA task differ from those in multi-source DA and CTTA.

Questions

Please refer to the weaknesses above.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The limitation is included in the paper.

Summary

The work introduces a new framework called CONTRAST, designed for dynamically combining multiple pre-trained source models during testing to adapt to changing target data distributions. For each test batch, CONTRAST learns the optimal combination weights of the source models, ensuring that the test error of the combined model does not exceed that of the best single source model. CONTRAST updates the parameters of the source models most relevant to the current test data, avoiding the forgetting of irrelevant model parameters, thereby maintaining good performance in long-term dynamic distribution adaptation. Theoretical analysis shows that CONTRAST optimizes the combination weights by balancing the distribution shift of the source models and the quality of pseudo-labels, minimizing the risk on the target distribution. Experimental results demonstrate that CONTRAST outperforms single source model adaptation methods on dynamic distribution setup.

Strengths

1. The method of the work is clear, intuitive, and easy to implement. 2. The writing is clear, making it easy to understand and read.

Weaknesses

1. Using multiple source models and performing ensemble is a feasible and reasonable approach to improve test-time adaptation. However, since previous techniques only utilized a single source model, this ensemble approach introduces an unfair comparison in the experiments. 2. In the current task setting, dynamic test-time adaptation requires sequential model updates. However, this may not be a reasonable design or baseline. After the first update, we will have two models: the original model and the model updated on Task 1. Therefore, in practical scenarios, when a new task arrives, we wouldn't simply update the model that was updated on Task 1. Instead, we would at least try to update both the Task 1 updated model and the original model then ensemble them, or even discard the Task 1 updated model and start updating from the original model again.

Questions

1. It is recommended that the authors add two baselines: one using the original TTA method with multiple source models, which can be achieved by ensembling multiple TTA-updated models. Another baseline is using the TTA method, independently updating from the original model for each task. 2. It is suggested that the authors discuss the computational complexity of the learning rate selection method. The experiments in the appendix indicate that the effective learning rate range is quite large. The difference between the proposed method and the optimal fixed learning rate is only 0.4. Therefore, to evaluate the necessity of using the proposed method, it is important to consider the additional computational overhead it introduces.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are discussed in the corresponding section.

Summary

The manuscript address continual learning in the context of adaptation to multiple data distributions. The method employs a model ensemble for unsupervised domain adaptation to the dynamically evolving distributions. The weights denoting the contribution of each model are calculated through optimization. The bounds for the optimal solution are provided. The combination of different models weights for the model ensemble model are calculated and updated during the continual learning. Only the model that correlates the most to the new data is adapted.

Strengths

* The manuscript address continual learning in the context of adaptation to multiple data distributions. * During the test time, multiple source models are adapted such that it optimally blends the sources * The theoretical foundations for the proposed methodology are provided * Extensive experimental results are provided in the manuscript and in the supplemental materials

Weaknesses

* Many references have missing bibliographic information. For example reference [36] should be marked as being presented at MIDL 2023 and published in PMLR. Reference [3] was presented at the International Conference on Learning Representations (ICLR), 2021. Also, pages are missing in many references, including [1,2,4,5,6,7] among many others.

Questions

What happens if many models are correlated with the new data to be learnt?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Broader Impact and Limitations are discussed in Section 6 of the manuscript.