Follow-up to Reviewer Utu5 [1/2]
We thank the reviewer for their comments and for acknowledging our previous response.
> I'm still wondering why the entire test set is not included in the experiments
The test sets of ImageNet-C, CIFAR10-C, and CIFAR100-C consist of 15 different corrupted versions of the original test set of each dataset—these 15 corrupted versions represent different variations of the same “clean” test images. Therefore, we found it more natural to form an out-of-distribution test set that contains a single instance of a specific image, rather than using all 15 versions of the same original image. Additionally, we used shorter adaptation streams to demonstrate our approach’s ability to achieve faster adaptation compared to baseline methods, as shown in Figure 2 (bottom right). We apologize for any confusion and hope this explanation clarifies our initial choice.
To address the reviewer's concern, we have now conducted experiments using the entire test set of CIFAR10-C and CIFAR100-C, which includes all 15 corrupted versions of each test image. The results are described hereafter.
> I understand you need a subset from the source domain for EATA and POEM, but this can be small and taken from the source data, right?
Yes, the holdout set can be small and should include unlabeled source samples. Since we use an off-the-shelf pre-trained model, we selected these holdout samples from the test set of the original dataset, representing the source domain. To ensure a fair out-of-distribution test set, we made sure our method (and EATA) does not have knowledge of the clean versions of the corrupted images in the test set. This is why we removed the holdout images from the corrupted test data, as the corrupted images are merely variations of the clean ones.
**Long-range adaptation experiments on CIFAR-10C and CIFAR100-C:**
In the following experiment, we reserved 2,500 images to form a holdout set for EATA and our method. We followed the same experimental protocol described in the global response to all reviewers and ran each adaptation method on a test set containing 112,500 samples (15 versions of 7,500 images). The results are summarized in two tables, presented in a separate comment below. These tables show that our proposed method is competitive with the baseline methods in terms of adaptation accuracy. Notably, the runtime of our method is twice as fast as SAR and comparable to EATA and TENT. Importantly, we **do not** use a model-reset mechanism (as done by SAR) or anti-forgetting loss (as done by EATA), highlighting the stability of our approach. By contrast, we found that TENT is highly sensitive to the choice of learning rate. We sincerely thank the reviewer for raising this point.
> I am not entirely convinced of entropy matching being a robust alternative to entropy minimisation
Beyond the theoretical aspects of our work and the monitoring capabilities we introduce, our experiments demonstrate several practical advantages over entropy minimization:
1. The proposed method maintains the performance of the source model while avoiding overconfident predictions under in-distribution settings, a crucial advantage over entropy minimization methods.
2. Short-term adaptation: Our approach achieves faster adaptation than entropy minimization methods, e.g., as shown in Figure 2 (bottom right). This is attributed to our betting scheme that quickly reacts to distribution shifts. While the reviewer emphasizes the problem of long-range adaptation, it is important to recognize the critical role of adaptation speed as well. Rapid adaptation is especially crucial, amid various strategies proposed for stabilizing long-range adaptation, such as resetting the self-trained model to its original state when specific heuristic conditions are met (as employed in SAR) or incorporating an anti-forgetting component into the entropy loss (as used in EATA).
3. Long-term adaptation: In extended test periods, our new experiments with 112,500 test examples show comparable adaptation performance to strong baseline methods, demonstrating the robustness of our method. Moreover, if stable long-range adaptation is the main concern, we could integrate model-resetting or anti-forgetting mechanisms. Notably, our monitoring tool can detect when unfamiliar corrupted data arrives, allowing for rigorous decisions on model resetting, for example, to prevent aggressive adaptation from a diverged state. This capability highlights another unique and practical advantage of our method.
Once again, we apologize for any confusion and hope this discussion resolves the reviewer’s concerns. Please let us know if there are any questions, comments, or concerns left.