Response to Reviewer sauS
Thank you for the constructive feedback and helpful suggestions for our manuscript. Regarding the comparison with state-of-the-art methods, please refer to our “Response to common concern: comparing with the SOTA”. For your other comments, below is our point-by-point response.
**Move Appendix Fig. 5 to the main paper, poor paper writing, and missing objective function in Sec 3.3.**
We appreciate the reviewer for highlighting these presentation issues. We will fix them as suggested, and try to identify and rewrite sections that are dense or difficult to follow as well. If you have other suggestions on areas that could be improved to help us understand where to focus, it would be really helpful.
**Class extrapolation is unreasonable when there are large conceptual gaps between base and novel classes.**
We agree the quality of class extrapolation highly depends on the semantic similarity between base and novel classes. We do observe faithfully extrapolated class examples when the semantic gap is small (Fig. 3 for example), but less faithful ones otherwise.
Among the many downstream datasets we have experimented on, most of them demonstrate shared regularities between classes (unlike CIFAR10), which gives rise to good extrapolation quality. Examples include datasets with a large number of conceptually close classes like FGVCAircraft and StanfordCars, as well as datasets with only a few but pattern-sharing classes like EuroSAT (satellite images) and DTD (texture images). The well-extrapolated OOD data on these datasets yield consistent, and sometimes large, gains on novel class performance, see Fig. 6(a).
On the other hand, there are datasets UCF101 and Caltech101 that have relatively large semantic gap between classes. Their data size is also small (13k and 9k respectively). Both of the two factors will negatively impact the class extrapolation quality. However, we still observe from Fig. 6(a) convincing gains on novel classes for UCF101 and Caltech101. The reason is twofold: 1) the abundance of extrapolated class examples alleviates overfitting on small datasets by scaling up the negative data, 2) extrapolated data from known classes, despite being noisy, can still serve as ``semi-hard'' negatives to regularize decision boundaries. One natural direction for improvement is to enrich the data used for extrapolation, e.g. by combining a living memory of diverse web data with the few training classes on downstream datasets.
**Limited novelty and performance gains of ALMT.**
In response, ALMT is not the main innovation of this paper, and we view ALMT as a surprisingly simple add-on to reduce overfitting. Note behind the adaptive window size in ALMT lies a well-grounded rationale, i.e. to reduce both overfitting and underfitting during model distillation (see texts before Eq. 7). Such MT adaptation does not require extensive hyperparameter tuning to gain improvements, which is non-trivial.
Performance-wise, Tables 3&6 verify that, ALMT in its simple form, can indeed improve both base (ID) and new (OOD) class performance. Notably, the improvement on base class is larger ($ \approx$ 1\%), which is particularly useful in our case since our OOD feature generator may slightly degrade the base class performance when improving OOD generalization. Overall, ALMT serves as an optional booster that helps to strike a good balance between the base and new class performance. We will add such discussions in the revised paper, with ALMT's role to be toned down accordingly.