Dear reviewer XjUF,
Thanks for your responses and we would like to make some further clarifications:
> Is "Frozen CLIP head" taken from pretrained model with large-sacle data the dominant factor?
Concerning whether conclusions of Fig. 4b are dominated by “frozen CLIP head”:
1. It is important to emphasize that directly comparing the correlation values of models trained from scratch with those of a frozen pre-trained CLIP head is not fair, nor do we intend to make such a comparison in the figure. This is because the CLIP head is trained on 400M data points, while the other methods are trained on at most 3M data points. Due to limited computational resources, we could only scale our data to the million level for these studies. However, in Fig. 4b, we observe a clear trend: as the data scale increases, the correlation consistently decreases, indicating that data scale is indeed a significant factor in robustness.
2. Furthermore, in Fig. 1, where we scale the data from 3M to 2.5B, it becomes even more apparent that increasing the data amount leads to lower correlation values. It's important to note that *all models in Fig. 1 are trained from scratch*, meaning that the pre-trained head is not a contributing factor to the results in Fig. 1. Interestingly, the low correlation observed when using a frozen CLIP pre-trained model actually supports our hypothesis that robustness improves significantly with larger data scales. This implies that with substantial amounts of data, the trend toward increased robustness becomes more pronounced.
3. The reason we incorporated frozen CLIP head into Fig. 4 is to illustrate that the feature space learned by CLIP from large-scale data can serve as a teacher to finetune models with a smaller scale of training data while achieving robustness *for free*. Still, even in this setting, it can be observed that robustness can be enhanced with more training data.
Concerning whether conclusions of Fig. 5b are dominated by “frozen CLIP head”:
4. In Fig. 5b, we use "frozen CLIP head" to 1) estimate the behaviors of a large-scale trained CLIP when only small-scale controlled data is available, and 2) verify how effective the data factors are given knowledge from large-scale data (exactly your major concern). The results from the "frozen CLIP head" and the "from scratch" setting should not be compared directly. Instead, they should be interpreted separately, as both approaches can independently verify the effect of data imbalance, diversity, and distribution shift. The results of Fig 5b show that tendencies of "from scratch" and "frozen CLIP head" are coherent, indicating these data factors are consistently effective, regardless of whether knowledge from large-scale data is available. Fig. 3 under this setting is also discussed in Fig. 17.
We apologize for the confusion and will revise the figure to clarify the presentation. Additionally, we will separate the study into individual figures to enhance clarity.
> Is “no distribution shift” equal to “=freq”?
Sorry for the confusion, the revised figure in our PDF response might be clearer than the original one. Results between LAIONet subsets have no distribution shift, and only difference in intra-class image diversity (if using different thresholds) or level of data imbalance (if using same threshold, different class distribution). "=freq" refers to the LAIONet variant that has the same class distribution with IN-Caps, it should be compared with IN-Caps and the only difference between them is distribution shift.
> For heavy-tailed classes in both 1 and 0-shot, is relying on large-scale pretrained CLIP head our main contribution?
No, using pre-trained "frozen CLIP head" is already a common practice in open-world recognition (more similar to the setting in Sec. 4.1) and long-tailed classification, and not our novel contribution. But note that, none of them considered introducing the training mechanisms of CLIP (vocabulary subsampling) to the downstream task. Our contribution is to show adding vocabulary subsampling is important to acquire CLIP knowledge when its head is reused.
> The authors' responses do not include the results of correlation
Please note that Sec. 4.1 considers a very different setting than Sec. 3, aiming to let models better acquire CLIP knowledge when pre-trained CLIP head is used, and not to study factors behind CLIP's robustness. In this setting, the expected property is higher tail/novel-class accuracy. Correlation values indeed can be computed, but note that many classes have the same 1/0 frequency, and the correlation is less indicative in this case.