Further responses to your new concerns
Thanks for your reply. We are glad that our responses partially solved your concerns.
For your concern, the enlargment of the gap and the generalization performance are still open problems in the literatures. Currently, reseachers, including us, only empirically observe such phenomena, and find that generalization performance (zero-shot, few-shot etc.) is associated with the gap. All of us try to figure out and explain these problems from different perspectives.
In the following, we will try to address your new concerns, based on the insightful discussions with Reviewer Ms7f.
> Gap enlargement & generalization performance
For your concern about why enlarging the gap improves the generalization performance, __inspired by the [comments from Reviewer Ms7f](https://openreview.net/forum?id=JWLiK3kKWQ¬eId=HRvuDHQQfk#:~:text=Based%20on%20the,different%20transformation%20heads)__, we find that overfitting and regularization may be a good perspective to explain. Specifically, __enlarging the gap reaches a balance between learning discriminative representations and achieveing better generalization performance__.
According to Fig. 3(a), it is easy to observe that the enlargement of the gap is constrained in a small range. Two cases are worth discussion here:
- If the gap is narrowed, the similarity between prototypes and instances increases. However, the validation loss also increases. __Such a phenomenon indicates that narrowing the gap may result in overfitting__. As a demonstration, we can also observe the overfitting phenomenon happens in some cases from Fig. 27 (f), (g), and (j) (__the orange line__).
- If the gap is infinitely enlarged, the validation loss will also increase since the similarity between the prototypes and instances decreases.
Thus, from this perspective, __the enlargement of the gap can be seen as a regularization that drives the model to explore a balance between learning better representations and achieving better generalization performance__.
> Why NCC + 2 heads does not work well and how symmetric cross-entropy loss address this issue.
Based on the above findings, we find that we can give a more reasonable explaination to this issue as well. For your concern about why NCC loss along with 2 different transformation heads does not work well, we further conduct an experiment here to see if 2Heads+NCC has an overfitting issue. For simplicity, we only applied two different transformation heads on URL method with the random seed 42 and ImageNet dataset (due to the time constraint, but we will include more experiments for this in the updated version of our paper).
|Methods|Train Acc|Test Acc|
|---|---|---|
|__URL+2heads+NCC__|94.93|56.67|
|__CoPA__|90.87|57.92|
According to the results, in the "URL+2heads+NCC" case, we observe __higher training accuracy and lower test accuracy__ compared to CoPA. __This is a phenomenon of overfitting__ (compared to CoPA). However, applying the symmetric cross-entropy loss alleviates such a phenomenon to some extent (see the training and test accuracy of CoPA).
Thus, based on the explanation above (thank Reviewer Ms7f for this valuable insight!), __the symmetric cross-entropy loss itself might implicitly regularize the model during the learning process and achieve a balance between the discriminability and the transferability of the learned representations__.
In summary, we think that these explanations are interesting, and we will merge these discussions in our updated version.