Concerns about Different Architectures and Pure Text Embedding Models
We appreciate your further feedback. We’d like to address your further questions point-by-point.
**On the architecture.**
(i) We agree with you that CNN and Transformers are different in visualizing attention maps, but they are similar in incorporating with LICO due to it only depends on the latent representations, i.e., the representations before the final classification head.
(ii) LICO does not affect the calculation of attention maps (the last self-attention) in ViTs. For image encoder of ViTs, LICO can also guide the representations of class tokens through proposed $L_{\text{OT}}$ and $L_{\text{MF}}$.
(iii) In this paper, we follow the previous studies of interpretation, which focused on interpretation methods based on simple CNN backbones. **LICO can effectively overcome their common difficulty of improving interpretability and classification performance simultaneously**. Hence, we will treat incorporating LICO with ViTs as our future work in that there needs more effort to investigate how to obtain more explainable self-attention of ViTs, suitable quantitative metrics for interpreting ViTs, whether there exists trade-off between interpretability and classification performance, etc.
(iv) Thanks for recommending the publication [1], which is a wonderful work that utilized partial distance correlation (DC) to measure similarity of different networks. The DC is helpful for generating improved attention maps via conditioning on another network, which benefits from its beautiful properties of end-to-end optimization and measuring feature spaces of different dimensions. This work inspires us to consider the relationships among features of different models and to further facilitate interpretation studies.
- [1] On the Versatile Uses of Partial Distance Correlation in Deep Learning, ECCV 2022.
**On pure text embedding model.**
In addition to Word2Vec (W2V), we further applied pre-trianed BERT to testify the effectiveness of LICO: (i) The pure language BERT [1], (ii) The text encoder of vision-language BERT, i.e., BERT-ALIGN, ALIGN [2] is also a framework that aligns image and language features in latent space, which differs from CLIP in training with a noisy image-text dataset that is larger than that in CLIP. For both BERT [1] and BERT-ALIGN [2], we take learnable class-specific prompts as the inputs of text encoders.
Based on the Table R1, we further provide the Table R3 as follows. We can see that BERT [1] surpassed W2V and W2V-P due to the generalizability of stronger pre-trained model. However, BERT still cannot achieve better performances than CLIP. Furthermore, BERT-ALIGN performs competitive and even better than CLIP, which attributes to its larger training set with more noisy image-text pairs. Consequently, from the results in Table R3 and this paper, we conclude that LICO works better with those vision-language pre-trained text encoders. The pure text encoders like W2V, even the pre-trained BERT with frozen parameters, are inferior in image-text alignment in LICO due the pre-trained parameters are not sensitive to visual features. This deficiency may be addressed by utilizing some transfer learning and domain adaptation tricks.
**Table R3**: Comparison of different text encoders on CIFAR-10
| Encoder | CLIP | W2V | W2V-P | BERT[1] | BERT-ALIGN[2] | None |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Full | $\textbf{95.8}$ | 95.6 | 94.9 | 95.7 | $\textbf{95.8}$ | 95.6 |
| 4000 | 81.5 | 81.0 | 80.2 | 81.3 | $\textbf{81.7}$ | 80.9 |
- [1] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, ACL 2019.
- [2] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, ICML 2021.
Based on your valuable comments and suggestions, in our future work, we will comprehensively discuss different architectures and pre-trained models and try to unify the interpretation of CNNs and ViTs.