CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss

Summary

The paper proposes a novel continuously weighted contrastive loss function for multi-modal models. The proposed method takes consideration of 1) similarity of samples in the same batch 2) the non-binary nature of similarity between input samples, and therefore utilizes a continuous weight matrix in learning alignment between modalities. In general I think that the ideas on this paper are novel. The explanation of the methods is explicit and concrete. I think it is enough to be reproducible. However the paper may benefit from some revision regard to how the work is presented and focus of experiments.

Strengths

* The proposed idea is novel in resolving two crucial problems in standard contrastive loss. * The authors considered modalities of both text-image and speech-image and presented some supportive zero-shot evaluation results.

Weaknesses

* The experimental results are not complete, it would be good to see a more complete set of results on how this method would perform on the downstream retrieval tasks (T4 compares between CL and CWCL, without indicating performance compared to baseline models such as LiT or CLIP). * The experiments regarding the transfer from text to speech are nice however the results in Section 4.2.1 is not clearly presented and is a bit hard to follow. * Figure1 is not really informative and maybe can be combined with Figure 4 so that more interesting and qualitative results can be presented.

Questions

* Is it possible for you to provide results for retrieval compared to the baseline models? This would address a major weakness in the evaluation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

* There is no explicit limitations section in the paper.

Summary

The authors propose a composite loss in contrastive learning in order to preserve structure between two embedding spaces for different modalities. In the setting in which they study, two unimodal encoders are aligned by freezing one of them and computing a "continuous" contrastive weight between samples $(i, j)$ as $0.5 + \frac{q_i^T q_j}{2}$. With these weights an additional loss for structure preservation is used in addition to the standard self-supervised loss for the unfrozen encoder. The proposed approach outperforms standard approaches such as LiT, SimCLR on 0 shot classification tasks.

Strengths

- The method is simple and efficient requiring a small $\mathcal(O)(k^2)$ computation for pairwise weights per batch - Performance improvements are strong across the experimentation

Weaknesses

- There is a lot of setup while the entirety of novel contributions seem to be buried in section 2.3. - This paper makes more sense for a workshop, the contributions are limited and is a small modification of previous works such as (https://arxiv.org/pdf/1905.12837.pdf, https://arxiv.org/pdf/1811.01459.pdf, https://arxiv.org/pdf/2103.14003.pdf)

Questions

- Which continuous pairwise weights did the authors try? Given this is the main contribution it would be helpful to understand performance as this weighting changes in the 0.5 bias used or for nonlinear weighting wrt to the inner products.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

In this work, the authors consider the problem of cross-modal zero-shot transfer and propose a new loss function called continuously weighted contrastive learning (CWCL) that extracts better supervision from pretrained models in a single modality and leads to better alignment between two modalities. They run experiments on two modality pairs, image-text and speech-text, and CWCL yields significant improvements compared to standard contrastive learning (with up to 20-30% absolute improvement on a speech-to-intent classification task).

Strengths

- The authors present a simple and effective contrastive learning variant that makes effective use of pretrained models in individual modalities. - They achieve a significant performance boost in zero-shot image classification when compared to existing contrastive learning objectives. - They also demonstrate superior results on a zero-shot speech-to-intent classification task.

Weaknesses

The main contribution of this work (CWCL) can be grounded better in the context of existing work. This work highlights an important shortcoming in existing formulations of contrastive learning, namely that similarity has a binary interpretation. All positive examples and all negative examples are treated equally. This has been identified in prior work, albeit for a single modality, which the authors fail to cite. E.g., "Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification", V. Suresh and D. Ong, EMNLP 2021. There are also works on multimodal learning that highlight the importance of using a weighted contrastive loss to downweight faulty positives/negatives such as "Robust Audio-Visual Instance Discrimination", Morgado et al., CVPR 2021.

Questions

- Did the authors experiment with other weighting functions for w_ij's in Equation 3? Some insights on how w_ij = q_i^T q_j/2 + 0.5 was arrived at would be useful for the reader. - As in Figure 5, does CWCL exhibit robustness to templates for the speech-to-intent classification task as well? - From Table 1, CWCL outperforms the supervised Resnet50 baseline on ImageNet-V2 as well as ImageNet-R, ImageNet-A and ObjNet by substantially larger margins. Is Resnet50 the appropriate supervised baseline to show for these datasets? - From Table 2: - RoBERTa+S performs consistently better than BART+Y on SLURP and STOP, but the trend flips for KWS. Can the authors comment on why that is? - CWCL with RoBERTa+S appears to outperform the text-intent upper bound for STOP (87.87 vs. 84.78). Please explain this result. - Just a comment: If the authors are looking to make some space in the draft, Figure 1 can be omitted without loss of clarity.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations have not been explicitly listed.

Summary

They propose a simple but effective method to align the representation space of two self-supervised models using pairs of examples from two modalities. They propose a CWCL loss where they reweight the contrastive loss of example pairs based on the similarity measured in one modality (equation 2). Specifically, let $(v_1, u_1)$ and $(v_2, u_2)$ be two example pairs, the CWCL loss encourage the representation of $v_1$ to be close to $v_2$ if $u_1$ is close to $u_2$. They experiment with a task requiring text-image and a task requiring text-audio alignment in zero-shot settings. The results show that adding CWCL loss to the original contrastive loss outperforms previous methods greatly.

Strengths

1. The proposed approach is very simple, but the results show that it brings significant improvements. Especially for the image classification task, their model outperforms the supervised-trained model in the zero-shot setting. 2. They also show that, compared to a previous method, LIT, their model is more robust to the choice of prompt templates used to generate the text representation for the zero-shot image-classification experiment. 3. The experiment of audio-text alignment also shows that adding the CWCL loss improves the performance.

Weaknesses

1. The ablation study comparing pure contrastive loss vs contrastive loss plus CWCL is only conducted for the audio-text alignment task. 2. In section 4.2.1, the author selected the text model very carefully. It thus seems that this method is very sensitive to the choice of the text model. 3. It may provide more insight if there are some qualitative analyses between the model trained without CWCL and the model trained with CWCL.

Questions

1. The performance of LiT reported in Table 1 is much lower than the performance reported in the LiT paper. Could you explain what might be the reason for this discrepancy? 2. For the zero-shot keyword spotting task, because this task seems to be very much at the lexical level instead of the semantic level, I wonder what would be the benefit of using a contextualized pretrained language model? 3. Instead of having CWCL loss in addition to the original contrastive loss, what about having CWCL loss only but you set the weight of negative pairs to be 0? There seem to be some subtle design choices made for CWCL. I would like to know more.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

I didn't find the authors address the limitations of this work.

Response to rebuttal

Thanks to the authors for their detailed response and clarifications to my specific questions. I'm raising my score from 5 to 6.