Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities

Summary

This paper shows that CLIP-style pairwise contrastive objectives on multiple modalities (more than two) fail to capture the total dependencies of different modalities, and proposes to use total correlation, a higher-order generalization of mutual information as the new objective to optimize multimodal contrastive learning. The authors provide derivations of the lower bound on total correlation, the optimal scoring function, the practical objective, and proof of statistical sufficiency. On a simulated dataset, a newly created multilingual dataset, and a Chest X-ray dataset, the authors illustrate the limitation of existing CLIP-style pairwise objectives and the effectiveness of the proposed approach.

Strengths

Clarity: the paper is very well written. The motivation, problem formulation, derivation, and novelty of the paper are clearly stated. Figures 1 and 2 are helpful for the reader to understand the concepts. Quality: the theoretical derivations seem valid to the reviewer upon checking. The theoretical results of the lower bound, optimal scoring function, and statistical sufficiency make this work concrete. The derivations in Appendix A seem correct, and the reviewer did not check the derivations in other Appendix sections. This paper also introduces a new multilingual dataset, Symile-M3, including 33 million (audio, image, text) samples. Significance: the paper points out that pair-wise CLIP losses cannot capture conditional dependencies between multiple variables, and provides a simple and effective solution to address this. The problem of leveraging contrastive learning on more than two modalities is worth studying.

Weaknesses

Originality: the problem of capturing dependencies of multiple variables beyond pairwise mutual information is well-studied. There are many solutions proposed already, albeit not in an explicitly multimodal setup, as discussed by the authors in Lines 226-231. However, the reviewer reckons that the methods such as Bai et al. can be extended to the setting involving three or more modalities, e.g., substituting the $x^1, ..., x^m$ by different modalities in the Sample-based TC estimator from Bai et al. The authors did not include any such baseline results on the datasets. The reviewer would appreciate clarifications on this. Quality: the authors did not justify clearly why they curated a new dataset (and a multilingual one) instead of using existing multimodal datasets, e.g., VQA, GQA, How2, HowTo100M, etc. Lacking comparisons on standard datasets weakens the submission.

Questions

There are no error bars in Figures 5 and 6, will the authors intend to provide error bars for them? There is a quite related paper on capturing total information and conditional independence in a multimodal setup. The reviewer would appreciate some comparisons with it: Liang, Paul Pu, et al. "Factorized contrastive learning: Going beyond multi-view redundancy." NeurIPS 2023.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors include limitations of the work.

Summary

The paper introduces Symile, a new contrastive learning objective designed to accommodate any number of modalities, addressing the limitations of pairwise CLIP which fails to capture joint and conditional information between modalities. Symile targets total correlation, capturing higher-order dependencies among multiple variables. The paper derives a lower bound on total correlation using a generalized form of inner products and demonstrates that Symile representations form sufficient statistics for the remaining modalities. Experiments on a large multilingual dataset and clinical data show that Symile outperforms pairwise CLIP in cross-modal classification and retrieval tasks, even with missing modalities.

Strengths

1. The paper proposes a model-agnostic contrastive learning objective that captures higher-order dependencies among any number of modalities, addressing limitations of existing pairwise CLIP methods. It effectively identifies and addresses a significant limitation of CLIP, presenting a novel and important contribution to the field. 2. The paper starts with a simple one-dimensional problem to illustrate the failure mode of pairwise CLIP, which aids in understanding and intuition. 3. Theoretical analysis is provided to further support that Symile learns sufficient statistics.

Weaknesses

I wonder if more explanation could be provided on the rationale of the MIP. For example, if we have 3 vectors and consider one coordinate, imagine two cases: (1) the values are x= -1, y= -1, z= 1, whose product is 1, and (2) the values are x= -1, y= -1, z= -1, whose product is -1. In this example, case (2) apparently has higher similarity between the 3 values than case (1), but case (1) ends up with a larger MIP value, which seems undesirable to me. For the experiments, since I am not very familiar with the state of the art in learning with three or more modalities, I'll leave it to other reviewers to judge whether the benchmarks used and the comparisons are sufficient.

Questions

See the question in Weaknesses.

Rating

7

Confidence

2

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed certain limitations in section 6.

Summary

This paper proposes Symile, a new contrastive learning objective that addresses the challenge of contrasting multiple modalities simultaneously by considering total correlation. Symile captures the statistical dependence between multiple variables and uses a generalized inner product to derive a lower bound on total correlation. The representations learned by Symile form sufficient statistics for other modalities, outperforming pairwise CLIP in cross-modal classification and retrieval. This is demonstrated through experiments on a multilingual dataset with 33 million samples and a clinical dataset. === After response == I initially assigned a score of 4, but after reviewing the manuscript further and clarifying some misunderstood details addressed in the rebuttal, I decided to adjust my score. Following a discussion of other reviews, I increased my score to 6 (weak accept).

Strengths

Introducing the limitation of pairwise summation of contrastive losses for multiple modalities is interesting, as demonstrated by both a synthetic example and theoretical justification. Addressing this limitation makes the work more principled. Additionally, handling missing data is crucial for practical scenarios.

Weaknesses

Limited experiments: - The paper only compares the proposed method with CLIP. More recent works, especially those focused on representation learning with multiple modalities, should be included for a comprehensive comparison. ImageBind, with its publicly available weights, would be considered as a baseline. - The formulation of the Symile model requires complete triplets (e.g., image, text, audio) for training, which makes it difficult to scale up the training data. The current approach does not accommodate missing values in the triplet elements, which is a significant limitation for practical applications. An extension to handle missing values is needed.

Questions

#1. It's a great idea to include a simple and clear example to show why summing pairwise contrastive losses might not be best for multiple modalities. However, the failure analysis seems very theoretical. In real-world datasets, the assumption that random variables are jointly dependent but pairwise independent might not be true. Can you provide more realistic examples where real-world datasets fit this assumption? #2. What is the reason for using the coordinate-wise sum of the element-wise product of a set of vectors instead of the simple dot product for the scoring function? #3. The only baseline used to compare the proposed method is CLIP. More recent works, especially those focused on representation learning with multiple modalities, should also be included. In my opinion, ImageBind would be selected as its weights are publicly available. #4. The negative sampling with O(N) compared to O(N^2) seems to be a very crude approximation. Does it affect the performance? From a theoretical perspective, can we analyze the difference between the linear and quadratic sampling schemes in terms of the total correlation lower bound? #5. Minor comments What do the color shades in Figure 1 represent? In line 138, the element of Z_n​ should be written in lowercase as z_n. In related works, the authors mention that contrastive learning has been popularized since the release of CLIP. However, in my opinion, the credit for popularizing contrastive learning should go to SimCLR.

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have mentioned the limitations, aside from the lack of comparisons. Additionally, in my opinion, there is no potential negative societal impact of this work.