Identifiable Contrastive Learning with Automatic Feature Importance Discovery

Summary

In this paper, the authors proposed a new contrastive learning scheme (tri-contrastive learning) with a learnable diagonal matrix, which can capture the feature importance. To show the significance of the proposed method, the authors conducted both theoretical and experimental analysis, where some interesting findings are observed. However, it is not clear how the proposed method advances existing contrastive learning-based methods, and experimental results are not very convincing. The detailed comments are below.

Strengths

The idea is simple and interesting, with reasonable theoretical analysis based on eigenspace.

Weaknesses

1. The idea of analyzing feature importance for contrastive learning is not new [a]. While the proposed method is different from [a], the submission lacks analysis how the proposed method outperforms [a], especially based on the generalization capability the submission focused on. [a] Accurate and Robust Feature Importance Estimation under Distribution Shifts, aaai21 2. The visualization example in fig 1 is not clear. Noted that the contrastive learning is performing on the feature space, it is not clear to me based on the paper how the feature space linked to the image space. 3. The generalization bound is not very clear, the upper bound depends on the term \alpha, which relies on data augmentation. How different augmentations inclunce the term \alpha. In what condition that the sample and its augmented view could have different labels? 4. The improvement on three benchmark datasets are very marginal (almost no improvement), and the results can’t justify the generalization capability claimed in this paper, convergence analysis is also not presented.

Questions

Please see above.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

n/a

Summary

The paper proposes a new contrastive triCL method, which extends the existing pair-wise contrastive framework. The core motivation is the inability of CL methods to deliver identifiable representations. To tackle this, the paper proposes factorizing the pair-wise similarity measure x^T@y into a bilinear x^T@S@y form, where learnable matrix S is a diagonal feature importance matrix; and theoretically proves the identifiability property. The paper also demonstrates possible automated feature importance discovery by analyzing the learned feature importance matrix.

Strengths

1. The paper addresses an important problem of identifiability, which is often overlooked in contrastive learning. 2. The proposed method is elegant in its simplicity. Also, due to its simplicity, it does not introduce any major computational overheads. 3. Feature importance discovery can be an important component to speed up the inference (by excluding features of low importance)

Weaknesses

1. In my opinion, the term tri-contrastive (triCL) does incorrectly reflect the essence of the method. The proposed method still is based on pair-wise contrast in a nutshell, although the contrast is in the factorized form. 2. It seems that BarlowTwins tackles almost the same problem as the paper states in L33-34. On the technical side, the L_dec term Eq (8) seems to be a special case of the Barlow twins' objective. In that sense, the L_dec term in triCL can be considered not as just a regularization, but as a main source of the supervisory signal. 3. The ablation is needed to verify the importance of the L_dec term in the triCL objective. I would be interested to see the results for L_dec, L_tri, and L_dec + L_tri objectives separately. 4. Experimental evaluation is limited to relatively small-scale datasets and only two baseline contrastive learning methods. UPD: The rebuttal addressed most of my concerns.

Questions

1. Why is the softmax in S? Wouldn't the softmax limit the flexibility of each feature dimension, when computing x^T@S@y similarity? Have authors tried other normalization approaches, e.g. sigmoid for each diagonal element of S? 2. Does identifiability of representation always boils down to reproducibility within the scope of this work?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I suggest adding a paragraph on limitations, which would highlight the scope of application of the proposed approach (or is the proposed method expected to work better in all circumstances?)

Summary

Traditional CL calculates the similarities between two samples while this paper proposes to insert a learnable diagonal matrix between two feature vectors multiplication and name it triCL. The authors claim and show the proposed method leads to the exact interpretability of learned features. Some theoretical analysis is provided.

Strengths

The paper is well written and the proposed idea is simple and may be useful for explaining the CL.

Weaknesses

1. Lack of novelty: The proposed method is essentially a bilinear function, and it can be rewritten the same as conventional CL, i.e., $f_1^TSf_2 = (\tilde{S}f_1)^T(\tilde{S}f_2)$ where $S=\tilde{S}^T\tilde{S}$ since S is a nonnegative diagonal matrix, i.e., a kernel matrix. From this point, the new parameter $S$ can be integrated into the neural network training, without explicit consideration. $S$ may be useful for the interpretation of CL, but at this point, I do not see how different using $S$ is from using traditional features with some statistical pro-processing such as norm. At least, I do not see the necessity of the proposed method in terms of interpretability. 2. Weak experiments: From Table 1, the improvement is marginal and even worse than the baseline. This is probably because introducing bilinear forms makes the training easier stuck in local minima. Also, these results verify my thoughts about bilinear form in learning above. I suggest the authors to investigate more in training, e.g., plotting training curves and doing some comparisons.

Questions

see my comments above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

no discussion on limitations is provided, and i suggest the authors to add this part

Summary

The paper proposes a new contrastive learning objective that learns an additional diagonal matrix which forces the features to exhibit different importance. Theoretical results are provided to show that the new algorithm achieves similar downstream linear probe accuracy but also better interpretability / identifiability. Empirically, the authors test the new objective on several standard benchmark datasets and show that learned features exhibit better k-nn accuracy when a subset of dimensions are used, and the downstream linear probe accuracy can also be improved in certain cases.

Strengths

- The motivation is clear and the paper is written well. - It's nice that the learned representations can have a clear subset of dimensions that captures most of the semantic information. This can be especially useful in many retrieval use cases where the dimensionality is the key bottleneck for scaling up the size of data. With this modified representation, people can easily select a subset of dimensions to construct less informative but more computational efficient representations for downstream use cases. - The proposed method is justified with corresponding theory.

Weaknesses

- Only small datasets are used. Could include larger ones like ImageNet. - Theorem 5.1 seems to be exactly following that in [HaoChen et al], so I'm not sure there's enough novelty in this specific theorem.

Questions

Is there a easy way to project the full feature from SCL or SimCLR to a 20-dimensional subspace and still match the same k-nn performance of the top 20 dimensions from tri-SCL? I wonder whether doing some PCA can achieve that.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Thanks for authors’ rebuttal

Q1. This addresses my concern. Q2. I am not so convincing about the answers. I think the authors should show something that supports the proposed method is necessary for learning feature importance in training together with contrastive learning. Maybe there is another experiment that is needed: Feed the features from a learned contrastive learning model to a feature importance learning algorithm as a post-processing step, and compare the results. I agree with Reviewer aafx that i do not see the evidence in the experiments to support the theoretical claim on generalization, and the rebuttal seems not to provide more evidence neither.

Further Discussion

Dear Reviewers, The open discussion phase of the paper is nearing its end, and the authors have provided more detailed elaboration, ablation studies, and explanations of the principles in the rebuttal phase, mainly in response to the proposed algorithms. In order to ensure the smooth running of the conference, we would like to receive the response of each reviewer to the authors' rebuttals as soon as possible. Therefore, we kindly ask you to submit your feedback as early as possible, if possible. Once again, we thank you for your time and look forward to your valuable comments. Best wishes, AC of Paper ID10613