Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning

Summary

This paper questions a common assumption on data augmentations in contrastive unsupervised representation learning, specifically label-preserving, where data augmentations do not change the label of the original sample. Indeed, this assumption does not always holds in data augmentations used in contrastive unsupervised representation learning in practice as explained in Sec. 2. The authors claim that such a label-destroying data augmentation can be seen as feature-dropout, and this would be useful for universal representation learning, where the learnt feature extractor can give good discriminative representations for multiple downstream tasks. The authors perform such label-destroying data augmentations are useful on synthetic datasets and Viewmaker networks, which are previously proposed generative models to improve downstream tasks' performance of contrastive unsupervised representation learning. To give a theoretical understanding of this label-destroying data augmentation, the authors provide a theoretical justification of the usefulness of noise under a linear contrastive model with Gaussian-based data distribution in Sec. 5.

Strengths

Originality: Questioning the commonly used assumption on data augmentation sounds novel problem. Quality: The empirical and theoretical results give a better view of the problem (commonly used assumptions should be revisited) Clarity: The paper is well-written and clearly explains the motivation of the problem with concrete examples. Significance: Unsupervised representation learning for many downstream tasks is a desired algorithm in practice; this result gives another research direction to create a better "view" for such representation learning.

Weaknesses

## W.1 Details of experiments and results are missing - I could not find how to perform the transfer accuracy reported in Tables 1 and 2. The details should be given in the manuscript or Appendix. - The audio results in Sec. 3 are in neither Figure 4 nor others. ## W.2 Assumptions on the theoretical analysis The empirical and theoretical results have a gap in data, model, and data augmentation. For example, the analysed linear model has a sparse connection and is independent of the other weights as in Fig. 3(b). Still, empirically used models are non-linear and the weights are non-independent via a fully connected or pooling layer.

Questions

Q1: Could you give the details of experiments as pointed out by W1? Q2: L84: Is there any concrete example paper using "brightness shifts repeatedly"? In my understanding, SimCLR (Appendix A. in Chen et al. [2020]) applies it to a sample at most once. Q3: Clarification on data augmentation: L129 says > data augmentations for the image datasets [9] Could you clarify which data augmentation was used in SimCLR v1 paper? SimCLR paper used slightly different data augmentation depending on ImageNet or CIFAR-10. Concretely, B.9. in the SimCLR paper for CIFAR-10 says > "For data augmentation ... leaving out Gaussian blur" In my understanding, the implementation of this paper uses the ImageNet version: not excluding Gaussian blur according to `load_default_transforms` and `load_default_unnorm_transforms` in `feature-dropout-code/src/datasets/datasets.py`. Q4: Can Figures 4, 6, 8, and 9 be in the main body? Their explanations are in the main body to discuss the proposed analyses but I needed to see them in the Appendix. ### Typos and notation errors: - L83: `\citep` for `Chen et al. [9] and He et al. [23]`? - Fig. 4: Fig 4(b)'s caption is "Letters", but a label in the legend is "Digits accuracy". Similar typos are in Fig. 4(c): caption is "Digits", but the label is "Letters accuracy". - L261: aren't -> are not - L265: didn't -> did not

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

As mentioned in Weaknesses, the analysed linear model is a more restricted model class than widely used linear models in contrastive learning paper.

Summary

**Summary**. The authors question the common assumption in contrastive learning that good augmentations should be label-preserving, and suggest that dropping features out (for example, with data augmentation) actually is helpful in practice because it helps learning of "weaker" features: features that may be helpful on downstream tasks, but are not necessary for the contrastive task. To do this, they create synthetic datasets, use view-maker augmentations, and consider a linear toy example. **My position.** I think this work is interesting: I find it interesting to read, but there are some limitations in the work that make me question it's central thesis. Overall, I marginally would like to see it at NeurIPS because I found it interesting and think it would lead to an interesting discussion, but some technical concerns limit my enthusiasm about the paper.

Strengths

1. I enjoyed reading and thinking about this work, I think the question is interesting, and in general, I like the approach of questioning common assumptions in the field. The choice of data augmentation is clearly very important for contrastive learning, so that's great. 2. The paper is reasonably well written.

Weaknesses

# Major Weaknesses / Concerns 1. The paper emphasises "multiple downstream tasks", but what this means precisely is unclear, and **I am not sure whether the experimental setup reflects the problems we actually care about.** In the experiments, the features for the added digit/shape, etc are basically completely independent of the rest of the image features. I'm not sure whether this holds in real life settings of multiple downstream tasks e.g., if classifying both digit rotation and digit identity, the features of the "object" are shared; does this count as multiple downstream tasks here? The paper emphasises "the need for analyzing the interaction between multiple downstream tasks" (L19), but I'm not sure, precisely, what this might mean. 1. Instead of using synthetic experiments where another feature like a shape is overlaid on top of the base image, it would be better to use for example, a latent variable generative model, and downstream classification where different sets of features only are important for the class label. I think this would be closer to real world scenarios. 2. The paper quotes Tian et al. "a good set of views are those that share the minimal information necessary to perform well at the downstream task". I think this is a slight strawman (i.e., simplification of the prevailing view), because it is well known that "heavy" data augmentation is necessary for good performance (e.g., experiments from the original SimCLR paper). I suggest making the nuance more clear, and make clear the aim of the work is to help clarify this confusion. 3. There are no error bars on the experiment results, though multiple seeds are run. 4. I'm not convinced that the toy example with a linear model captures the important aspects of the problem. I don't think that learning the last linear layer after a feature is what is challenging about feature learning, but rather, it is learning the complex, non-linear representation. I am more interested in how dropping out features would affect learning of the representation, rather than learning of the feature layer. (See also Suggestion 2). 5. I am concerned about sensitivity to the viewmaker pertubation budget, $\epsilon$. As far as I can tell, the viewmaker objective basically always tries to make the contrastive task as hard as possible, which is to remove as much information about the base image identity as possible (i.e., remove information about the shape as well as the base image), because the contrastive task is to identify the base image identity from augmented pairs of images. If the budget is too large, all the images would become noise. The objective directly incentivises feature dropout! Therefore, the choice of $\epsilon$ is really important. I want to understand how it is that you chose this, and how sensitive your results and findings are to it. 6. I don't think the expert augmentations are really "expert" augmentations in the sense that they don't preserve label information for the synthetic datasets. Some of the augmentations will destroy information e.g., by flipping a 9, it no longer becomes a digit. I would like to see augmentations where you only flip e.g., the digits that are invariant to flipping. Then, the comparison between viewmaker and expert is more fair. 7. The results in Table 1 I find hard to interpret. Is the claim that the view-maker does feature dropout, and it performs better, suggesting that feature dropout is good? Because the difference in performance for shape are small, the performance for both the object and image class is poor for digit, and for letter, the expert augmentations outperform viewmaker. Also, should it not be possible that we can add the synthetic data without losing performance on the CIFAR-10 task? 8. "Two clear modes appear for viewmaker, but not for expert augmentations" (L181/182). Some of the expert augmentations seem to have very small probability of the correct label, suggesting that they perhaps always dropout the label? I was surprised by this. (This is ~Fig. 6 in the appendix). Can you explain this please 9. It seems many of the experimental results assume that viewmaker is doing feature dropout, and the expert augmentations are not i.e., the expert augmentations are good. I'm not sure this is actually the case. 10. WRT the theoretical analysis, I have some concerns. (i) the setup is not like contrastive learning. Here, we have $K$ features in a higher dimension space, but really the feature space is low dimensional. But contrastive representations live on a hypersphere, with the contrastive loss encouraging points to spread out across the hypersphere. (ii) "A good representation will capture ground true features ... such that $\theta_k$ is aligned with $\mu_k$ ..." I am confused by this; isn't there permutational and rotational invariance here? i.e., $\theta_1$ could be aligned with $\mu_5$. This, as a result, undermines your theoretical results. 11. In figure 4 in the appendix, the synthetic feature dropping out only improves the accuracy of the main image class quite a small amount, and dropping the feature out also drastically reduces the accuracy of the synthetic task. What one should do in this scenario is quite unclear. Perhaps we would do better by just constructing new, strong data augmentation that preserves label information. That seems to be the fairest comparison. ## Other suggestions 1. L22: pitching the paper as foundation models seems like overselling because the work focuses on contrastive learning. 2. If the hypothesis is feature dropout, one could also try dropping out features that are learnt during contrastive pre-training e.g., adding dropout regularisation before or after the projection head. This would (approximately) correspond directly to feature dropout, and would be somewhat similar to the linear contrastive example you use. Do you think this would work? Have you considered this? 3. Figure 4 and Figure 6 should not be in the appendix if they are mentioned in the main paper as prominently as they are. 4. I suggest streamlining Figure 1 e.g., including only one row, making it more visually clear which image is the base image and which images are augmentations. 5. I would like to see the accuracy of the contrastive task (i.e., how accurate is the linear layer when the class templates are the mean representations of augmente images) for the different viewmaker augmentations.

Questions

1. Do you think feature dropout changes the optimal solution, or just makes the optimisation process easier? 2. L84: "repeated application of some augmentations causes challenges for all downstream tasks". Are people using repeated augmentation in practice? I'm not sure this is a real problem. 3. Are the expert views really expert for CelebA? 4. For Fig. 4, how are the features dropped out? They just aren't added to the image?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

I would have liked to see further discussion of the limitations of the work that I mentioned above.

Summary

The authors of this work challenge the belief that label-preserving augmentations are necessary for contrastive learning by demonstrating that label-destroying augmentations can be useful for learning representations useful for a variety of downstream tasks. The authors treat augmentations as feature dropout, and demonstrate how straightforward contrastive learning techniques struggle to learn multiple features that are present in a visual scene. This was demonstrated through synthetically created datasets. Furthermore, the authors discover that feature dropout helps learn representations that encode multiple competing features - the authors believe that this is what explains why Viewmaker networks can help learn a wide range of features well.

Strengths

The strengths of this work lies in its thorough experimentation and its theoretical justification regarding the benefits of learning representations of multiple features when noise is added to corrupt a particular feature. The work was clearly and well-presented, and convincing arguments were made when challenging previously-held beliefs regarding augmentations for contrastive learning. Furthermore, linking the idea of stochastic feature dropout with the success of Viewmaker networks is an interesting discovery.

Weaknesses

In the introduction, the authors suspect that augmentations serve as a form of feature dropout that prevents any one feature from becoming a shortcut feature and suppressing the learning of other features. This idea has been noted before in prior work, as also mentioned in the Related Work section (in “Feature Suppression”). In the original SimCLR paper [Ting et. al, 2020], it was discovered that certain combinations of augmentations improved the representations learned. In particular, color jittering was found to consistently improve performance; it has been speculated that the addition of color jittering as a data augmentation helps to destroy the color histogram values for pixels from the same patch, which can easily shortcut the recognition of two positives. Therefore, it is already generally accepted that augmentations in contrastive losses serve the effective purpose of feature dropout. On the other hand, the authors utilize a set of synthetic datasets to determine if certain features suppress the learning of other features; this line of work was also previously done in prior work [Ting, et. al, 2022]. It must be noted that the original authors of [Ting et. al, 2022] did not attempt stochastic dropout of the secondary features. The paper can be made better with improvements to the formatting - for example, the entirety of page 5 is devoted to a singular figure. I believe most of the entire work is predicated on the feature suppression effect, which go through great lengths to show even though it is a well-known property of contrastive losses. I agree that the authors explore this idea thoroughly in their work, extending it to multiple modalities, adding theoretical justification, and explaining the potential reason VIewmaker networks work well, but fundamentally I believe the work lacks substantial core novelty beyond what was already established in prior work. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020. Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.

Questions

It is encouraging that Viewmaker networks seem to overcome it due to feature-level dropout. Then, do the authors believe that utilizing Viewmaker networks with contrastive losses are the best way to learn multi-feature aware representations?
 What are some ways this idea can be demonstrated for more natural-looking images? Currently, the synthetic datasets created indeed highlight the feature suppression affect inherent in contrastive learning techniques. However, are there any practical cases of natural images where we would like multiple features to be learned from a given image that could showcase the approach outlined in the paper?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

No negative societal impact.

Response to the authors rebuttal

Thank you for your response, especially for providing additional experimental results for Q3, which is really helpful. The first weakness would be addressed in a self-contained way: explaining the details of the experiment and adding the audio's result to Fig. 4. I'd be happy to raise my rating to 6 from 5.