Self-supervised Transformation Learning for Equivariant Representations

Summary

This paper is concerned with the learning of expressive representations in a self-supervised fashion. In particular, it aims at learning representations that

Strengths

- Presentation: The paper is clearly written, well-structured, easy to follow, and overall is in a good state - Proposed approach: the idea proposed by the authors is not ground-breaking but simple, makes sense, and is efficient. It applies the same reasoning used in invariant self-supervised learning to equivariant representation learning. - Evaluation pipeline: the authors evaluate their method quite thoroughly, evaluate semantic classification across a large number of datasets, evaluate object detection, and check the sensitivity to hyper-parameters and transformation prediction. - Experimental results: results show that the approach effectively learns representations that perform well on both semantic and non-semantic information in comparison to baselines. Sensitivity to hyperparameters seems limited.

Weaknesses

- My main concern regards the baselines chosen for evaluation. Authors explicitly mention that SIE and SEN are excluded from the evaluation due to the need for these methods to have access to transformation labels which STL does not require. While I see how these methods require more "supervision" than the proposed method, STL still requires weak supervision (i.e., access to pairs of images that have undergone the same transformation). Cases, where one would have access to such weak supervision but not access to transformation labels, are quite rare (e.g., AugMix as mentioned by authors), as a consequence I think the evaluation should include these baselines, and results should be analyzed considering this light difference in supervision level. My score will be increased if this concern is addressed. - Experimental results are computed for a single seed, for object detection results, the performance gap between EquiMod and STL is narrow hence additional seeds would make the evaluation more robust. - In Table 1, STL+AugMix can be compared to SimCLR *with* AugMix augmentations which I do not believe is reported in this table. Minor: - line 40: typo

Questions

No questions but a suggestion for improvement, I think section 4.3, first part, is intriguing and it is not clear to me why equivariance and transformation learning are _complementary_. Discussing further the intuition behind these results would I think improve the paper.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed and in particular, the main limitation I see to this work (i.e., the framework requires access to a pair of images that have undergone the same transformation, this weak form of supervision is not always accessible in the wild), is mentioned.

Summary

The authors propose Self-supervised Transformation Learning (STL) to learn equivariant representations. The core of the method is to not use augmentation information but instead use transformation representations obtained from pairs of data. In this sense, instead of knowing the transformation information, it is necessary to have pairs of data with the same transformation applied to them. This allows STL to leverage more complex augmentation schemes such as AugMix to reach higher performance on classification and detection benchmarks.

Strengths

The non reliance on knowledge of the group elements leads to a method with different assumptions than previous works which can thus be used in different scenarios. Assuming knowledge of pairs of data with the same transformation can be a weaker or stronger assumption than knowledge of the group element depending on the domain, which makes STL complementary to existing works. Assuming that images are sampled randomly, using a pair of images to compute the transformation representation of another one ensures that information does not leak between the semantics of the image and the transformation representation. At inference time, applying the transformation does not require knowledge of pairs. The performance gained using STL over existing methods is convincingly shown, across a wide range of tasks. We can however notice that aiming for equivariance over all augmentations does not lead to the best performance (Table 6),reinforcing previous findings that the benefits of invariance/equivariance can be task dependant

Weaknesses

1\) An important reference to [1] is missing from the paper. It was proposed in this paper to use pairs of data transformed by the same (but unknown) group element (see equation 4 for example) to learn equivariant representation. Although the considered experimental setting is different, and the way to leverage the pairs differs, it remains a very related method. 2\) With how the loss is designed, the representations are both aimed at being invariant and equivariant at the same time (albeit with a projector in between the loss and representations). As invariance is a perfectly valid solution when wanting equivariant solutions ($\theta = Id$), it is possible that the learned representations are invariant to augmentations in the end, as illustrated in figure S2 of [2] for example. Currently, the metric defined in equation 15 doesn't provide enough information as to whether or not the predictor indeed applied transformation well (and thus whether or not the representations are equivariant). Intuitively (correct me if my understanding is wrong), the proposed metric looks at how much closer the predicted representation is to the target compared to its starting point. A value higher than 1 thus means that the prediction is closer to its target than its starting point. But here the values are barely around 1.1 which would suggest that while the predictor is indeed doing something positive, it's far from applying the transformations perfectly. Completing this analysis with other common metrics such as the ones used in EquiMod[3] or metrics such as Hit at rank K, MRR, PRE[2,4,5] for example would provide more compelling evidence of the equivariance or not of the representations. 3\) Performance seems to not be reported on the pretraining dataset but only on other downstream tasks. It is important to report those numbers to understand the behaviour of the models both in and out of domain. [1] Shakerinava, Mehran, Arnab Kumar Mondal, and Siamak Ravanbakhsh. "Structuring representations using group invariants." Advances in Neural Information Processing Systems 35 (2022): 34162-34174. [2] Garrido, Quentin, Laurent Najman, and Yann Lecun. "Self-supervised learning of split invariant equivariant representations." arXiv preprint arXiv:2302.10283 (2023). [3]Devillers, Alexandre, and Mathieu Lefort. "Equimod: An equivariance module to improve self-supervised learning." arXiv preprint arXiv:2211.01244 (2022). [4]Kipf, Thomas, Elise Van der Pol, and Max Welling. "Contrastive learning of structured world models." arXiv preprint arXiv:1911.12247 (2019). [5]Park, Jung Yeon, et al. "Learning symmetric embeddings for equivariant world models." arXiv preprint arXiv:2204.11371 (2022).

Questions

Line 42-43 "with a transformation label, each transformation is treated independently, disregarding interdependency [...] each component in color jitter is treated distinctively although they are related to each other." I am not sure to see the point here, as this argument can be made for any image augmentation. Giving all transformation parameters (as well as their order if it isn't fixed) gives full information about the transformation. The transformations are fully independent of each other and at best they do not commute. Line 47-48 "After all, the reliance on transformation labels limits the performance gain in equivariant representation learning". This claim seems unsubstantiated, are there any references or concrete evidence to support this claim ? In Equation 2 the notation lacks preciseness. If $\mathcal{L}$ is an InfoNCE criterion, it cannot consider samples in isolation as is written in Equation 2, but instead needs knowledge of the full batch to be computed. When using Color Jitter for AugSelf and Equimod, is the order of the transformations fixed or randomly selected for each call, as is default in torchvision for example ? If it is not then the same labels can be associated with different transformations which will hinder their learning. When considering a large set of transformations, it is possible that the transformation representations only represent some of the transformations and not all of them, or that the predictor only uses partial information (e.g. applies hue change but not brightness change). Did you perform analyses on individual transformations to study the equivariance properties on individual transformations ?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors address the limitations in section 6, notably how using pairs of transformed data is not a panacea, and can have issues to be extended to even larger transformation sets than considered with STL.

Summary

The paper introduces STL, a method for learning self-supervised equivariant representations. It suggests replacing transformation labels with representations derived from data pairs. The proposed pretext task promotes learning invariant and equivariant representations alongside transformation-related information. The method's effectiveness is showcased through various classification and object detection tasks.

Strengths

- The paper argues that existing methods treat each transformation independently as they require transformation labels, which disregards interdependency among transformations. The main strength of the paper lies in their method, which does not require transformation labels. - The complexity of transformations does not constrain the proposed method STL, as they demonstrate learning equivariance representations with complex transformations like AugMix. - Authors show empirically that the learned representations are competitive for classification and detection. They also show that STL learns equivariant representations with relational information between transformations.

Weaknesses

- Experimental comparisons with some key existing equivariant SSL methods are missing [1, 2] and datasets (3DIEBench). - Missing several related works [3-6]. The authors should thoroughly review the relevant literature. - The approach seems computationally demanding since there are multiple (three) InfoNCE losses for each iteration. Additionally, the authors haven’t provided results/discussions on computational costs. - Information for reproducibility is limited, especially when extending STL to BarlowTwins, BYOL, and SimSiam. [1] Garrido, Quentin, Laurent Najman, and Yann Lecun. "Self-supervised learning of split invariant equivariant representations." *arXiv preprint arXiv:2302.10283* (2023). [2] Park, Jung Yeon, et al. "Learning symmetric embeddings for equivariant world models." *arXiv preprint arXiv:2204.11371* (2022). [3] Gupta, Sharut, et al. "Structuring representation geometry with rotationally equivariant contrastive learning." *arXiv preprint arXiv:2306.13924* (2023). [4] Xie, Yuyang, et al. "What should be equivariant in self-supervised learning." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022. [5] Guo, Xifeng, et al. "Affine Equivariant Autoencoder." *IJCAI*. 2019. [6] Gupta, Sharut, et al. "Learning Structured Representations with Equivariant Contrastive Learning."

Questions

- It’s known that InfoNCE loss can be prone to dimensional collapse [7]. Having multiple InfoNCE losses (especially when invariance and equivariance losses can be contradictory) in your method, have you observed dimensional collapse happening? - What influence does the size of the hypernetwork $f_T$ has over the downstream tasks? - How does different hyperparameter configurations of $\lambda_{inv}(\cdot)$, $\lambda_{equi}(\cdot)$, and $\lambda_{trans}(\cdot)$ affect the rate of convergence while pre-training? - How do you extend STL to asymmetric methods like BYOL and SimSiam? Are the InfoNCE criteria still kept for equivariance and transformation-related losses? - Any insights into how much contribution $\lambda_{equi}$ and $\lambda_{trans}$ have for learning the relational information between transformations? [7] Jing, Li, et al. "Understanding dimensional collapse in contrastive self-supervised learning." *arXiv preprint arXiv:2110.09348* (2021).

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors have provided some limitations. It would be beneficial to include a discussion on computational costs.

Summary

This paper proposes a new way to learn equivariant representations by directly learns the transformation representation. They enforce that different transformations have their own input-agnostic representations. To obtain this, they learn an encoder that takes pairwise representations of the same image to extract the transformation representation. To avoid trivial solutions, they use different examples in transformation representation extraction and later transformation module, which helps disentangle the sample and transformation features. They further add an regularization on the sample consistency to encourage it to be sample-invariant. They combine it with invariant learning loss for training, and demonstrates its benefits on several tasks.

Strengths

- I generally like the idea of disentangling sample and transformation representations in equivariant learning, and the paper proposes a clever patching-based objective to enforce it. - And the transformation representation they obtain in Figure 1 does make sense intuitively since it reflects the relative distance between different augmentations. - The evaluation demonstrates its advantages over previous works on linear probing and object detection tasks. The propose method is particularly superior on the transformation prediction task. I think it's because they successfully disentangle the features.

Weaknesses

- The evaluation is not complete. I didn't find linear probing results (often the most important ones) on ImageNet-100 or STL-10. - Lack the comparison to a few relevant E-SSL baselines, such as [1,2]. - Lack more in-depth analysis of the learned transformation representations. I have some doubts about it. Since there are many variations in each augmentation (eg cropping with different positions, ratios), what is the relationship of their representations in the latent space? Is there an Arithmetic relationship? [1] [Equivariant Contrastive Learning. ICLR 2022.](https://arxiv.org/abs/2111.00899) [2][ Residual Relaxation for Multi-view Representation Learning. NeurIPS 2021.](https://arxiv.org/abs/2110.15348)

Questions

no

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Thanks

Thanks the authors for the rebuttal. I find the new results on linear probing, reproduced baseline of E-SSL, and intra-relationship to be promising and help strengthen this work. Therefore, I would like to increase my score to 6.

Answer to Rebuttal

Thank you to the authors for their efforts in answering my concerns. The additional experimental validation provided by the authors addresses my concerns as I believe they strengthen the validation of the proposed method. I have adjusted my score accordingly.