Combating Representation Learning Disparity with Geometric Harmonization

Summary

The authors tackle the problem of contrastive learning in the context of imbalanced datasets where some classes have many more samples than other classes. Since no label information can be used, importance sampling is not an option and without any interventions, contrastive learning tends to overrepresent majority classes in the latent space, thereby hurting the readout accuracy on the minority classes. The authors propose a Geometric Harmonization technique in which they propose to estimate the class labels with clustering and then recalibrate the learned embeddings such that majority classes occupy the same feature space as minority classes. The authors show experimental results on CIFAR100-LT, ImageNet-LT and Places-LT.

Strengths

- I appreciate the complexity analysis of the proposed method in Section 3.4. - The authors compare to several other benchmarks and show superior results with their method. - I think in general the method makes sense and it is intuitive that it should work in the proposed case.

Weaknesses

- There are many typos and grammatical errors in the text which makes the paper hard to understand. Some mathematical definitions are wrong and inconsistent which makes me doubt the claims of the theoretical guarantees since there are even errors in the Lemmas / definitions: - Line 94: “Let N denote the sample number and R = max_i n_i/min_i n_i denote the imbalanced ratio, where n_i denotes the number of samples in class i.” I am confused, the max and min are calculated per class i, which means that max_i n_i = min_i n_i = n_i, therefore R would be 1 for each class. Since I do not understand how R is defined, I cannot understand / judge the results in Table 2. - Lemma 3.2 is inconsistent with itself. If n_L_H=n_H and n_L_H=n_T, then n_T = n_H and n_H / n_T -> infty is impossible because n_H / n_T = 1 by definition. - Eq. 5: Using M both as the geometric structure and the size of the collection of negative samples is confusing and should not be done. - Considering the experimental results, I am not convinced that the authors have looked into the most important baselines. CIFAR100-LT, ImageNet-LT and Places-LT are well-established benchmarks and the baseline results that the authors aim to improve seem to be very low to me. I will elaborate on the different datasets in the following: - For ImageNet-LT (https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt), the currently best number for a ResNet50 architecture is an accuracy of 70.1 (https://arxiv.org/pdf/2111.13579v4.pdf) with extra training data and 67.2 (https://arxiv.org/pdf/2111.14745v1.pdf) without extra training data. The best result the authors report in this paper is about 38% which is far below what is currently state of the art according to the benchmark list. - For Places-LT, https://paperswithcode.com/sota/long-tail-learning-on-places-lt, the best numbers for a ResNet50 are above 45% while the authors here report numbers below 35%. -> I believe that in order for this method to be relevant to the community, the authors need to show results on the superior models with higher baseline accuracy. In the current state, it is not clear whether their results would generalize to the better models. - The motivation for the studied problem does not become clear to me from the introduction. The authors write: “However, the real-world natural sources usually exhibit the long-tailed distribution [31], and directly learning representation on them might lead to the distortion issue of the embedding space, namely, the majority dominates the feature regime [45] and the minority collapses [28].” Using the modal verb “might” here indicates a possibility but no further evidence is presented. Citing fairness research as an application is not enough in my opinion. From reading the introduction, I am not convinced that the problem the authors want to study actually exists in SSL. I would advise the authors to provide concrete examples where using SSL actually harms performance on minority classes. Currently, the main question of study posed in line 39 (“Why the conventional contrastive learning underperforms in self-supervised long-tailed context?”) does not seem well supported. - The first part of the first contribution is misleading and I believe wrong: “To our best knowledge, we are the first to investigate the drawback of the contrastive learning loss in self-supervised long-tailed context.” This drawback is investigated in several other papers which the authors discuss in their related work section, e.g. SDCLR specifically tackles this problem with a different method. - Figure 1: The spheres in the middle and right Figure look like ovals which is confusing. These should be spheres/ circles. - Abstract, line 5: „The attribution is that the vanilla SSL methods that pursue the sample-level uniformity easily leads to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse.” -> This sentence is hard to parse and understand. - Table 1: What does IR stand for? - Line 119: “and π ∈ RK+ refers to the the marginal distribution constraint.” -> "The" twice and what do you mean under “marginal distribution constraint”? - The used datasets and models must be cited. - Line 162: “This phenomenon indicates all representations of the minority will collapse completely to one point without considering the category discrepancy, which corresponds to our observation regarding the passive collapse of tailed samples in Figure 1.” This statement is not accurate because the representations of the minority classes do not collapse to a single point in Figure 1. Lemma 3.2. covers an extreme case where there are infinitely more samples in the head classes compared to the tail classes, so Figure 1 does not represent this scenario and thus, using “corresponds” here is not accurate. - It is confusing that Table 7 comes before Tables 5+6, please fix. - Does Table 5 show the computational cost per epoch? - Line 218: “For hyper-parameters of GH, we provide a default setup across all the experiments: set the geometric dimension K as 100 and the temperature γGH as 0.1. In the surrogate label allocation, we set the regularization coefficient λ as 20 and Sinkhorn iterations Es as 300. Please refer to Appendix E.3 for more experimental details.” There are many hyperparameters that need to be set for this method, and it is not clear whether they were chosen on the test set, or how they were selected. It is also not clear how sensitive the algorithm is to these hyperparameters. - Line 223: “For comprehensive performance comparison, we present the linear probing performance and the standard deviation among three disjoint groups, i.e., [many, medium, few] partitions [25].” Please explain what the partitions into many/medium/few mean as it is not possible to understand the results otherwise. - Line 290: “To justify the intuition in Section 3.2” which intuition? Please be more specific. - I find the results in Fig. 3b unintuitive. The NMI scores show that GH is better aligned with ground truth labels compared to all other methods, but then why is the readout accuracy of GH only 1-2 percent points better compared to the other methods? Please discuss this as I am not sure how to interpret this result and whether it is meaningful to compare NMI scores between GH and the other methods.

Questions

- How sensitive is GH to the choice of the hyperparameters? - Can you please comment on how the presented results on Places-LT and ImageNet-LT compare to the state-of-the-art results on https://paperswithcode.com/sota/long-tail-learning-on-places-lt and https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The authors could think a bit more about the impact of their work on applications related to long-tailed data distributions where minority and majority classes are present. How would applications situated in fairness research impacted by their work?

Summary

The paper investigates the SSL in a long-tailed distribution setting, where traditional contrastive learning may not work well. The authors attribute that issue to the "sample-level" learning of SSL and propose Geometric Harmonization (GH) to regularize the "category-level" during training. GH can be a plug-and-play loss to the existing SSL frameworks without modification. Empirical results of linear probing accuracy show the effectiveness of the proposed method in the long-tailed data over the baseline SimCLR.

Strengths

+ Contribute to SSL capability in real-world scenarios with imbalanced classes which may be useful for unsupervised representation learning. + Some theoretical analyses are provided to support the problem and method. + The idea is reasonable + Writing is clear enough

Weaknesses

+ SSL has been developed with various methods to facilitate the representation learning for unlabelled data, this paper provided the experiments for one method, i.e. SimCLR, which is not a very strong baseline and not a state-of-the-art CL framework as claimed. From the presented results, SimCLR still works not too bad compared to SimCLR+GH in all settings. I would expect to see the effectiveness of GH in more strong SSL baselines including contrastive learning frameworks such as BYOL, MoCo-v3, etc, and maybe a more recent branch of SSL, namely mask autoencoder (MAE). To make the proposed method more complete, those baselines should be considered. + The "passive collapse" of tail classes is mentioned in the paper but seems that except for some quantitative results showing GH improves baseline to some extent, there are no results (either visualization or quantitative metric) to show such collapse happens and that GH can mitigate or solve that problem.

Questions

+ For transfer learning to downstream tasks, which dataset is pre-trained for SSL? + From the concept/hypothesis in Fig. 2 bottom, when applying GH, the samples of each class are clustered well even though it is the imbalanced case. Is there visualization (t-SNE for example) for the real data that has been conducted in this paper to show that works as the figure of concept in Fig.2?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

They have discussed the limitations.

Summary

- The paper proposes to tackle the task of self-supervised learning on datasets sampled from long-tailed distributions. - Typical SSL approaches tend to perform a lot worse for the minority classes, thus hurting performance. - The proposed approaches termed Geometric Harmonization encourages category-level uniformity leading to better performance on various downstream tasks. - The approach is shown to be complementary to various existing SSL approaches.

Strengths

- The paper tackles an important problem in SSL - that of training on data sampled from long-tailed distributions which is common in the real world. - The proposal of geometric harmonization is interesting. The idea is simple and easy to implement. While the approach is motivated by recent clustering-based works in SSL, the proposed approach seems novel. - The paper is well written for the most part (some concerns are pointed out in other sections) - Authors perform extensive analyses and ablations to understand the working of the approach. - Experiments on various LT datasets show the benefit of using the approach.

Weaknesses

Following are some of my concerns with the paper: - Clarity: Given that geometric harmonization is the key contribution of this work, I think the authors should give the readers more idea about the intuitions for the section on "surrogate label allocation". While the Figure 2 looks good, the caption is not very helpful to drive home the intuition. I see a similar issue with Definition 3.1 : meaning of K only becomes clear later in the paper. - Effect of batch-size : The authors have provided an analysis on the effect of batch size. I think such analysis will be especially interesting for datasets which have lot more labels : like ImageNet. - Discussion/comparison missing on some related works: [a], [b] [a] The hidden uniform cluster prior in self-supervised learning [b] Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data

Questions

- Minor suggestions : - L42 : reword "as proof in [38].." - "sample number" : I suggest using "number of samples" or a similar alternative instead to avoid any confusion. - Some more discussion and analysis on the observation in L298-300 would be really interesting. For this experiment, what happens if there is no support for a certain class in that mini-batch ?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and societal impact of their work.

Summary

This paper concentrates on the long-tailed distribution problem in SSL representation learning. To overcome the challenge in vanilla SSL methods, where head classes with the huge sample number dominate the feature regime, the authors propose the Geometric Harmonization (GH) method to encourage category-level uniformity. Specially, GH measures the population statistics of the embedding space, and then infer an fine-grained instance-wise calibration to constrain the space expansion of head classes while avoid the collapse of tail classes. Extensive results show the effectiveness of GH with high tolerance to the long-tailed distribution problem.

Strengths

-1- This paper investigates the drawback of the contrastive learning loss in SSL long-tailed context, and shows that the resulted sample-level uniformity is an intrinsic limitation to the representation parity, as shown in Figure 1. The motivation of pursuing category-level uniformity is clear. -2- This paper is well-written and easy to follow. The proposed GH method that incorporates the geometric clues from the embedding space to calibrate the training loss, is interesting and solid. -3- The proposed GH loss is versatile and can be easily plugged into existing SSL methods. The extensive experiments and ablation stidies demonstrate its effectiveness in learning robust representation.

Weaknesses

-1- About the surrogate label allocation. What is the advantage to choose the discriminative clustering [1], what if we choose other clustering method such as K-means. How does the quality of the geometric label affect the final training results? When verifing the assumption that the constructed surrogate geometric labels are mutually correlated with the oracle labels, can the authors provide more intuitive results? -2- In E.q. 4, do you need to design how to balance these two losses? -3- Table 6 shows the results on class-balanced data. Can you explain why in some cases, incooperateing with GH will lead to the performance drop?

Questions

Please refer to *Weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper studied self-supervised representation learning with implicit class-imbalance data, in contrast to most prior works that assume class-balance. To do this, this paper proposed to augment the existing contrastive learning method with a novel geometric harmonization (GH). Intuitively, GH pulls samples toward the closest prototype in the embedding space, which differs from previous cluster-/prototype-based methods in the learnable prototypes and marginal prior. Through extensive experiments on several standard long-tail classification benchmarks, the authors demonstrated the efficacy of the proposed method with different contrastive learning methods as start points.

Strengths

1. The paper is generally well-written and easy to follow. 2. The paper is well-motivated. Class imbalance occurs in most real-world scenarios, and existing contrastive learning methods fail to take this into account. This paper provides a good exploration of this direction. 3. The proposed geometric harmonization regularization is interesting. Though prototype-based constraints are already studied in contrastive learning [1, 5, 12], GH carefully considers the class imbalance by its design and is thus robust to such imbalance. 4. The experiment results are strong compared with the baselines on imbalanced data. Moreover, the proposed method even performs on par with the baseline with balanced data.

Weaknesses

1. The geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space to estimate the assignment and prior. While this condition can be satisfied on curated datasets like CIFAR and ImageNet, it might be a different case for the uncurated datasets, e.g., YMCC or a subset of it. Some analysis on this end would further strengthen this paper. 2. Lack of transfer learning experiments. In all the experiments, the models were pretrained and then finetuned on the same datasets. It is unclear whether the learned representations are general to transfer to other data distribution and different tasks. Possible options are adding cross-dataset experiments (e.g., YFCC-to-ImageNet-LT, ImageNet-Lt-to-CIFAR-LT) and transfer learning on MS-COCO object detection as in [5].

Questions

See the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The major limitation is that geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space, which may not hold for real-world data.

Summary

This paper addresses the class-imbalance problem under the SSL setting. It shows that the Vanilla SSL methods pursuing the sample-level uniformity easily lead to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse. It proposes Geometric Harmonization (GH) to encourage the category-level uniformity in representation learning. The GH loss can be easily integrated into existing SSL methods, and GH loss improve the performance over baselines on the imbalance class problem under the SSL setting, based on the experimental results.

Strengths

1.This paper define the category-level uniformity in the embedding space that SSL learns. The motivation to using category-level uniformity for imbalance class problem is clear. 2.The proposed Geometric Harmonization (GH) loss improve the performance over baselines on the imbalance class problem under the SSL setting, based on the experimental results. 3.The presentation of this paper is generally clear and easy to follow.

Weaknesses

1. In the experiments of Section 4.3, it is better to conduct experiments on large-scale ImageNet pertaining using the proposed methods, and then transfer to other datasets (This is the so called “representation transferability”), rather than only: SSL pre-training and finetuning on the same datasets. 2. I personally believe the theories and analyses both are specific to contrastive learning, rather than on other SSL methods (e.g., the non-contrastive SSL methods, BOYL, SiamSiam). Even though this paper conducts an experiment on other SSL methods in Section 4.4, it is not clear how the theory/analyses work for these SSL methods? Do the theoretic analyses still work for other SSL methods? This paper should at least mention it. 3. At last, I have concerns on whether the imbalance problem under the SSL is an important research topic. I definitely agree with that the class imbalance problem is very important. However, it is weird when considering the class imbalance problem under the SSL scenario. As everyone knows, SSL is considered as pre-training under large scale unlabeled data (class-free), and then transfer to downstream tasks (not specific for the classification problem). I believe the contributions are significant if this is the first paper to propose the imbalance problem under SSL setting. However, with several exists works as shown in the paper, I donot recognize the significance this paper contributes to the ML community. I think this paper will be stronger, if considering transfer to the class-imbalance object detection/semantic tasks.

Questions

Address Weakness 2 and 3.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Response to Author Rebuttal

Thanks for the response that addressed my concerns, and I will keep my positive score.

Response to the rebuttal

Dear authors, thank you for your replies to my concerns. It is indeed that I am not familiar with the literature on self-supervised long-tailed (SSL-LT) learning, as well as with the field in general, as is reflected by my low confidence. Given that other much confident reviewers write that the experimental results are strong, I will lift this concern. I will increase my rating to a '6', but will not increase my confidence. I encourage the authors to improve the clarity of the paper, rectify the definitions etc. My low rating was also due to me not understanding several important equations because they were wrong / very ambiguous. Best, Reviewer R4di

Thanks for the rebuttal

Thanks for the very detailed rebuttal. The explanations and new experiments on CC3M are very helpful. I have no additional concerns at this moment.

Response to the authors

Thanks for the great effort in the rebuttal. It resolves my concerns about the premise and the transferability of this method. I have no additional concerns now and would like to raise the score.

Response to authors' rebuttal

I acknowledge the response of the authors and I have read the authors responses. My concerns on “representation transferability" are addressed by the authors' additonal experiments. Besides, the results of additional experiments pretrained on CC3M also increases my interest in the imbalance class under SSL setups. I raised my score from 5 to 6, and hope the authors can include the additional experiments in the revised version.