Rank-N-Contrast: Learning Continuous Representations for Regression

Deep regression models typically learn in an end-to-end fashion without explicitly emphasizing a regression-aware representation. Consequently, the learned representations exhibit fragmentation and fail to capture the continuous nature of sample orders, inducing suboptimal results across a wide range of regression tasks. To fill the gap, we propose Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by contrasting samples against each other based on their rankings in the target space. We demonstrate, theoretically and empirically, that RNC guarantees the desired order of learned representations in accordance with the target orders, enjoying not only better performance but also significantly improved robustness, efficiency, and generalization. Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare verify that RNC achieves state-of-the-art performance, highlighting its intriguing properties including better data efficiency, robustness to spurious targets and data corruptions, and generalization to distribution shifts. Code is available at: https://github.com/kaiwenzha/Rank-N-Contrast.

Paper

References (52)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer 2Kp16/10 · confidence 2/52023-06-29

Summary

The paper proposes the framework Rank-N-Contrast in order to learn regression-aware feature representations. Authors claim that this representation learning mechanism captures the continuous nature of sample orders and helps achieve better performance in downstream regression task.

Strengths

1. The paper provides both experimental results and theoretical proof to support their method. 2. The paper includes relatively comprehensive ablation study and analysis section.

Weaknesses

1. The paper is in lack high-level intuition and in-depth analysis of why the method work and only have low-level interpretation on the experimental results. Following are some example questions I hope authors could have addressed (and go one-step further than just stating their claims): Why preserving order in feature representation help to learn continuous nature? In fig. 3, why discernible pattern is better? What are the theorems trying to convey (my understanding is that feature embedding follows order inherited from the labels)? 2. Although authors have conducted experiments on 4 different datasets (where is the results of IMDB-WIKI b.t.w.?) and with 7 different regression losses, the datasets mostly have scalar responses (except for MPIFaceGaze) and seem repetitive to me. I think the experimental results will be more convincing if authors include experiments with higher-dimensional responses.

Questions

See Weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

I don't think authors have adequately addressed their limitations and weaknesses in the paper.

Reviewer dBkT8/10 · confidence 4/52023-07-04

Summary

The paper introduces a deep learning method, Rank-N-Contrast, for regression tasks. This method aims to capture continuity in data, something existing methods struggle with. The authors define a concept of $\delta$-ordered feature embedding and show theoretically that if Rank-N-Contrast loss is minimized, feature embeddings will be $\delta$-ordered. The proposed method excels in several regression tasks.

Strengths

This paper provides a novel perspective to regression tasks using deep learning models, a well-studied and widely-acknowledged problem setting. The paper's unique approach lies in its application of contrastive learning methods, which is a departure from conventional techniques that alter the loss function or incorporate specific regularization. The paper sets itself apart by offering an extensive comparison to existing methodologies, using both theoretical and empirical results from regression and representation learning tasks. This comprehensive approach underlines the proposed method's potential and relevance. The ubiquity of regression tasks in the realm of deep learning models implies that this paper's approach could contribute meaningfully to the field. Furthermore, the paper is well-composed, making it easy for readers to grasp the fundamental concept of the study.

Weaknesses

The paper positions "regression-aware representation" as vital, but a slight disparity between the theoretical and experimental results is apparent. Theoretically, it suggests that a deep learning model can fulfill "regression-aware embedding" characteristics by minimizing the RNC loss appropriately. Experimentally, the paper exhibits consistent performance enhancements over existing approaches in both regression tasks and representation learning frameworks. Furthermore, it confirms that the implementation of RNC provides resilience against data corruption, improves performance with limited training data, and boosts transfer learning. While the paper implies the potential to secure $\delta$-ordered feature embeddings by reducing RNC loss, and evidences steady performance improvement in real-world data contexts, the precise relationship between the continual characteristics of the "regression-aware representations" and the observed performance improvements is not fully clear yet. Additional discussion around this subject could offer more insights and, thereby, enrich our comprehension.

Questions

1. **About Embedding Dimensions:** The learned representations in Fig. 1 are impressive, but the embedding dimension isn't specified. Were these embeddings trained directly in two dimensions, or were they first embedded in a high-dimensional space and then visualized with methods like UMAP? If UMAP was used, are the apparent clustering and lack of continuity in L1 and SupCon embeddings an artifact of the visualization process, or do these characteristics persist in the high-dimensional embeddings? 2. **About the Construction of RNC loss:** In l.104, just before Equation 1, it's stated that the normalized likelihood of $v_j$ "can be written as …" Is Equation 1 derived from some definition or axiom? At least the normalized likelihood is “defined” in the reference [41]. 3. **About Trainability:** Theorem 3 claims that if RNC loss is sufficiently minimized, a desirable representation can be obtained. But how much can RNC loss be practically minimized? Can any insights be provided from a theoretical standpoint on the trainability of the proposed loss?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

One potential limitation is the difference in the number of training epochs between the one-stage and two-stage methods. According to Appendix l.629, the one-stage methods were trained for 400 epochs, whereas the two-stage methods, including the proposed approach, were trained for a total of 500 epochs. This longer training period for the proposed method could potentially be contributing to the improvements observed in table 2. A further analysis or an ablation study might help to confirm whether these enhancements are genuinely due to the method itself and not merely a consequence of the extended training period.

Reviewer wKSy7/10 · confidence 4/52023-07-04

Summary

The authors present a new loss for representation learning in regression, RNC (Rank-n-contrast). RNC can be seen as the SupCon loss adapted to the regression setting, where the labels given are not hard class labels but rather continuous regression labels. In SupCon the negatives for each example are members of other classes and the positives come from members of the same class. In RNC positive pairs are formed with every example to the anchor, and the negatives are those examples with larger label distances to the anchor than the positives. By doing this they are able to train a representation for the data that is continuous in nature, which is hypothesized to represent the data better.

Strengths

1. I believe the method is sufficiently original as a nontrivial adaptation of the SupCon methodology to the regression setting. The designation of what the positives and negatives are for the RNC contrastive loss are well motivated and I can see why the loss would give the kind of continuous representations that it does. 2. The method seems to be pretty simple to implement. I particularly like that the RNC loss is the only loss the authors pretrained with and it wasn't some highly tuned composition of many different losses. It gives me more confidence that the RNC loss is providing the gain in performance. 3. The theory is simple and well-motivated in that it proves the loss is doing what the authors are claiming it does. The theory combined with the visualization of the representations give me confidence that the representations are in fact ordered continuously. 4. The experimental evaluation is fairly thorough. The standard questions are answered, i.e. standard deviation error bars, how does it perform against other pretraining tasks, how does a plain two-stage training compare, whether augmentation is important, etc.

Weaknesses

1. I think the experimental evaluation can be improved to improve the generality of the method. Currently the datasets that are evaluated are on computer vision regression tasks, which confines the evaluation to just ResNet-based models when we compare against the different losses and training methods in the paper. In particular, one area where regression tasks are popular is the tabular setting. I would be very interested to see a comparison between RNC and tree-based methods on popular tabular methods. For examples of models and datasets to compare on the authors can check https://arxiv.org/abs/2106.11959 (not my paper btw) 2. It is great to see that the method is able to consistently perform well when compared against other baselines. But as someone who is not familiar with these particular regression tasks I am not sure whether the magnitude of improvement is substantial. Could the authors help me understand the scale of improvement? 3. Returning to my point on how the evaluation is restricted to ResNet-based architectures, I wonder if given that ResNet was designed to classify images that it has some inductive bias on clustering sets of images based on semantic similarities, which is what causes the disjointed representations in Figure 1 (left). Expanding past the computer vision domain would be very valuable in making the contribution more impactful and general.

Questions

Questions and possible directions of improvement are listed under "Weaknesses".

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I believe this was sufficient.

Reviewer D7S98/10 · confidence 3/52023-07-06

Summary

The authors discuss the benefits of contrastive learning for learning structured representations in a regression setting. While contrastive losses are typically formulated in terms of “similar” and “dissimilar” examples, the authors make use of the extra information conveyed by the continuous target label. They show how adding the RnC contrastive term to training is beneficial for a suite of high-dimensional regression tasks.

Strengths

This is a technically strong paper that, in my opinion, makes a contribution towards the important and understudied problem of representation learning for regression. The qualitative results [Fig 1] are compelling. The proposed method is intuitive and the paper is well written. The theoretical analysis and extensive empirical results suggest to me that the proposed method will be of interest to NeurIPS attendees. I was especially interested in the suggestion that contrastive training yields more robust representations [lines 259--279], although I think further work will be needed in the future to validate this finding in more sophisticated OOD generalization settings.

Weaknesses

To my knowledge the technical contributions here are sound. However, after reading the paper, I found myself a bit concerned about how neural net-based regression models are going to be used in the near future. And I think the paper would benefit from a head-on discussion of the potential ethical issues at play. Regression is notoriously difficult for neural networks (for example, many baseline methods convert the regression into a classification problem). A technological advance here could open up entirely new application areas. However, the use of datasets involving human faces suggest to me that some of these new applications could bring up new ethical concerns as well. While Predicting age and gaze direction from face images (as done in this paper) might seem reasonable, there are existing critiques in the literature [e.g. https://ir.lawnet.fordham.edu/cgi/viewcontent.cgi?article=1804&context=iplj, https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a, https://dl.acm.org/doi/pdf/10.1145/3375627.3375820] arguing that predicting other target variables, such as a continuous proxy for emotion, are ethically fraught. In the authors’ opinion, are there application areas where RnC should *not* be applied? I think the paper would benefit from a discussion of these issues. While the lack of such discussion (the broader impact section is, frankly, rather boilerplate) represents a weakness in my view, it probably should not stand in the way of the paper’s acceptance, since it is more of a critique of the whole subfield rather than this specific paper. However, I am going to request an ethics review in order to get a second opinion about this. The related works are generally well covered. One exception is the recent C-mixup paper [https://arxiv.org/abs/2210.05775], which also uses label similarity in regression to group “similar” data points. However their approach uses these similarities for data augmentation rather than contrastive learning. A discussion of how the two approaches differ would benefit the reader.

Questions

* How is the temperature parameter \tau tuned? [line 106]

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations and broader impacts are discussed [Sec. 5.4, App. H]. However, given that images of faces are used frequently in the experiments, I think that a more complete discussion about broader impacts, especially as it relates to task definition for regression, should be included (see “Weaknesses” above and my request for an ethics review below).

Reviewer 6JYz7/10 · confidence 4/52023-07-13

Summary

In this paper, the authors proposed a novel framework that learns continuous representations for regression problems, by contrasting samples against each other based on the rankings induced by the target values. The proposed method is evaluated on several regression tasks and the results show that the proposed method can achieve competitive performance compared to the state-of-the-art methods.

Strengths

1. This paper tackles a very interesting problem of representation fragmentation in deep regression models. The proposed method is simple and very effective to learn continuous representations that fit the regression task. It is very interesting to see that the traditional methods learn representations clustered based on spurious targets (i.e., camera location in the SkyFinder dataset) while the proposed method learns a nice continuous representation that captures the target values. 2. The authors present a theoretical analysis of the proposed loss function and show that it can learn delta-ordered feature embeddings when sufficiently trained. 3. The proposed loss function is conceptually simple and easy to implement. The proposed method is thoroughly evaluated on several regression tasks and the results show that the proposed method can achieve competitive performance compared to the state-of-the-art methods. The authors further conducted ablation studies to show the effectiveness of the proposed method under data corruption, unseen or spurious targets.

Weaknesses

1. While it is nice to see the proposed method leads to delta-ordered feature embeddings. It would be nice if the authors can further theoretically connect the delta-ordered feature embeddings to the final performance of the regression task. This would help to justify that the delta-ordered feature embeddings are indeed useful for the regression task. 2. For the remark for Thm 3, to achieve delta-ordered features for the entire dataset, is it that we need to optimize all batches to achieve low enough loss? Is there any guarantee/insight that there is a feasible solution that can achieve low enough loss for all batches? 3. It seems the IMDB-WIKI performance is missing in Table 2.

Questions

See the comments in "weaknesses".

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes.

Reviewer D7S92023-08-10

author rebuttal

I read the rebuttal and the other reviews. For now I will keep my score the same, which reflects my belief that the paper would be a very nice addition to the conference. If an ethics review is added later I will read and consider its contents.

Reviewer 2Kp12023-08-10

Raise score to 6

Thanks for the authors making clarifications and further explaining the details of the experiments. After carefully reading the authors' rebuttal and other reviewers' comments, I am generally satisfied with the responses to my concerns. I decide to raise my score to 6.

Authorsrebuttal2023-08-12

Thank you very much for your feedback

Thank you very much for your feedback. We are glad to learn that our response has addressed your concerns and that you decided to raise your score to 6. We noticed however that the score and review haven’t yet been updated. Thus, we would like to kindly request that you update your score and review in the system to reflect your decision to raise the score to 6. Once again, thank you very much for your time and effort, and please do not hesitate to let us know if you have any further questions or comments about the paper.

Reviewer wKSy2023-08-11

The rebuttal has addressed my concerns.

I have raised my score accordingly.

Reviewer 6JYz2023-08-20

Response

Thank you for the clarifications. I have read the response and other reviews. I keep my current score unchanged.

Reviewer dBkT2023-08-22

Response

Thank you for giving us comprehensive responses to all the questions. Roughly all my questions were addressed by the authors' feedback. I would leave one point of slight concern. Regarding the use of UMAP visualization, the authors stated as follows > The clustering and lack of continuity are unlikely to be artifacts, because if the points are clustered / far apart in the UMAP visualization, it suggests that those points were close to / distant from each other in the high-dimensional space as well [29]. I would like to leave a different point of view on this comment. Indeed, dimensionality reduction methods such as UMAP are designed to preserve distance relationships in the original space as much as possible. In practical situations, however, UMAP visualization can often reveal superficially spurious cluster structures, even in artificial data that is inherently random and structureless. Since such a phenomenon depends on the number and dimension of the data and the hyperparameters of the UMAP, it would be good to have a simple ablation study that would eliminate such a possibility. Overall, my assessment of this paper remains the same. This paper is well-written, and its value is clear. I believe it is well worthy of being accepted, while one minor point I mentioned above still remains. Therefore, I would like to keep my initial score.

Authorsrebuttal2023-08-22

Further response to the UMAP question

Thank you for your feedback and for your opinion that the paper is worthy of being accepted. As suggested by the reviewer, we will include an ablation study of the number / dimension of the data and the hyperparameters of the UMAP in the revised paper. Here, due to the limited time, we provide some preliminary results that show the structure is **not** due to artifacts: - **Ablation of the number of data samples**: In Fig. 6 of the main paper, we show the UMAP visualization using a 10-webcam subset of SkyFinder dataset, whereas Fig. 1 uses the full dataset, which contains 44 webcams. The structure of the visualization in Fig. 6 is consistent with Fig. 1, indicating that the difference in their number of data samples did not eliminate the structure. - **Ablation of the dimension of feature embeddings**: In the PDF attached in the Global Response, we generated the same plots for L1 and RnC as in Fig. 1, but using ViT-Small as the backbone, whose feature dimension is 384. The structure of the visualization in the PDF is consistent with Fig. 1, despite their difference in the number of dimensions (384 vs. 512). - **Ablation of the hyperparameters of UMAP**: Following your suggestions, we explored the impact of major UMAP hyperparameters, such as the number of neighbors (`n_neighbors`) and the minimum distance between embedded points (`min_dist`). Specifically, we tried `n_neighbors` from {5, 10, 20, 50, 100} and `min_dist` from {0, 0.25, 0.5, 0.8, 0.99} (In Fig. 1 we use the default parameters in UMAP, where `n_neighbors = 15` and `min_dist = 0.1`). Unfortunately, we are not able to include the figure at this stage, however, we would like to let you know the structures of visualizations are still consistent with Fig. 1 among all of these hyperparameters: L1 and SupCon embeddings are fragmented while RnC embeddings are continuous. Once again, we thank the reviewer for the constructive feedback and insightful suggestions. We hope the above results will address your concerns and help you be more confident about our paper. We will stress this point and provide a comprehensive ablation study in the revised paper.

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC