Summary
The paper introduces a deep learning method, Rank-N-Contrast, for regression tasks. This method aims to capture continuity in data, something existing methods struggle with. The authors define a concept of $\delta$-ordered feature embedding and show theoretically that if Rank-N-Contrast loss is minimized, feature embeddings will be $\delta$-ordered. The proposed method excels in several regression tasks.
Strengths
This paper provides a novel perspective to regression tasks using deep learning models, a well-studied and widely-acknowledged problem setting. The paper's unique approach lies in its application of contrastive learning methods, which is a departure from conventional techniques that alter the loss function or incorporate specific regularization.
The paper sets itself apart by offering an extensive comparison to existing methodologies, using both theoretical and empirical results from regression and representation learning tasks. This comprehensive approach underlines the proposed method's potential and relevance.
The ubiquity of regression tasks in the realm of deep learning models implies that this paper's approach could contribute meaningfully to the field. Furthermore, the paper is well-composed, making it easy for readers to grasp the fundamental concept of the study.
Weaknesses
The paper positions "regression-aware representation" as vital, but a slight disparity between the theoretical and experimental results is apparent.
Theoretically, it suggests that a deep learning model can fulfill "regression-aware embedding" characteristics by minimizing the RNC loss appropriately. Experimentally, the paper exhibits consistent performance enhancements over existing approaches in both regression tasks and representation learning frameworks. Furthermore, it confirms that the implementation of RNC provides resilience against data corruption, improves performance with limited training data, and boosts transfer learning.
While the paper implies the potential to secure $\delta$-ordered feature embeddings by reducing RNC loss, and evidences steady performance improvement in real-world data contexts, the precise relationship between the continual characteristics of the "regression-aware representations" and the observed performance improvements is not fully clear yet. Additional discussion around this subject could offer more insights and, thereby, enrich our comprehension.
Questions
1. **About Embedding Dimensions:** The learned representations in Fig. 1 are impressive, but the embedding dimension isn't specified. Were these embeddings trained directly in two dimensions, or were they first embedded in a high-dimensional space and then visualized with methods like UMAP? If UMAP was used, are the apparent clustering and lack of continuity in L1 and SupCon embeddings an artifact of the visualization process, or do these characteristics persist in the high-dimensional embeddings?
2. **About the Construction of RNC loss:** In l.104, just before Equation 1, it's stated that the normalized likelihood of $v_j$ "can be written as …" Is Equation 1 derived from some definition or axiom? At least the normalized likelihood is “defined” in the reference [41].
3. **About Trainability:** Theorem 3 claims that if RNC loss is sufficiently minimized, a desirable representation can be obtained. But how much can RNC loss be practically minimized? Can any insights be provided from a theoretical standpoint on the trainability of the proposed loss?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
One potential limitation is the difference in the number of training epochs between the one-stage and two-stage methods. According to Appendix l.629, the one-stage methods were trained for 400 epochs, whereas the two-stage methods, including the proposed approach, were trained for a total of 500 epochs. This longer training period for the proposed method could potentially be contributing to the improvements observed in table 2. A further analysis or an ablation study might help to confirm whether these enhancements are genuinely due to the method itself and not merely a consequence of the extended training period.