Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications

Summary

The paper demonstrates a root cause of the over-pessimism issue of existing distributionally robust optimization (DRO) methods: excessive focus on noisy samples. To mitigate this issue, the authors proposed a novel DRO method called Geometrically-Calibrated DRO (GCDRO). They introduce the free energy implications of their method (Section 3.1) and approximate optimization method (Section 3.2). Finally, they empirically validate their method for both simulation data (Section 4.1) and real-world data (Section 4.2).

Strengths

- While this paper is a theoretical work (for the ML community), I believe that the problems addressed in this paper can be applied in practical situations. Specifically, existing debiasing methods [1, 2] focus on minor group samples with loss/gradient information. Similar to existing DRO methods (as the paper shows), these debiasing methods will focus excessively on noisy samples. In this case, GCDRO (or a more computationally scalable approximation version) can be applied to mitigate this issue. - [1] Nam, Junhyun, et al. "Learning from failure: De-biasing classifier from biased classifier." *Advances in Neural Information Processing Systems* 33 (2020): 20673-20684. - [2] Ahn, Sumyeong, Seongyoon Kim, and Se-young Yun. "Mitigating Dataset Bias by Using Per-sample Gradient." *arXiv preprint arXiv:2205.15704* (2022). - The free energy perspective of DRO using duality seems novel and interesting.

Weaknesses

- The organization of this paper is somewhat puzzling and hard, although it deals with theoretical topics that are difficult for non-experts to understand. For example, it would be recommended to highlight the key difference between DRO and ERM in Section 2. As ERM does not assume graph structure G_N, readers unfamiliar with DRO would expect a detailed explanation/usage of the graph in L74. - It would be recommended to add a confidence interval for the results in Figure 2. Also, a vector image format provides more clear results as it does not blur when zooming in.

Questions

Can you provide the overall computation cost in terms of wall clock time? What is the computational bottleneck in the proposed method? Can the k-NN graph construction used in Appendix A be applied to debiasing settings in [1,2]?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not discuss the limitations of their method.

Summary

In this work, the authors propose a novel approach called Geometry-Calibrated Distributionally Robust Optimization (GCDRO) to address the over-pessimism issue in traditional Distributionally Robust Optimization (DRO) methods. DRO aims to optimize worst-case risk within an uncertainty set to mitigate the effects of distributional shifts in machine learning algorithms. However, DRO often leads to low-confidence predictions, poor parameter estimations, and limited generalization. The authors analyze a possible cause of over-pessimism in DRO, which is the excessive focus on noisy samples. To mitigate the impact of noise, they incorporate data geometry into calibration terms in DRO, resulting in GCDRO specifically designed for regression tasks. They demonstrate that their risk objective aligns with the concept of Helmholtz free energy in statistical physics, and this free-energy-based risk can be extended to standard DRO methods. To optimize the GCDRO objective, the authors leverage gradient flow in the Wasserstein space and develop an approximate minimax optimization algorithm. This algorithm guarantees a bounded error ratio and standard convergence rate. The authors further explain how their approach alleviates the effects of noisy samples in the optimization process. To validate the effectiveness of GCDRO, the authors conduct comprehensive experiments. These experiments demonstrate that GCDRO outperforms conventional DRO methods in terms of prediction accuracy, parameter estimation, and generalization performance. Overall, the proposed GCDRO approach addresses the over-pessimism issue in DRO by incorporating data geometry into the calibration terms. The authors provide a theoretical analysis and empirical evidence to support the superiority of GCDRO over conventional DRO methods, showing its potential for improving the robustness and performance of machine learning algorithms in the face of distributional shifts and noisy samples.

Strengths

Novel Approach: The paper introduces a novel approach called Geometry-Calibrated Distributionally Robust Optimization (GCDRO) to address the over-pessimism issue in traditional Distributionally Robust Optimization (DRO) methods. This new approach incorporates data geometry into calibration terms, offering a unique perspective on mitigating the impact of noisy samples and improving the performance of machine learning algorithms. Theoretical Analysis: The paper provides a theoretical analysis of the over-pessimism issue in DRO and establishes a connection between the risk objective in GCDRO and the Helmholtz free energy in statistical physics. This theoretical analysis deepens the understanding of the problem and the proposed solution, providing a solid foundation for the proposed method. Optimization Algorithm: The paper leverages gradient flow in Wasserstein space to develop an approximate minimax optimization algorithm for GCDRO. This algorithm guarantees a bounded error ratio and standard convergence rate, providing a reliable and efficient optimization framework for GCDRO. Empirical Evaluation: The paper includes comprehensive experiments to evaluate the performance of GCDRO compared to conventional DRO methods. The experimental results demonstrate the superiority of GCDRO in terms of prediction accuracy, parameter estimation, and generalization performance. The empirical evaluation strengthens the claims made in the paper and highlights the practical benefits of the proposed approach. Practical Significance: The paper addresses an important problem in machine learning—dealing with distributional shifts and noisy samples—and offers a practical solution that can improve the robustness and generalization of machine learning algorithms. The proposed GCDRO method has the potential to be applied in real-world scenarios where distributional shifts are prevalent, making it highly relevant and valuable.

Weaknesses

Limited Comparison: The paper compares GCDRO only with conventional DRO methods, without exploring a broader range of state-of-the-art approaches or alternative methods for addressing the over-pessimism issue in DRO. Including a more diverse set of baselines would provide a more comprehensive evaluation and better contextualize the performance of GCDRO. Lack of Real-world Applications: The paper focuses on theoretical analysis and empirical evaluations using synthetic or benchmark datasets. However, the absence of real-world applications or case studies limits the understanding of how GCDRO would perform in practical scenarios. Extending the evaluation to real-world datasets and applications would enhance the applicability and relevance of the proposed method.

Questions

None

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weaknesses

Summary

This work investigates the impact of noisy samples on Distributionally Robust Optimization (DRO) algorithms. Using a simple model and empirically, they show how DRO variants tend to give to much importance to those samples, which make them overly pessimistic in finding the worst case distribution shift. To solve this problem they propose a new method called Geometrically Calibrated DRO (GCDRO), which adds two new calibration terms. One of this term is looking at the relationship between samples (provided by a graph $G_N$) and penalizes shifted distributions $\mathbf{q}$ which assign high probability mass to connected samples when those have very different prediction losses.The other entropy term favors shifted distributions $\mathbf{q}$ with larger entropy, penalizing $\mathbf{q}$ which overly focus on a subset of samples. The authors provide an algorithm to approximately solve the inner maximization loop which they use to apply their algorithms to several benchmarks in which GCDRO is outperforming prior methods. Moreover, the authors draw a parallel between their method and Helmholtz free energy.

Strengths

Originality: While the two added calibration terms are not entirely novel on their own, I find some novelty in combining them and (i) demonstrating a parallel with Helmholtz free energy, as well as (ii) showing how those can tackle the specific problem of noisy samples in DRO. Clarity: Albeit dense, the paper is clear enough. I would maybe recommend clarifying early on how to get the graph $G_N$. Quality & significance: This work is well motivated and I can see it being impactful. Especially, the link between DRO methods and statistical physics could motivate interesting future directions.

Weaknesses

The method requires a graph $G_N$, which might not be easy to get. There should be a paragraph on how this graph was obtained for each dataset in the experiments section. In the supplementary, it is mentioned that $k$-NN graphs are used and the authors show that the results do not depends so much on $k$. While $k$-NN graphs might work on simple regression datasets, I doubt they would be efficient on more complex ones. For instance, DORO has been applied to CelebA, I'm not sure a $k$-NN graph would work there. Some more complex regression datasets can be found as part of WILDS [1]. I understand the goal of the paper is not manifold learning, yet better understanding the limitations associated with requiring $G_N$ seems important to grasp how useful the proposed method can be in practice. Would proposition 3.3 still hold considering the error obtained from the inner maximization? References: [1] WILDS: A Benchmark of in-the-Wild Distribution Shifts (https://www-cs-faculty.stanford.edu/people/jure/pubs/wilds-icml21.pdf)

Questions

- Would proposition 3.3 still hold considering the error obtained from the inner maximization?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the method are somewhat discussed in the supplementary. I like that the authors did experiments with several $k$-NN graphs, but I believe this analysis can be more convincing by testing the method on more complex regression datasets.

Summary

This paper introduces a novel approach to tackle the issue of overconservativeness in conventional DRO models. The proposed method incorporates data geometry properties into the design of the objective function and ambiguity set. The authors leverage the discrete geometric Wasserstein distance, initially presented in Chow et al. (2017), as a probability metric to construct the ambiguity set. Additionally, they enhance the model's performance by introducing the graph total variation quantity to the objective function. This modification effectively diminishes the influence of detrimental data points, thus mitigating their impact on the overall optimization process. The effectiveness of the proposed approach is extensively validated through experiments conducted on synthetic and real datasets. The results demonstrate its superiority over conventional DRO models, exhibiting improved accuracy and robustness.

Strengths

I acknowledge the significance of addressing the challenge posed by noisy examples in machine learning, and I commend the motivation behind the design presented in this paper. The proposed method is intriguing, and the introduction of new calibration terms appears to be a reasonable approach. The experimental results showcase promise, particularly on synthetic datasets, and also demonstrate competitiveness on real datasets.

Weaknesses

1.The Discrete Geometric Wasserstein Distance restricts the worst-case distribution to have the same support as the training dataset. If this is the case, then the method you propose is fundamentally distinct from Wasserstein DRO. I am interested in understanding the specific scenarios in which Wasserstein DRO outperforms divergence-based methods. As far as I comprehend, GCDRO aims to enhance phi-divergence DRO methods by incorporating graph information. 2. I kindly request verification of the assumption made in Proposition 3.3, which states that F(\theta) is L-smooth. It appears that the convergence results were obtained based on this assumption alone.

Questions

See weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA