Wasserstein Distributionally Robust Optimization Through the Lens of Structural Causal Models and Individual Fairness

Summary

The authors propose a novel framework to enhance individual fairness guarantees under a Wasserstein distributionally robust optimization strategy. For such purposes, they employ counterfactuals based on the underlying causal structure of the model at hand. They further propose an alternative with theoretical guarantees for when the causal structure is unknown.

Strengths

S1 - The issue being investigated is of significant importance. S2 - The introduction and abstract effectively substantiate all the claims made, including the contributions put forth by the authors. These assertions find validation through a thorough description of the methodology employed and the theoretical results provided. S3 - The paper demonstrates a strong mathematical foundation, supported by numerous theoretical results. S4 - The paper demonstrates commendable attention to reproducibility by providing detailed information regarding the experimental setup.

Weaknesses

W1 - The text lacks clarity in several areas. It becomes quite technical at times, and it would benefit from including more examples alongside the technical explanations, particularly in the introduction, to help readers develop a better intuition. Additionally, there are instances where the ideas presented lack cohesion and do not clearly relate to one another, making the overall narrative difficult to follow. Improving the flow and connection between ideas would enhance the readability and comprehension of the text. W2 - The paper does not cite many significant works on individual fairness and algorithmic fairness in general. For instance, citing [1] would be relevant, as it shares many similar points with the presented work. It would be beneficial to outline both the similarities and differences between them. Additionally, the related works section overlooks a significant body of literature on algorithmic fairness that is based on causality. These works are numerous and, in my opinion, should be acknowledged. W3 - It is not very clear which are the main advantages and benefits of the proposed approach with respect to existing works in the literature. W4 - As I understand it, the authors propose modifying only the sensitive attribute while leaving all other attributes unchanged (please correct me if I am wrong). However, this approach can create unrealistic twins. For example, consider a male instance with a height of 1.85m, which is a typical height in a European country. If we change the gender to female without adjusting the height, the resulting instance would be an outlier and not representative of a typical female, making the two instances not equivalent. In the presence of such unrealistic twins, the classifier's decisions regarding these instances may not be informative. This issue arises because bias exists not only in the sensitive attribute but also in the non-sensitive attributes; for example, gender and income can be highly correlated. Therefore, when we change the sensitive attribute of an instance, in order to create an ‘equivalent’ instance of the other gender, non-sensitive attributes should as well be modified. That is, to ensure the relevance of their analysis, the authors should verify that the twins they consider are realistic and plausible instances. For more insight on this matter, see [1]. W5 - The empirical evaluation of the proposal is poor. It does not include popular benchmark fairness-enhancing interventions, and only a few classification tasks are considered, despite the availability of numerous widely-used datasets in the algorithmic fairness literature. Besides, there is no deep discussion regarding the results. W6 - The model lacks any discussion or insights regarding its computational complexity or cost. W7 - The acronym SCM is used before it is defined: it is first used in line 49, but it is defined in line 89. W8 - (minor) I suggest moving the related works section into a separate section, as it represents a distinct aspect of the work. W9 - (typos), line 60 (Our → our), line 113 (variables[50] → variables [50]), line 204 (space 9 ??) [1] De Lara, L., González-Sanz, A., Asher, N., Risser, L., & Loubes, J. M. (2024). Transport-based counterfactual models. Journal of Machine Learning Research, 25(136), 1-59.

Questions

Q1 - Which are the main benefits/advantages that are provided by this proposal with respect to existing works in the literature? Q2 - Does the approach use the conventional Wassterstein-ball based uncertainty set or is the uncertainty set considered the one from equation (16)? Is there any equivalence between them? Q3 - Recognizing that assuming knowledge of the underlying causal structure for a given classification task is generally unrealistic, the authors propose an alternative approach that operates under more realistic conditions, where only a set of samples/instances is available. They offer guarantees as long as the assumptions outlined in Assumption 2 are met. However, it is unclear how realistic these assumptions are. Are they typically satisfied in real-world applications? Can you provide real examples where these assumptions hold true? Q4 - Could this method be employed in classification tasks beyond tabular data? Q5 - What do you mean by ‘We further estimate the regularizer in more general cases and explore the relationship between DRO and classical robust optimization.’? Where is this claim validated in the main text?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors discuss the limitations of their method in Section 6.

Summary

This paper uses wasserstein distributionally robust optimization to address individual fairness concerns with causal structures and sensitive attributes.

Strengths

The problem is well-motivated and novel to my knowledge. The formulation is clear. The solution is novel.

Weaknesses

It does not seem easy to scale up this method. minor issue: the DRO objective in line 142 should be sup_{Q in B_delta(P)} En_Q [l(Z,theta)].

Questions

How are the causal relationships determined for the experiments? If one assumes no causal relationship (e.g. i.i.d. formulation), does that impair the performance significantly?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

adequately addressed

Summary

This submission studies the connection between Wasserstein Distributionally Robust Optimization (DRO) and individual fairness in certain Structural Causal Models (SCMs). Namely, it is first shown that, in the case that the SCM at hand is an Additive Noise Model (ANM) with known structural equations, one may define a Causally Fair Dissimilarity Function (CFDF) on the feature space in a canonical manner. With this, the remainder of the paper concerns the problem of DRO of the risk function (i.e. minimization of $\mathcal R_{\delta}(\mathbb P,\theta)$ the worst-case risk over all distributions which lie in a (Wasserstein) ball of radius $\delta$ from $\mathbb P$). Notably, a dual form for $\mathcal R_{\delta}$ is provided by extending known results; this effectively converts the infinite-dimensional maximization problem defining $\mathcal R_{\delta}$ to a finite dimensional minimization problem. In the case that the center $\mathbb P$ for the Wasserstein ball in $\mathcal R_{\delta}$ is an empirical measure from $N$ samples, $\mathbb P_N$, it is shown that, under certain assumptions on the loss function, $\mathcal R_{\delta}$ can be recast exactly in terms of the standard empirical risk or the objective from the counterfactually robust optimization problem depending on the size of the diameter of the set of sensitive attributes. Under weaker assumptions on the loss and that the SCM is linear, a first order (in $\delta$) expansion of $\mathcal R_{\delta}$ is also provided. Next, it is demonstrated that standard adversarial optimization can be used to approximate DRO. Finally, the rate of convergence of $\mathcal R_{\delta}(\mathbb P_{\star},\hat \theta_N^{\mathrm{dro}})$ to $\inf_{\theta\in\Theta}\mathcal R_{\delta}(\mathbb P_{\star},\theta)$ is characterized. The paper concludes with a numerical study of the described causally fair DRO (CDRO). Namely, a comparison between the CDRO and other common approaches is provided on real-world and synthetic datasets. It is shown empirically that the CDRO exhibits slightly lower accuracy than the other models, but yields a lower unfair area (this is especially evident in the COMPAS and LIN datasets).

Strengths

The paper is well-written and its contributions are clearly identified relative to the existing body of work. Although the connection between DRO and individual fairness has been considered before (in the linear SCM case), I believe the extension to the ANM case is of interest. Furthermore, the section on duality and corresponding representations of $\mathcal R_{\delta}$ provide a nice interpretation nice interpretation for this approach.

Weaknesses

1. It is difficult to get a sense for how strong some of the assumptions made in this work are. Although most of the assumptions are coupled with some examples of cases where they apply, I believe it would be relevant to provide some rationale for why these assumptions are necessary or describe primitive classes of examples where these assumptions hold rather than just some specific examples. 2. Assumption 2 (iii) requires estimation of the CFDF; it would be useful to expand a bit more on this assumption keeping in mind the above point or at least provide some heuristics for what rates one can expect in general. 3. More generally, the derived results would benefit from some additional discussion regarding their implications.

Questions

1. I believe there is a small mistake regarding assumptions (i)-(ii) and the given examples. Notably the quantile loss is 1-Lipschitz, but $|h(t_0+t_k)-h(t_0)|=|\gamma t_k|$ if $t_0\geq 0$ or $|(\gamma-1)t_0+\gamma t_k|$ otherwise. In either case the limit from assumption (ii) is $\gamma\neq 1$. Perhaps there is a sign mistake? 2. In Theorem 4 it is stated that the necessary condition for the existence of an infinite DRO solution is that [...]. Should it not be a finite DRO solution?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the limitations of their work in the conclusion and the broader impact is addressed in the appendix.

Summary

This paper proposes a novel framework called Causally Fair Distributionally Robust Optimization (CDRO) to address individual fairness in machine learning. It combines causal modeling with distributionally robust optimization, using a causally fair dissimilarity function (CFDF) to measure individual similarity while considering sensitive attributes. The framework provides a strong duality theorem, enabling efficient computation of worst-case losses under distributional uncertainty. It offers explicit solutions for the regularizer in linear Structural Causal Models (SCMs) and estimates it for non-linear SCMs, mitigating overfitting and ensuring fairness. Additionally, the framework provides finite sample guarantees for convergence even with unknown SCMs, enhancing its practicality. Empirical evaluations on real-world and synthetic datasets demonstrate CDRO's effectiveness in reducing unfairness while maintaining accuracy compared to other methods.

Strengths

- Introduces a new framework that integrates causality, individual fairness, and adversarial robustness into DRO, providing a comprehensive approach to address fairness concerns in machine learning. - Offers several theoretical advancements, including a strong duality theorem, explicit regularizer formulations, and finite sample guarantees, contributing to the theoretical foundation of fair and robust machine learning. - The framework is designed to be practical for real-world applications, even when the underlying causal structure is unknown, making it a valuable tool for addressing fairness in various domains.

Weaknesses

- Please define the abbreviation SCM before its first use in line 49. - The experimental setting description could be improved. Consider providing an algorithm for the proposed approach to enhance clarity. - The paper appears to have considerable overlap with Ehyaei et al. (https://arxiv.org/pdf/2310.19391): - The first contribution claimed in Section 1.1 (line 64) seems to have been previously established by Ehyaei et al. - Definition 1 and Proposition 1 in Section 3 appear to closely resemble Definition 2 and Proposition 1 in Ehyaei et al. Please clarify the novelty of these elements. - The framework's reliance on an additive noise model assumption may limit its applicability in complex real-world scenarios. Could you discuss potential impacts on CFDF accuracy and fairness guarantees, and any plans to address this limitation?

Questions

Given that the proposed approach demonstrates lower prediction accuracy compared to existing methods, could the authors provide insights into potential factors contributing to this outcome? Additionally, how might this trade-off between prediction accuracy and other performance metrics be justified in the context of the method's overall objectives?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Reply to rebuttal

I thank the authors for their clarification. I would like to maintain my score.

Request for Reviewer Engagement During Rebuttal Process

We respectfully wish to express our concern regarding the lack of response from one of the reviewers during the rebuttal process. We invested significant effort in thoroughly addressing the concerns raised, providing a detailed response of approximately 6,000 characters to ensure that each issue was adequately covered. We were hopeful that the reviewer would engage with our rebuttal, as this dialogue is crucial for ensuring that all points are fully understood and resolved. Given the importance of this interaction, we kindly ask whether the reviewer feels that their concerns have been satisfactorily addressed or if there are any remaining questions that we can clarify during this stage. We believe that this feedback loop is essential to the integrity of the review process and the fair evaluation of our work.