A Causal Lens for Learning Long-term Fair Policies

Q1

Thanks for your insightful comments and for raising this important question. In addition to the RL setting, we think our work shares a nuanced connection with [Hu & Zhang, 2022]. While we address a similar problem to that of [Hu & Zhang, 2022], we employ different methodologies that lead to novel insights. To provide a clearer comparison, let’s first explain the connection between our work and [Hu & Zhang, 2022] and then highlight the differences. In [Hu & Zhang, 2022], long-term fairness is defined as the difference in qualification rates at a future time point $t^*$, i.e., $\mathbb{E} [ P(y^{t^*}=1|x^{t^*},do(s^+) ] - \mathbb{E} [P(y^{t^*}=1|x^{t^*},do(s^-) ]$. This definition can be represented using our qualification gain function if we define: $g^{s}(x^{t},x^{t+1}) = P(y^{t^*}| x^{t+1}, do(s)) - P(y^{t^*}| x^{t}, do(s))$ and introduce a virtual initial time step with a fixed qualification gain $P(y^{t^*}| x^{0}, do(s))$. With this setup, the state value function $V_{do(\pi,s)(x)}$ becomes equal to $\mathbb{E} [ P(y^{t^*}=1|x^{t^*},do(s) ]$. This analysis demonstrates the connection between our proposed qualification gain and the qualification rate used in [Hu & Zhang, 2022]. However, our work employs a different method to analyze the relationship between long-term and short-term fairness. In [Hu & Zhang, 2022], for short-term fairness, they focus on sensitive feature unawareness in the decision (e.g., demographic parity) by analyzing the paths from the sensitive feature to the decision in the causal graph. Their conclusion is that there is an inherent conflict between long-term and short-term fairness. In contrast, our work reveals a theoretical connection between long-term fairness and benefit fairness by proposing a novel causal decomposition technique. Specifically, we examine the impact of the policy on the $outcome$ of the decision (i.e., the qualification gain in our context), whereas the decision outcome is not explicitly modeled in [Hu & Zhang, 2022] (note that the decision outcome is different from the decision itself). Regarding our findings, our analysis shows that benefit fairness may not necessarily conflict with long-term fairness, implying the possibility of reconciling short-term and long-term fairness objectives. This contrasts with the conclusions of [Hu & Zhang, 2022], thereby providing new insights into how fairness can be achieved over time without inherent conflicts. In summary, although we tackle a similar problem as [Hu & Zhang, 2022], our distinct methodologies, including the incorporation of RL settings and the causal decomposition, lead to different conclusions and novel insights. We believe these differences not only highlight the novelty of our work but also contribute valuable perspectives to the ongoing discussion on long-term fairness. For other references, performative prediction is a different framework focused on model-dependent distribution shifts in machine learning models. Although it is a powerful approach that can offer theoretical convergence guarantees, it has its limitations. For instance, due to its complexity, it often considers decoupled performative risk, where the data distribution is influenced by parameters different from those being optimized. Equal improvability investigates a related but different problem setting, where the model learns to apply additional efforts to improve individuals' qualifications. After reviewing these papers, we believe that our work offers unique contributions to the field of long-term fair machine learning.

Q2

The Spurious Policy Effect (SPE) is inherently independent of the policy $\pi$ and, consequently, remains influenced by fairness mechanisms. Regarding the Indirect Policy Effect (IPE), its lower sensitivity to fairness mechanisms may arise from the inherent uncertainties within the environment, making IPE more challenging to optimize effectively.

Q3

Regularization is a well-established and easy-to-implement technique for formulating objective functions. It is widely used in the literature, including in the works you cited (e.g., [[Jia et al., 2024, Guldogan et al., 2022]). Regarding our work, one way to interpret it is as a re-weighting of the total reward based on the penalty for violating benefit fairness. This formulation encourages the model to satisfy fairness constraints, leading to more equitable outcomes while still optimizing performance.

Q4

The reason we didn’t include the error bars in Figures 3 and 4 is that they would make the figures unreadable. We have re-plotted the figures with the error bars, which are shown in Figure 9 in the appendix of the revised PDF of the paper.

W1

For the penalty coefficients $\beta^{KL}$ and $\beta^{\Lambda}$, the former influences the variance of the model, while the latter affects the strength of enforcing benefit fairness. We have conducted additional experiments to empirically evaluate their effects, as shown in Figures 7 and 8 in the appendix of the revised PDF of the paper. Notably, in Figure 8, we observe that the loan rate for $\beta^{\Lambda} = 2$ is more balanced than that for $\beta^{\Lambda} = 0$, further implying that benefit fairness may not conflict with long-term fairness.

W2

We agree that our proposed method does not include a formal guarantee for achieving long-term fairness like the Pareto optimality, largely due to the complexity of analyzing the convergence properties of complex reinforcement learning algorithms like PPO.

W3

Thank you for your comments. We agree that performative prediction is an important framework for studying dynamic systems. However, we would like to clarify that performative prediction is a different framework focused on model-dependent distribution shifts in machine learning models. Although it is a powerful approach that can offer theoretical convergence guarantees for algorithms like Repeated Loss Minimization, it has its limitations. For instance, due to its complexity, it often considers decoupled performative risk, where the data distribution is influenced by parameters different from those being optimized. Compared with performative prediction, we believe that our work also has its own merits in advancing the discussion on long-term fairness.

W4

The connection between long-term fairness and conventional fairness metrics like demographic parity (DP) and equal opportunity (EO) has been examined in previous theoretical studies, e.g., [1]. These works generally conclude that enforcing DP or EO can negatively impact long-term fairness in typical scenarios. Our analysis, on the other hand, reveals that benefit fairness does not necessarily conflict with long-term fairness, suggesting that it may be feasible to align benefit fairness with long-term objectives. [1] Zhang, Xueru, et al. "How do fair decisions fare in long-term qualification?." Advances in Neural Information Processing Systems 33 (2020): 18457-18469.

Q1

We appreciate the reviewer’s question regarding our choice to use a fairness penalty instead of re-weighting in the optimization problem. In fact, incorporating a fairness penalty into the objective function can be viewed as a form of re-weighting applied to the overall reward. Traditional re-weighting methods typically adjust the weights of individual rewards, which can be challenging to implement in our case when distributing fairness constraints across each individual component. Therefore, we opt to use a fairness penalty to effectively integrate fairness constraints directly into the optimization process, ensuring a more cohesive and manageable approach to achieving fairness.

Q1

Thanks for your comments. I believe you are referring to Figures 3 and 4. Figure 3 presents the utility comparison. The absolute value of the utility depends on the utility function. Specifically, in our context, the bank’s initial cash and the interest rate. In our experiments, we set the initial cash to \\$10,000 and the interest rate to 10\%. That is why the difference between the worst and best approach is a mere 0.2%. We can easily increase the relative difference between different methods to more than 100\% by setting the initial cash to \\$0.

Q2

[1] is one of the earliest works in studying long-term fairness in MDP. While both our work and [1] address fairness in dynamic decision-making systems, there are significant and nuanced differences in the fairness notions we employ. In [1], fairness is defined as the equality of the cumulative "rewards" received by different groups. In contrast, our work decouples the reward from the qualification gain, focusing on the individual's progression. For example, in a lending scenario, the reward is the profit the bank earns, while the qualification gain is defined as the increase in the individual's credit score. This distinction is crucial because it highlights the trade-off between the utility of the decision-maker and the long-term advancement of the individuals affected by the decisions. By separating these two aspects, our framework can more effectively address systemic inequalities and focus on the intrinsic connection between long-term and short-term fairness. [1] Wen, Min, Osbert Bastani, and Ufuk Topcu. "Algorithms for fairness in sequential decision making." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

W1&2

Thank you for recognizing that our method aligns with well-accepted methodologies. However, extending causal decomposition to the MDP framework is not trivial. The causal decomposition of the return or advantage function is a non-trivial problem, as highlighted in recent research [1]. In our context, existing methods, either in causal fairness for static settings or in advantage causal decomposition, cannot be directly applied because our objective is to elucidate the intrinsic connection between long-term and short-term fairness and to derive insights that inform algorithm design. To address this gap, we have introduced a novel decomposition of long-term fairness into direct and delayed impacts, which not only advances theoretical understanding but also provides practical guidance for algorithm development. [1] Pan, Hsiao-Ru, and Bernhard Schölkopf. "Skill or Luck? Return Decomposition via Advantage Functions." The International Conference on Learning Representations (2024).

W3

While we acknowledge that our proposed method does not include a formal guarantee for achieving long-term fairness, which is largely due to the complexity of analyzing the convergence properties of complex reinforcement learning algorithms like PPO, we believe our work makes significant contributions in other critical areas. Specifically, our research provides a novel perspective by unveiling the intrinsic connection between long-term and short-term fairness in reinforcement learning, a link that has not been thoroughly explored in existing literature. Regarding the mention of performative prediction by some reviewers, we would like to clarify that performative prediction is a different framework focused on model-dependent distribution shifts in machine learning models. Although it is a powerful approach that can offer theoretical convergence guarantees, it has its limitations. For instance, due to its complexity, it often considers decoupled performative risk, where the data distribution is influenced by parameters different from those being optimized. Compared with performative prediction, we believe that our work also has its own merits in advancing the discussion on long-term fairness.

Q1&2

Q1 Thanks for pointing out this issue. It might be a slight misuse of notations but not a typo. The two x’ have different meanings. To avoid confusion, we have revised one of the x’ to x’’. Q2 No, the Indirect Policy Effect (IPE) cannot be expressed in terms of benefit fairness. This is one of the insights gained from our analysis, which shows that the IPE is less sensitive to benefit fairness, and enforcing benefit fairness may not result in a significant change in the IPE.