Fair Canonical Correlation Analysis

Summary

This paper proposes a fair CCA algorithm that aims to find fair CCA projection matrices. The authors claim that it is necessary to develop an appropriate algorithm for CCA with fairness guarantee. In the presence of sensitive attribute and unfairness in observed data, the proposed algorithms successfully reduces certain unfairness of the learned projection matrices. Specifically, the two proposed algorithms, MF-CCA and SF-CCA, are optimized using gradient descent algorithms on Stiefel manifolds. The convergences of these algorithms are also theoretically guaranteed, and several experiments conducted provide empirical support for the proposed algorithms.

Strengths

- The paper is generally well-written and easy to follow in overall. - To the best of my knowledge, this work is the first to address the issue of unfairness issue in CCA, which could be a milestone. - The fairness metric targeted (i.e., Correlation Disparity Error in Definition 1) is well-defined and fits well with other general fairness notions used in fair prediction tasks (e.g., demographic parity). - The proposed algorithms (MF-CCA and SF-CCA) align well with the theoretical studies presented (Theorems 4 and 5). - The roles of the two algorithms well-specified. MF-CCA finds the optimal solution by minimizing CCA error and unfairness losses simultaneously, while SF-CCA provides an advantage in controlling the trade-off between CCA error and unfairness losses. - Empirical results validate the efficacy of MF-CCA and SF-CCA, not only on synthetic data but also on real datasets.

Weaknesses

- There is no theoretical guarantee for fairness provided. If the authors could theoretically demonstrate that the solutions to equations 7 and 9 have low $\mathcal{E}^{k} (U, V),$ as has already been empirically shown, the contribution would be more novel. - To illustrate the ability of SF-CCA in controlling the trade-off between error and fairness, a visualization such as Pareto-front lines (commonly used in fair classification problems [1, 2]) would be beneficial. Table 1 only presents results using a single $\lambda$ selected from the set [1e-2, 1e-1, 1, 10, 100]. [1] https://arxiv.org/abs/1802.06309 [2] https://arxiv.org/abs/2103.06503

Questions

- Which penalty function, $\phi$, was used in the experiments? - Is the computation time more significantly impacted by the size of the training data or the dimension of the input feature? - Could the authors provide the definition of a 'componentwise Lipschitz continuous function' as stated in Assumption A?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Unlike the vanilla CCA which solves the objective by optimizing two matrices, $U$ and $V$, the proposed algorithms require more matrices to be optimized, the number of which increases with the number of sensitive attributes, $K.$ - Naturally, the computational cost of these proposed algorithms is higher than that of the vanilla CCA due to the fairness objectives they must minimize. Any future work aimed at reducing this computational cost would certainly be novel.

Summary

This paper addresses fairness and bias in Canonical Correlation Analysis (CCA). The authors propose a framework that minimizes correlation disparities associated with protected attributes, reducing unfairness without compromising accuracy. Experimental evaluation validates the effectiveness of the approach. The findings emphasize the importance of fairness in CCA applications.

Strengths

- Novel Contribution: The paper introduces a framework to address fairness and bias concerns in Canonical Correlation Analysis (CCA), making a valuable contribution to the field. - Practical Relevance: By focusing on CCA, a widely used statistical technique, the paper addresses a real-world problem and emphasizes the importance of considering fairness in data analysis. - Experimental Validation: The authors conduct experiments on both synthetic and real-world datasets, providing empirical evidence of the effectiveness of their proposed framework in reducing unfairness without compromising the accuracy of the CCA model. - Clear Presentation: The abstract provides a concise overview of the paper's objectives, approach, and findings, making it easy to understand the key contributions of the research.

Weaknesses

- Lack of Detailed Metrics: The paper could benefit from providing more specific details about the metrics used to measure unfairness and bias in CCA. This would enhance the transparency and reproducibility of the experimental evaluation. - Limited Comparison: The paper does not explicitly compare the proposed framework with existing fairness-aware CCA methods. Including such comparisons would provide insights into the relative performance and advantages of the proposed approach. - Scope and Generalizability: While the paper addresses fairness concerns in CCA, the focus is limited to this specific technique. It would be beneficial to discuss the potential implications of the findings for other statistical methods or broader machine learning applications.

Questions

- In the experimental setup, what considerations were made in selecting the synthetic and real-world datasets? Were there any specific characteristics of these datasets that influenced the results or generalizability of the findings? - While the paper focuses on fairness and bias in CCA, could you elaborate on the potential implications of the findings for other statistical methods or broader machine learning applications? How transferable do you believe the proposed framework is beyond the scope of CCA? - Are there any additional factors or considerations that should be taken into account when applying the proposed framework in practical settings? For instance, how would the framework handle missing data, outliers, or high-dimensional datasets? - Given the goal of reducing unfairness, how does the proposed framework balance the trade-off between fairness and overall predictive accuracy? Were there any cases in the experiments where the framework significantly sacrificed accuracy to achieve fairness?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

- Fairness Metrics: The paper could provide a more detailed discussion on the fairness metrics used to evaluate the proposed framework. Further elaboration on the choice and justification of these metrics would enhance the clarity and interpretability of the experimental results. - Generalizability: The generalizability of the findings may be limited by the specific characteristics and distribution of the datasets used in the experiments. The authors could discuss the potential challenges or variations that may arise when applying the framework to other datasets or domains. - Trade-off between Fairness and Accuracy: The paper briefly mentions that the proposed framework minimizes unfairness without compromising the accuracy of the CCA model. However, a more in-depth analysis of the potential trade-off between fairness and accuracy would provide a clearer understanding of the framework's limitations and its impact on prediction performance. - Real-world Application Challenges: While the framework demonstrates effectiveness in reducing unfairness, the paper does not extensively discuss the practical challenges that may arise when applying the proposed approach to real-world scenarios, such as handling missing data, complex feature distributions, or scalability issues.

Summary

This paper investigates the concept of Fair CCA, focusing on addressing the potential bias that arises when analyzing the relationship between two sets of variables using CCA, a widely utilized statistical technique. The conventional application of CCA fails to account for the impact of sensitive attributes like gender or race, leading to potential biases. In response, this study aims to bridge this gap by integrating fairness principles into CCA. The authors introduce the fairness issue within the context of CCA and propose two distinct methods to tackle it: a multi-objective approach and a single-objective approach, each offering unique strengths. The effectiveness of the proposed methods is substantiated through empirical and theoretical analyses, confirming their value in addressing the fairness concerns in CCA.

Strengths

1. The problem addressed in this paper holds significant importance. Given the increasing influence of machine learning algorithms and methods on individuals and society, it becomes crucial to delve into the study of fairness within this domain. By mitigating bias issues in machine learning, we can contribute to a more equitable outcome and benefit vulnerable groups. While numerous works have explored fairness in machine learning, the majority of them focus on the supervised learning scenario. In contrast, this paper ventures into uncharted territory by examining the fairness issue in CCA, an unsupervised learning approach. This unique perspective underscores the urgency and significance of studying fairness within the context of CCA. 2. The concepts and methods presented in this paper exhibit a high degree of novelty. To the best of my knowledge, this is the first study to explore fairness within the context of CCA. Fairness, being a multifaceted concept, encompasses various definitions. In the realm of supervised learning, researchers have proposed different definitions such as demographic parity, equalized odds, and group sufficiency. Therefore, establishing a practical and reasonable definition becomes crucial. This paper introduces the notion of fairness criteria through correlation disparity error, which takes into account both global and group-wise correlations. The resulting fairness definition is intuitive and reasonable. The incorporation of fairness as additional objectives (in the multi-objective framework) or constraints (in the single-objective framework) is accomplished seamlessly, aligning with the natural progression of the problem. Furthermore, the authors introduce the Riemannian manifold in their solutions, which promotes convergence and facilitates computation, thereby introducing a novel aspect to the research. 3. The paper exhibits good writing quality, characterized by clarity and soundness. It effectively guides readers through its content, ensuring easy comprehension from the motivation and definition to the methods and solutions. Notably, Figure 1 provides a clear and intuitive visualization that enables immediate understanding of the proposed method's functionality. The effectiveness of the methods is supported by robust experimental results on synthetic and real data. Furthermore, Figure 3 serves as a compelling validation of fairness, as it visually demonstrates the improved proximity between the two groups after the projection using the proposed methods. Overall, the paper is meticulously crafted, maintaining a high level of clarity and rigor throughout.

Weaknesses

1. Limited discussion on multiple modalities: While CCA is not restricted to two modalities, the paper primarily focuses on this scenario. It would be beneficial to discuss a more general setting involving multiple modalities and computing correlations under the fairness setting. 2. Figure 4 lacks an obvious trend: The authors could consider including Figure 9 from the supplementary file in the main body, as it provides a more intuitive demonstration of the method's effectiveness.

Questions

1. Clarification on "critical Pareto" (line 164): The definition of "critical Pareto" is unclear. It would be helpful to provide a more intuitive explanation for better understanding. 2. Elaboration on optimization problem (8) and steepest descent direction (lines 174-175): The statement regarding the optimization problem and steepest descent direction requires further elaboration to enhance clarity.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The method is currently limited to two modalities. Even though this is the most common scenario in CCA, it would be interesting to see how the method can be extended to more than two modalities. Overall, this is a well-written and informative paper that makes a significant contribution to the field of machine learning. The proposed methods are novel and effective, and the experimental results are convincing. However, the method is currently limited to two modalities, and it would be interesting to see how it can be extended to more than two modalities.

Summary

This paper addresses a fairness issue that arises in CCA, proposing a fair CCA that well trade-offs correlation disparity errors w.r.t. sensitive attributes against correlation w.r.t. global projection subspaces. It introduces two optimization frameworks (multi-objective and single-objective), then developing corresponding efficient algorithms based on the generalized Stiefel manifold together with convergence analysis. Experimental results both on synthetic and real datasets are provided to validate the theoretical findings.

Strengths

S1. The paper is very well written. Many illustrations (e.g., Fig. 1) are greatly helpful in grasping the ideas. S2. The proposed optimization frameworks are convincing, and the corresponding translation techniques enable the use of efficient algorithms in the manifold literature. In addition, the theoretical analysis (Theorems 3 and 4) provides convergence guarantees of the algorithms. S3. Experimental results emphasize the efficacy of the proposed approach, and the discussions therein are insightful.

Weaknesses

W1. The translations for efficient algorithms, (8) and (9), can further be detailed for those who are not familiar with the manifold literature. W2. For Theorems 3 and 4, the proof sketches (or technical contributions if any) are preferred to be included in the main body.

Questions

See Weakness in the above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See Weakness in the above.