Summary
This paper introduces a method for fair representation learning based on Partial Least Squares (PLS). The proposed approach employs supervised learning, requiring input data along with sensitive attributes $S$ and target label $Y$. The goal is to project the input data into a $k$-dimensional subspace that maximizes the covariance with $Y$ while minimizing the covariance with $S$. The method is implemented as a linear projection optimized via gradient descent and is further extended to non-linear kernel projections using Hilbert-Schmidt Independence Criterion (HSIC).
Weaknesses
I have the following questions and concerns regarding the contribution and evaluation of the work:
1. **Motivation for PLS-based Framework:** It would be helpful for the authors to clarify the motivation behind selecting PLS as the foundation for their framework. There are various types of approaches for fair representation learning, such as adversarial learning [1], disentanglement [2], and distribution alignment [3]. What are the specific advantages of a PLS-based approach in comparison to these methods?
2. **Applicability and Practical Constraints:** The proposed method requires annotations for both the target label $Y$ and the sensitive attribute $S$, which can limit practical applications. Compared to existing approaches in unsupervised fair representation learning or those that do not rely on sensitive attribute annotations (such as [4]), it would be beneficial for the authors to further clarify the unique advantages of their method, potentially in terms of efficiency, theoretical guarantees, or effectiveness.
3. **Evaluation and Comparison:** The evaluation of the proposed method is based on relatively small datasets and lacks comparisons with related approaches. The current experiments seem to focus on applying the learned representations to various classifiers (like in Figure 1) rather than comparing with alternative representation learning methods. The lack of thorough evaluation and comparison makes it challenging to validate the effectiveness of the proposed method.
4. **Extension to LLMs:** It's helpful that the authors discussed in Section 4.3 about extending their method to Large Language Models (LLMs), where their method could decompose the CLS-embedding from a transformer encoder for fairness constraints. However, it's suggested that the authors could provide a more detailed discussion (with math formulation) in this section and conduct experiments to validate this extension.
5. **Coupling Issue in Fair Representation Learning:** Fair representation learning often involves a trade-off between fairness constraints and downstream task performance. It would be insightful if the authors could discuss how their method might address or mitigate this issue.
6. (Minor point on notations) In the previous context, the authors use x to refer to the input data, but in Section 4.3, x refers to the transformed data in the latent space. The authors may use a different symbol to avoid confusion.
[1] Madras, David, et al. "Learning adversarially fair and transferable representations." International Conference on Machine Learning. PMLR, 2018.
[2] Balunovic, Mislav, Anian Ruoss, and Martin Vechev. "Fair Normalizing Flows." International Conference on Learning Representations. 2022.
[3] Creager, Elliot, et al. "Flexibly fair representation learning by disentanglement." International conference on machine learning. PMLR, 2019.
[4] Chai, Junyi, and Xiaoqian Wang. "Self-supervised fair representation learning without demographics." Advances in Neural Information Processing Systems 35 (2022): 27100-27113.