Summary
This paper addresses the crucial issue of out-of-distribution (OOD) generalization in machine learning models, where the testing data distribution differs from the training data. To tackle this limitation, the authors introduce the concept of the Probability of Sufficiency and Necessary Causes (PNS), representing the likelihood of a feature being both necessary and sufficient. They propose a novel optimization approach, PNS risk, to enhance representations with high PNS values, effectively capturing the information of sufficient and necessary causes. The paper provides theoretical analysis and experimental results on synthetic and real-world datasets.
Strengths
* The authors propose a novel PNS risk and build the theoretical connections between PNS risk and generalization errors under some assumptions.
* An approximate algorithm is derived to optimize the proposed risk.
* Experimental results validate the effectiveness of PNS risk.
Weaknesses
I have the following comments:
* The difference between the proposed PNS risk objective and other invariant learning methods: Although the PNS risk objective is newly incorporated in invariant learning, I would like to know the key difference from existing metrics/objectives. Since the objective is also to minimize the upper bound of PNS. I am confused about this.
* The exact meaning of the derived bounds: In Equation (7), the authors provide an upper bound of the generalization error on the target distribution. I have some concerns about the RHS: when $k\rightarrow \infty$, $\beta_k$ will be extremely large, and it seems that the proposed method could not address the covariate shift problem. Therefore, I double the real effect of this bound.
* Approximation of algorithms: In algorithms, conditional distributions are parameterized by some parameters, could you analyze the approximation error of such parameterizations? It seems that this is a very theoretical work, but there is a gap between theories and algorithms. Also, the authors do not mention how to calculate $\mathbb E_{c\sim P(C|X)}$, does this require multiple implementations $C$ for one given input $X$? Also, in Equation (8), why the minimization is taken over $\phi, w$?
* Mismatch between algorithmic design and experiments: I think this algorithm & theories are not designed for covariate shifts, which could be seen from the remaining $\beta_k) term in Theorem 3.2. But for image classification tasks, the shift patterns are mainly covariate shifts, since one image is not likely to be given different labels in different domains. Also, for image datasets, the supports of different domains are likely to be disjoint, then why the proposed method could achieve good performance in these settings, which are not well-supported by theoretical analysis.
Questions
Please refer to weaknesses.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Please refer to weaknesses.