Enhancing node-level Out-Of-Distribution (OOD) generalization on graphs remains a crucial area of research. In this paper, we develop a Structural Causal Model (SCM) to theoretically dissect the performance of two prominent invariant learning methods -- Invariant Risk Minimization (IRM) and Variance-Risk Extrapolation (VREx) -- in node-level OOD settings. Our analysis reveals a critical limitation: due to the lack of class-conditional invariance constraints, these methods may struggle to accurately identify the structure of the predictive invariant ego-graph and consequently rely on spurious features. To address this, we propose Cross-environment Intra-class Alignment (CIA), which explicitly eliminates spurious features by aligning cross-environment representations conditioned on the same class, bypassing the need for explicit knowledge of the causal pattern structure. To adapt CIA to node-level OOD scenarios where environment labels are hard to obtain, we further propose CIA-LRA (Localized Reweighting Alignment) that leverages the distribution of neighboring labels to selectively align node representations, effectively distinguishing and preserving invariant features while removing spurious ones, all without relying on environment labels. We theoretically prove CIA-LRA's effectiveness by deriving an OOD generalization error bound based on PAC-Bayesian analysis. Experiments on graph OOD benchmarks validate the superiority of CIA and CIA-LRA, marking a significant advancement in node-level OOD generalization. The codes are available at https://github.com/NOVAglow646/NeurIPS24-Invariant-Learning-on-Graphs.
Paper
Similar papers
Peer review
Summary
The authors propose a novel regularization and alignment mechanism for invariant graph learning with the goal of obtaining better out-of-distribution (OOD) generalization. Besides proving that existing methods tackling OOD generalization in other domains do not transfer to graphs, they propose CIA (environment-label dependent) and CIA-LRA (environment-label independent) to tackle the challenge of OOD generalization on graph data. Both, theoretical and empirical evidence is provided showing that the proposed methods provide benefits in OOD generalization.
Strengths
- theoretical derivation of failure of IRM and VREx on graph data - theoretical analysis of CIA an CIA-LRA - empirical evidence that the proposed methods achieve OOD generalization significantly better than IRM and VREx - detailed discussion of further findings and proofs in appendix
Weaknesses
## Methodology - in Fig. 1 it is not clearly described what the variables exactly refer to (e.g. $G$ is not mentioned in the text) - while I appreciate the theoretical findings, the GNN defined in Eq. 3 is a linear model, hence the theoretical analysis is somewhat limited. However, one should keep in mind that the analysis of non-linear and highly complex networks is hard - line 204: it is a bit counter-intuitive why invariant node features of nodes with the same class should significantly diverge when they are far apart in the graph. Also the empirical evidence in Fig. 7/8 does not really support this statement. - a brief definition/discussion of concept shift and covariate shift would be beneficial for clarity - in line 232: why is a sum operator employed here instead of a mean? Won't a simple sum be highly influenced by the graph size and connectivity/density? - line 248: inconsistent naming ($x_{cau}$ and $x_{inv}$) - in theorem 4.2 it is unclear what $\alpha$ is and what the inequation $0 < \alpha < \frac{1}{4}$ means. Further a bit of a discussion on the goodness/tightness of the given bound would be good to assess the usefulness of that bound - proof G 1.1.: the proof only shows that $\theta_2 = 0$ is **a possible** solution, but not a unique solution, isn't it? (Didn't check the proofs too deeply though) - Eq. 34: the entire term does not depend on $\theta_2$, so it is unclear to me why $\theta_2 = 0$ is **the** solution to that equation. Wouldn't any arbitrary value be a valid solution then? - Eq. 35: in App. F.1 it is stated that the expectation of $\epsilon^e$ is 0. Looking at Eq. 35, won't all mutliplicative terms entailing $\epsilon^e$ cancel out in the expectation and thus in the IRM loss? This would make the IRM loss independent of the environment **in expectation** (not for some specific environment though). Thus, I don't see how the conclusion is reached that the IRM loss depends on $\epsilon^e$ if, in expectation, it is 0. - line 995: which property are you referring to exactly? ## Experiments - since the authors present an alignment-based invariant learning strategy for graph data, it would be good to see some more recent baselines taking a similar approach such as [1] and [2] - line 296: why does CIA-LRA improve upon CIA? To me it is counter-intuitive since CIA-LRA has no access to the environment labels, thus I'd expect this task to be harder ## General notes - in the Introduction there should be more on related methods based on representation alignment such as [1] and [2] ## References [1] Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data. Zhu et al. NeurIPS 2021. [2] Stable Prediction on Graphs with Agnostic Distribution Shifts. Zahng et al. KDD 2023.
Questions
- how often did you repeat the experiments? (Maybe it's written somewhere and I didn't see it)
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
- major parts of the analysis are rather limited as they rely on the definition of the GNN in Eq. 3 which is a linear model - a comparison to some more recent baseline would increase the value of the paper
Summary
The paper investigate the failure case of IRMv1 and VRex for graph data, and develop a novel approach CIA for node-level OOD generalization. To adapt for the scenarios without environment labels, the authors propose CIA-LRA for the datasets without environment labels.
Strengths
1. The paper is well-written and provides clear motivation. 2. The paper is solid in both theoretical and empirical aspects. 3. The evaluations are fair and convincing.
Weaknesses
1. The author may want to clarify the reasoning behind the two SCMs corresponding to the two types of distribution shifts, as this connection is not entirely clear. 2. In Equation 9, it should be $\phi_\theta(j)$. 3. Assuming the sample label corresponds to the same invariant factors, it is unclear why $r_{\text{same}}$ is necessary for reweighting. 4. It appears that CIA-LRA requires all labels for reweighting alignment. However, the typical node-level classification setting is semi-supervised, which limits the applicability of this method. 5. Line 231: "to pairs with" should be corrected to "to pair with". 6. Line 854: In the proof of the non-graph case, $(A^e)^k$ appears, which seems to be an oversight. 7. The general idea of sampling nodes from the same target label and different environments is similar to CIGA [1], which performs supervised contrastive learning to identify invariant subgraphs. The objective in CIGA may be a specific form of $d(\cdot)$. The authors may want to compare and contrast their approach with CIGA. __reference__ [1] Chen et al., Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs, Neurips 2022.
Questions
1. CIA-LRA learns masked subgraph, is the experiment inductive or transductive? 2. In line 318-321, why CIA will be less performant than CIA-LRA, it seems not reasonable, as CIA will adopt environment labels while CIA-LRA doesn’t. 3. $\theta_m$ is separated from $\theta$, how to ensure it will learn the invariant subgraph, under what assumptions?
Rating
5
Confidence
2
Soundness
3
Presentation
4
Contribution
2
Limitations
N/A
I appreciate the reviewer’s great efforts for the response. However, I still have some concerns. > Q9 I understand the author’s explanation regarding concept shift and covariate shift. However, I believe that the SCM presented in Figure 1 merely represents an underlying data generation process, which may not directly correspond to the types of distribution shifts. This could be potentially misleading. > Q10,12 * Although there is no edge from C to S in Figure 1b, according to the definition of FIIF, the condition $I(S;Y|C)=0$ still holds. * FIIF and PIIF represent data-generating processes, and therefore do not directly correspond to concept shift. * CIGA primarily addresses covariate shift, where the conditional distribution $P(Y|G)$, i.e., the underlying mechanism, remains unchanged, while $P(G)$ shifts due to spurious features. * I recognize the contribution of this work in addressing excessive alignment. However, the basic idea aligns with supervised contrastive learning, which is why I perceive similarities with previous studies like CIGA. * The utilization of techniques such as neighborhood label distribution is indeed novel, but they are not directly applicable to graph-level OOD. This is why I consider CIA-LRA to be an adaptation of supervised contrastive learning for node-level OOD. * Overall, I remain positive about this work due to its high-quality writing and rigor. However, I also have concerns about the level of novelty in the proposed method. Additionally, some design choices in the methodology appear heuristic (in Sec. 3.2), which also raises some concerns.
We thank reviewer yNy3 for carefully reading our response. We want to address your further concerns as follows: **** Q13: **However, I believe that the SCM presented in Fig. 1 merely represents an underlying data generation process, which may not directly correspond to the types of distribution shifts.** A13: Thanks for your advice! We acknowledge that Fig.1 are indeed data generation processes, but our deduction in A1 has demonstrated that under the data generation process in Fig. 1 (a)/(b), the concept/covariate shift will occur. The potential misleading could be that the current caption of Fig.1 is 'concept shift' and 'covariate shift'. We'll modify it to be the "underlying data generation process leading to concept/covariate shift" and add the derivation in A1 to our paper for clarity. Q14: **Although there is no edge from C to S in Figure 1b, according to the definition of FIIF, the condition $I(S;Y|C)=0$ still holds.** A14: We acknowledge that $I(S;Y|C)=0$ holds for both Fig. 1(b) in our paper and the FIIF. This means $S\perp Y|C$ holds for both FIIF and our Fig. 1(b), but it doesn't mean $S\perp C$ holds for both of them. In fact, according to the SCM of FIIF, $S$ correlates with $C$, but they are independent in Fig. 1(b). Hence, Fig. 1(b) and FIIF are fundamentally different. Q15: **FIIF and PIIF represent data-generating processes and do not directly correspond to concept shift.** A15: FIIF and PIIF are indeed data-generating processes, but we can still deduce that concept shift occurs under such data-generating processes. FIIF: under FIIF, $S=f_{spu}(C, E)$, so $p_{S|C}(s|c)$ changes across environments. Following the derivation in A1, to prove $p_{Y|X}(y|x)$ changes with environment (i.e., concept shift happens), we only need to prove $p_{Y | C, S}(y | c, s)$ is variant. $p_{Y | C, S}(y | c, s)=\frac{p_{S|C}(s|c)p_{Y|C}(y|c)p_C(c)}{p_{C,S}(c,s)}$ contains $p_{S|C}(s|c)$ in its numerator, so it also changes with environments. PIIF: The structure of PIIF is exactly the same as Fig 1(a), so the analysis in A1 can be directly applied to PIIF to show that it encounters a concept shift. Q16: **CIGA primarily addresses covariate shift, where the conditional distribution $P(Y|G)$, i.e., the underlying mechanism, remains unchanged.** A16: From the analysis in A15 we can infer that $p_{Y|X}(y|x)$ changes with environments (which is equivalent to the $P(Y|G)$ you refer to here) under the FIIF and PIIF assumption considered by CIGA, so CIGA is not guaranteed to solve covariate shift. Q17: **However, the basic idea aligns with supervised contrastive learning, which is why I perceive similarities with previous studies like CIGA. The utilization of techniques such as neighborhood label distribution is indeed novel, but they are not directly applicable to graph-level OOD. This is why I consider CIA-LRA to be an adaptation of supervised contrastive learning for node-level OOD.** A17: Although the methodology of representation alignment of CIA-LRA is somewhat similar to the supervised contrastive loss of CIGA, adapting such an idea to the node-level task is non-trivial: there may not exist a subgraph containing information about $Y$ for each node, so $\max I(\hat{G_s};Y)$ may not identify the spurious subgraph. Therefore, we propose to utilize the neighborhood label distribution to better distinguish spurious features and eliminate them. In this work, we currently focus on the challenging node-level OOD task, and we'll aim to improve graph-level OOD generalization in the future.
Thanks for the authors' efforts for constructing the response. Some of my concerns cannot be addressed in the rebuttal stage, such as: 1) in Q15, under FIIF, since $I(S;Y|C)=0$, which implies that $P(Y|C,S)=P(Y|C)$ when conditioned on $C$. 2) The claim that the approach would __solve concept shift is also problematic__. Most invariant learning methods ever since IRM would assume a stable relations between semantic objects and $Y$, hence they aim to solve covariate shift but not concept shift. One example would be a cow in a picture would always be mapped to $Y=1$, hence it is invariant; Otherwise, in concept shift, a cow would be mapped to a different label, indicating that there is no stable features, but instead, the underlying responsive mechanism changes, rather than the spurious features. The claim that this approach, which is a refinement of supervised contrastive learning, can solve concept shift is not convincing. 3) Some designs in the approach remain heuristic. For the above reasons, I won't raise my score, and won't lower my score given the strength of this work. However, I strongly suggest that the authors __reconsider the claim that CIA would solve concept shifts__, which is my biggest concern for this work.
We sincerely thank reviewer yNy3 for the effort and time during the review and discussion! We are happy to continue clarifying the following issues: **** Q18: **in Q15, under FIIF, since $I(S;Y|C)=0$, which implies that $P(Y|C,S)=P(Y|C)$ when conditioned on $C$.** A18: We acknowledge that $I(S;Y|C)=0$ holds and it leads to $P(Y|C,S)=P(Y|C)$ in both FIIF and Fig 1(b). However, this doesn't affect the fact that $C\rightarrow S$ holds in FIIF. Since $C\rightarrow S$ and $E\rightarrow S$, the spurious features $S$ will be determined by the causal features $C$, and this mapping from $C$ to $S$ changes across environments (this can also be seen from the Assumption 2.2 in CIGA: $S:=f_{spu}(C, E)$). Therefore, in FIIF a specific causal feature will correlate (for example, emerge simultaneously) with a specific spurious feature and this correlation changes across environments. However, in Fig. 1(b) of our paper, $C$ and $S$ are independent. As a result, FIIF and Fig. 1(b) represent different kinds of distribution shifts. Q19: **The claim that the approach would solve concept shift is also problematic**. A19: Sorry, it seems we may have had some misunderstanding regarding the definitions of the two types of distributional shifts. Concerning concept shift and covariate shift, we have followed the commonly used definitions in the OOD community (please refer to Sections 3.1 and 3.2 of [13]). High-level speaking, **covariate shift** refers to a situation where **spurious features have different support sets in the training and test environments, and there is no association between $Y$ and spurious features**. For example, in a binary classification problem involving cows and sheep, where the training environment has a grass background and the test environment has a desert background, the probability of cows and sheep appearing in both environments is 50%, i.e., there is no correlation between the classes and the background. **Concept shift**, on the other hand, refers to a situation **where there is an association between $Y$ and spurious features, and this association changes with the environment, while the spurious features have the same support set in the training and test environments.** For instance, there are two training environments, one with a grass background and the other with a desert background. The probability of cows appearing in the grass and desert backgrounds is 80% and 20%, respectively, and for sheep, it is 20% and 80%, respectively. In the test environment, the probability of cows and sheep appearing in both backgrounds is 50%, meaning that the spurious association between background and class changes during testing. According to the above definitions, IRM is capable of addressing the concept shift (as seen in the theoretical and experimental results of the IRM paper). Moreover, our Eq. (2) also aligns with the aforementioned definition of concept shift: there exists a stable correlation between $Y$ and the invariant features $X_1$, and there exist spurious correlations between $Y$ and spurious features $X_2$. We theoretically proved that CIA can solve this type of shift in Theorem 3.1. [13] GOOD: A Graph Out-of-Distribution Benchmark, NeurIPS 2022
Summary
This paper addresses OOD generalization in node-level graph neural networks. The authors theoretically analyze why popular invariant learning methods like IRM and VREx fail on graph data, and propose two novel solutions: CIA and its environment-label-free variant, CIA-LRA. These methods enforce intra-class similarities to improve OOD generalization. The authors provide theoretical guarantees, including an OOD generalization error bound under the PAC-Bayesian framework. Experimental results on graph OOD benchmarks demonstrate the effectiveness of the proposed methods, particularly CIA-LRA, in improving node-level OOD generalization performance.
Strengths
1. The paper provides a comprehensive theoretical investigation into why existing invariant learning methods struggle with graph data, supported by formal proofs. 2. The proposed CIA and CIA-LRA methods offer innovative approaches to tackle node-level OOD generalization, addressing limitations of previous methods. 3. The authors conduct comprehensive experiments on multiple datasets and compare against various baselines, demonstrating consistent performance improvements.
Weaknesses
1. Limited comparison to recent node-level OOD method [1]. 2. Lack of visualization of the overall method architecture to help readers understand intuitively. 3. While Appendix J briefly mentions GPU requirements, a more detailed analysis of the computational complexity and runtime comparisons of CIA and CIA-LRA against baselines would be valuable. 4. Insufficient citations of recent methods, such as [2] [3]. [1] Learning invariant representations of graph neural networks via cluster generalization [2] Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization [3] Individual and structural graph information bottlenecks for out-of-distribution generalization
Questions
N/A
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A
Looking forward to your comments
Dear reviewer LzeD, I hope this message finds you well. As the discussion deadline is approaching in a few hours, I kindly wanted to remind you of the pending feedback for our submission. We have tried our best to address the concerns and revised our paper accordingly. Your insights are invaluable to us, and we eagerly await your comments to improve our work further. Thank you for your time and consideration!
Summary
This paper analyzes the failures of standard invariant learning techniques for node classification. Through theoretical analysis, they argue that methods such as IRM and VREx will learn spurious features on graph data. Using this as motivation, they design a new method, CIA, that introduces additional invariance regularization to better guide the GNN to learn better representations. They verify the effectiveness of their method on multiple benchmark datasets.
Strengths
1. I think the authors do a good job of showing the failures of methods such as IRM and VERx on graph data. In particular, the theoretical analysis is interesting. 2. The proposed method is straightforward and follows naturally from the theoretical analysis. Furthermore, the theoretical justification in Section 4 is welcome. 3. The performance of the proposed method is good compared to baselines.
Weaknesses
1. The authors seem to leave out some baselines in their experiments. For example, see [Wu et al., 2024] and [Li et al., 2023b]. From Footnote 1 at the bottom of page 2, I surmise that this is because they are not "general solutions". However, I find this to be a weak reason for not including them as baselines. Why should that be a disqualifying measure? The authors should better motivate which baselines they compare against and which they don't. 2. Similarly, the authors should include the performance when using MatchDG. In the paper, the authors note the similarity of their method to MatchDG. As such, it reasons that it should be included in the experiments as a baseline. If not, the authors should provide sufficient reasons why to not include it. 3. There's a lack of discussion on the efficiency (time complexity) of the method. This is important, as the authors note that the performance of CIA-LRA peaks at around 6-10 hops (see line 674). I'm curious as to how efficienct it is to consider such a large number of hops, especially on larger graphs. Naively, this seems to me that it can be a drawback of applying this method on larger graphs.
Questions
1. What's IRMv1? You begin by mentioning the failure of IRM but then start calling it IRMv1. What's the difference? 2. On line 135, the authors mention that the GNN used in their analysis resembles SGC. I'm curious how this effects the analysis. How different would the conclusions be if we considered a more standard GNN like GCN? To be clear, I'm not asking for a full analysis here or anything, I'm just curious of the practical limitations of considering SGC in the analysis. 3. Could you share the time complexity of your method?
Rating
6
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Yes.
| Additional Baselines + MatchDG Thanks for the new experiments. These definitely enhance the results in the paper. | To show the running time of CIA-LRA, we show the time cost Interesting, thank you. | IRMv1 refers to the practical implementation of the original challenging IRM objective I appreciate the clarification and formulation. | When we consider a GCN rather than an SGC... Thanks for the discussion. If you have time, I think it would be worth the effort to include these results in the appendix. I appreciate the detailed response. I've raised my score to a 6.
Thank you
Thank you for raising the score! We’ll add all new results of rebuttal to our paper. Thank you again for your efforts spent and constructive comments!
Answer to Rebuttal
Thank you for the detailed rebuttal! Although I did not check A6 in detail, it explains where $\alpha$ is coming from. The rebuttal clarified all other points, and the results of the additional experiments look good. Thus, I raised my score from 5 to 7.
Thank you
Thank you for appreciating our work and rasing the score! Thank you for your time, effort, and valuable feedback during the review process!
Thank you for the response. While your reply addresses some of my questions, I have a few points of disagreement with certain aspects of your response. >**SCM and the types of distribution shifts** * For concept shift, the authors aim to show $p_X(x)$ is invariant across environments, however, $X$ depends on $S$, which is affected by $E$. Hence this claim appears to be invalid. * Similarly for covariate shift, $P(y|x)$ won't be invariant across environments, as $X$ depends on $E$. >**Comparison with CIGA** Figure 1(a) is equivalent to PIIF in CIGA, and Figure 1(a) is equivalent to FIIF, hence CIGA can solve both situations in Figure (a) and (b). >**Collapsed learning** I'm also not convinced that with environment labels, the learned representations will collapse. With environment labels, the method could facilitate better differentiation among various groups, enhancing alignment. >**Similarity with previous work** I still think this approach is quite similar to CIGA, which is an adaptation for node-level tasks, with some reweighting scheme for a single graph.
Further response to reviewer yNy3
Thank you for your further comments. We'd like to address your concerns as follows: Q9: **SCM and the types of distribution shifts** A9: Thank you pointing out the careless mistake. The accurate claim should be: **under concept shift, $p_C(c)$ is invaraint; under covariate shift, $p_{Y|C}(y|c)$ is invariant.** However, this misrepresentation doesn't affect the correctness of the derivation in A1, because it doesn't rely on the assumption that certain distribution is invariant across environments. Q10: **Comparison with CIGA** A10: I guess you might mean "Fig. 1(a) is equivalent to PIIF, Fig. 1(b) is equivalent to FIIF". However, this is not the case. Although PIIF is equivalent to Fig. 1(a) in our paper, FIIF is not equivalent to Fig. 1(b) in our paper because $C\rightarrow S$ exists in FIIF, causing $C$ to correlate with $S$. However, in Fig. 1(b), $C$ and $S$ are independent. In summary, both FIIF and PIIF represent concept shifts where invariant features correlate with spurious ones, as highlighted in Section 2.2 of the CIGA paper [1]: "$S$ is directly controlled by $C$ in FIIF and indirectly controlled by $C$ through $Y$ in PIIF". However, our paper also considers the covariate shift where $C$ and $S$ are independent. Q11: **Collapsed learning** A11: We kindly remind you that we are not saying "with environment labels, the learned representations will collapse", we actually mean **excessive alignment will lead to the collapse of the invariant features.** We use the intra-class variance of the representation corresponding to invariant features (averaged over all classes) to measure the degree of collapse of invariant features. Base on this measurement, the excessive alignment can be caused by: 1. **Using a $\lambda$ that is too large.** Evidence: on the toy dataset of Fig. 2, a larger $\lambda$ leads smaller intra-class variance of invariant representations Table D: the intra-class variance of invaraint representations at epoch 50 | CIA | $\lambda=0.05$ | $\lambda=0.1$ | $\lambda=0.5$ | | -------- | -------------- | ------------- | ------------- | | variance | 0.061 | 0.039 | 0.011 | 2. **Aligning the representations of too many nodes.** Evidence 1: we add an experiment to show that **aligning fewer node pairs can prevent the collapse of invariant representation, even with environment labels.** By modifying CIA to align local pairs (same-class, different-environment nodes within 1 hop), termed "CIA-local", the results in Table E show that when by aligning **local** pairs instead of **all** pairs, CIA-local avoids the collapse that CIA suffers and achieves better performance. Table E: accuracy and the variance of the invariant representations on the toy dataset of Fig. 2. at epoch 200 | | CIA | CIA-local | | ------------------- | ------ | --------- | | Concept, Acc. | 0.253 | 0.354 | | Concept, variance | 0.0003 | 0.2327 | | Covariate, Acc. | 0.250 | 0.312 | | Covariate, variance | 0.0002 | 0.1699 | Q12: **Similarity with previous work** A12: Additional differences between CIA-LRA and CIGA: * **CIA-LRA solves both concept and covariate shift but CIGA is only guaranteed to solve the concept shift (PIIF and FIIF).** As we pointed out in A10 and A4, CIA-LRA can solve both concept (Theorem 3.1, 4.2) and covariate shift (Propsition B.3, Theorem 4.2). CIGA relies on the assumption that the spurious subgraph $G_s$ and the invariant subgraph $G_c$ "can share certain overlapped information about $Y$" to identify them, which means it only work well under concept shift. Their theoretical guarantee is under the PIIF and FIIF assumptions (Theorem 3.1 in [1]). * **CIA-LRA utilize the fine-grained graph-specific features (neighborhood label distribution, NLD) to better eliminate spurious features but CIGA ignores such information and only consider loss-level regularization.** CIA-LRA propose to utilize the NLD to identify node pairs with large discrepancies in spurious features and similar invariant features to better eliminate the former and avoid the collapse of the latter. However, CIGA ignores such fine-grained information and solely minimize the loss of using an spurious subragph $\hat{G}_s$ to predict $Y$ ($\max I(\hat{G}_s;Y)$) and enforcing the loss of $\hat{G}_c$ is smaller than $\hat{G}_s$'s. * **CIA-LRA additionally consider the issue of the collapse of invariant features while CIGA doesn't.** We reveal the collapse of invariant features caused by excessive alignment (Fig 2 Mid and Table D, E in A11) and propose the localized alignment and $r^{same}$ to address this. CIGA focus solely on indentifying the invariant subgraph. Our work provides new insights that excessive alignment can lead to the collpase of invariant representations and hurt generalization, which can help the community better understand the dual role of representation alignment in OOD generalization.
Decision
Accept (poster)