Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization

Summary

This work addresses the issue of graph out-of-distribution (OOD) generalization by considering the data generation assumption of the presence of a causal subgraph and a spurious subgraph within each graph, and the assumption of label and environment causal independence. The paper proposes an adversarial training approach that leverages accessible environment information in the dataset. By jointly optimizing over these properties, the method aims to discover causal subgraphs for invariant prediction. Experimental evaluations on synthetic and real-world datasets demonstrate the method's superior performance compared to existing approaches.

Strengths

1. The paper is well-written, organized, and effectively conveys its ideas. A notable distinction from prior work is the assumption of access to environmental information, but the authors provide a thorough evaluation of this assumption's fairness compared to existing methods. 2. This paper introduces a novel approach to address the graph OOD problem by integrating environment and label independence, allowing for the comprehensive utilization of environmental information. The proposed method is accompanied by theoretical guarantees, enhancing its credibility. 3. The empirical evaluation is carefully designed, incorporating diverse datasets, relevant baselines, and comprehensive sanity checks. The authors further enhance their study with sensitivity analysis and ablation studies, providing a robust evaluation framework.

Weaknesses

1. The paper introduces assumptions about the data generation process but lacks a formal definition of the OOD task within the paper. It would be beneficial to clarify details such as the training data distribution, test data distribution, and the specific differences between them. Specifically, is the difference solely attributed to the environment variable E, and are new E values observed in the test distribution? Additionally, it would be helpful to address whether the support for $P^\text{tr}(E)$ and $P^\text{te}(E)$ should be the same. 2. Although the paper has theoretical analysis of the proposed approach, it does not directly connect to the OOD generalization error. It is not explicitly stated how the proposed model's performance will be in various OOD settings theoretically. 3. The paper explores three different distribution shift assumptions, but it appears to place more emphasis on the Covariate SCM assumption. In the experimental section, it would be advantageous to clarify the specific SCM that the data satisfy and provide further details regarding how the data satisfy the SCM.

Questions

1. As mentioned in the weakness section, what is the theoretical definition of the graph OOD problem that this paper is trying to tackle? 2. As far as I understand, the theoretical statements claim that if the model is optimized optimally, then the subgraph selector $f_\theta$ should select the optimal causal subgraph in the training graph. However, for the test data with a different distribution, how to guarantee the subgraph selector is still able to find the correct causal subgraph? 3. In line 160, it is said “for any $G_p$ such that $G_s\subset G_p$, $I(G_S;Y) \leq I(G_p; Y)$”. It is unclear whether this statement is an assumption or a proven fact. Since in the line 599, it is said “$I(E; G_C)\geq I(E; G_p)$ is trivial since $G_p$ is a subset of $G_C$”. Why is it trivial? If it is trivial, does it mean “for any $G_p$ such that $G_s\subset G_p, I(G_S;Y) \leq I(G_p; Y) $” is also trivial? 4. In the synthetic dataset section, it seems the dataset follows the Covariant SCM setting. Is it possible to add synthetic experiments for dataset following the FIIF and PIIF settings? 5. Is it possible to know which SCM does each real-world dataset follow? 6. Most real-world datasets show comparable performance across methods, including the ERM baseline. In many cases, the OOD validation performances are better than in-distribution performances. These observations raise questions regarding the difficulty of the OOD task investigated in this section. Can you provide any insights or explanations for these findings? 7. The experimental section primarily focuses on the GOOD benchmark dataset, where the environment information is readily available. Although the paper suggests that obtaining environment labels is feasible by applying the algorithms from GOOD to other datasets, it is worth considering the value and applicability of the environment information in datasets beyond the GOOD benchmark. Clarifying whether the proposed approach can still benefit from the environment information in datasets outside of the benchmark would enhance the understanding of its generalizability. 8. Typo: There are multiple instances of the word "casual" in the paper, which should be corrected to "causal" for accuracy and clarity.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

1. The components of the method are not particularly novel on their own since it is standard adversarial training. The theoretical guarantee of OOD generalization performance is not fully justified. 2. The accessibility and quality of environment information in general graph datasets are not thoroughly addressed in the paper. Exploring the possibility of incorporating environment information into other existing methods would provide a more comprehensive evaluation and a fairer comparison across different approaches.

Summary

This paper proposes a novel adversarial approach called LECI, for OOD generalization. The paper leverages label and environment information to identify the causal and invariant graph, thereby enhancing the accuracy and reliability of causal and invariant subgraph identification. The authors conduct extensive experiments on various real-world datasets under different distribution shift and demonstrate the effectiveness of the proposed approach.

Strengths

(1) The paper proposes an effective learning strategy to incorporate label and environment causal independence for tackling the graph OOD problem. (2) Detailed theoretical guarantees for casual subgraph discovery are provided. (3) Extensive experiments on synthetic and real-world datasets, which provide insightful empirical findings.

Weaknesses

1. Limited novelty. Since environment inference and environment exploitation are often combined, and previous methods have used environment-based approaches specifically for graphs[1-2], it is puzzling why the author propose that previous methods did not explicitly mention graph-specific designs in line 26. Moreover, if representations are obtained by both structural and feature information using message passing, it begs the question of whether invariant learning on such representations has already addressed by previous works. The novelty of the article requires more presentation. 2. Tables 1 and 2 do not include a comparison with GIL[2] as mentioned in line 266. Furthermore, based on my limited knowledge, GIL also involves capturing and optimizing invariant subgraphs, utilizing invariant learning for optimization. It is recommended to provide an explanation regarding the explainability of subgraphs between LECI and GIL. It appears that the paper requires additional elaboration on its novelty. 3.Does the presence of three relatively independent adversarial training criteria in the main model pose significant challenges to the model training, such as difficulties in convergence similar to DANN? From Figure 4 on the left, a similar trend can be observed. It seems that this model requires dedicated hyper-parameters such as learning rate and number of epochs in order to achieve better results. [1] Chen, Yongqiang, et al. "Learning causally invariant representations for out-of-distribution generalization on graphs." NIPS, 2022. [2] Li, Haoyang, et al. "Learning invariant graph representations for out-of-distribution generalization." NIPS, 2022.

Questions

see Weaknesses

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

na

Summary

The paper proposed a novel learning strategy to incorporate label and environment causal independence (LECI) for tackling the graph OOD problem. Enforcing such independence properties can help exploit environment information and alleviate the challenging issue of graph topology shifts. With good empirical results on several datasets and related theoretical analyses, the paper justifies the effectiveness of the proposed LECI.

Strengths

1. The paper proposed an effective method for tackling the graph OOD problem. The author proposes to solve the problem by introducing label and environment causal independence, which is a good idea. 2. The logic is clear and solid, including the well-defined research problem and challenges, the proposed solutions, and theoretical analysis. 3. The gained empirical improvement is quite significant with the proposed LECI framework. Several ablation studies are shown.

Weaknesses

1. I am puzzled for "graph-specific and "distribution shift on graph topology". Compared with other graph-specific OOD baselines, why LECI can solve the problem, as the author mentioned on line 19 and line 20. 2. It would be better to perform analysis on the complexity analysis, and some details are not explained so clearly of the two discriminators. 3. Due to the adversarial mechanism, the proposed method cannot scale to large-scale datasets and also it is very hard to train.

Questions

In addition to the questions given above, I also want to ask some additional questions. 4. The question is related to the adversarial mechanism. Take EA as an example. For both equations 6 and 7, the adversarial principles are easy to understand. However, From the mathmatical perspective, I am lost. For equation 6, the inner max can be naturally converted to min L_E, as that in Equation 7. In this way, with the same inner function, the outer operation in Equation 6 is min while that in Equation 7 is max. Is this a contradiction? 5. From figure 4, the curves of LECI drop first and then increase, which is marked by the effect of the two independence constraints. Why do the two independence constraints take effect too late?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors provide detailed analysis on limitations in the supplementary materials.

Summary

This paper proposes a new graph learning model to incorporate label and environment causal independence (LECI) for solving the graph OOD generalization problem. The authors first introduce two causal independence properties to distinguish causal and spurious subgraphs. Enforcing environment causal independence properties and utilizing the environment label information, LECI is able to identify causal subgraphs for invariant predictions. The authors further leverage an adversarial learning strategy to learn the label and environment casual independence. In contrast to the previous graph OOD methods which commonly assume non-existence of the environment information, LECI instead fully utilizes environment information and introduces a graph-specific environmental exploitation algorithm. Extensive experiments on GOOD and DrugOOD benchmark datasets demonstrate the superiority of LECI among other baseline methods.

Strengths

1. The illustration of graph OOD generalization from a causal perspective is clear and easy to understand, and the derived two causal independence properties could be used to distinguish between causal subgraphs and spurious subgraphs for graph OOD future works. 2. The adversarial learning implementation of LECI is technically sound, and the derivation of the adversarial objective is rigorous. 3. The theoretical results are convincing. 4. Extensive experiments on GOOD and DrugOOD benchmark datasets demonstrate that LECI significantly outperforms baselines, and the authors also conducted comprehensive experiments for sensitivity analysis and ablation studies.

Weaknesses

1. LECI assumes the environment information is known, which is a tight requirement. The authors argue that the environment labels can be accessed by applying simple groupings or deterministic algorithms, but in this case, the obtained labels are not accurate. From my perspective, these labels can be served as auxiliary information or as a proxy for ground-truth environment labels, but directly regarding them as environment labels may be not appropriate. However, I do believe LECI can be combined with the existing environment inference methods. I wonder if the authors have made some attempts. 2. The authors did not discuss the computational complexity of LECI in the paper.

Questions

1. Questions in Weaknesses part. 2. As environment labels may only be observed partially in some real-world scenarios, I wonder if LECI could utilize the observed environment labels to infer the rest of the labels using the adversarial implementation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations and future work in the Conclusion section.

Summary

This papers considers the graph OOD generalization problem with given environment information. It proposes a novel invariant learning method based on adversarial loss which is derived from independence conditions of the causal graph. Experimental results show the proposed method is very effective.

Strengths

- Compared with previous work, this work proposes a more graph-specific environmental exploitation algorithm, which includes a subgraph selector. - The proposed method comes with theoretical results to show that the derived optimal paramteres can lead to learning causal subgraph only for invariant prediction. - The proposed method outperforms existing methods on various benchmarks by a large margin. Ablation studies show the proposed method is benefiting from each component.

Weaknesses

- In fact, the proposed method can also be not that graph specific. We only need to replace the subgraph selector f_{\theta} with a feature selector, then we can apply the method to invariant rationalization [1]. So, the claim may not be 100% true. - The theorems only provide equivalence between optimal paramters and learning causal subgraphs, but they did not mention what can be guaranteed if we cannot achieve optimal parameters (e.g., due to hidden variables). [1] https://arxiv.org/abs/2003.09772

Questions

- Since Y is not necessarily independent of G_s in FIIF and PIIF, what is the motivation to even use a soft version of the independence property? - For model selection, we know most invariant learning methods do not work with in-domain validation set [1] for image datasets. I am curious about what is the overall improvement when using OOD validation set over in-domain validation set in the graph OOD problem. [1] Gulrajani, Ishaan, and David Lopez-Paz. "In search of lost domain generalization." arXiv preprint arXiv:2007.01434 (2020).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- This work only considers causal features and spurious features as subgraphs. But there can other types of high-level graph features that can be causal features. - This work relies on invariant learning, which itself can be limited. For example, when the invariant features are not highly correlated with label, then the prediction accuracy can be quite low.

Response to authors

Thank you for the authors' response. The authors have addressed some of my concerns; however, there is still room for improvement regarding the novelty aspect. The idea of using the environment to enhance generalization has already been explored in various previous works.

Re: Rebuttal by Authors

Thank you for the rebuttal. The authors addressed some of my concerns, as they evaluated LECI together with a K-means cluster-style environment inference (EI) approach on several datasets. However, I would like to keep my score as is, since the results in the rebuttal are preliminary, and the authors also acknowledged that a deep exploration of EI methods is needed for future study. Moreover, thank the authors for answering my question. I agree that class-incremental tasks are often encountered and specialized techniques should be designed.

thanks for the response

Thanks for the response, authors addressed most of my concerns. > Response: For model parameter selection, we've incorporated a table delineating the performance difference from ID to OOD model selections of leading-edge methods across real-world datasets like GOODHIV, GOODSST2, and Twitter. Should it be deemed essential, we stand ready to complete and append a comprehensive table to our appendix. - For the table shown by the authors, it looks to me the Twitter dataset is an outlier, any insight for that?