A Foundation Model for Zero-shot Logical Query Reasoning

Summary

This paper considers the inductive setting of complex logical query over incomplete knowledge graph, in which unseen entities and relations exist. To address this challenge but the particularly important setting, this paper generalizes the foundation model of knowledge graph completion Ultra and proposes UltraQuery. Experiments show that UltraQuery trained on FB15k-237 can achieve competitive performance across 22 out-of-distribution KG datasets and can zero-shot answer the logical query on other KG.

Strengths

1.This paper introduces a foundation model which can answer the logical queries over unseen KG. The foundation model is trained on the FB15k-237 and achieve the competitive performance comparing with SoTA baseline. This results is quite impressive. 2. This paper is well organized and well written. 3. This discussion is solid and the experiments well supports the foundation model for logical query answering.

Weaknesses

1. I am concerning about the novelty of the proposed methods. While the results of UltraQuery is impressive, the zero-shot ability is naturally derived from the foundation model of KG completeness, Ultra. This extension reminds the extension NBF-NET to GNN-QE.

Questions

1. The feature used in UltraQuery is the intersection between entity and relation, which deprecates the representation of the specified entity (relations) and can generalize the unseen entity (relation). I am wondering if UltraQuery can exploit the feature from entity (relation) like attributes, types, and so on to improve the performance by fine-tuning. 2. Will you present the results of the setting providing in your further work? In terms of the query patterns, the results can be directedly evaluated without modifying the models. [1] Hang Yin, Zihao Wang, Weizhi Fei, and Yangqiu Song. EFOk-cqa: Towards knowledge graph complex query answering beyond set operation. arXiv preprint arXiv:2307.13701, 2023. [2] Hang Yin, Zihao Wang, and Yangqiu Song. Rethinking complex queries on knowledge graphs with neural link predictors. In The Twelfth International Conference on Learning Representa- tions, 2024. URL https://openreview.net/forum?id=1BmveEMNbG. [3] Pablo Barcelo ́, Tamara Cucumides, Floris Geerts, Juan Reutter, and Miguel Romero. A neuro- symbolic framework for answering conjunctive queries. arXiv preprint arXiv:2310.04598, 2023.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

None

Summary

This paper proposes a new framework for the generalization of complex logical question answering (CLQA). Specifically, the authors handle an extreme situation where the knowledge graph at the test time is completely different from the training time, which requires the model to adapt well to new entities and relations. The authors implement a foundation model that can construct a relation graph for any type of relation for execution. The authors examine the foundation model on generalization between extremely different knowledge graphs, which shows advantages when understanding new relations is important.

Strengths

This paper establishes solid work for a unified CLQA foundation model that has strong adaptability to completely new knowledge graphs. I feel this to be an important work that benefits the training of CLQA models under low-resource situations and continuously updating knowledge graphs. The authors also widely conduct experiments to support their claims which show a strong generalization performance.

Weaknesses

While I am optimistic about accepting this paper, I find it hard to get the novelty of the proposed method, whose main difference from the previous ones is the ability to handle new relations. However, the contribution on this part is not clearly presented. I feel the main problem is the lack of a detailed description of the ULTRA method and how it is adapted to the foundation model. Also, the contribution beyond the ULTRA framework needs further explanation. Still, I am glad to accept this paper for its contribution of a generally generalizable CLQA model.

Questions

Please refer to the weakness part

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper presents ULTRAQUERY, a groundbreaking model for zero-shot logical query answering on knowledge graphs. It introduces a novel approach that combines inductive reasoning with non-parametric fuzzy logics to generalize to new entities and relations without additional training. The model demonstrates competitive performance across various datasets and query types, setting new benchmarks in the field.

Strengths

1. The paper introduces ULTRAQUERY, a novel foundation model for zero-shot logical query answering on knowledge graphs (KGs), which is a significant advancement in the field of complex logical query answering. 2. One of the key strengths of ULTRAQUERY is its ability to generalize to new entities and relations in any KG without the need for retraining, which addresses a major limitation of existing CLQA methods. 3. The paper provides extensive experimental results, demonstrating that ULTRAQUERY outperforms existing baselines on multiple datasets, showcasing its effectiveness across various query types.

Weaknesses

1. While the model's generalizability is a strength, the complexity introduced by the inductive reasoning might make it challenging to scale or adapt to very large KGs. 2. The paper does not present formal theoretical results or proofs to support the empirical findings, which could have strengthened the contribution. 3. The reliance on a pre-trained model could lead to overfitting on the training dataset, potentially affecting the model's performance on unseen data. The paper could have provided more details on the computational efficiency of ULTRAQUERY, especially in the context of large-scale KGs.

Questions

1. How does ULTRAQUERY handle extremely large KGs, and what are the scalability challenges? 2. Can the authors provide more insights into the decision-making process behind the choice of fuzzy logics for logical operations?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Applications to more general scenarios.

Summary

This paper proposes ULTRA QUERY, a foundation model for zero-shot logical query reasoning on knowledge graphs (KGs). Existing complex logical query answering (CLQA) methods are either transductive or only partially inductive, requiring training on each specific graph. ULTRA QUERY overcomes this limitation by deriving projection and logical operations as vocabulary-independent functions that generalize to new entities and relations in any KG. The model is pre-trained on a simple KG completion task and then fine-tuned on a single complex query dataset, enabling it to perform CLQA on unseen KGs in a zero-shot manner. Experiments show that ULTRA QUERY achieves competitive or better performance compared to baselines trained specifically on each dataset, setting a new state of the art on 15 out of 23 tested datasets.

Strengths

1. ULTRA QUERY is the first foundation model for inductive CLQA that can zero-shot generalize to any KG with new entities and relations. This is a significant breakthrough as existing CLQA methods are either transductive or only partially inductive. 2. The model design is well thought out, combining an inductive projection operator based on ULTRA with non-parametric fuzzy logical operators. The fine-tuning approach on a single dataset is effective in adapting the pre-trained projection operator to the multi-hop query answering framework. 3. The ability to perform CLQA on any KG without retraining is highly valuable, potentially saving significant compute resources and time. The performance of ULTRA QUERY on various datasets demonstrates its effectiveness and practical impact.

Weaknesses

1. While the experiments cover a range of datasets, a more in-depth analysis of the model's behavior under different conditions (e.g., varying graph sizes, query complexity) could provide further insights. 2. The paper does not discuss the model's scalability to very large KGs, which is an important aspect for practical applications. 3. And the scalability of the model (parameters and performance) can be further discussed.

Questions

How the model scales as the pretraining data scales up?

Rating

6

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have made an effort to discuss the limitations of their work, particularly regarding the model's reliance on the pre-trained inductive KG completion model and the multi-source propagation issue.

Response to Authors

Thanks for providing more details about the novelty and difference with important previous works. From your clarification, I get how your framework and implementation are novel against baselines. A possible way is to add which component in your framework addresses which issue in Table 1. Please consider adding these details to the main content of the paper. Based on the rebuttal from the authors and opinions from other reviewers, I am glad to improve my confidence and score for a **clear acceptance** since no major issue is mentioned. I appreciate the authors' effort in establishing a solid work and defending it during the rebuttal period.

The response

Thank you for your response! I don't have any concern.