UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction

Summary

This paper presents a framework denoted UrbanKGent, for finetuning an LLM to assist the construction of knowledge graph construction (specifically triplet extraction relation prediction). The gist of the study is to construct an ad-hoc corpus for finetuning. In this process, a method of trajectory refinement is proposed to enhance both the effectiveness and explainability of the framework.

Strengths

1) An interesting and important application domain of LLM 2) A generally well-developed workflow for tackling the target problem 3) The evaluation is more or less adequate for showing the superiority

Weaknesses

1) It seems that the generation of the corpus, which is a key ingredient of the framework, lacks a systematic and scientific methodology for its generation. 2) The transferability is questionable – is this method useful in cities that are not finetuned? 3) The description of the roles of GPT4 and the smaller LLM that is finetuned is vague.

Questions

See weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

See weaknesses.

Summary

The paper introduces UrbanKGent, a comprehensive framework utilizing a large language model (LLM) for constructing urban knowledge graphs (UrbanKGs). The key components of UrbanKGent involve creating an instruction set for tasks like relational triplet extraction and knowledge graph completion, which are tailored to urban data's heterogeneity and spatial characteristics. An iterative refinement module is employed to improve and adjust trajectories derived from the GPT-4 model, enhancing the quality of the knowledge graph. The framework is then fine-tuned using these enhanced trajectories on the Llama 2 and Llama 3 models, resulting in UrbanKGC agents, available in 7/8/13B versions.

Strengths

1. Intensive efforts have been dedicated including crawling and preprocessing raw data, empirically validating the insufficiency of GPT-4, and instruction fine-tuning the UrbanKGC agent. 2. The paper focuses on real-world scenarios with domain knowledge infused in UrbanKGC agent construction and carries out comprehensive evaluations on real-world datasets using human and GPT-4 self-assessment.

Weaknesses

1. The computational efficiency of the proposed algorithms and their scalability with the size of the dataset is not thoroughly discussed.

Questions

1. As mentioned in Appendix Section D.1, human annotators are employed to fill out an evaluation form. Is it possible to provide more detailed information on human annotators, such as an exact number of human annotators?

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

As acknowledged in appendix section F, the paper has limitations on the further application demonstration of construction UrbanKGs and the evaluation method in this work is cost-intensive.

Summary

The paper presents UrbanKGent, a unified large language model agent framework for urban knowledge graph construction. The framework consists of knowledgeable instruction generation, tool-augmented iterative trajectory refinement, and hybrid instruction fine-tuning. The authors evaluate the framework on two real-world datasets using both human and GPT-4 self-evaluation. The experimental results show that UrbanKGent outperforms 31 baselines in urban knowledge graph construction tasks and achieves state-of-the-art performance.

Strengths

1. The paper proposes a unified framework for urban knowledge graph construction, addressing the challenges of heterogeneous relationship understanding and geospatial computing. 2. The knowledgeable instruction generation module and tool-augmented iterative trajectory refinement module are innovative and practical methodologies. 3. The experimental evaluation is comprehensive, including both human evaluation and GPT-4 self-evaluation. 4. The results show that UrbanKGent outperforms 31 baselines in urban knowledge graph construction tasks and achieves state-of-the-art performance.

Weaknesses

1. This paper focuses on urban knowledge graph construction tasks. However, the details of the urban knowledge graph are not clearly defined. For example, the ontology of urban KGs, entities types, and relations types are not discussed. This limits the understanding of the proposed framework for readers who are not familiar with urban knowledge graphs. 2. The KGC tasks only focus on the geospatial relations. Is there any specific reason to select these relations? Can we predict other relations using the same framework? 3. The motivation of the KGC task is not clearly explained. If the geospatial information is given, can we directly use the tool to get their relations? The paper should provide more insights into the motivation behind KGC task and the necessity of using LLMs for it. 4. The ablation study is not clearly described. The paper should provide more details on the datasets, settings, and the performance of the final model used in Table 10. Otherwise, it is hard to evaluate the effectiveness of each module. 5. The iterative trajectory self-refinement module is not well explained. Based on what criteria is the trajectory refined? How many iterations are performed? How does the performance improvement as the number of iterations increases? Moreover, if LLM lacks the knowledge about urban KGs as the motivation of this paper, how can it be a good judger? Is there any analysis about the correctness of the comments and refined trajectory? 6. The cost for fine-tuning data construction is not discussed. How much data is required for fine-tuning the model? What is the cost of collecting and labeling the data for fine-tuning? 7. In experiments, the paper should compare UrbanKGent with existing UrbanKG construction methods rather than purely focusing on the LLMs which are not tailored for this task.

Questions

1. What is the ontology of urban KGs? 2. What is the motivation for selecting geospatial relations for KGC tasks and using LLMs for this? Can we predict other relations using the same framework? Can we directly use the tool to get relations if geospatial information is given? 3. Please provide more analysis of the iterative trajectory self-refinement module. 4. Settings of the ablation studies. 5. Cost for fine-tuning data construction. 6. Can we compare UrbanKGent with existing UrbanKG construction methods?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper proposes a unified large-scale language model agent framework called UrbanKGent, specifically for the Construction of Urban Knowledge Graph Construction (UrbanKGC). UrbanKGent utilizes instruction generation of heterogeneous and geospatial information, as well as an iterative trajectory optimization module based on GPT-4, to further enhance the ability to extract critical knowledge from multi-source city data and effectively reduce the significant manual labor of traditional methods. By fine-tuning the hybrid instructions on the Llama 2 and Llama 3 series models, the researchers developed the UrbanKGC agent, including the UrbanKGent-7/8/13B version. Experiments on two real-world datasets show that the UrbanKGent family not only significantly outperforms 31 benchmark models on the UrbanKGC task, but is also more than 20 times more cost-effective than GPT-4, while being able to build a knowledge map of cities with hundreds of times richer relationships with less data.

Strengths

1. UrbanKGent proposed in this paper provides a unified solution for building urban knowledge graphs, which automates the process of extracting key information from multi-source urban data and reduces the need for human intervention. 2. The authors propose instruction generation methods of "heterogeneity awareness" and "geospatial information fusion", which can better capture the characteristics of urban knowledge graph construction tasks and make up for the shortcomings of ordinary LLM in understanding complex heterogeneous relationships and geospatial computing capabilities. 3. Experimental results on two real-world data sets show that UrbanKGent not only significantly outperformed 31 benchmarks on the UrbanKGC task but also outperformed the state-of-the-art LLM GPT-4 by more than 10% at about 20 times lower cost, which has great potential for efficiency and economy in practical applications.

Weaknesses

1. As the size of the dataset grows, the algorithmic efficiency and scalability of UrbanKGent may become an issue. Although not mentioned in the paper, in practical applications, the processing of large-scale data sets may require more computational resources and time. Although the paper notes limitations, the specifics may require more elaboration. For example, under what conditions may the model perform poorly and how these limitations affect the quality and reliability of the final knowledge graph? 2. UrbanKGent may face privacy and fairness challenges, especially when handling personal or sensitive information. I wonder what the author thinks about privacy and fairness. Although these issues are not discussed in detail in the paper, in actual deployment, if not properly addressed, they can lead to violations of personal privacy or unfairly affect specific groups. 3. The UrbanKGent framework is mainly aimed at urban knowledge graph construction tasks. I wonder if the author has considered the applicability of other types of knowledge graph construction tasks. Can the UrbanKGent framework proposed in this paper be further extended to the knowledge graph construction in other fields?

Questions

NA

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Thanks

I would like to thank the authors for the response. I would like to keep my rating as I find the argument about transferability is not that convincing.