End-to-End Ontology Learning with Large Language Models

Summary

This paper proposes a new methodology of Ontology Learning using Large Language Models and ontology paths which is called OLLM. After introducing two ontology-datasets for Wikipedia and ArXiv, they train an LLM using the titles, summaries, and ontology paths to allow for sub-graph generation. The method is additionally improved using post-processing techniques for summing and pruning the subgraphs with the purpose of building a complete ontology. New metrics are also presented for facilitating the comparison of the generated ontology with the ground truth and other baseline techniques.

Strengths

• This work presents a novel technique leveraging the power of LLMs and ontology paths for ontology learning. • Two different ontology-datasets (Wikipedia and ArXiv) are constructed and used for the experiments. • New metrics for ontology evaluation are proposed based on semantic similarity. • OLLM seems to outperform most of the baseline methods in the experiments. • Overall, the paper is well-written.

Weaknesses

• While the OLLM results are better than most of the baseline methods that is not the case using the motif distance metric (Wikipedia: Finetune = 0.05 while OLLM = 0.080, Arxiv: Memorisation = 0.037 while OLLM = 0.097). That is not well explained in the paper. • Performance seems to drop significantly when using a smaller dataset such as arXiv. • It is unclear whether OLLM can generalize well to different domains without having an initial ontology for extracting the paths.

Questions

• Usually training sets consist of the biggest part of a dataset than validation and test sets. Why did you choose such small portion of the datasets for training? Could that be one reason for the initial poor generalisation? • It is also unclear how much the overlapping parts of training/validation/test sets contribute to the final result.

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The main limitation, to my opinion, is that OLLM cannot generate a complete ontology for a different domain without already having an initial ontology for extracting the paths first. This means that there is still the need of reusing or constructing some part of an ontology.

Summary

The paper is well-written, addresses a valuable real-world problem, and proposes a simple, intuitive, and effective method. The experiments are rigorously designed with comprehensive evaluation metrics. Overall, this is a high-quality work with notable contributions. However, there are concerns about whether the improved performance is due to end-to-end modeling or LLM capabilities, the assumption that each document is associated with at least one concept, the consistency of the generated ontology, and the post-processing procedures.

Strengths

- S1: The paper is well-written and easy to follow - S2: The investigated ontology learning problem is of great value for real-world applications - S3: The proposed method is simple, intuitive, yet problem-oriented, novel, and effective - S4: The experiments are well-designed, in particular, the baselines and the data-splitting strategy are rigorously - S5: The set of evaluation metrics is comprehensive and covers various aspects of performance

Weaknesses

- W1: The paper's main claim is that end-to-end modeling is better than pipelined methods. However, it is unclear whether the improved performance is credited to the end-to-end modeling approach or the capabilities of LLMs. Although I am convinced that end-to-end + LLM is a good solution, I am curious if applying LLMs for the subtasks in the pipeline will work even better, especially considering LLMs are good at decomposing problems and solving them step-by-step. - W2: The authors assumed each document is associated with at least one concept in the ground truth ontology. However, in real-world applications, it would be very possible that the whole document does not contain any relevant edges or even concepts. Can the proposed OLLM say no if a random irrelevant document is given? - W3: Consistency is a crucial property of ontologies, which is the prerequisite to run most logical reasoners. Is the generated ontology guaranteed to be consistent? - W4: I am concerned about the post-processing procedure. Are there any entity (concept) resolution mechanisms involved to merge semantically equivalent concepts? For example, if “English” and “english” are both there, are then treated as the same concept? And for “English Language” and “The Language Named English”, are they merged? - W5: Why using 3 prompts is generally worse than 1-shot?

Questions

Please refer to W1-W5.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes.

Summary

The paper introduces a novel method called OLLM (Ontology Learning with Large Models) for automating the construction of ontologies using large language models (LLMs). Ontologies are crucial for structuring domain-specific knowledge, but their manual construction is labor-intensive. The authors aim to address the limitations of partial ontology learning approaches by proposing an end-to-end method to build ontologies from scratch. This paper demonstrates OLLM's effectiveness through quantitative and qualitative results on Wikipedia data, showing that it outperforms subtask composition methods in producing more semantically accurate ontologies while maintaining structural integrity. Additionally, OLLM's adaptability to new domains like arXiv is showcased, requiring only a small number of training examples for effective adaptation.

Strengths

1) The authors propose a custom regularizer that reweights concepts based on their frequency of occurrence, which helps to mitigate overfitting on common concepts and enhances the model's ability to generalize to new, unseen data. 2) The paper introduces a novel suite of evaluation metrics that use deep learning techniques to measure semantic and structural similarity between ontologies. These metrics provide a more robust and nuanced assessment compared to traditional methods. 3) The authors demonstrate the effectiveness of OLLM through comprehensive experiments on Wikipedia and arXiv datasets. The results show that OLLM outperforms existing subtask-based methods, indicating its practical utility.

Weaknesses

1) The paper focuses on building ontologies with concepts and taxonomic relations, which is only a part of the full spectrum of ontology components. A comprehensive ontology includes not only hierarchical relationships but also properties, semantic relations, constraints, and other elements that provide a richer and more explicit semantic content. The paper does not address how these additional aspects are captured and integrated into the learned ontologies. 2) The paper uses Wikipedia categories and arXiv taxonomy as the basis for ontology learning. However, these structures are more accurately described as taxonomies or folsonomies rather than full-fledged ontologies, which can lead to confusion about the capabilities and outputs of the proposed method. 3) The paper does not provide a clear and detailed explanation of the end-to-end ontology learning model's training process. Key aspects such as the construction of the training dataset, filtering of irrelevant categories, and the specific model architecture and training regimen are not thoroughly described. For example, it is not clear how the model deals with Wikipedia categories that have little semantic content but are used for page management. 4) The paper does not include a comparative analysis with other ontology learning methods, such as the LLMs4OL (LLMs4OL: Large Language Models for Ontology Learning, https://arxiv.org/abs/2307.16648.). Without such comparisons, it is difficult to assess the relative advantages and disadvantages of the proposed method in terms of effectiveness, efficiency, and scalability.

Questions

See weaknesses.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

See weaknesses.

Summary

The paper aims to address the challenge of constructing ontologies, which traditionally require substantial manual effort. Ontologies are structured representations of domain knowledge used for automatic machine processing. Previous methods for ontology learning (OL) using large language models (LLMs) focused on solving individual subtasks, but this approach failed to capture the interactions between these subtasks. Experimental results on Wikipedia demonstrate that the authors' proposed approach, called OLLM, outperforms traditional subtask composition methods, producing more semantically accurate ontologies while maintaining structural integrity. The model also shows effective adaptation to new domains, such as arXiv, with only a small number of training examples, indicating its scalability and domain-independence.

Strengths

This is an interesting problem, and could serve as the foundation for important neurosymbolic applications in the future. The authors introduce OLLM, a general and scalable method that builds ontologies from scratch by fine-tuning an LLM. Instead of focusing on individual relations between entities, OLLM models entire subcomponents of the target ontology, reducing overfitting on high-frequency concepts through a custom regularizer. The principles of the approach are moderately innovative. The paper also introduces new metrics for evaluating the quality of the generated ontology, measuring its semantic and structural similarity to the ground truth. This is laudable, but in going through the metrics, I noted some significant potential flaws that I note below.

Weaknesses

I would question the validity of the fuzzy F1 metric. Using embedding similarity is not a good fit here, as for difficult samples, it would likely be wrong. There is also bias that comes from pretraining (so the metric can't really be used in 'unusual' domains for which such embeddings are not available; but that's where the true applications of this proposal would be!) The gap between the literal and fuzzy F1 in the performance table should have been a warning to the authors. The experiments could have been much stronger. A more difficult benchmark than either of the ones the authors selected would have been more valuable for assessing the true potential or limitations of the method. The baselines could similarly have been stronger. It doesn't look like the authors considered some form of sophisticated (e.g., chain of thought) prompting, which could have made a big difference here.

Questions

Given that there are so many ontologies and corpora out there, why only limit to these two very well-known datasets? I feel that the approach could have been better evaluated with multiple ontologies, some of which are unusual and hence not captured as well by LLMs.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The study focuses on constructing simple ontologies that include only concepts and taxonomic relations. To extend OLLM to produce non-taxonomic relations, tags indicating the relation type could be added to each edge during the linearization of subgraphs for sequence modeling. New evaluation metrics may also be needed to handle multiple types of relations. Another limitation is that the taxonomic relations in the generated ontologies are not always transitive due to the presence of cycles, a common issue in many ontology learning methods. There are existing algorithms for cycle removal to clean hierarchies. Additionally, the study could not fully control for data contamination since the pretraining dataset of Mistral 7B is not publicly known. However, the generated ontologies were found to be sufficiently different from the ground truth, suggesting that OLLM does not directly remember samples from its pretraining stage.