Construction and Application of Materials Knowledge Graph in Multidisciplinary Materials Science via Large Language Model

Summary

This paper introduces the Materials Knowledge Graph, a pioneering graph database designed for materials science. It leverages advanced NLP methods and LLMs to extract and organize a vast amount of high-quality research into structured triples. It streamlines the discovery process by organizing information into nodes and edges, enhancing data integration and reducing the need for traditional experiments. The MKG also employs algorithms to predict material applications, offering a significant advancement in accelerating materials research.

Strengths

1. This paper demonstrates a thorough and systematic methodology, including data preparation, model training, entity resolution, and graph construction, ensuring a robust and credible knowledge graph. 2. The MKG shows a cutting-edge approach to parsing and structuring vast amounts of scientific literature, offering a significant advancement in accelerating materials research. 3. The application of link prediction algorithms for predicting material applications is a robust method for identifying new potentials in the field. 4. Experiments shows the effectiveness of the MKG and the employed models, enhancing the credibility and reliability of the results.

Weaknesses

1. As the field of materials science evolves, maintaining the currency and accuracy of the MKG could become increasingly complex, requiring continuous updates and curation.

Questions

1. How can your methods be adapted for other scientific fields? Are there specific modifications needed? 2. What strategies do you have for continuously updating and curating the MKG to ensure its relevance and accuracy?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have made a commendable effort in addressing the limitations of their work. They acknowledge the dependency on manual annotation for data preparation, which could limit scalability and timeliness. However, the paper could benefit from a more detailed discussion on maintaining the currency and accuracy of the MKG as the field of materials science evolves. Continuous updates and curation will be crucial, and outlining specific strategies for this would strengthen the paper.

Summary

This paper presents an innovative way on leveraging the power of Large Language Models for the construction of a Material Knowledge Graph (MKG) and link prediction. The method includes annotating few scientific articles (abstracts) related to material science which are used for training and finetuning LLMs. After that step and using additional articles and a finetuned LLM, triples are generated. Entity resolution is performed by using different Natural Language Processing (NLP) techniques such as ChemDataExtractor, mat2vec and an expert dictionary. Finally, the MKG is constructed and used for link prediction with the aid of network-based algorithms and graph embeddings. This approach is compared with another technique called MatKG2 and experiments were conducted for finding the LLM that provides the best results and for evaluating the link prediction of the MKG.

Strengths

• This is an innovative work for the fast and automatic construction of a Material Knowledge Graph with minimal annotation that could have a broad impact in the advancements of material science. • It reuses effectively new technologies such as Large Language Models and other NLP techniques. • It does not require too many annotated documents or other manual tasks.

Weaknesses

• More experiments would have supported better this work. The result of this method could have been compared with some baseline experiments of simply using LLMs for the triples generation. Moreover, since this method is compared with MatKG2, it would have been useful to compare the results of precision, recall and F1 score of MatKG2 with the results of MKG. • While the work seems quite interesting it not well written and several typos and syntactic errors have been found which are detailed below. Proof-reading would have been beneficial before submission. - Line 64: “ A user-friendly databases.. “ -> “User-friendly databases..” - Line 80: Acronyms NER and RE are used but they are introduced later in the text (line 85) - Line 89: “… through query the MKG.” -> “… through querying the MKG.” - Line 93: “the elaborate workflow” -> “the elaborated workflow” - Line 98: “NERRE” -> I believe this refers to NER and RE - Figure 1 (b): ChemDataExactor -> ChemDataExtractor - Figure 4: FMKG -> the material knowledge graph has been referred in the whole paper MKG. The acronym FMKG is introduced only in this figure. - Line 233: “indicate”->”indicates", “achieve”->”achieves”, “contribute”->contributes”. - Line 240: “ChemDataExactor” -> ”ChemDataExtractor” - Line 252: “The result shows in.. “ -> “The result is shown in…” • Figure 3 and network-based link prediction is not well explained in the article. • The code and Knowledge Graph is not available for further evaluation. The authors state that they will only be available if the paper gets accepted.

Questions

Limitations of this work are not detailed in the paper. - What would you consider limitations of this work? - How would you ensure that MKG will be up-to-date with the state of the art in material science?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

There are some concerns about the scalability and maintainability of MKG which are not addressed in the paper.

Summary

The study presents an innovative pipeline for Knowledge Graph (KG) construction, specifically designed for efficient extraction of triples from unstructured scientific texts. The methodology enables fine-tuning of Large Language Models (LLMs) with limited annotated datasets, which is then utilized to extract structured information from extensive corpora of unstructured text. The authors have constructed a Material Knowledge Graph (MKG) that captures relationships between materials and their associated entities, such as properties and applications, derived from abstracts of 150,000 peer-reviewed papers.

Strengths

Originality: The paper introduces a novel pipeline for KG construction that departs from predictive modeling, enhancing the authenticity and traceability of extracted structured data. Quality: The authors demonstrate the effectiveness and credibility of the MKG through ablation experiments and similarity analyses based on node similarity and graph embedding. The results indicate the substantial predictive capacity of the MKG, with 48.5% of 'material-application' predictions validated within nine years, which is impressive. Clarity: The paper provides a clear and comprehensive explanation of the methodology, including the fine-tuning of LLMs, extraction of structured information, and construction of the MKG. Detailed results and analyses support the authors' claims. Significance: The MKG has significant potential in extending the depth of structured information extraction, improving entity labeling precision, and adapting the pipeline to other scientific fields.

Weaknesses

Some figure captions, like "Fig 1(a)" and "Fig 3," are unclear. Captions would benefit from additional context. Certain acronyms, such as "ER-NF" on line 239 and "NER” / “RE" on line 80, appear before being defined. Typo on line 140: "task"s->task’s."

Questions

Does the inclusion of DOIs affect the resource consumption such as memory or performance in any noticeable way? It is common practice to begin with abstracts. Do the authors intend to extend their work to encompass full texts in the future?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors did addressed the limitations, and mentioned that strict normalization and entity resolution process can loss some correct entities.

Summary

The paper on constructing and applying a materials knowledge graph (MKG) in multidisciplinary materials science via a large language model (LLM) is valuable and well-written. However, it primarily focuses on application rather than strong technical contributions, with issues in experimental design, lack of non-LLM baselines for comparison, and insufficient comparison with sophisticated knowledge graph completion methods. It may be a good dataset track paper but not the main track.

Strengths

- The studied problem is of great value in the real world - The paper is well-written and easy to follow - The produced MKG could be very helpful for the computational and experimental material science

Weaknesses

- W1: This paper is towards applications of LLMs in an engineering flavor instead of having strong technical contributions. The process includes prompt engineering, basic model fine-tuning, and human-in-the-loop entity resolution. And this paper looks like an extended application of Darwin, instead of an independent research work. - W2: The experimental design is flawed. In particular, why the normalization is only applied to Darwin? Is it possible that other base models + normalization can perform better? And is the normalization only working well for Darwin? - W3: Non-LLM baselines for NER and RE should be included for comparison. - W4: The modified Jaccard similarity method is claimed as a specified KG completion algorithm for material science. Therefore, the experiments should include comparisons with more sophisticated KG completion methods, instead of only comparing with TransE, which is outdated. - W5: Minor issues include but are not limited to: Typos in Figure 4, what is FMKG?

Questions

Please refer to W2 and W5

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.