Generative Hierarchical Materials Search

Summary

This paper proposes a hierarchical generative system for material search, which consists of a language model to translate user input into intermediate textual representation, a diffusion model which then generates crystal structure, and finally a property predictor used for sample selection. The works leverage data at different levels from different sources: the high-level knowledge in web texts and low-level crystal information in specialized databases. This motivates the factorization of the text-to-crystal generation process into the high-level text-to-symbolic step, and the low-level symbolic-to-crystal step. The generation results are filtered/ranked by a heuristic module for each step. The approach is supported with experiments on several families of material.

Strengths

The proposed framework effectively utilize data from multiple data sources. The collaboration of the generation modules and scoring modules in the framework is a reasonable approach and might benefit other works. The discussion on future works are also useful.

Weaknesses

The baselines seems to be weak. If the LLMs are finetuned on CIF data, probably the baseline performance can be improved.

Questions

it seems the baseline is few-shot, but the GenMS language prompt is zero-shot. Why? it is not easy to apply few-shot on GenMS or did you find it's not helpful compared to zero-shot?

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

the language model used in this paper, as well as potential ways to improve this module (such as prompting strategies, few-shot, finetuing) are not well explored.

Summary

This work leverages the powerful capabilities of large language models to propose a generative framework that transitions from natural language to material structures. The framework divides material generation into high-level semantic information, which can be expressed as chemical formulas in text, and low-level structural information, which requires atomic types and coordinates for expression. High-level semantic information is generated by the large language model based on user instructions, while the low-level structural information is completed by a conditional diffusion model. Both high-level and low-level information are optimized and ranked using empirical formulas. Finally, a property-predicting Graph Neural Network (GNN) is used to further optimize the generated material structures. The proposed model accomplishes end-to-end material generation, particularly starting from user instructions, which makes this research highly practical.

Strengths

1. Leveraging the capabilities of large language models to generate materials based on user instructions is a highly imaginative approach, with significant potential for future applications. 2. The overall structural design is comprehensive, including various component models and empirical functions. 3. There are technical innovations, such as the use of compact crystal representations to accelerate inference.

Weaknesses

1. The claim of "end-to-end" is confusing. The overall model seems more like a combination of different components (LLM, diffusion model, GNN). These components are not inherently connected and are not trained together, which seems to create a gap with the typical "end-to-end" meaning in the machine learning field. 2. The work seems more like an engineering contribution. While I do not deny the significance of such work in the current LLM era, the extensive discussion on multi-objective optimization (Section 2) feels somewhat redundant. This part appears to involve simple rule-based pruning, making the emphasis on multi-objective optimization as a contribution rather unconvincing. 3. The work lacks ablation studies. For instance, as mentioned in point 2, many rules are designed based on heuristic experience (including how to filter chemical formulas and molecular structures). I am uncertain about the actual impact of these heuristic rules and their robustness. 4. The core comparison baseline in this paper is the ability of a generic LLM without fine-tuning to generate materials. This is evidently unfair when compared to the proposed complete framework, which includes the diffusion model. Even adding fine-tuned LLMs does not seem convincing. The authors should at least compare their work with some specialized material generation models based on diffusion models or other methods. 5. Similar to point 1, although it claims to be end-to-end, it seems that only the diffusion model is trained. I believe there is a lack of detailed explanations regarding the LLM and the property-predicting GNN used.

Questions

1. Lacking domain knowledge, I am curious about the guiding capability of the chemical formula in this context. Why is it that a chemical formula generated by an LLM based on user instructions is considered so meaningful in providing guidance? 2. Using compact crystal representations to accelerate inference is understandable, but I am still curious why the authors chose a U-Net architecture as the denoising model. Typically, in non-image domains, common backbones are GNNs or Transformers.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

1. While generating molecules through user natural language instructions is a very exciting prospect, it seems that the role of the language model in this context is merely to generate a chemical formula. This makes it feel like the potential of the LLM is quite limited.

Summary

The paper presents a language-to-structure generation model for crystal structures. They propose a hierarchical approach that uses a language model to generate intermediate textual crystal information and then generate low-level crystal structures using diffusion models. They demonstrate their ability to generate crystal structures from language instructions and compare it to baselines. Results show that GenMS outperforms the baselines in generating valid and low-energy structures that satisfy user requests.

Strengths

1. The paper proposes a technically sound approach that decomposes the problem into two step and uses separate models for each step based on the task characteristics. 2. The empirical results are good. They demonstrate their effectiveness in generating valid and low-energy structures satisfying user requests and they outperform baselines.

Weaknesses

1. The baseline is relatively weak as it is only a few-shot prompted LLM. For example, the ablation of each of their proposed modules (e.g., LLMs, retrieval) should be treated as baselines and the results should be put in the main table with more comprehensive evaluation results than the current ones. 2. The overall workflow is a combination of several widely-used models (e.g., LLMs, diffusion models, retrieval models). The pipeline makes sense but combining these models may not be enough to make significant contributions for a research paper.

Questions

Can you add stronger baselines and show more comprehensive evaluation results?

Rating

5

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

the authors adequately addressed the limitations

Summary

This paper presents Generative Hierarchical Materials Search (GenMS), a novel framework for material search using LLMs. GenMS consists of a language model that takes high-level natural language as input and generates intermediate textual information about crystals (e.g., chemical formulae). Specifically, it uses a RAG architecture to make an LLM incorporate information from Wikipedia articles and other retrieved articles from the Internet. The generated formulae are filtered and re-ranked by a heuristic function involving domain knowledge, such as a formulae regular expression checker. A diffusion model then generates material structures conditioned on the filtered formulae. Experimental results show that GenMS is a solid framework that can generate structures satisfying users' requirements, demonstrating that using LLMs for structure search is a promising direction.

Strengths

1. The paper is easy to follow, even for readers not familiar with materials search. 2. The proposed method is interesting, particularly in its use of RAG to enable LLMs to follow the formula in articles. The ablation study shows RAG's effectiveness. 3. Using LLMs for material search or other scientific problems is an important but underexplored domain. This paper presents a solid framework for using LLMs in material structure search, showing that this is a promising direction for scientific applications.

Weaknesses

1. The architecture involves several steps requiring ad-hoc decisions, such as using Wiki-API for information retrieval or heuristic operations for selecting formulae. For example, lines 117-120 describe ad-hoc operations to prune intermediate formulae, making the method hard to reproduce and requiring more human effort. There might be more elegant, machine-learning-based ways to incorporate these operations, such as using LLMs for formula checking. 2. The heuristic function design involves significant domain knowledge, requiring more ablation studies or analysis to justify the heuristic choices.

Questions

1. Considering the complexity of the whole system, what is the average inference latency for each query? 2. How does using Wiki-API affect performance? Is there any ablation study on combining Wiki-API and RAG differently?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The design of the heuristic function requires a lot of domain knowledge, making the method difficult to reproduce.

Thank you for the response

Thank you for the response! I've read all the responses and the other reviews. The response does not address my concern about the technical contributions of the paper, as the overall workflow still seems like a combination of several widely-used models. However, because I am not an expert in this field and this work may be interesting to researchers in this particular domain, I would keep my relatively positive rating as is.