FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions

Summary

This paper proposes using a second-step Riemannian flow matching (RFM) model, separately trained, to improve the quality of crystal structures generated by a pre-trained large language model (LLM), closely following Gruver et al. [1]. Specifically, the authors first follow Gruver et al. to fine-tune an LLM to enable the generation of crystal structures, and then train an RFM model from scratch to improve the stability, or in other words, quality, of the generated crystals. The motivation of this work is clear. The reason for introducing this kind of post-processing machine learning model is to address the issue that LLMs cannot deal with real values, while atom positions and lattice parameters in crystals are usually real values. The experimental results show that introducing this second-step RFM model can improve the stability of generated crystals. However, it faces limitations, such as the inability to generate materials with specific properties, as mentioned in the limitations section. Additionally, there are other well-established pipelines to refine the generated crystal structures that may need to be compared and discussed. References: [1] Nate Gruver et al. “Fine-Tuned Language Models Generate Stable Inorganic Materials as Text”. In: arXiv preprint arXiv:2402.04379 (2024).

Strengths

## Strengths: 1. Clear motivation: LLMs face challenges when dealing with real values, and crystal structures are usually represented by lattice parameters and atom fractional coordinates that are real values. Introducing a post-process module to refine the generated structures of LLMs is reasonable. 2. Good performance when compared with models without such second-step refinement modules. The experimental results show that introducing such a second-step RFM model will increase the overall stability of generated crystals.

Weaknesses

## Weaknesses: 1. Missing discussions with other second-stage refinement strategies that are well-established. The generated crystal structures from LLMs can also be refined by machine learning force fields (MLFF), such as M3GNet, CHGNet (which the authors have used when calculating stability), or MACE-MP. For these MLFFs, one can directly use them without training a separate RFM model from scratch to refine the generated crystal structures. 2. The contribution may be a little limited. Given that there are well-established methods for the proposed issue in this paper, the contribution of introducing an RFM model to increase stability is somewhat limited. 3. The stability rate drops significantly when removing the duplicates (to SUN rate), up to more than three times (from 17.8 to 4.92). This means a majority of the generated stable crystals are similar or the same as each other. 4. Inability to generate materials with specific properties as mentioned in the limitations section. If the ability of this model is limited to generating stable crystal structures, there are other computationally cheaper models like DiffCSP that the authors have compared with for this task. Thus, it would be beneficial to add more discussions.

Questions

## Questions: Most of my concerns are listed above in the weaknesses. Addressing the weaknesses points and the following questions may impact the final score. 1. Why not just use MLFFs like CHGNet to increase the stability of generated crystals from LLMs? 2. Are a majority of the generated stable crystals similar or the same as each other? If not, why does the stability rate drop significantly when removing the duplicates (to SUN rate), up to more than three times (from 17.8 to 4.92)? 3. Why only compare with methods without this kind of second-step refinement module?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed in the main paper.

Summary

The paper proposed the FlowLLM, a hybrid approach for material generation that combines LLMs and RFMs, effectively leveraging their complementary strengths. Namely, this method generates samples from the parametric distribution using a two-step procedure, using 1) LLM and 2) RFM. By adopting this hybrid approach, experiments demonstrated that it exhibited better generation capabilities especially in terms of SUN and stability compared to the existing baseline.

Strengths

1. Since the LLM has been trained on material data, the proposed method can greatly simplify the denoising process. 2. The approach of utilizing both LLM and RFM, two generative model approaches, to complement each other’s weaknesses is intriguing.

Weaknesses

1. Since the output of the LLM is used as the starting point for the RFM denoising process, the generation time is longer compared to using a simple data distribution if there is no prior sampling of a large number ($N_{tr}$) of examples from the base distribution in advance. I would appreciate a comparison of the generation times between using 1) RFM with a simple base distribution and 2) FlowLLM without prior sampling from the LLM. Additionally, since the fewer required integration steps in the structural validity-number of integration steps plot could be attributed to the benefits of RFM, it would enhance credibility to compare this with FlowMM. 2. Using the trained LLM as the base distribution for the RFM necessitates additional LLM training costs. 3. The Preliminaries section of the paper overlaps significantly with that of the FlowMM paper. The structure and content of the Preliminaries section are very similar, which raises some questions. 4. It is insufficient to verify the performance of the proposed method on only the MP-20 dataset. I believe it would be fairer to cover all the datasets addressed in other baselines. 5. This appears to simply combine the models of CrystalLLM and FlowMM, making the novelty seem somewhat weak. If there are contributions beyond this combination, please let me know.

Questions

1. Are there any experiments supporting the claim that LLMs are weak with continuous type data and denoising models are weak with discrete type data? 2. In FlowMM, bit representation was applied to atom types. Is there a specific reason for not using this approach? 3. Approximately what is the rate of generating invalid crystals? 4. Could you explain the process of how $\lambda_f$ and $\lambda_s$ were determined? 5. Could you please explain in more detail about the invariance of flat tori and fractional coordinates?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to the Weaknesses section.

Summary

This paper presents a generative model for crystals that uses a LLM as a base distribution and a flow matching model to refine the 3D structure. Both the LLM and flow matching parts are mostly borrowed from prior studies. However, the authors demonstrate a significant improvement in the stability of generated materials by using LLM generated crystal as the prior for the flow matching model.

Strengths

The main strength of the paper is the strong empirical result that shows FlowLLM significantly improves the percentage of stable crystals by replacing the prior distribution used in FlowMM with a LLM prior. The results are validated with DFT calculations and thus convincing. Ablation study is conducted to isolate the contribution of LLM prior. They show that the atom type is the most important factor that contributes to higher stability.

Weaknesses

Despite the significant improvement in empirical results, the method presented is a straightforward combination of 2 prior works. The method is also unprincipled in some cases. For example, the LLM prior doesn’t always generate valid crystals in the defined domain. It may also generate lattices with angles outside the 60-120 range. The authors say that they simply reject invalid samples from LLM. It would be nice to report the rejection rate. In Table 1, the SUN rate is significantly lower than the stability rate for FlowLLM. It means either the model is generating non-novel or non-unique crystals. Can the authors also report the novel and unique rates? The wdist for density is significantly higher than DiffCSP and FlowMM. It indicates that the density of the generated crystals is less accurate.

Questions

Have the authors looked at the RMSD of the generated crystals to their equilibrium structures? The wdist for density is high for FlowLLM. It might indicate that the model is generating structures far away from equilibrium

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors noted the key limitation of FlowLLM is that it lacks end-to-end differentiability. However, the authors didn’t discuss potential negative societal impact of the model.

Thanks for your response

Thanks the authors for detailed response and additional results. They've address most of my concerns, so I'd like to increase my score from 5 to 6. The improved RMSD is a nice result. It would be helpful if the authors can provide additional insights on the source of improvements. Can it be attributed to the better prior or the better flow matching process? Although I understand that authors use CHGNet to get a fast result on RMSD, DFT result is preferred for the final version of the paper. I also appreciate the authors adding a paragraph about potential negative societal impact.