Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation

Summary

This paper presents a novel approach for generating crystal materials using language models. The key innovation lies in the Mat2Seq method, which converts 3D crystal structures into 1D sequences while ensuring SE(3) and periodic invariance. This approach addresses the challenge of representing crystal structures in a way that is unique and invariant under different mathematical descriptions.

Strengths

1. The paper is well-written, providing clear descriptions of the problem background and proposed methods. 2. It tackles an interesting problem by using language models to generate crystal structures and introduces the consideration of SE(3) and periodic invariance for the first time. 3. The experiments are comprehensive, covering standard benchmarks and attempting to discover crystals with specific properties, demonstrating the robustness and versatility of the method. 4. Although the paper ensures the invariance of the generated structures, enforcing this invariance at the tokenizer level might limit the diversity of the generated structures to a small subspace. To obtain structures with different rotations, additional post-processing may be required, unlike models based directly in the spatial domain which can generate various rotational variants naturally.

Weaknesses

1. The paper should discuss the computational complexity differences between language models and specialized crystal models, as language models generally have higher parameter counts and computational demands. 2. The comparison of language models is limited. The experiments only compare with CrystalLLM, despite mentioning other methods. 3. The rationale for comparing only with CrystalLLM in Section 4.2 and not with other methods from Section 4.1 should be clarified.

Questions

1. For crystalline materials, isn't the original data already the most primitive unit? If we first apply Niggli cell reduction before using CIF files to describe unit cells, can we ensure a consistent representation? 2. While generating continuous crystal structures in 3D space seems natural, I am curious whether this token generation method can ensure that the reconstructed 3D crystal structure is continuous. Specifically, can the generated sequences always be converted back into a meaningful crystal representation?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

1. When performing conditional generation, it appears that generating structures for each specific property requires fine-tuning under specific conditions. This could result in high complexity in practical applications.

Summary

In this paper, the authors focus on the application of language models in the field of material generation. Starting from CIF files that represent crystal unit cell structures, they primarily utilize the Niggli reduction method to organize structures under different translations, rotations, and unit cell expansions into a unique representation. This representation is then used for crystal generation and conditional generation tasks. Compared to CrystaLLM, the proposed method demonstrates better generalization capabilities across multiple tasks.

Strengths

1. The authors introduce a new process, Mat2Seq, which uniquely represents crystal structures using Niggli reduction. This approach ensures SE(3) invariance, periodic invariance, and completeness, avoiding data augmentation problem and shortening token length in language model. 2. The paper showcases Mat2Seq’s ability to generalize to novel crystal structures that were not seen during training. This is a crucial aspect for practical applications and demonstrates the robustness of the method.

Weaknesses

1. The authors utilized a unique representation method for crystal structures, which theoretically promises to become an essential step in data preprocessing. This representation method primarily aims to improve the model's ability to recognize equivariant transformations in materials. However, the evaluation does not demonstrate this method's generalization capability to equivariant structures. It remains unclear from the authors' validation whether this method help language model learns from SE(3) equivariance or if the improved performance is due to the new token representation. 2. In the experimental validation in Section 4.2 and the conditional generation in Section 4.3, the evaluation of capabilities is very limited. The authors only provided the proportion of generated structures without detailing whether new structures can be generated, how many of the generated structures are valid, and accuracy of these generated structures. The generated structures need detailed explanation and evaluation.

Questions

1. In Section 3.1, the authors list two conditions for uniqueness and three types of transformations: translation, rotation, and unit cell expansion. It is suggested that Figure 1 should clearly illustrate all these transformations. Additionally, the "Change Lattice" operation in Figure 1 is confusing. Which type of transformation does this correspond to? Simply changing the length in the a-direction should not alter the atomic positions. Clarification is needed on whether the authors intended to illustrate unit cell expansion or some other operation. 2. In Table 1, the authors demonstrate Mat2Seq's ability to recognize crystal uniqueness, but the specific evaluation details are not provided. How were the values obtained, especially the 30% proportion? The methodology and criteria for this evaluation should be clearly explained. 3. In Section 4.3, while band gap is indeed an important property for semiconductors, it should be noted that whether the band gap is zero is a classification task distinguishing metals from non-metals. The authors' use of <0.5 as a threshold is inappropriate, especially when they mention that "values from 0 to 0.5 are marked as 0," which exacerbates the issue. This oversimplification could lead to misleading results. 4. In Section 4.2, in evaluating the ability to generate structures of new materials, the authors only indicate whether the generated structures match the given chemical formulas. It is recommended to include an evaluation of the atomic distance differences between the real and generated structures using RMSE to better assess the generation accuracy. 5. In Section 4.3, compared to commonly used machine learning models for material screening, generative models have the potential to produce out-of-distribution (OOD) structures, aiding researchers in discovering new structures with desired properties. However, the authors do not inform readers about the chemical formula repetitions, structure repetitions, or generation errors in the generated structures. A standard should be defined to evaluate the generated structures based on validity, uniqueness, and novelty.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors utilize an existing unique unit cell representation method in the data preprocessing stage for language models. This method holds promise as a necessary step for future work. However, the evaluation does not provide sufficient evidence to demonstrate its application value. Additionally, the conditional generation applications mentioned by the authors are also quite limited, based on the evaluations provided in the paper.

Summary

There are several challenges when developing LMs for materials: 1) each crystal structure consists of an infinite number of atoms and a unique and invariant unit cell must therefore be selected for each crystal 2) the unit cell can be represented in a one-dimensional (1D) sequence that maintains invariance under arbitrary rotations and ensures completeness. To tackle these challenges, the paper proposed a method called Mat2Seq that systematically transforms 3D crystal structures into 1D sequences. This is achieved by first identifying SO(3) equivariant unit cells and subsequently converting these into SE(3) invariant sequences. The experimental results in crystal structure prediction and crystal discovery with desired properties validate the efficacy of Mat2Seq.

Strengths

- I understand why it is important for the inherent symmetry of materials to be reflected in the text when designing a language model for material generation. - The importance of unique representation and completeness from the structure of the material to the sequence is emphasized.

Weaknesses

- Overall, the novelty and contribution of the paper are low. Instead of proposing new algorithms, they simply apply technologies that are already widely used in materials science to the process of representing materials as 1d sequence text. More specifically, the method for determining the equivariant unit cell is derived from Niggli cell reduction rather than being an original contribution of this paper. This concept is already widely used in the field of materials science, and the authors’ application of it to the text representation of materials does not seem like a significant contribution. - It is unclear what Section 3.3 is trying to convey. The authors claim to achieve text representation reflecting SE(3) symmetry following SO(3) symmetry, but it is doubtful whether this is well conveyed in this section. - The evaluation metrics used are limited. To demonstrate efficiency more effectively, the paper should present values for various metrics such as Validity, Stability, and S.U.N., which are commonly addressed in other studies.

Questions

- Could you provide a more detailed explanation with examples regarding the input for the language model (LM)? - Why is the target property value inserted at the beginning of the text input when generating materials for a specific target property? - Does “irreducible atom sets” refer to the atom sets contained within the minimum unit cell?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Please refer to the Weakness section.

Summary

This article propose a novel method, known as Mat2Seq, to tackle this challenge. Mat2Seq to converts 3D crystal structures into 1D sequences and ensures that different mathematical descriptions of the same crystal are represented in a single unique sequence, thereby probably achieving SE(3) and periodic invariance. Experimental results show that, with language models, Mat2Seq achieves promising performance in crystal structure generation as compared with prior methods. Overall this work gives new insights and should be accepted for the conference after minor revisions.

Strengths

1. Mat2Seq to converts 3D crystal structures into 1D sequences 2. it reports the framework for creating unique and complete crystal sequence representations, followed by the construction of a material LLM capable of generating novel crystal structure with desired properties of interest.

Weaknesses

1. Lack of experimental data validation for the generated data.

Questions

1. Author needs to verify the generated data with more experimental results as currently it have multiple limitations. 2. Author needs to compare the accuracy with already existing CIF data of crystals.

Rating

8

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations of the current Mat2Seq include: (1) it cannot be directly used for other atomic systems, like molecules and proteins; (2) the extension to model disordered materials remains a challenging frontier; and (3) large-scale training with more stable crystal structures can potentially enhance the robustness and performance when more computational resources are available.

We appreciate the chance to answer any additional questions

Dear Reviewer 5Kst, Thank you again for the valuable time you invested in reviewing this work. For your previous questions and concerns, we have provided detailed clarifications along with extensive additional experiments. We tend to believe these clarifications and additional experiments could thoroughly address your concerns and questions. As the author-reviewer discussion period is closing soon, we would greatly appreciate the chance to answer any additional questions you may have. Yours sincerely, The Authors