Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer

Summary

This paper propose DOSTransformer, a transformer architecture for the task of DOS prediction, that takes energy levels as input instead of predicting a list of energies for different energy levels. The performances of this proposed DOSTransformer is better than MLP, GNN, and E3NN instances they implemented.

Strengths

- The proposed method is the first one to take energy levels as input instead of directly predict a list of energies for different energy levels. - Writing: clear and easy to follow. Figures are informative.

Weaknesses

- Novelty is limited, core ablation study is missing: the major novelty beyond previous works is taking energy levels as input instead of directly predict a list of energies for different energy levels. To do this, they proposed the multi-modal module that considers energy levels. But this point is not well supported by the performance ablation studies of E3NN. Also, the ablation study for their proposed model about this point (just directly predict a list of energies) is missing. - Missing comparisons with previous crystal neural networks. Previous crystal neural networks including CGCNN, MEGNET, Matformer, ALIGNN, M3GNet can all directly be used for this DOS task, by just changing the output dimensionality from 1 (single scalar value prediction) to M. Just ignoring all these methods is not convincing. Comparing with these methods (at least one or two current SOTA methods) can better demonstrate the importance of taking energy levels as conditions instead of predicting a list of energies. - Missing implementation details and what level of irreducible representations is used in E3NN. Simple E3NN with only irreducible representations of rotation order zero or 1 is not powerful enough. Using representations with rotation order of 2 will boost the performances significantly. Given the not significant performance gains beyond e3nn, further implementation details is needed.

Questions

As listed in above weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

They listed the limitations in the main paper.

Summary

This paper proposes a prompt-based Transformer network for predicting the density of states of crystalline materials. The prompts are used to represent and control the energy and the additional structure information of the materials. Experiments show the proposed method can perform very well on two datasets under different settings.

Strengths

- The whole system and method are motivated well and designed carefully. The model design considers the characteristics of both the task and the deep learning methods, e.g., prompt-based Transformer. - The experiments are carefully designed to show the benefits of the proposed methods. - The paper is well-written and easy to read.

Weaknesses

- The datasets and experiments are a bit simple. The variations of the material type and structure type in the datasets are restricted. In this sense, although prompts are used to formulate the general variation of the energy and structure, the datasets contain very simple and limited variations. It is a bit limited in evaluating the potential of the proposed method and the techniques on more realistic datasets and applications. - For example, the prompt is used to represent the structure. There are only seven discrete options that will be given during testing. May the model handle the structure variation without relying on the input during testing? - In Table 4, the prompt based method is compared with the one-hot based input. Is the one-hot binary encoding directly used to replace the embedding of the prompt? If so, it might be unfair. How does the one-hot encoding work with a fully connected linear layer (i.e., feeding the one-hot encoding as the input to the FC layer), where the weights in the FC layer are corresponding to the embeddings for each type of the system? - In the worst case, the proposed prompt based method may work similarly to the linear embedding based approaches, on the simple datasets. - The authors may analyze the learned prompts to indicate whether the leaned embeddings can reflect and are consistent with the physical characteristics of the system types or structures indicated by the prompts, such as whether some similarity and relationship between these characteristics can be reflected in the learned prompts. - It seems that fine-tuning do not improves the performance significantly, especially given the identical MAE performance comparing Table 2 and Table 3.

Questions

Questions are left with the weakness points.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The limitations are discussed and can reflect the main limitations of the proposed method – the proposed method cannot encode the properties of both materials into a single model.

Summary

The paper proposes a new transformer-based method DOSTransformer for predicting density of states of crystalline materials. Different from previous methods, the energy level is additionally modeled as an input modality, i.e. the model takes in material configuration and energy level as input to predict DOS(material, energy). DOSTransformer uses cross-attention layers to relate the material and energy representations. Self-attention layers are also used to integrate information across different energy levels. Learnable prompts are introduced to provide information about the crystal system type, which helps capture structure-specific interactions between material and energy. Experiments on phonon and electron DOS datasets show DOSTransformer outperforms existing methods like graph networks and E3NN, in both in-distribution and out-of-distribution scenarios. Ablations and sensitivity analysis demonstrate the benefits of the different components like energy embeddings, global/system losses and crystal system prompts.

Strengths

- Novel problem formulation: The authors provide a new perspective on DOS prediction by treating it as a multi-modal learning problem with material and energy as inputs. This proves to work better for DOS prediction compared to prior material-only models. - Neural net design: The multi-modal transformer architecture with cross-attention and prompt-guided self-attention is well-suited for the problem. It allows capturing complex relationships between material and energies. - Empirical results: The model is evaluated extensively on two datasets — Phonon DOS and Electron DOS — for both in-distribution and out-of-distribution scenarios. The physical validity of predictions is also analyzed through derived material properties. In particular, the introduced energy embeddings are shown to be useful in reducing prediction errors. - Ablation studies: The contributions of different components like the dual losses and crystal system prompts are quantitatively demonstrated through ablation studies.

Weaknesses

- Simple prompts: The crystal system prompts are relatively simple vector embeddings. More sophisticated prompt learning methods could be explored to reason about structural information of different systems.

Questions

N.A.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

N.A.

Summary

This work proposes a transformer architecture for predicting density of states of crystalline materials for different energy levels. The distribution of states is spectral property that is approximated as a function of both the material and energy levels. The architectures proposed consists of a multi-modal transformer with self and cross attention layers, and a GNN for embedding. Experiments compare the proposed architecture with state-of-the-art E3NNs, demonstrating a decrease in MSE.

Strengths

The main strengths of this work include: 1. The approach considers the crystalline material, energy levels, and structural properties. 2. The problem and methods are well-defined, and the architectures are implemented using state-of-the-art libraries. 3. The results include out-of-distribution experiments, ablation studies, and sensitivity analysis.

Weaknesses

The main weaknesses of this work are that: 1. It needs to be clarified what MSE's are required for the approach to be useful in the real world. For example, in proteomics, and RMSD of less than 2-3Angstrom is considered valid compared to experimental results. What would be the MSE's required to achieve such a validation in this domain. 2. The terminology and definition used for fine-tuning and especially prompting in this work differs significantly from the standard definition and should be clarified.

Questions

What are the MSE values of bulk, band, and ferm. are considered useful for real-world applications? Are there 38,889 crystalline materials after or before filtering for non-magnetism?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are adequately addressed.

Summary

This work proposes a transformer model to predict the density of states of given materials. The model takes in all energy levels that are desired for the Density of States calculation as a 'prompt' and information about the materials structure, the output is a prediction of the DOS at the given energy levels.

Strengths

+ The DOSTransformer is evaluated on both in-domain structures and out of domain structures against relevant baseline models and previously reported models. + The DOSTransformer model design makes use of domain specific knowledge, such as the crystal structure type and the energy levels. + Authors demonstrate that the task of predicting DOS is heavily added by providing energy levels for targeted prediction via relevant ablation studies + The model is clearly described. I was able to understand all the components of the proposed model.

Weaknesses

- More could be done to describe the task of predicting DOS for downstream applications, especially for a non-materials science audience. How are the DOS outputs for predicting bandgap energy / electrical conductivity? How many grid points are needed to gain an 'accurate enough' prediction of these properties? What is kind of error in the DOS prediction is reasonable? Are there certain energy levels where it is more critical to predict the DOS correctly? - I would appreciate some more statistics of the datasets used (PhononDOS and ElectronDOS) in the main text (e.g. # atom types, # crystal types).

Questions

- As I understand it one of the tasks in this work is to predict the output of a function (Density of States) on a fixed grid of energy levels. Are there other examples in the literature where transformers are used to predict the outputs of a mathematical function, and the fixed grid of inputs is provided as a prompt for the model? If there are, it would be great to learn about them in the background section. - Because the DOS is predicted over a finite grid, I am curious about the smoothness of the DOSTransformer model between finite grid points. If the DOSTransformer model was to be evaluated at energy levels between the provided finite points in the prompt, are the predictions smooth where you'd expect the function to be smooth?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- More description should be provided about the datasets used. How many different elements/ types of materials were included int he PhononDOS and ElectronDOS provided?

Responses to rebuttal

Thank you for your effort. The additional ablation studies and more comparisons with recent crystal neural networks enhanced the clarity and addressed my concerns. Therefore, I adjusted my score accordingly. It seems we cannot adjust the original review, but I will adjust my score to 6 if editable.

Response to rebuttal

Thanks for the authors' response. Although the discussions on the very empirical observations in response to 'W2' cannot fully convince me, I feel the response addresses most of my concerns. I increased the score to 6.