LLaMo: Large Language Model-based Molecular Graph Assistant

Summary

The paper introduces LLaMo, a Large Language Model-based Molecular graph assistant. LLaMo is a model that integrates a GNN encoder, a multi-level graph projector, and a language model. The projector uses a cross-attention mechanism to convert graph representations into graph tokens by abstracting outputs from each GNN layer and motif representations. Additionally, the authors use machine-generated molecular graph instruction data, leveraging GPT-4, for instruction-tuning. Extensive experiments show that LLaMo excels in some tasks.

Strengths

The framework incorporates a multi-level graph projector that effectively captures multi-scale information. This design mitigates the over-smoothing problem. A figure is provided to illustrate the significant impact of this design on model performance and attention distribution. The paper offers a comprehensive, step-by-step explanation of each component within the LLaMo framework. Additionally, a lot of typical examples are presented to clearly demonstrate the framework's design and functionality in practical applications.

Weaknesses

In order to perform instruction-tuning, the authors utilized GPT-4 to generate molecular graph- text instruction-following data using graph-text pair datasets with human-written instructions. Although this approach enhances the results, it lacks a robust evaluation of the generated data's quality and validity. It is crucial to develop methodologies to rigorously assess the accuracy and relevance of the GPT-4 generated instruction data to ensure the reliability of the model's performance improvements.

Questions

The two-step training process involves initially training the graph encoder in Stage 1, followed by fine-tuning the Large Language Model using LoRA in Stage 2. Why is the multi-level projector set as trainable during both training stages?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The experiments predominantly concentrate on a few tasks such as molecular description generation, property prediction, and IUPAC name prediction. This limited focus may not fully reveal the model’s versatility and potential shortcomings across a wider spectrum of molecular tasks.

Summary

This paper proposes a molecule-text model, LLaMo, which aligns text and molecule modalities to tackle diverse downstream tasks with a language interface. Through two-stage alignment: training a graph encoder and then tuning the LLM through instructions, LLaMo aligns text and molecular representations well, and achieves promising performance on diverse downstream tasks.

Strengths

1. The writing and presentation of this paper are clear. It is well-motivated and easy to read. 2. Code implementation and comprehensive details are provided, which ensures reproducibility. 3. The design of learnable query tokens is interesting. 4. I observe an interesting phenomenon in the experiments. Low-level information is sometimes overlooked for graph-level prediction, and it is mainly studied for node-level predictions. However, for molecule-text models, some tasks require both local and global-level information, and it presents more importance for tasks like IUPAC prediction.

Weaknesses

1. The implementation details of functional groups are not given in the main text nor appendix. Could you share more details on this part? 2. How is the graph modality treated in the instruction tuning phase? Is it fixed, and the graph token is augmented into the text token inputs? 3. I am curious about the design of the learnable query tokens. Why are the motif and graph-level representations modeled separately? 4. One factor that prevents me from giving a higher score is that this pipeline is widely adopted for vision-language models, so it does not surprise me.

Questions

Please refer to the questions given in the weakness part

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations and broader impacts are discussed in the appendix. Guidelines should be removed from the checklists.

Summary

This paper proposes a Large Language Model-based Molecular graph assistant (LLaMo), which can enhance the molecular graphs's general-purpose understanding and generation capabilities. By integrating molecular graph encoders with large language models, LLaMo enables instruction-following responses in the molecular domain. The paper conducts extensive empirical studies and justifies the effectiveness of the proposed method.

Strengths

- The proposed LLaMo has a clear and effective design, making it powerful in molecular description and property prediction. - Extensive experiments have been conducted to provide a good insight into the components of the proposed method. - The paper is generally well-written, with clear illustrations and tables.

Weaknesses

- The multi-level graph projector shares an idea similar to JKNet[1]. In addition, Graph Transformers [2] can also handle the over-smoothing problem, especially for small molecules. The author could consider discussing how the proposed techniques differ from these referenced works. - The paper uses function groups as prior information for training; would this cause information leakage? The downstream task is to predict these groups or generate molecular captions based on these motifs. - The GPT-4 generates samples for instruction-tunning, but the samples's qualities are not assessed properly. [1] Representation Learning on Graphs with Jumping Knowledge Networks. In ICML, 2018. [2] Representation Learning on Graphs with Jumping Knowledge Networks. In NeurIPS, 2022.

Questions

1. Could GPT -4 perform better with prior information instilled, such as function groups? 2. What is the purpose of using different LLM for LLaMo tasks (Tables 2 and 3)? Could better LLM result in better performance? Would there be some upper bounds? 3. Could LLaMo design or edit SMILES based on the user's instructions?

Rating

7

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The idea is novel and effective. It would be better if the author could make LLaMo able to design or edit SMILES based on the user's instructions.

Summary

This paper presents LLaMo, a novel enabling LLMs and instruction tuning in molecular domain. To bridge the gap between different modalities, the paper also proposes a projector that transforms the graph representations into graph tokens level by level. The authors conduct extensive experiments and compare LLaMo with several proper baselines. The results are convincing.

Strengths

1. The proposed method makes a good contribution to enable LLMs into the molecular domain. It successfully bridges the gap between language and graph modalities. 2. In terms of quality, the experiment setup is sound. Also, the authors conduct detailed and extensive experiments. It seems evident that LLaMo performs better than the baselines mentioned in the paper. 3. The paper is well-written and well-structured overall.

Weaknesses

1. Though the authors conduct several experiments, they still lack some comparison between different GNN and LLM backbones. In section 5.3(Impact of Multi-level Graph Projector), will different GNN backbones harm or improve the results? Also, under the same experiment settings, the paper lacks the comparison between different LLM backbones. Such as the settings in Table 2, will the results be better for LLama-7b? 2. With several layers of GNNs, the efficiency of this projector may not be good.

Questions

1. For line 138-139, could you explain in detail for the $\[p^{(l)}_1, .... , p^{(l)}_b\]$, such as what is the learnable prompts and how do you initialized it? 2. For the few-shot prompts part, how do you choose the few-shot examples?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors address the limitations of this work.

Official Comment by Reviewer P4Gm

Thank you for the clarification. It addresses my concerns, so I will raise my score to 7.