GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

Summary

This paper introduces GIMLET, a unified graph-text model for instruction-based molecule task pretraining. It leverages natural language instructions and decoupled graph encoding techniques to improve the interaction of graphs and texts. Through experiments and evaluations, the paper demonstrates the effectiveness and robustness of GIMLET in various downstream tasks, showcasing its potential for molecule zero-shot learning.

Strengths

- This paper proposed GIMLET, a novel unified graph-text model that leverages natural language instructions for pretraining. It takes each node as a token, alleviating the problem of losing structural information using separate graph encoding. - The authors collected a set of datasets for pretraining, which can be a useful resource for the community. - The empirical results look promising compared to previous methods. - The paper is overall well-written and easy to follow.

Weaknesses

- The authors didn't present the training details, e.g. total iterations, hyper-parameters, etc. - GIMLET was trained from scratch with a large dataset. What are the computational requirements (GPU/memory/time) for training such a model? It seems the training can be quite expensive, especially considering the graph encoding part. Minor: - Page 4, line 126: need to explain what is $o_m$. - Page 9, line 333: nature language -> natural language.

Questions

- For the pretraining dataset, I understand the authors isolated the tasks for training/testing, but are the molecules also isolated for train/test, i.e. will the same molecule appear in both the training and testing set? - For graph, node embeddings are used as input tokens. If the number of nodes varies a lot for different graphs, how do the authors handle the computational efficiency as a max length (padding) will be set for the input?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

I'm not an expert in biology/chemistry, so I'm not sure if the procedures for constructing the dataset are correct.

Summary

The paper proposes a new unified language model for graph and text data for instruction-based molecule zero0shot learning. The paper tries to address the problem of a supervised fine-tuning approach where labeled data by instruction tuning. The paper first treats both graph nodes and instruction tokens as input tokens. The paper then proposes a new relative position embedding to represent the edges in the graph and the relative position for text. The paper then retrains the model on Chembl and sometimes pm several tasks from MoleculeNet.The instructions are based on websites and databases. In a zero-shot setting, the model is compared against several baselines, including KVPLM, Galactica, and MoMu. The model is also compared to the graph-based models, including GCN, GAT, GIN, and Graphormer, in a supervised setting. The paper applies few-shot instruction-0based tuning for several datasets and compares it against KVPLM, MoMu, and GIN. The paper then conducts ablation and exploration studies.

Strengths

1. The paper proposes a new unified text-graph T5-based model to unify molecular graphs and task instructions. The idea of relative position embedding for molecular graphs is interesting. 2. The paper achieves strong performance against several zero-shot baselines with fewer parameters. The paper also conducts experiments over multiple different tasks. The model also performs well on the few-shot settings. The paper shows its model design by conducting an ablation study to analyze the effectiveness of unified inputs and decoupling encoding. The paper also shows the effectiveness of pertaining over two different pertaining dataset. The paper also examines the robustness of instruction by GPT3.5 paraphrase. Finally, the paper ablates the explanation of instructions in downstream tasks. 3. The paper includes a very detailed appendix to explain some parts of the paper. The paper also visualizes attention on molecular to help readers understand the model.

Weaknesses

1. The idea of encoding graphs with a seq2seq transformer has been introduced previously. This idea has been applied to graph-to-text tasks since 2021 (Ribeiro et al., 2021), despite using an unchanged T5 structure. One contribution of this paper is the new relative position embedding. However, the paper fails to include it in the ablation study. 2. The input of the model needs to be clarified. It would be better to include a sample input-output pair in the Appendix. Does the model use each node's chemical element as the molecular graph node's textual form? The datasets used for testing are hard to understand for readers who are not familiar with them. It would be better to include a small explanation for each testing dataset in the Appendix. 3. The paper fails to include a limitation section and an ethical section for potential social impact. Ribeiro, L., Schmitt, M., Schutze, H., & Gurevych, I. (2021). Investigating Pretrained Language Models for Graph-to-Text Generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI (pp. 211–227). Association for Computational Linguistics.

Questions

Does the model use each node's chemical element as the molecular graph node's textual form?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper fails to include a limitation section. The paper needs to include a social impact section to discuss the potential misuse of the current system.

Summary

This paper proposes a unified language model for both graph and text data with two main tech contributions: (1) a unified graph-text transformer encoder with a distance-based joint position embedding to encode graphs, and (2) textual instructions to enhance transferability among tasks. Zero-shot tests on classification and regression tasks are conducted to show its effectiveness.

Strengths

The high-level motivations of a unified graph-text model for molecular understanding and enhancing transferability with instructions are interesting.

Weaknesses

1. It doesn’t make sense to evaluate the model on classification and regression tasks. The model structure is based on T5, which is for generation/translation tasks. In fact, the results in Table 1 show a huge performance gap between GIMLET and GNNs. Why not use those GNNs and Graphormers with better performance and fewer parameters? I think additional experiments of fine-tuning GIMLET on the train datasets of Bio-activity, Toxicity, and Pharmacokinetic tasks in Table 1 are needed to make a fair comparison with supervised GNNs. 2. Claims need to be further clarified. For example, in line 150-154, I don’t understand “the dense vectors encoded by GNN have a limited capacity to carry structure information”. According to the results in your Table 1, those dense vectors encoded by GNN actually carry more information than GIMLET. And “training the additional module is difficult due to the increased layers”, actually, in MoMu and CLAMP (cited in line 146), the graph and text encoders are two-tower structure like CLIP, which I don’t think increased layers. 3. The writing needs to be improved.

Questions

Critical question: Do we really need a text-decoder model instead of conventional GNNs to perform classification and regression tasks?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See weakness part.

Summary

This paper presents GIMLET, a unified graph-text model for instruction-based molecule zero-shot learning. The proposed model uses natural language instructions to tackle molecule-related tasks in a zero-shot setting. GIMLET overcomes existing limitations and significantly outperforms molecule-text models. The paper includes experimental results that demonstrate the superiority of GIMLET over molecule-text models. The paper also explores the use of pretraining tasks and the robustness of GIMLET to instructions. Overall, the paper provides a comprehensive approach to molecule zero-shot learning using natural language instructions.

Strengths

This paper introduces GIMLET, a pioneering, integrated graph-text model specifically designed for instruction-based, zero-shot learning in molecular sciences. Leaping beyond the confines of current methodologies, GIMLET leverages natural language instructions to proficiently navigate molecule-related tasks in a zero-shot context. It surpasses the performance of conventional molecule-text models, reflecting substantial advancements in this field. Through rigorous experimental analysis, the superiority of GIMLET over conventional models is vividly illustrated.

Weaknesses

1.One thing I'm not quite sure about is why the authors chose only a few molecule-text models as baselines, with MoMu and Galactica not even being specifically designed for predicting molecular properties. I understand that the authors might have done this because the model proposed in the paper is based on both molecules and text, but for a fair comparison, shouldn't other types of models (like those involving only molecules and not text) that predict molecular properties also be used as baselines? Otherwise, how can we highlight the advantage of incorporating text into the model for this task? 2.The paper uses a graph-text position encoding method to encode molecular graphs. Given that molecules already have their serialized representations, such as SMILES and SELFIES, isn't this approach superfluous and potentially introducing unnecessary computations and noise? 3.Based on the description in this paper, I can't clearly understand the specific definition of molecule zero-shot learning, which doesn't seem to differ much from other pre-training-fine-tuning training methods? Perhaps I misunderstood, could the authors provide a more precise explanation? 4.In Figure 1, the authors don't seem to have clearly described the specific meaning of different colored lines and boxes, which makes it difficult for readers to interpret this image.

Questions

See weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

The paper addresses the challenge of molecule property prediction, especially the label insufficiency caused by costly lab experiments. It uses natural language instructions to handle molecule-related tasks in a zero-shot setting. The authors propose GIMLET, a model that unifies language processing for both graph and text data. GIMLET uses generalized position embedding to encode graph structures and instruction text without additional modules. It also decouples the encoding of the graph from task instructions, thereby improving graph feature generalization across different tasks. The authors create a dataset of over two thousand molecule tasks with corresponding instructions and pretrain GIMLET on these tasks, which allows the model to transfer effectively across tasks. GIMLET demonstrates good performance in instruction-based zero-shot learning, even closely matching supervised Graph Neural Network (GNN) models on tasks such as toxcast and muv.

Strengths

- The paper is well-written and clearly presented; - The authors create a comprehensive dataset for molecule tasks with instructions, which can be a valuable resource for future research. - The ideas of leveraging instruction plus graph structures are novel ways for molecule property pretraining and the proposed unified GIMLET for graph and text data could contribute to the field of cheminformatics. - The zero-shot performance of proposed methods is strong across baselines and tasks. Detailed ablations including pretraining, model design, and instructions are presented to illustrate the design choices.

Weaknesses

- While GIMLET demonstrates strong performance in zero-shot learning, how could it perform with more supervised learning scenarios (beyond 256 or even with full training set) is questionable. The instruction-tuned LLM demonstrates strong generalization performance on single-task finetuning in NLP, is this also true for GIMLET remains a question; - How could model size (only one size is presented), the scale of pretraining, and the diversity/quality of instruction / pretraining tasks affect GIMLET performance, it could also be valuable to see the performance of instruction-aided evaluation / qualitative examples on helf-on validation tasks (whether GIMLET could actually follow the instructions).

Questions

- How model size, and the scale of pretraining (not only the domain) contribute to the final performance of GIMLET; - Could GIMLET be extended to using frozen / parameter-efficient fine-tuned language models? / How to validate if the instruction-based pretraining / the proposed GIMLET model is scalable?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above

Thanks for the rebuttal

The authors have addressed my questions. I'm maintaining my previous score and recommend acceptance.