LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset

Summary

This paper proposes a new instruction-tuning dataset, SMolInstruct, that includes 3M+ data points across 14 chemistry tasks, divided into 4 main categories, to help improve the performance of large language models in chemistry. To this end, the paper describes the data collection, curation, and splitting processes used to obtain SMolInstruct, and empirically evaluates the performance of standard open LLMs fine-tuned on this dataset using LoRA against both (i) state-of-the-art (SoTA) models that narrowly focus on individual tasks, and (ii) state-of-the-art language models like GPT-4 and Claude 3. The paper finds that fine-tuning on SMolInstruct allows the base LLM to substantially outperform the best-performing LLMs and narrow the gap with the SoTA models for the individual tasks significantly. To support these findings, the paper also conducts ablation studies by changing the underlying molecular representation and fine-tuning on different instruction tuning dataset, observing that SMolInstruct is the key factor behind the observed empirical improvements.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The proposed dataset is large, well-curated, and a useful addition to the literature. - The empirical results shown in the paper are encouraging. Moreover, they are obtained with little computational resources, and thus there is strong reason to believe that a more rigorous computational exploration could unlock further improvements. - The discussion on underlying molecular representations and the importance of the fine-tuning process is insightful and important. Indeed, the argument about pre-performing canonicalization highlights that LLMs inherently struggle with more structure-related tasks, and further support the need for better-designed datasets, as well as more careful integration of LLMs with specialized components. - The empirical analysis (baselines, protocol) is comprehensive and insightful.

Reasons to reject

- The paper's argument about the differences in the fine-tuned base model performances is not rigorous enough. In particular, the hyper-parameter tuning conducted for these results is not complete enough to conclusively assert the performance differences. The point raised about the original training is plausible and pertinent. However, I would still tone this down to better reflect the significance of the results. - The paper does not evaluate on existing benchmarks for the tasks being studied, instead proposing its own test split on its own data, which makes findings less comparable with other approaches. There are well-established benchmarks for at least some of these tasks. For instance, molecular property prediction benchmarks based on MoleculeNet are very common, e.g., in the area of graph neural networks (GNNs), which the authors cite (see the OGB [1] paper for the specific benchmarks). Hence, I strongly advise the authors to re-design their splits to align with existing baselines, i.e., vetting the training split of SMolInstruct, which includes MoleculeNet data, to avoid leakage w.r.t, the established baselines, and then running similar experiments on the known benchmarks. This would not only provide a real-world reference task with relevant baselines, but also consolidate the generality of your results. [1] Hu et al. Open Graph Benchmark: Datasets for Machine Learning on Graphs, NeurIPS 2020.

Questions to authors

- Why did you not follow existing benchmarks (where available) for the tasks at hand?

Summary

This is a solid dataset paper, which introduces a new domain-specific text-based dataset for instruction tuning. Overall, this is a high-quality, somewhat niche, but thorough paper that shows how dataset papers ought to be done. It is very clearly written (except for a few typos here and there), provides the necessary context for a subject matter expert to understand the data generation procedure, and goes into meticulous detail of generation protocol, dataset statistics, comparison to other datasets, and experimental evaluation details. Given the importance that text-based benchmarks, even those in niche domains, have recently gained for general-purpose evaluation of LLMs, the impact of this paper is likely to be high in the community.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

* Very likely will have moderate-to-high impact in the LM community, even if only "an benchmark dataset" in an LM eval suite * Thorough description of data collection/curation pipeline * Careful control of data leaking for reciprocal tasks (crucial for LLM benchmarks!) * Modest but thorough baseline evaluation

Reasons to reject

* Claims dataset size as a strength, but it is only 3x compared to most similar dataset (Mol-Instructions) * Dataset is somewhat niche / narrow

Questions to authors

* Q: it's unclear whether the tagging strategy used (e.g. for <SMILES>) is textual or whether it relies on a dedicated embedding for the tag (e.g. alla tagLLM https://arxiv.org/abs/2402.05140). Please clarify. * S: label all tables, even those in the appendix, with arrows to indicate lower/higher is better * S: there's a few typos throughout the paper, including "spitted into training" (Sec 3.2)

Summary

This paper proposes SMolInstruct, a large-scale instruction-tuning dataset designed with over 3 million samples and 14 chemistry tasks. The authors use this dataset to finetune several open-source LLMs like Galactica, LLaMA, and Mistral, creating the LlaSMol series of models tailored for chemistry applications. Extensive experiments show that the LlaSMol models significantly outperform existing LLMs like GPT-4 and Claude on the chemistry tasks, though they still trail behind task-specific models. The authors conduct analyses on different modeling choices like canonicalization of SMILES, use of SELFIES vs SMILES representations, and comparison to a previous chemistry instruction dataset.

Rating

5

Confidence

5

Ethics flag

1

Reasons to accept

1. SMolInstruct is large enough and diverse enough for instruction tuning of LLMs, with 3 million carefully filtered samples over 14 downstream tasks. 2. Four backbone LLMs of different architectures are evaluated in the experiments of this paper, and the performance gains are significant. 3. Careful dataset splitting avoids potential data leakage across related tasks.

Reasons to reject

1. All the data collection, validation, and processing techniques have been used by previous studies. The novelty of this paper is limited. 2. The evaluation metrics like exact match do not fully capture the models' capabilities. For example, this results in too many zeros in Table 1. Some better metrics may help reveal the models' performances. 3. The paper does not explore the models' generalization abilities beyond the specific downstream tasks in the dataset. 4. While outperforming existing LLMs, the LlaSMol models fail to surpass the non-LLM models which have far fewer parameters.

Questions to authors

I don't have additional questions for the authors.

Thank you for the responses

Thank you for the responses and clarifications. My assessment of the strengths/weaknesses of the paper remains the same. I still think the dataset's scope is narrow (which does not mean it's not significant/relevant - my original review does not put that in doubt).

Reviewer response

I thank the authors for their response. I will keep my verdict.

Feedback

``` For the first time, PubChem and MoleculeNet are used for instruction tuning, which is not trivial. ``` **Response:** PubChem is used for instruction tuning in 3D-MoLM [1]; MoleculeNet is used in Mol-Instructions [2]. ``` Some relevant works are not appropriately acknowledged in the related work: ``` * *molca: molecular graph-language modeling with cross-modal projector and uni-modal adapter. In EMNLP 2023.* **It seems LlaSMol share many similar tasks with this earlier work, including 1) presents property prediction results on the MoleculeNet datasets (e.g., BBBP, Clintox, and SIDER); 2) presents name conversion results; and 3) present molecule caption results** * *Towards 3D Molecule-Text Interpretation in Language Models. In ICLR 2024.* This earlier work also employs Uni-Mol for molecule property prediction, and shares some other tasks. **Why are these relevant works not discussed or compared?** **Reference:** [1] Towards 3D Molecule-Text Interpretation in Language Models [2] Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models