Instructor-inspired Machine Learning for Robust Molecular Property Prediction

Summary

This paper proposes a framework InstructMol for utilizing unlabeled data to help molecular property prediction on out-of-distribution (OOD) domains. The framework combines (1) a molecular model *f* that predicts (pseudo)labels with (2) a binary classifier *g* as instructor that evaluates the probability of labels being pseudo and reweighs *f*’s loss. In the experiments, it is compared to several self-supervised learning (SSL) baselines in predictive accuracy and OOD generalization. The experiments also investigate the effects of unlabeled data size and the instructor model’s behavior as an uncertainty estimator.

Strengths

- Utilizing large, unlabeled, and potentially distribution-shifted data is relevant in chemical and materials sciences, and this paper presents an effective method for that. - This paper is well-written, the figures are informative, and the experiments are comprehensive.

Weaknesses

- Sec. 2’s review on related works is comprehensive but could be better organized. - The separation of pretraining and SSL is confusing: techniques like contrastive learning are often viewed as SSL, and some SSL algorithms work as pretraining. - The mention of methods “jointly learning multiple tasks” (Line 171) seems more appropriate in related works than method. - The paper presents strong empirical results but not enough interpretations. Compared to SSL, what extra information does InstructMol utilize, or why does it extract information more effectively, that leads to better accuracy/generalizability? Discussing these could provide more insights to future model development.

Questions

- In Line 33, domain knowledge being biased is raised as a problem. (1) Could some “bias” be helpful, e.g., by providing correct inductive bias to the model? (2) I suggest citing relevant literature on the scenarios where bias becomes a problem. - Line 78, “usually follow significantly different distributions” seems exaggerating. Is there justification? - Line 97, could uncertainty quantification methods, e.g., Bayesian NN, SNGP, evidential DL, address the problem for regression tasks? - Section titles of 5.3 and 5.4 (Visualization) are confusing. - The meaning of “confidence score” is unclear: in the main text (Lines 156, 301), 1 indicates pseudo-label, while Fig. 5 seems the opposite.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Discussed in Appendix.

Summary

The paper introduces InstructMol, an innovative learning framework designed to address the challenge of data sparsity in chemical and biological sciences by leveraging large-scale unlabeled data through reliable pseudo-labeling. Unlike traditional methods that rely on transferring knowledge between domains, InstructMol operates within a single domain, eliminating potential discrepancies between pretraining and fine-tuning stages. The authors demonstrate the algorithm's high accuracy using various real-world molecular datasets and out-of-distribution (OOD) benchmarks, demonstrating its effectiveness in enhancing machine learning applications in biochemical research.

Strengths

- Well-written and easy to follow - InstructMol effectively addresses the data scarcity issue in biochemical data by leveraging large-scale unlabeled data without the need for domain transfer. - Extensive experiments are conducted to demonstrate the efficacy of InstructMol.

Weaknesses

- The paper does not provide a detailed analysis of the computational complexity or resource requirements of the InstructMol algorithm. - While the paper showcases the superior performance of InstructMol, it does not sufficiently address potential overfitting issues that may arise due to the iterative use of pseudo-labels. - Although the paper compares InstructMol with several baseline methods, it does not include a thorough comparison with some of the latest advancements in semi-supervised learning and domain adaptation techniques.

Questions

- Due to limited discussion on computational complexity, it difficult to assess the practicality and scalability of the approach for very large datasets or in resource-constrained environments. - It was mentioned that determining 'k' in InstructMol is important; please show the experimental results and discussion related to this. - It would be beneficial to have an experiment to determine whether using a poor model for model $f$ can still result in performance compensation due to the confidence score, or if it leads to a decline in performance. - Why RMSEs in Figure 3 become worse as the training data increases? - What model and data were used for training on "Real-word Drug Discovery"?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed limitations in Appendix.

Summary

This paper targets the problem of label-scarcity in the domain of molecular property prediction. It can be seen as an improved version of proxy labeling. It utilizes a separate model that measures pseudo-labels’ reliability and helps the target model leverage large-scale unlabeled data. This method applies to both classification and regression tasks. The authors run numerous experiments on predicting molecular properties, OOD generalization, and combination with pre-training models.

Strengths

The target problem is both timely and important, with a well-justified motivation. The proposed method alleviates some issues by utilizing an instructor model to predict confidence scores. Overall, the method is clearly presented, and adequate experiments are conducted and documented.

Weaknesses

The improvement in model performance is relatively weak, with large standard deviations, and the method is compute-demanding due to the separate model and the iterative procedure. Additionally, some technical details and experimental results are unclear.

Questions

* In Line 201-202, how are the average increase in AUC-ROC and the average decrease in MSE defined and calculated? Why are there only three numbers for the six classification tasks? * I assume the architecture of the InstructMol model is not necessarily the same as the target molecular model. The article does not clearly explain how the InstructMol model is constructed.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors briefly touched upon the necessity of developing a self-supervised learning algorithm better aligned with InstructMol than existing methodologies. Could the authors briefly elaborate on the extra compute incurred by the instructor model?

Summary

The authors develop a method, called InstructMol, for adding pseudo-labels to any training task by including an "instructor" that is trained to discriminate real labels from pseudo-labels, and whose uncertainty is used to modulate the training loss for the primary predictors. The authors show that adding this instructor model improves overall performance over similar methods across a number of property prediction tasks. The authors also show that pretraining delivers state of the art results on the MoleculeNet benchmarks.

Strengths

- novel general pseudo-labeling method - novel loss for extracting signal from all pseudo-labels, even when the estimated uncertainty is high - top results amongst comparative models, especially the results in Table 1

Weaknesses

- Using GIN, GAT and GCN for the results in Table 1, but then GEM for those in Table 3, makes it feel like the results are cherry-picked, especially since the GEM+InstructMol results in Table 3 are comparable to and within the error of some of the other methods. - It is unclear how the results of Figure 4 were obtained. The reader assumes these are comparable to the GEM+InstructMol of Table 3, but this should be better explained. - The distance of the 9 molecules examined in the real world drug discovery section to the training set should be examined, even if the fact that these were patentable suggests they are dissimilar from known molecules. Also, there are likely many more examples like this that could have been provided more prospective evaluation, and only showing one makes the reader again wonder if the example is cherry-picked. - It would have been useful to see similar plots to those in Appendix C for other uncertainty estimation methods as a way of showing that InstructMol learns better separation between real and pseudo labels.

Questions

- I can't tell from the paper if the benefit that comes from InstructMol is due simply to training on more data or for more iterations, simply due to having more labeled data. In which of the experiments is this possibility controlled for? Is it Table 1 where other pseudo-label methods are compared to? If such a control exists, will you please make this explicit in the paper? If not, this seems a critical experiment to run. - Why not use TDC benchmarks instead of MoleculeNet? While the datasets are obviously similar, TDC has done some additional work to clean them, provide reasonable splits, etc.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Summary

The authors present "InstructMol" which does not require transferring knowledge between multiple domains, which avoids the potential gap between the pretraining and fine-tuning stages. and demonstrate it on real-world molecular datasets and out-of-distribution (OOD) benchmarks.

Strengths

Instructive Learning Framework helps the model the generalize better for out-of-distribution molecular property prediction task.

Weaknesses

The paper mostly focus on GNN as backbone but would be worth discuss more about transformer based model trained on SMILES representation.

Questions

How would Instructive Learning Framework works on transformer based model trained on SMILES representation?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper mostly focus on GNN as backbone but would be worth discuss more about transformer based model trained on SMILES representation.