We introduce GAUCHE, a library for GAUssian processes in CHEmistry. Gaussian processes have long been a cornerstone of probabilistic machine learning, affording particular advantages for uncertainty quantification and Bayesian optimisation. Extending Gaussian processes to chemical representations, however, is nontrivial, necessitating kernels defined over structured inputs such as graphs, strings and bit vectors. By defining such kernels in GAUCHE, we seek to open the door to powerful tools for uncertainty quantification and Bayesian optimisation in chemistry. Motivated by scenarios frequently encountered in experimental chemistry, we showcase applications for GAUCHE in molecular discovery and chemical reaction optimisation. The codebase is made available at https://github.com/leojklarner/gauche
Paper
Similar papers
Peer review
Summary
Gaussian processes are widely used for black-box optimization when data is scarce. On the other hand, effectively representing molecules, proteins, and chemical reactions is a dedicated research area in molecular machine learning. Although separate tools exist to address these two challenges, this paper introduces a novel library that integrates both. By doing so, it enables chemists without extensive knowledge of Bayesian optimization to harness its benefits.
Strengths
The library's objective is clearly defined and holds potential for the chemistry and machine learning research communities. The persuasive comparison with existing work in Section 5 strengthens its position. Sections 2.3 and 2.4 provide thorough explanations of molecular and chemistry reactions.
Weaknesses
I find the objective of the paper somewhat unclear. Here is my understanding and suggested improvements: * The paper's main aim, as summarized in Section 5, is to create a unified library called GAUCHE that combines chemistry libraries (for molecule/chemical reaction representation) with Bayesian optimization libraries. Such libraries exist separately, GAUCHE aims to unify them. * Sections 2 and 3 effectively describe the library's provided representations and kernels. * However, I fail to grasp the purpose of the experiments described in Section 4. The message seems to be that "the library works and enables Bayesian optimization on various chemistry benchmarks." This leaves room for improvement: what sets GAUCHE apart? Why should users choose it over manually combining a molecular representation with a GP library? Section 5 partially answers these questions – and should be presented earlier in the paper, in my opinion. * Overall, it appears that the library does not offer new theoretical insights or algorithms. While this is not necessarily a problem (as GAUCHE fills a relevant niche, allowing non-GP-expert chemists to utilize state-of-the-art black-box optimization for chemical reactions), **the paper should clearly emphasize the library's strengths**. These may include (a) ease of use, (b) a modular code structure enabling user extension, or (c) superior performance on public benchmarks. In that spirit, a helpful addition would be a table comparing different libraries, with the different libraries (containing GAUCHE) in rows and the different features (GP, molecular representation, etc) in columns, and cells indicating whether the feature is implemented or not.
Questions
* `l. 60`, the covariance $\sigma_y$ is not introduced. I understand that this assumes a Bayesian linear model of the form $f({\bf x}) = \phi({\bf x})^\top {\bf w} + \sigma_y \epsilon$ where $\epsilon$ follows a standard normal distribution and $K({\bf x}, {\bf x}') = \phi({\bf x})^\top \phi ({\bf x}')$, as in [1, Eq. 2.21]. I am not familiar with the BO literature, and perhaps it's the only model considered, but it would be worth mentioning that you are considering a Bayesian linear model, along with a proper citation to justify the formula. * `l. 68`: again, adding a reference e.g. [1, Eq. 5.8] seems necessary to justify the NLML formula. * Fig. 1 is not informative for someone who does not have GP or chemistry background. After reading sec. 2.{3, 4, 5}, we understand that the 3rd row show the possible representation for the different applications considered in the 2nd row (even though "SMARTS" in not described in sec. 2. 4). In any case, the figure would benefit from a more detailed caption, so that the message it is trying to convey appears clearly. * Table 1 and 2: why the font is so small? It would be convenient to have the same font size as the rest of the document. * `l. 227`, I don't know the 3 metrics for measuring the quality of the uncertainty estimates. A word of introduction explaining what they measure, what are their strength and limitation would be welcomed here. **Related literature.** * You may be interested in [2] which provides an convolutional kernel network for graph-structured data. **Typos.** * `l. 55`: $m(\mathbf{x'})$ instead of $m(\mathbf{x})$ * `l. 80`: extra "the" **References.** [1] Rasmussen and Williams - 2006 - Gaussian processes for machine learning [2] Convolutional Kernel Networks for Graph-Structured Data – 2020 – Dexiong Chen, Laurent Jacob, Julien Mairal
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The limitations identified by the authors regarding specific algorithms for certain problems are noteworthy but can be left for future research. Aside from that, my main concerns are summarized in the "Weaknesses" section mentioned previously.
Summary
The authors discuss Molecular, Reaction, and Protein Representations and provide a unified framework for these models. Python's GPyTorch library is used to train the Gaussian processes. The authors define certain kernels for Gaussian processes to fit and perform several experiments to evaluate the performance. The paper also contains a brief overview of Gaussian processes and Bayesian optimization.
Strengths
The authors properly analyze the related kernel functions used in chemistry. Indeed, they provide an overview of the applications and representations available in the GAUCHE library. The library seems easy to follow and can adapt to proteins, molecules, and chemical reactions.
Weaknesses
The major weakness is the contribution. The training task uses GPyTorch, a powerful library in Python for Gaussian processes. The main contribution of the paper is to provide some important kernels in chemistry. It seems some of the kernels have been developed in other libraries or their scripts are available on the internet. The theory part of the paper is not rich enough. Section 3 only explains the relevant kernels and does not provide novel ideas or solutions for the methodological challenges. The Gaussian process methods and kernels are already there without new inventions, and the paper does not describe how to adapt them to chemistry problems. It would be great if the authors describe the major benefits of the proposed solution compared to the other libraries related to the Gaussian processes, except the kernels.
Questions
What are the major limitations of the available packages in R/Python so that they can not be used in the chemistry problems and how this package addresses them? If the kernels are omitted, what is the main advantage of the library compared to the others?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors addressed the limitations of the paper.
Summary
This article presents a library for Gaussian process-based inference with a special focus on chemistry applications. At heart, the library contains two classes of objects: kernel, and data loaders. The article introduces Gaussian processes and chemistry-specific kernels and discusses the interfacing of the library with other frameworks, in particular relating to Bayesian optimisation.
Strengths
The proposed library seems to fill a gap in the open-source GP stack, providing specialised building blocks for use in chemistry. The code cleanliness is rather high and the code is well-tested. The paper is fairly clear (see however my questions in the weaknesses), and provides a reasonable introduction to the problem and to the GP literature around it, some illustration of the superior performance of GPs (compared to Grakel) is adequately shown. The relation to prior works and existing solutions is also well documented.
Weaknesses
As it stands, there is no clear indication in the article that the library has had any impact on the practice of data-driven chemistry. If possible, the authors should include examples of real-life uses of GAUCHE, and if there is none, it then would mean that the library is likely not mature enough to warrant a full-size article (although it would likely be a good workshop paper). I have found it hard to understand which "20+ bespoke" kernels were in fact implemented. For instance, the graph kernels are described in the article, but the code provided shows nothing under "gauche/kernels/graph_kernels". Similarly, I cannot count 20 kernels in the code and would like for the authors to specifically list these. The future of the library is also not mentioned: what is the governance model and what are the next steps? Is it to implement more kernels? When new kernels are implemented, then what? As it stands it seems that the authors suggest they will "wait and see", depending on the feedback they receive from the practitioners. This makes me believe that publishing an article on the library itself is then fairly premature.
Questions
As discussed in the weaknesses, my main reasons for the negative rating are (i) that the supported methods are not well documented within the article (ii) the future of the library is really unclear: for instance > We seek to further grow our userbase and solicit feedback from laboratory practitioners on the most common use-cases for BO and GP modelling in molecular discovery campaigns. is an admission of stale development and of an unclear vision for the future of the library. Some clarity on this would be much appreciated. A remark on my confidence level: I am by no means in the position to judge the relevancy of introducing a package for chemistry-specialised kernels and do not really understand the end applications. I will therefore fully defer to other reviewers when it comes to this.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
NA
Summary
This paper presents a library for Gaussian processes on chemistry data. In this library, a number of kernels are implemented over chemical representations such as graphs, strings and bit vectors. Regression and Bayesian optimization experiments are shown using the library.
Strengths
- GP has been widely used as a ML tool for Chemistry. - A reliable library can lower the barrier for chemistry experts on using ML tools.
Weaknesses
- All of the implemented kernels are known in the literature. - Neurips is probably not the right venue for publishing software libraries.
Questions
Neurips is probably not the right venue for publishing software libraries.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The limitation of the proposed method has been explicitly discussed.
Summary
The authors present a framework called GAUCHE with comprehensive exploration of Gaussian Processes (GP) and their application to molecular machine learning. The authors thoroughly examine different ways of representing molecular structures - through hand tuned fingerprints, string notations (SMILES/SELFIES/Protein sequences), and undirected graphs. The authors evaluate the proposed kernels and representation schemes across several diverse benchmarks involving regression tasks, uncertainty quantification, and Bayesian optimization (BO). The experiments use diverse datasets, representing a span of property prediction from single molecule and reaction yield prediction from sets of molecules. Finally, the authors evaluate the best-performing GAUCHE kernels in a Bayesian optimization framework using three distinct datasets. Results reveal that GAUCHE's kernels outperforms random baseline, especially in low data regime, highlighting its practical utility in aiding chemists to prioritize synthesis candidates. Overall, this is an impressive work paper with strong theoretical framework with rigorous empirical evaluation.
Strengths
1. This paper introduces a novel, theoretically robust kernel design for Gaussian process regression tailored for molecular data types. 2. The authors demonstrated the utility of the GAUCHE kernels in Bayesian optimization tasks which highlights their real-world applicability, particularly in the context of supporting chemists in candidate selection for synthesis. 3. The authors ensure reproducibility by providing clear explanations and thorough empirical evaluations as well as a very well designed codebase. 4. The paper uses a variety of datasets ranging from properties applicable to drug discovery and material discovery domain, which showcases the adaptability of the GAUCHE kernels across different data. 5. The authors evaluate the GAUCHE kernels across a wide array of molecular input types ranging from fingerprints, strings and graphs highlighting the versatility of the proposed framework.
Weaknesses
This paper presents a strong methodological framework and robust empirical evaluation, but it does fall short in some key areas. 1. The authors have shown that GAUCHE framework has been proven to be effective in a number of experiments, the comparison against SOTA neural network is notably lacking. For instance, chemprop, and ChemBERTa. 2. The authors emphasizes GAUCHE's applicability for a low data regime, which is certainly a critical need in many scientific domains. However, in cases where large, high-quality datasets are available. For instance, internal databases at big pharma companies or public repositories like BindingDB. It would be great to see experiments around how the computational requirements, training time, and model performance would be impacted when scaling up the dataset size. 3. It would be great if the authors could explore comparisons between RDKit/FragPrints and proprietary fingerprint techniques such as Dragon FP and Schrödinger, provided the licenses and costs are manageable.
Questions
1. How does the splitting strategy affect the results? For example, what if the data were split based on molecular scaffolds, which would ensure the structural diversity of the test set, instead of a random split? 2. In the context of the Uncertainty Quantification benchmark, how does the predicted variance of the model relate to OOD samples? Could you provide a plot of the Tanimoto distance to the training dataset versus the predicted variance? 3. The authors touch upon the potential use of embeddings of molecules from pretrained models as inputs for the kernels, but there doesn't seem to be any results for this approach. Is this something the authors are considering for future investigations? 4. How would the performance of the kernels be impacted with the changes in the hand tuned fingerprint radius? Have any ablation studies been conducted to investigate this?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
1. The GPR models can be computationally intensive, especially for larger datasets if used in conjunction with large dimensional fingerprints etc. This could limit the applicability in scenarios where abundant high-quality data is available. 2. The performance of predictive models in low data regimes is heavily affected by the choice of dataset splitting method. The results obtained using random splitting might not hold if other splitting methods like scaffold or temporal splitting are used.
Thank to the authors for their rebuttal. The table comparing GAUCHE to other frameworks is compelling. I believe that any libraries that streamline the utilization of advanced machine learning tools are valuable. **I am willing to raise my rating from 5 to 6**. I still have reservations about fully assessing the library's contribution.
On GAUCHE's Contribution
Thank you for the quick response! We are happy to hear that you found the table compelling and that you appreciate the value of a library that lowers the barrier for applying advanced molecular machine learning tools in practical research settings. > I still have reservations about fully assessing the library’s contribution. To address your remaining concern and further clarify the contributions of our work, we would like to expand on the points in our initial rebuttal to: 1. Highlight some of the technical innovations that were necessary to extend the existing open-source GP/BO stack to work for discrete kernels that operate over molecular representations (i.e. bit vectors and graphs). 2. Outline the concrete impact our work has had by listing instances in which GAUCHE has already been applied by other researchers in a range of practical settings. ## __Substantial Technical Contributions__ ## One important and major limitation of existing GP frameworks is that they are built with continuous data in $\mathbb{R}^d$ in mind. For instance, the kernel base class of GPyTorch assumes that custom kernel sub-classes are based on Euclidean distance metrics. In GAUCHE, we provide a parallelizable and batch-GP-compatible alternative to this base class that can be easily extended to implement arbitrary bit and count vector kernels. As another example, we would like to point out that current GPU-enabled GP libraries do not natively support non-tensorial inputs, necessitating a substantial amount of engineering work to extend them to graph-structured input spaces. The resulting SIGP class is, to the best of our knowledge, the first open-source GP implementation that enables GPU-accelerated and autodiff-based end-to-end learning over graph kernel hyperparameters, including all kernels in the GraKel library. While certain kernels - such as the Weisfehler-Lehman (WL) kernel - have concrete feature functions that could be fit directly by a GP, many others - such as the random walk and shortest path kernel - don’t, which was our motivation for designing a more general wrapper framework. ## __Real-World Impact__ ## By combining these technical innovations with a range of easy-to-adapt data loaders, featurization functions and notebook tutorials, we made it as easy as possible to integrate molecular GP and BO models into real-world research workflows. Specifically, we would like to mention that GAUCHE has already successfully contributed to a range of academic and industrial research efforts. We are currently aware of at least three application domains in which GAUCHE has featured as a core component of published work: 1. Additive screening for chemical reaction optimization. 2. Catalyst discovery 3. Self-driving laboratories Additionally, GAUCHE has been a core component in enabling novel Bayesian optimization methodologies to be evaluated on molecular datasets. Published work in this direction has included: 1. The evaluation of a novel multiobjective Bayesian optimization scheme on the task of identifying molecules with favourable cell permeability for drug delivery. 2. The evaluation of a novel method featuring Bayesian quadrature on a) the task of identifying molecules with anti-malarial properties and b) the task of identifying molecules with promising solvation capabilities for use in lithium-ion battery electrolytes. We have consulted the AC and SAC on the best way to provide these references without compromising the double-blind review process and are currently waiting to hear back. ## __Summary__ ## We hope that these clarifications are helpful in allowing you to fully assess the contributions of our work and are more than happy to provide additional details and answer any follow-up questions! Sincerely, The Authors
Rebuttal acknowledgement
Thank you for the clear response (both to me and other reviewers). Because sharing the examples is pending AC approval/guidelines, I will refrain from updating my score just yet. As it stands I am however inclined to increase it substantially (subject to the examples provided) > For the graph kernels alone, the total count subsumes the total number of kernels available in the GraKel library (18) I believe this should be very clearly stated rather than the fact that you provide 22+ kernels. The fact that your library immediately extends another one with no real overhead is a good thing. > Examples of Real-World Use-Cases I am eager to see these. > Plans for Future Development I would highly recommend making the short and long-term plans clearer in the conclusion of the article: if anything to prevent the kind of reaction I personally had thinking it was "an admission of stale development and of an unclear vision for the future of the library".
Many Thanks for the Prompt Response and Additional Feedback!
Thank you for your response! We are happy to hear that you found our clarifications helpful and sincerely appreciate the additional feedback. ## __Additional Kernels__ ## > I believe this should be very clearly stated rather than the fact that you provide 22+ kernels. The fact that your library immediately extends another one with no real overhead is a good thing. Thank you. We agree that this should indeed be stated more clearly in the paper and we will ensure that the graph kernel section (3.3) is adjusted accordingly. Motivated by your comment, we have additionally implemented 12 new bit vector kernels based on the following similarity measures from [1] and [2]: 1. Dice 2. MinMax 3. Sokal-Sneath 4. Russell and Rao 5. Sogenfrei 6. Forbes 7. Intersection 8. Faith 9. Otsuka 10. Rogers-Tanimoto 11. Braun-Blanquet 12. Rand All of these measures yield provably symmetric positive-definite kernels [3] and we provide parallelisable and batch-GP-compatible implementations with associated unit tests. We have also evaluated each of these kernels on the Photoswitch dataset, finding that they often slightly outperform the more standard Jaccard-Tanimoto kernel - particularly in the case of the Sorgenfrei kernel (number 5). ## __Real-World Use-Cases__ ## We are still waiting to hear back from the AC/SAC regarding the best way to share external references and will follow up with them shortly. > I am eager to see these. To provide you with as much information as possible whilst we wait, we think that it is safe to share the specific application domains in which GAUCHE has featured as a core component of published work. These include: 1. Additive screening for chemical reaction optimization. 2. Discovery and optimization of novel catalysts. 3. Self-driving laboratories and computational experiment planning. Additionally, GAUCHE has been a core component in enabling novel Bayesian optimization methodologies to be evaluated on molecular datasets. Published work in this direction includes: 1. The evaluation of a novel multiobjective Bayesian optimization scheme on the task of identifying molecules with favourable cell permeability for drug delivery. 2. The evaluation of a novel method featuring Bayesian quadrature on a) the task of identifying molecules with anti-malarial properties and b) the task of identifying molecules with promising solvation capabilities for use in lithium-ion battery electrolytes. We hope that these additional elaborations are helpful and provide useful context to inform your decision-making process. ## __Governance Model__ ## > I would highly recommend making the short and long-term plans clearer in the conclusion of the article Thank you. We agree that the conclusion should more clearly convey our short-term development and long-term governance plans and will adjust it accordingly. Regarding the latter, our aim is to maintain a lean, well-tested, and up-to-date main codebase. Following the maintenance model of GPflow, which has proved successful, we aim to invite community-driven contributions principally as PRs in the form of notebooks (as opposed to extensions to the main codebase) that reflect the needs and considerations that researchers come across in practice. In this fashion, we may support more advanced features without bloating the codebase and increasing maintenance requirements. We thank you again for your feedback and are happy to answer any further questions! Sincerely, The Authors ## __References__ ## [1] Choi, Seung-Seok, Sung-Hyuk Cha, and Charles C. Tappert. “A Survey of Binary Similarity and Distance Measures.” Journal of Systemics, Cybernetics and Informatics 8.1 (2010): 43-48. [2] Todeschini, R., D. Ballabio, and V. Consonni. “Distances and Similarity Measures in Chemometrics and Chemoinformatics.” Encyclopedia of Analytical Chemistry. RA Meyers, 2020. 1-40. [3] Nader, Rafic, et al. “On the Positive Semi-Definite Property of Similarity Matrices.” Theoretical Computer Science 755 (2019): 13-28.
Follow-up on References of Real-World Use Cases
Thank you again for taking the time to respond to our rebuttal. Following up on the best way to share references of real-world use cases, we have reached out to the Program Chairs, whose recommendation was to > provide the references directly to the AC, which can confirm for the reviewer that the library is being used. We have reached out to the AC again and will provide them with these references as instructed. In the meantime, we hope that the more detailed descriptions of the specific application domains and objectives we presented in our previous response - as well as the outline of our governance model and the additional kernels - already provide some useful information to address your remaining concerns. Sincerely, The Authors
Update after SAC comment https://openreview.net/forum?id=vzrA6uqOis¬eId=1wstuXSBjW
I am very happy to see that GAUCHE has been used by a variety of researchers. This removes the possible suspicion (mine and potential readers) that this is a paper about a personal toolbox. Please do include these examples as part of the introduction of your paper! I have increased my score from 4 to 7 accordingly. I will not go above because I don't think this library makes anything possible that was not before but rather simplifies the inferential workflow.
Many Thanks for the Response and Constructive Feedback during the Review Process
We would like to close out the discussion period by thanking the reviewer once again for all their work in responding and offering constructive feedback during the review process! Sincerely, The Authors
I appreciate the authors' comprehensive response to the feedback and the steps taken to address the main points raised in the review. The rebuttal addresses my concerns thoroughly, especially in regards to computational requirements for larger datasets, comparison with SOTA methods and the comparison to proprietary fingerprints. However, specific details on why direct comparison with some neural networks might not be applicable could strengthen your argument. Also, the discussion on the computational requirements is comprehensive, and the rebuttal carefully outlined the approach to proprietary fingerprints. Regarding the questions, the authors have provided mostly satisfactory answers, with clear acknowledgments and plans for further investigation. My current scores for this paper remain the same at this stage.
Additional Experiments and Clarification
Thank you for your response! We are happy to hear that our rebuttal thoroughly addressed your questions and concerns. In the following, we would like to clarify our response regarding the direct comparison to deep neural networks and share results from three additional experiments motivated by your suggestions. # __Direct Comparison to Deep Learning Algorithms__ We would like to refer to Appendix A to point out that we do already perform an extensive empirical comparison to a range of state-of-the-art uncertainty-aware deep neural networks [1-4]. We apologise for any confusion and will make sure to feature these results more prominently in the main text of our manuscript, as we agree that they provide an important reference point. In our previous response, we were referring to the fact that the predictions of deep learning frameworks such as ChemProp and ChemBERTa do not provide native estimates of their predictive uncertainty. However, we discovered that ChemProp does provide an ensembling functionality that produces empirical uncertainty estimates that we can directly compare against our GPs. We have thus carried out an additional empirical evaluation of ChemProp on the Photoswitch dataset. Specifically, we created 20 random train-test splits, for each of which we performed 100 iterations of `hyperopt`-based hyperparameter search using the functionalities provided in ChemProp. Using the best hyperparameter combination, we then trained an ensemble of five GNNs and evaluated it on each held-out test set, using the mean and variance of the ensemble predictions to compute the root-mean-squared error (RMSE) and negative log-predictive density (NLPD) to quantify the models' predictive accuracy and the calibration of their predictive uncertainty estimates. As is apparent from the table below, we found the ensembling approach of ChemProp to underperform a Tanimoto-kernel GP. | Method | RMSE (↓) | NLPD (↓) | | -------- | -------- | -------- | | ChemProp|30.35±1.30|4.53±0.83| | Tanimoto GP |**20.9±0.7**|**0.22±0.03**| # __Results for Scaffold Splits__ > How does the splitting strategy affect the results? For example, what if the data were split based on molecular scaffolds, which would ensure the structural diversity of the test set, instead of a random split? To answer this question, we have re-run parts of our experimental evaluation with 80-20 Bemis-Murcko scaffold splits instead of random splits. As only the lipophilicity dataset exhibits sufficient scaffold diversity to perform such an analysis (the skewness of the scaffold distribution in the others makes an 80-20 split impossible), the following results focus on the predictive accuracy (RMSE) and calibration (NLPD) of GP models in this setting. While this more challenging evaluation setup leads to slightly higher RMSEs and NLPDs, we note that one can observe the same trends as with random splits: Tanimoto-based GPs generally outperform Scalar Product ones, while string kernel-based GPs are better than both. |Kernel| Representation | RMSE (↓)| NLPD (↓)| |:---:|:---:|:---:|:---:| |Tanimoto|Fragprints| 0.86 ± 0.01 | **1.02 ± 0.04** | ||Fingerprints| 0.88 ± 0.01 | 1.12 ± 0.04 | ||Fragments| 0.89 ± 0.01 | 2.10 ± 0.13 | | Scalar Product | Fragprints | 0.89 ± 0.01 | 1.75 ± 0.08 | ||Fingerprints| 0.95 ± 0.01 | 1.99 ± 0.09 | ||Fragments| 1.00 ± 0.01 | NaN | |String|SMILES| **0.82 +- 0.01** | 1.08 +- 0.04 | # __Ablation over Fingerprint Radius__ > How would the performance of the kernels be impacted with the changes in the hand tuned fingerprint radius? Have any ablation studies been conducted to investigate this? Motivated by your suggestion, we have performed an ablation study over the hand-tuned radius parameter of extended-connectivity fingerprints (ECFPs). Specifically, we trained Tanimoto-kernel GPs on the Photoswitch dataset using a series of five increasing radius parameters. In the table below, we report the mean and standard error of the RMSE and NLPD over 50 different 80-20 train-test-splits. Intriguingly, these results show a strong negative correlation between the fingerprint radius and predictive performance. We hypothesize that this is caused by the fact that an expanding feature space leads to lower and less informative Tanimoto similarity scores, making it more difficult to train generalisable models. | Fingerprint |RMSE (↓)|NLPD (↓)| |:---:|:---:|:---:| |ECFP4|**22.65±0.55**|**0.41±0.05**| |ECFP6|23.50±0.55|0.47±0.05| |ECFP8|24.43±0.54|0.52±0.05| |ECFP10|25.17±0.54|0.56±0.04| |ECFP12|25.70±0.53|0.58±0.04| We thank you again for the very helpful feedback and hope that these additional clarifications and experimental results are helpful in addressing any remaining concerns. Please let us know if you have any further questions. Sincerely, The Authors
Thanks authors for an another great response. I am impressed by your ability to conduct the necessary experiments so rapidly and respond comprehensively to the points raised in my previous response. In light of these ablation studies, I am updating my Soundness rating for this work to a 4. Thank you for the thorough and responsive engagement with the review process.
Thank You for the Extensive and Constructive Feedback during the Review Process
We wish to take this opportunity to thank the reviewer for their extensive constructive feedback during the review process, for raising interesting and relevant questions for the work, and for highlighting a number of action points which have served to strengthen the paper and will benefit users of the GAUCHE library! Sincerely, The Authors
Thanks a lot for the author's comprehension answers to the concerns and questions. I checked the rebuttal and also other reviewers' comments. Also, the list of papers that used this library was useful. Although I still have concerns about the contribution and theoretical novelty of this work, I want to raise my rating from 4 to 5.
Thank You for the Response and Feedback
Many thanks once again for taking the time to review the paper and offer feedback! Sincerely, The Authors
As the author discussion period is coming to an end, we would like to thank all reviewers once again for the time and effort they have dedicated to reviewing our manuscript and engaging with our rebuttal. We were pleased to see that our rebuttals and subsequent discussion led to a significant increase in scores and greatly appreciate the detailed and helpful feedback that reviewers have provided. Sincerely, The Authors
Decision
Accept (poster)