Virtual screening, which identifies potential drugs from vast compound databases to bind with a particular protein pocket, is a critical step in AI-assisted drug discovery. Traditional docking methods are highly time-consuming, and can only work with a restricted search library in real-life applications. Recent supervised learning approaches using scoring functions for binding-affinity prediction, although promising, have not yet surpassed docking methods due to their strong dependency on limited data with reliable binding-affinity labels. In this paper, we propose a novel contrastive learning framework, DrugCLIP, by reformulating virtual screening as a dense retrieval task and employing contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data without explicit binding-affinity scores. We also introduce a biological-knowledge inspired data augmentation strategy to learn better protein-molecule representations. Extensive experiments show that DrugCLIP significantly outperforms traditional docking and supervised learning methods on diverse virtual screening benchmarks with highly reduced computation time, especially in zero-shot setting.
Paper
Similar papers
Peer review
Summary
This paper proposes a new virtual screening framework DrugCLIP, which aims to identify molecules most likely to bind to a target. Inspired by the recent multi-modal learning approach CLIP, the authors propose to utilize contrastive learning for learning representations of molecules and proteins. Authors reformulate the drug binding problem based on the representations into an information retrieval task.
Strengths
- By adopting a self-supervised multi-modal learning approach to binding prediction problems, DrugCLIP outperforms previous work that utilizes labels which is costly in time and finance. - Augmentation technique inspired from domain knowledge is proposed, which is novel. - Extensive experiments have been conducted, including human evaluation.
Weaknesses
- More detailed explanation of the experimental setting will be helpful for the readers. - In line 109, the authors mentioned "an end-to-end manner," which may mislead the readers that the whole framework is. It should be clarified more since the DrugCLIP is a two-phased method. - In biological data, many missing data exist, which should not be considered a negative sample, as done in section 3.3. Recent self-supervised learning approaches deal with such false negatives that can also be adopted in this framework.
Questions
In my knowledge, the inherent principle in self-supervised learning is that "multiple views of the image should be consistent in representation space." This principle can also be applied to multi-modal learning: "representation of text and images that share the same semantic should be consistent in representation space." But I'm not sure why DrugCLIP works in the sense that binding molecules and proteins should be closer for binding. Do you have any intuitive explanation for that?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Provided in Weakness section.
Summary
The authors recast virtual screening as an information retrieval problem: by learning appropriate representations of both proteins and molecules, and taking contrastive loss based on binding affinity between protein-molecule pairs, the aim is to learn a model where protein queries passed through one encoder can be used to retrieve a molecule from a large-scale library. The design of the model allows training data from a range of sources such as PDBBind, but also ChEMBL and BioLip, as well as augmentation with protein structures based on protein homology across evolution. The method is benchmarked using DUD-E and LIT-PCBA, and by human evaluation of comparative examples from Glide, a commercial docking system. The method is demonstrated to outperform most finetuned deep learning methods on the benchmarks and produce better binding molecule sets than Glide in 80% of cases, as judged by human experts.
Strengths
The authors suggest a good method for obtaining aligned representations for proteins and molecules, where the alignment is induced by protein binding affinity, using a contrastive loss. The representations of molecules are undertaken in 3D which is still not the default in this field, despite its clear importance in protein-affinity, and uses a biologically-plausible data augmentation. The results in the DUD-E benchmark are strong, particularly in a zero-shot setting, significantly out-performing other methods. Results on LIT-PCBA are also strong, with enrichment factors far above the performance achieved with commercial docking software. The results obtained on the time taken to virtually screen large libraries are impressive -- there are clear advantages over commercial docking software, and machine-learned scoring functions. In addition, the clear ablation studies and human evaluation add nice details to the work. Overall this work is a very strong addition to the techniques available for screening and it will be exciting to see it in use.
Weaknesses
There are no significant weaknesses in this paper.
Questions
No questions
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
4 excellent
Limitations
No limitations
Respond to authors' rebuttal
Please, look at the authors' rebuttal and the other reviewers' comments and indicate if you would like to change anything in your review.
Summary
The authors proposed a contrastive learning framework, DrugCLIP, for the drug virtual screening task which identifies potential drugs from vast compound databases to bind with a particular protein pocket. It reformulates virtual screening as a dense retrieval task and employs contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data without explicit binding-affinity scores. Specifically, the framework computes a contrastive loss between two separate pre-trained encoders to maximize the similarity between a protein-molecule pair which can binding together and minimize it otherwise. Besides, the authors introduce a biological-knowledge inspired augmentation method, HomoAug, which creates protein-molecule pairs based on protein homology evolutions. Experiments on two challenging virtual screening benchmarks, demonstrate that zero-shot performance of this model surpasses most deep learning baselines that carefully finetune on labeled data.
Strengths
1. The proposed DrugCLIP framework reformulates virtual screening as a dense retrieval task and employs contrastive learning to align representations of binding protein pockets and molecules from a large quantity of pairwise data, which provides researchers a new perspective for virtual screening. 2. The designed contrastive loss relieve the dependency on explicit labeling of binding affinity, and facilitates the usage of large-scale unlabeled data beyond densely annotated small datasets (such as PDBBind). 3. The dense retrieval characteristic of DrugCLIP brings high efficiency to online inference and promising high-throughput virtual screening on billions of molecules. 4. A biological-knowledge inspired augmentation method named HomoAug are proposed, which creates protein-molecule pairs based on protein homology evolutions. The shortage of public data is indeed obvious in drug discovery, and this augmentation method may help alleviate it. 5. The organization of this paper is very clear and easy to understand.
Weaknesses
1. The main contribution of this paper is to directly apply the contrastive learning CLIP to the virtual screening scenario, which is maybe insufficient to support a poster published in NeurIPS. 2. The proof of Proposition 1 seems insufficient to support training-test consistency. Firstly, the model ability of f_theta and k_theta is ignored. The docking methods (k_theta here) can directly capture protein-ligand interactions and maybe more natural to adapt to conformation perturbations. Secondly, the binding conformations of ligands are different from their free states probably, and thus the optimal Rotation R and translation t cannot be found. Thirdly, many SOTA docking methods employ two-tower frameworks, such as Equibind, TANKBind, DiffDock, etc. 3. The model lacks further experiments on the screening power compared to docking methods in CASF-2016 which is an important benchmark for in silico drug discovery. 4. Lack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix. These conventional approaches offer visualizations that elucidate the binding mechanism between a pocket and a molecule, which is intuitive and reliable for chemistry researchers to check the rationality of receptor-ligand interactions and modify the molecule structures.
Questions
The questions are basically mentioned in the “weaknesses” part above. 1. The proof of Proposition 1 seems insufficient to support training-test consistency. Firstly, the model ability of f_theta and k_theta is ignored. The docking methods (k_theta here) can directly capture protein-ligand interactions and maybe more natural to adapt to conformation perturbations. Secondly, the binding conformations of ligands are different from their free states probably, and thus the optimal Rotation R and translation t cannot be found. Thirdly, many SOTA docking methods employ two-tower frameworks, such as Equibind, TANKBind, DiffDock, etc. 2. The model lacks further experiments on the screening power compared to docking methods in CASF-2016 which is an important benchmark for in silico drug discovery. 3. Lack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix. These conventional approaches offer visualizations that elucidate the binding mechanism between a pocket and a molecule, which is intuitive and reliable for chemistry researchers to check the rationality of receptor-ligand interactions and modify the molecule structures. Try to give some correlation analysis and demonstrate some interaction patterns learned by DrugCLIP and not just show the performance. That would be more convincing for biochemistry scientists.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Lack of interpretability. Although this model demonstrates enhanced effectiveness and efficiency, it falls short in terms of interpretability compared to docking methods, as the authors mentioned in the appendix.
Respond to authors' rebuttal
Please, look at the authors' rebuttal and the other reviewers' comments and indicate if you would like to change anything in your review.
Summary
## DrugCLIP: Contrastive Protein-Molecule Representation Learning for Virtual Screening The authors present DrugCLIP, a contrastive learning method for virtual screening. DrugCLIP is designed to align representations of small molecules and protein binding pockets. By using a contrastive learning framework, the authors are able to use data without associated affinity measurements, which are relatively scarce. Additionally, the authors present a data augmentation technique based on identifying homologous proteins.
Strengths
The authors employ an interesting framing of virtual screening and achieve good performance across a small panel of benchmark datasets with improved speed compared to more expensive traditional methods. The authors compare their method to several existing approaches and perform additional analysis around the practical usability and behviour of the model.
Weaknesses
* Clarity and explanatory detail could be improved throughout * The data augmentation method seems dubious. Whilst it clearly provides more data, selectivity is obviously a vital aspect of developing therapeutics. By using these augmentations it does not seem to this reviewer that the ability of the model to determine selective binding is well-served and really serves to smooth what is fundamentally not a smooth function due to cliff effects. * It would be interesting to explore performance on benchmarks of highly related targets and ligands, such as the KIBA and Davis Kinase inhibitor datasets. ### Minor Comments * Title: Typo in "Contrasive" * This reviewer disagrees with the notion that the key issue is to identify which molecules bind to a particular target. While this is an important subproblem, it fails to address downstream problems. Co-crystallised structures and affinity measurements will still be required in experimental campaigns, even if we had a reliable binder identification method. These are the time-consuming and expensive steps. Docking and affinity prediction methods address both implicitly address the binder identification problem, whilst also providing a proxy for one of these expensive measurements. While docking is time consuming, it produces a richer output: both pose and score. * L20-22 This is clearly true but perhaps lacks some nuance. Library composition must also have a significant impact. * The authors state to the best of their knowledge this is the first information retrieval based framing of the virtual screening. I think this claim holds water but there does exist very similarly motivated prior work on a related application that should be discussed and referenced for completeness [1]. * L71 value judgments such as "Innovative" and "superb" (L73) should be avoided * L127 the authors should be explicit about the type of noise added in the corruption processes. * L197 it would provide better clarity if the authors could use technical terms such as apo and holo * It would be helpful if the authors could be more descriptive about the noising procedure using RDKit. The authors refer to simulation but it is not immediately clear on what this is in reference to. [1] https://www.biorxiv.org/content/10.1101/2022.04.26.489341v1.abstract
Questions
* How are ligands combined with the pocket in the augmented dataset?* * Do the authors perform any pre-processing of the structures, such as relaxation? * I'm a little confused by the description of SE(3) Equivariance. The task is suited to invariance and the input appears to be invariant (based on discussion of distance-based encodings of structure). * Masking atom types -> Is this masking the element type or the atomic identifier (e.g. the 37 standard atoms in Proteins) * Do the authors apply any quality thresholding (e.g. based on pLDDT) for the data augmentation technique? * I would appreciate if if the authors could expand on the purpose of adding the central node ([CLS])? * Why do the authors filter out proteins with only one known pocket in the ChEMBL dataset? * Why do the authors not report BEDROC on DUD-E Finetune? * The authors state they remove all targets present in DUD-E from the training data. How exactly is this performed? * For the human evaluation, it would be interesting to present docked structures to the experts as well as the molecular structures.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Technical weaknesses discussed above. No ethical concerns.
Respond to authors' rebuttal
Please, look at the authors' rebuttal and the other reviewers' comments and indicate if you would like to change anything in your review.
Thank you for author rebuttal
We sincerely thank the authors for their effort in addressing my concerns. Also, domain knowledge on **why contrastive learning on molecules and proteins** would be helpful for readers in understanding the effectiveness of contrastive learning in molecule-protein representation learning. Therefore, I will raise my score to 7.
Many thanks to the authors for their response and incorporating my suggestions in their additional experiments . Re table 1: why do the authors think their model suffers a performance hit upon a stricter thresholding (the other ML method, Planet, does not). The 90% threshold also seems much too high to me. Could the authors provide some justification or additional experiments at more stringent cutoffs? It seems the EF score are quite affected. Re table 2: Which specific dataset is this evaluation performed on? I would like to be able to contextualise these results with what is reported in the literature.
## Response to Re Table1 Thank you for highlighting this crucial facet of evaluating machine learning techniques. We deeply regret that while we conducted supplementary experiments, we are constrained from sharing the outcomes during the current discussion phase. Consequently, we have augmented our explanation with more theoretical insights. To begin with, we emphasize the prevalent performance challenges that afflict the majority of machine learning methods, as extensively discussed in preceding research. To elucidate, a comprehensive study by [1] meticulously assessed an array of machine learning methods, unveiling that **incorporating akin structures** within the training dataset notably **enhances performance**. Additionally, it was observed that training on a specific protein family and subsequently testing on **dissimilar proteins** consistently results in **subpar performance** [2]. Regarding methods like Planet, it is worth noting that empirical outcomes beyond utilizing a 90% threshold for testing were not evident. Therefore, **in alignment with the Planet paper**, we opted to mirror comparable configurations by employing MMSeqs2 to expunge homologous sequences from our training data. Furthermore, **our performance across all configurations consistently surpassed alternative machine learning methods**, an accomplishment that bears significance. In terms of the machine learning aspect, contrastive learning offers a substantial advantage by ensuring that our method experiences **a lesser decline in performance compared to other supervised learning approaches**. Rather than directly modeling the relationship between protein-pocket pairs and ligand structures, contrastive learning focuses on extracting distinctions between strongly binding instances and negative samples. This distinction becomes evident in **the t-SNE visualizations from Section 4.4**, which underscore the potential pitfalls of utilizing conventional supervised learning techniques. These methods tend to yield representations that are notably imbalanced, and the clustering has proven that the **ML baselines memorize the pocket templates and ligand structures**. This outcome serves as compelling evidence that our approach adeptly captures the fused embeddings of both pockets and ligands. In contrast, competing methods fall short in this regard, resulting in a more pronounced deterioration of performance when faced with scenarios where similar pockets or ligands are excluded from the training set. Within the framework of pretraining and fine-tuning, our methods exhibit heightened resilience in **capturing intricate pocket features**. This resilience arises from the fact that many proteins, despite sharing similar sequences, exhibit dissimilar pockets characterized by divergent atom-level structures [3], e.g. single mutant residue. Notably, our model undergoes pretraining to accurately predict atom types and precise atom coordinates. This inherent capability empowers our model to effectively discriminate between proteins that possess analogous sequences—a competence that surpasses approaches that merely extract protein embeddings on a more generalized level. This finer differentiation is pivotal in rendering our method more suitable for virtual screening. [1] Su. et al, Tapping on the Black Box: How Is the Scoring Power of a Machine-Learning Scoring Function Dependent on the Training Set?, J. Chem. Inf. Model. 2020 [2] Wang. et al, Yuel: Improving the Generalizability of Structure-Free Compound–Protein Interaction Prediction, J. Chem. Inf. Model. 2022 [3] Davis et al, Comprehensive analysis of kinase inhibitor selectivity, Nature Biotechnology, 2011 ## Response to Re Table2 We sincerely thank you for providing some valuable benchmarks in the review section. Though KIBA and Davis Kinase inhibitor datasets are well-known kinase selectivity datasets, they are relatively out of date, as the number of kinase inhibitors grows dramatically in this decade. Furthermore, these datasets lack the crucial provision of associated structural information. This omission poses a challenge for structure-based methodologies like ours, which inherently rely on such data for effective application. Therefore, we build a novel dataset that provides protein structures of the kinase and the molecule structures of the inhibitors. To assemble this dataset, we meticulously curated 154 kinase structures sourced from the KLIFS database (https://klifs.net/). Complementing this, we harnessed the power of data mining techniques to derive a collection of 9423 inhibitor structures from the ChEMBL database with reported bioactivity data. In our experiments, the goal is to identify the correct kinase within the top 1/5/10 ranked pocket structures for the given inhibitor structure input. In summary, A new test dataset is built by collecting kinase structures from the KLIFS database and inhibitors from the ChEMBL.
Many thanks for providing the additional explanations. Without any additional baselines to compare on this new dataset I am not yet inclined to update my score.
## Response to the Kinase Experiment We sincerely appreciate your suggestion regarding the comparison of our model with additional baseline methods in the context of kinase inhibitor screening. Your advice has illuminated an intriguing avenue for future exploration. To begin, we would like to clarify our objective for including the experiment results in Table 2 within our rebuttal. Our intention is to demonstrate that **while our data augmentation introduces some level of noise, it does not adversely affect the performance on closely related protein pockets**. Therefore, we have not presented other baseline models for comparison at the first time. In addition, we have conducted supplement experiments to address to your concern. Recently, we curated a smaller subset with 116 molecule structures from our original test set and assessed the various methods' ability to correctly identify selective inhibitors for a given kinase protein. This subset was formed by intersecting our aforementioned dataset with FDA-approved drugs, as the protein binding of drugs has been extensively explored by researchers. It's noteworthy that **DrugCLIP surpasses the performance of both the molecular docking method (AutoDock Vina) and the Unimol binding affinity prediction model**. Given the constraints of time and reproducibility, we selected Unimol as a representative due to its being open-sourced and its achievement of state-of-the-art excellence in various ligand-pocket tasks, including the affinity prediction task, outperforming alternatives such as Planet. We fully intend to incorporate these additional findings into the revised version of the paper. We are truly grateful for the time you dedicated to reviewing our work and for your valuable contributions to this insightful discussion. | Method | Top5 acc | Top 10 acc | Top 15 acc | Top 20 acc | Time Consuming | |----------------------|----------|------------|------------|------------|------------| | Vina(docking) | 8.51% | 14.89% | 24.47% | 34.04% |4 h 10 min| | Unimol regression(ML)| 7.45% | 18.09% | 23.40% | 32.98% | < 1min| | DrugCLIP | **14.89%** | **23.40%** | **28.72%** | **37.23%** | < 1min|
Decision
Accept (poster)