Protein-ligand binding prediction is a fundamental problem in AI-driven drug discovery. Prior work focused on supervised learning methods using a large set of binding affinity data for small molecules, but it is hard to apply the same strategy to other drug classes like antibodies as labelled data is limited. In this paper, we explore unsupervised approaches and reformulate binding energy prediction as a generative modeling task. Specifically, we train an energy-based model on a set of unlabelled protein-ligand complexes using SE(3) denoising score matching and interpret its log-likelihood as binding affinity. Our key contribution is a new equivariant rotation prediction network called Neural Euler's Rotation Equations (NERE) for SE(3) score matching. It predicts a rotation by modeling the force and torque between protein and ligand atoms, where the force is defined as the gradient of an energy function with respect to atom coordinates. We evaluate NERE on protein-ligand and antibody-antigen binding affinity prediction benchmarks. Our model outperforms all unsupervised baselines (physics-based and statistical potentials) and matches supervised learning methods in the antibody case.
Paper
Similar papers
Peer review
Summary
In this paper, the authors developed an energy-based model for unsupervised binding affinity prediction. The energy-based model was trained under SE(3) denoising score matching where the rotation score was predicted by Neural Euler’s Rotation Equation. Experiments on protein-ligand binding and antibody-antigen binding are conducted.
Strengths
1. The paper is well-written and easy to follow. 2. The proposed method outperforms all unsupervised baselines and supervised baselines in the antibody case. 3. The code and data are provided. 4. Systematic ablation studies are performed to show the effectiveness of modules.
Weaknesses
1. The idea of using unsupervised generative models for binding affinity prediction is not new. The authors fail to discuss or compare with related works [1,2] [1] Luo et al., Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction, ICLR 23 [2] Guan et al., 3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity Prediction, ICLR 23 2. The authors only calculate the Pearson correlation coefficient. For the binding affinity prediction, other metrics such as RMSE, MAE, AUROC, and Pearson correlation coefficient. 3. The authors fail to consider the flexibility of proteins and ligands. The sidechains of proteins are also ignored. However, these factors are quite important for protein-ligand/protein binding.
Questions
please see the weakness.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The limitations are well discussed.
Summary
The authors propose an energy-based model for unsupervised binding affinity estimation, which is trained by SE(3) denoising score matching (DSM). Different from standard DSM, they add noise by random rotations and translations on the ligand. Utilizing Euler's rotation equations, the rotation matrix can be derived from predicted forces on each atom, making it possible to directly supervise on the predicted forces instead of the rotation matrix. Experiments on protein-ligand binding and antibody-antigen binding demonstrate the superiority of the proposed method over existing unsupervised method.
Strengths
1. Using Euler's rotation equations to formulate the supervision on the rotations as supervision on the predicted atomic forces is novel and clever. 2. Unsupervised learning on binding affinity is important and meaningful as precise affinity labels are hard to obtain in practical. 3. The docked setting is interesting and reals the potential applications of the proposed method in real scenarios where crystal structure is often hard to obtain. 4. It's very interesting to see that pair-wise energies are mostly lower at CDR-H3 and CDR-L3, which perfectly aligns with the domain knowledge.
Weaknesses
1. The learnt probability distribution $p(A, X)$ might not be well aligned with the actual $p(A, X)$ because the actual one is more complicated than the prior distribution of rotation and translation noises. In this sense, for the learnt $p(A, X)$, $p(A, X) \propto \exp(-E(A, X))$ might not hold.
Questions
1. Under the docked setting, the RMSD is quite large for the generated antibody-antigen complexes (median 19.4), however, the correlation is similar to the crystal setting. Does that mean the model actually did not learn useful geometric interaction information? Otherwise, with such badly docked structures, the model should have much worse performance because the structures can not provide accurate geometric information. 2. Will the correlation be improved if you first denoise the input structure (regardless whether it is crystal structure or docked structure) with the model, then output the energy from the denoised structure?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have mentioned that the limitation might be only considering the alpha carbon in the antibody-antigen setting. Perturbations on the side chains can be designed with a similar strategy in the future.
Summary
The authors address the problem of protein-ligand binding. Specifically, the authors reformulate binding energy prediction as a generative modeling task: they train an energy-based model on a set of unlabelled protein-ligand complexes using denoising score matching and interpret its log-likelihood as binding affinity. The key contribution of this work is a new equivariant rotation prediction network for SE(3) DSM called Neural Euler’s Rotation Equations (NERE).
Strengths
* The work is easy to follow and ideas are presented in a clear way * The proposed approach (with certain particularities) generalizes to both small and large molecules
Weaknesses
NERE DSM relies on cocrystal structures for training and cocrystal/docked structures for binding prediction of a new protein-ligand / Ab-Ag pair, which can be limiting in terms of experimental/computational resources. In addition, NERE DSM does not predict binding affinity directly, hence the model is evaluated using correlation instead of RMSE. The utility of NERE DSM in real-world scenarios where we seek to predict affinities for a large set of protein-ligand / Ab-Ag pairs seems limited. Moreover, the extremely slow rate at which new cocrystal data becomes available is an additional limiting factor. Could the authors comment on this?
Questions
### Questions * I’m curious, why were only the top 50 antigen residues chosen instead of all residues within a certain distance, as done for the protein in the small molecule case? * Could you provide more details on the nature of the test splits? Are they IID w.r.t. the training set, or are they split by sequence similarity/some other criteria? ### Questions regarding protein-ligand binding: * Why did the authors choose to compare their model to supervised models trained on 19K labeled affinity datapoints? Did I understand correctly that the 4806 datapoints used to train NERE DSM and the 19K are datapoints used to train the supervised models are disjoint? Is there no cases where affinity data is available together with cocrystal data so both models could be trained on the same dataset? * How long does AutoDock Vina take to dock a protein-ligand pair on average (in your case)? ### Questions regarding Ab-Ag binding: * It is not entirely clear to me, are the supervised pre-trained? * I have the same question here as for protein-ligand binding, is there no cases where affinity data is available together with cocrystal data so both (unsupervised and supervised) models could be trained on the same dataset? * How long does ZDOCK take to dock a protein-ligand pair on average (in your case)? ### Questions regarding ablation studies: * In lines 292-294, the authors say “Interestingly, we find that the model pays the most attention to CDR-H3 and CDR-L3 residues. In most test cases, their energy is much lower than other CDR residues. This agrees with the domain knowledge that CDR-H3 and CDR-L3 residues are the major determinant of binding “. To support this claim, the authors only show one Ab-Ag example which does not guarantee that the same trend is observed across all samples in the test set. Could you report a global metric, for example, the average difference in the test set between energies in CDRs vs energies in remaining Ab regions (plus statistical significance)? ### Minor comments * In lines 72-73, “Due to scarcity of binding affinity data, very few work has explored supervised learning for antibody binding affinity prediction.” Should be “Due to scarcity of binding affinity data, very few works have explored supervised learning for antibody binding affinity prediction.” * Experiments were repeated with 5 random seeds but only the average values were reported in table 1, could you also report the standard deviation with ±?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
In my opinion, the authors have not entirely addressed the limitations of their work, specifically, in relation to my comment in the "Weaknesses" section I set my assessment to "borderline reject" but my opinion can change based on the author's answers and comments
Summary
The paper introduces an unsupervised learning approach for predicting protein-ligand binding energy, called NERE (Neural Equivariant Rotation Estimation). The authors employ SE(3) denoising score matching to train an energy-based model on a dataset of unlabeled protein-ligand complexes. The model's effectiveness is demonstrated through evaluations on protein-ligand and antibody-antigen binding affinity benchmarks from PDBBind and the Structural Antibody Database (SAbDab).
Strengths
1. The paper innovatively applies SE(3) score matching to train an energy-based model, positioning it as an unsupervised energy ranking model. 2. The experimental results highlight the model's effectiveness, particularly when compared with other unsupervised models or physics-based models. 3. The paper is well-structured and easy to comprehend, making the proposed method and its implications clear to the reader.
Weaknesses
1. The proposed method only considers rigid transformations (translations & rotations) in the diffusion process, neglecting the degrees of freedom in small molecule ligands, such as additional rotatable bonds. 2. The method employs a residue-level representation for proteins, which might omit fine-grained information that could be crucial for accurate binding prediction.
Questions
1. Why was the SE(3) diffusion process chosen over the standard elucidating diffusion process? While the standard diffusion process may generate nonsensical conformations, the proposed diffusion process might not explore the entire conformation space. 2. In section 4.1, is the center of mass, denoted as $\mu$, computed based on atomic weight? 3. How does the model perform in the "Docked structure" scenario when the docking error is high? 4. Why does contrastive learning underperform compared to using score matching to train the energy-based model? Further elaboration on this would be beneficial. 5. Are there any experiments conducted on the docking benchmark of the CASF test set? It seems that the proposed method is more suitable for identifying the correct binding pose (ranking among multiple poses of a specific molecule) rather than ranking between binding poses of multiple molecules. 6. Different noise levels appear to lead to different energy functions, while only the minimal noise level can approximate the energy function of the real data distribution. How does the proposed method handle multiple noise scales?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The proposed method appears to be sensitive to docking error. However, in real-world scenarios, finding the true binding pose is a challenging problem. This sensitivity could limit the method's practical applicability.
Summary
This paper introduces NERE, an unsupervised method for predicting protein-ligand binding affinity. The authors propose a generative modeling approach that utilizes an energy-based model (EBM) to capture the characteristics of protein-ligand complexes. The EBM is trained by maximizing the log-likelihood of crystal structures from the PDB database. NERE incorporates random rotations and translations of the separate parts of the molecule complexes and predicts the optimal rotation and translation as the EBM's output. To train the EBM, SE(3) denoising score matching is employed. The effectiveness of the proposed approach is evaluated through experiments on both protein-ligand and protein-protein binding. The results reveal a positive correlation between the predicted energy and binding affinity. The authors demonstrate that their method surpasses all unsupervised baselines and even outperforms supervised baselines in the case of antibodies.
Strengths
Strengths of the review paper: - The paper introduces a novel approach for unsupervised protein-ligand binding energy prediction by utilizing an equivariant rotation prediction network. This innovative approach contributes to the existing body of knowledge in the field. - The paper proposes a unique method to predict rotation and translation as part of an unsupervised data augmentation technique. This approach enhances the understanding and modeling of protein-ligand interactions, as well as the correlation with molecule interaction energies. - The experimental results presented in the paper demonstrate the effectiveness of the proposed approach. The approach outperforms all unsupervised baselines, showcasing its superiority in predicting protein-ligand binding energies. Notably, the method even surpasses supervised baselines in the case of antibodies, indicating its potential for practical applications. - The paper exhibits clear and well-written content. The authors effectively convey technical concepts and methods, ensuring the readers can grasp the details of the proposed approach and its implementation. The organization of the paper enhances readability and comprehension. - The authors demonstrate their awareness of the limitations of their work.
Weaknesses
- The paper lacks visualizations of the correlation between the predicted energies and binding affinities. Visual representations, such as scatter plots or correlation matrices, would enhance the understanding of the relationship and provide a clearer picture of the results. Additionally, providing information about the distribution of outliers would give insights into the robustness of the proposed approach. - The authors could consider spearman correlation and p-value as additional evaluation metrics, as the ranking between different ligands might be more useful for drug discovery and virtual screening instead of absolute binding affinity values. - I am curious about the potential impact of the proposed unsupervised method on downstream tasks when serving as a pre-training strategy. It would be valuable to investigate whether the learned representations from the unsupervised approach can be beneficial for supervised downstream tasks. - The paper lacks an explicit discussion on the significance and relevance of an unsupervised method in the context of protein-ligand binding energy prediction. It would be beneficial to design experiments or provide theoretical arguments that illustrate how the utilization of unlabeled data, in comparison to traditional supervised methods, contributes to the understanding and performance of the model. Demonstrating the utility of unsupervised learning in this specific area would provide a stronger rationale for the proposed approach.
Questions
Please see weaknesses.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
None.
Thanks
Thanks for the response. However, I'm still concerned with the antigen-antibody experiments on docked structures. A median RMSD of 19.4 actually means that the orientation and relative position of the antibody to the antigen are barely correct. Though with larger epitope the correct interaction pairs might still be included in the calculation of the energy, they are not providing the correct geometric information! For example, an actually interacting pair might has long distance in the docked structure, and if the model learnt the true correlation between the geometry and the energy (e.g. interacting pairs with shorter distances might contribute more to the binding affinity), it should fail on such cases because the provided geometric information is wrong! Therefore, could you plot the correlation between the docking RMSD and the error of the predicted energy? It is better if you could compare the pair-wise energy contribution predicted by the model to those calculated by physical softwares (e.g. Rosetta).
Additional analysis
Thank you for your comments. To analyze how docking RMSD affects the performance of NERE DSM, we group the SAbDab test set into multiple bins based on RMSD and report the performance of NERE DSM for each bin. The result is shown in the table below. Here are our main findings: 1. We realize that 42.8% of the test cases have RMSD less than 10Å. According to CAPRI [1], a docking pose with RMSD less than 10Å is considered correct. While the median RMSD is 19.4Å, we still have a fair number of docked instances that are correct. 2. We find that the model performance decreases as docking RMSD increases. When RMSD is above 40Å, the performance is almost zero, which shows that the performance of our model is influenced by the docking error. 3. We notice that the model performance is surprisingly decent when docking RMSD is between 10Å and 30Å. It seems that the learned energy function is not sensitive enough to geometrical changes. We agree that some geometric information seems to be lost in the learned representation, which reveals the limitation of the encoder architecture we used in this paper (frame-averaging neural network [2]). We will explore alternative geometric encoder architectures in the future to improve our method. 4. We did not compare NERE's pairwise energy contribution with Rosetta because the performance of Rosetta is quite poor on our test set (correlation = 0.025). We were afraid that the Rosetta pairwise energy itself may be inaccurate. In summary, our results show that the performance of NERE is influenced by the docking error, but it is not sensitive as we expected. Please let us know if you have any more questions. | RMSD range | # test cases | Correlation | |------:|-------------:|------------:| | 0-10 | 238 | 0.506 | | 10-20 | 47 | 0.441 | | 20-30 | 69 | 0.410 | | 30-40 | 59 | 0.268 | | 40-50 | 51 | 0.030 | | >50 | 91 | -0.031 | [1] Janin et al., CAPRI: a Critical Assessment of PRedicted Interactions, Proteins 2003 [2] Puny et al., Frame Averaging for Invariant and Equivariant Network Design, ICLR 2022
Thanks for your response
Thanks for your careful response! I think the additional analysis has addressed my concern. I will keep recommending the acceptance of the paper.
Thanks
I thank the authors for their thoughtful responses. The authors addressed all my questions. I have 2 remaining comments: 1. Regarding response to Q1: "In terms of experimental resources, our method requires co-crystal structures for training. We agree that it can be a limiting factor but we can alleviate this issue by utilizing a larger set of protein-protein complexes to pre-train our model. There are 208,347 co-crystal structures in the Protein Data Bank in total and its growth is about 10,000 per year. We will explore this direction in our future work." There is no guarantee that pretraining on PPI data would help, particularly for the Ab-Ag prediction case. In any case, pretraining does not remove the model's limitation of only being trainable with co-crystal structures. Any thoughts on adapting the model to be trainable with docked structures? 2. Regarding Q5, was there a reasoning/justification behind the 40% similarity cutoff?
Thanks
Thank you for your thoughtful comments. Q1: Any thoughts on adapting the model to be trainable with docked structures? Our model can be directly trained on docked structures if they are accurate (close to crystal structures). In the case of protein-ligand binding, we can leverage docked structures from the CrossDocked2020 dataset [1], which has 22.5 million docked poses. We can use their binding pose classification model (accuracy = 0.95 AUC) to evaluate the accuracy of each docked structure and add only accurate structures to our training set. The same approach works for antibody-antigen binding. For example, we can use state-of-the-art protein complex structure prediction models like AlphaFold-multimer (AFM) [2] to generate a large set of antibody-antigen complexes. AFM gives a confidence score for each predicted structure that indicates how likely a docked pose is correct. The docking success rate of AFM on the standard antibody-antigen docking benchmark [3] is 28.4%, but the success rate becomes 80% if we only look at docked complexes with confidence above 0.8. Therefore, we can add these highly confident complexes into the training set without contaminating it with too many incorrect poses. Q2: Was there a reasoning/justification behind the 40% similarity cutoff? We used the 40% similarity cutoff because we saw that some prior work on antibody generation [4, 5] used the same similarity cutoff for their experiments. In addition, we found that most of the similar instances are filtered at the 70-90% threshold. For example, in the protein-ligand case, the original training set had 4806 complexes. With the 90% cutoff, the training set size became 3352. With the 40% cutoff, the training set size became 3064. The filtering speed was 1450 (per 10%) in the beginning and 60 (per 10%) after that. Based on these two observations, we thought that the 40% similarity cutoff was reasonable. References 1. Francoeur et al. 3D Convolutional Neural Networks and a CrossDocked Dataset for Structure-Based Drug Design, 2020 2. Evans et al. "Protein complex prediction with AlphaFold-Multimer." biorxiv 2021 3. Guest et al. "An expanded benchmark for antibody-antigen docking and affinity prediction reveals insights into antibody recognition determinants." Structure 2021 4. Jin et al. "Iterative refinement graph neural network for antibody sequence-structure co-design." ICLR 2022 5. Luo et al. Antigen-Specific Antibody Design and Optimization with Diffusion-Based Generative Models for Protein Structures. NeurIPS 2022
Additional experiment with docked structures
Thank you again for your insightful comments. We understand your concern about the limitation of using co-crystal structures for training. Based on your suggestion, we tried training our model on docked structures in the context of protein-ligand binding because there is a large number of docked structures readily available in the CrossDock 2020 dataset [1]. Specifically, we selected 45000 docked complexes that are close to crystal structures (with RMSD < 2Å, as suggested by CrossDock). We further filter this dataset by removing any proteins with over 40% sequence similarity to any of the test set proteins. After this filtering step, our final training set has approximately 30000 docked complexes spanning 2800 different proteins. We compare our model trained on this CrossDock data against our original model trained on the PDBBind co-crystal complexes. As shown in the following table, training on a larger set of high-quality docked structures slightly improves the model performance. Though the improvement is not that significant, it still demonstrates that our model is not restricted to only co-crystal structures. We hope this experiment addresses your concerns. Please let us know if you have any additional questions before the discussion period ends. Thank you very much! | | CASF (Similarity split) | |:-------|:---------| | NERE trained on PDBBind (crystal) | 0.634 ± 0.009 | | NERE trained on CrossDock | **0.640 ± 0.006** |
Reply to the authors
I have read the reply and appreciate the author's reply. Thanks!
Thanks for your response. I appreciate your efforts to address my concerns, but I remain concerned aboout the reasonability of varying noise levels. Specifically, introducing different noise levels into the data distribution results in diverse corrupted distributions, each of which may correspond to a distinct energy function. An extreme case would be the addition of extremely high levels of noise, which could obliterate all information from the original distribution. Yet, in the training objective, a single energy function is employed to model different noise levels simultaneously. Could this approach have some potential problems?
Multiple noise level
Thank you for your comment. Our approach is inspired by Noise Conditional Score Network (NCSN, Yang et al 2020), where they sampled different noise levels during training. The difference between our approach and NCSN is that our model is not conditioned on the noise level. As a result, our current training objective is matching our energy function to a mixture distribution, i.e., a mixture of perturbed distributions given by different noise levels. This mixture distribution is a valid distribution (analogous to Gaussian mixture models), but could be sub-optimal because it is not close enough to the original data distribution. Currently, we are not adding extreme levels of noise and that's why our model performs reasonably well. In principle, there are two solutions to address your concern. First, we can train our model with only one (small) noise level. We can tune this hyper-parameter on the validation set to find the optimal noise level. Second, we can condition the energy function on the noise level by appending the input with some encoding of $\sigma$. Similar to NCSN, we are learning multiple energy functions (with shared parameters) at the same time, each corresponds to one noise level. At test time, we can choose the one with the best cross-validation performance. We will explore this direction in our future work.
Thank you for your response
Thank you for your detailed response. I think the results are more convincing than before and this an interesting task so I have raised my score accordingly.
Decision
Accept (poster)