FABind: Fast and Accurate Protein-Ligand Binding

Summary

The paper presents a new method for protein ligand docking based on regression of the pocket center and the ligand atomic positions. Although the none of the components of is highly novel, they are well justified and tied together providing strong empirical performance with very fast inference speed. Although there are some adjustments to be made during the rebuttal process (see below) once these are completed I believe the paper will be ready for acceptance.

Strengths

Although most of the main components of the method were present in previous regression based docking methods, the authors tied these together adding a few key ingredients (e.g. center constraint, pairwise distance loss, cross attention) managing to significantly increase the performance over previous methods.

Weaknesses

Main weaknesses: 1. Some parts of the method section lack clarity (see next section) and should be improved. 2. As FABind selects the pocket with a regressive approach (unlike for example TANKBind) it seems that it is critically affected by uncertainty such as symmetries in the receptor, as the residues of the symmetrically opposite pockets would receive the same likelihood and a random independent selection of these would cause the predicted pocket center to be approximately in between the two symmetric pockets. These will apply also in other settings where symmetries are not present but the ligand simply has multiple binding sites. These were artificially removed by the test setting by EquiBind (taking only chains within 10A, not possible in reality without knowing already the binding pocket) but are often present in real blind docking scenarios. 3. Although the method is repeatedly presented as end-to-end, the approach seems to be disjoint with the pocket and docking phases with separate losses that are (almost) independent Some claims need to be adjusted: 1. line 72: the authors claim improvements in performance based on the mean ligand RMSD metric (the only place in the manuscript where this metric is actually highlighted), a metric which is known not to be reflective of docking performance. 2. line 341: “FABind outperforms existing methods by a large margin with higher efficiency.” “large margin” in performance seems an unjustified claim based on the results (in many cases on par or worse than DiffDock or other methods)

Questions

There are some critical steps of explanation of the method that seems to be missing from both paper and appendix, for example: 1. How is the ligand position initialized and placed during docking? During the pocket prediction since there are message-passing layers between the ligand and the receptor how are they placed relative to one another? During docking how are the ligand nodes initialized during docking? For both: is equivariance to the initial orientation of the receptor guaranteed? 2. Do the layers in the docking and pocket prediction share any parameters? Are the gradients propagated during training between the pocket and docking components? If no to both questions, why is the training of the two components joint (a single loss function) and not separate? Unclear components in the evaluation: 1. In the appendix 1 it is mentioned that approx. 500/19k complexes are further removed because of the ligand size or number of contacts. Does this filtering also affect the test set? 2. Appendix 2: how is the alignment between the ESMFold and the crystal structures performed? 3. Some of the baselines have reported different numbers compared to their peer-reviewed manuscripts (e.g. TANKBind, DiffDock), is it because of a slightly different test set? If not I recommend the authors to clarify the situation with the authors of the respective publications. 4. A detailed description of the hyperparameter selection process should be added to the supplementary material

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

I was not able to run the provided code to reproduce the results of the paper due to a lack of documentation.

Summary

This paper proposes an end-to-end model, FABind, which is a pocket prediction and docking end-to-end model considering the prediction accuracy and speed at the same time. By gradually adding the predicted pocket into the docking process, the model improves protein-ligand binding while minimizing differences between training and inference. An equivariant layer is also added in the model. According to empirical tests, FABind performs significantly better than the alternatives.

Strengths

1. This work considers limitations for sampling-based methods and regression-based methods and combines pocket prediction and docking to achieve accurate and fast protein-ligand binding. 2. This approach employs a particular ligand to identify the precise pocket that the ligand binds to, as compared with conventional pocket prediction, which just uses protein as an input and predicts numerous probable pockets.

Weaknesses

1. Why you choose 10 \angstrom as the cut-off distance, have you tried other distances as the hyperparameter? 2. The comprehensive training loss includes the pocket losses and docking losses, which each have two different kinds of losses, there are many weight hyperparameters that should be determined, how you choose the weight hyperparameters? The pocket prediction and docking seem to be two different tasks? Did you train only one model for these two tasks by the comprehensive training loss? 3. In Table 1., the performance of the proposed model, FABind seems not good enough, as the DiffDock(40) has many superior results, what is the reason? But for the flexible blind self-docking, FABind performs much better than DiffDock, what cause these differences? 4. You have pointed out the limitations for the sampling-based methods and regression-based methods, low efficiency and decreased accuracy, and the variation in protein sizes, I guess it may be better to show what is based on sampling-based methods, and what is based on the regression-based methods in the results, Table 1 and 2. Some models are tested on CPU, the comparison seems not fair, as you add the operation of refinement, which will increase the time. How you tackle the problem of the variation in protein sizes as you have mentioned?

Questions

See above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper proposed a graph model named FABind for protein-ligand binding prediction. The major difference from existing work is that FABind integrate both protein pocket prediction and protein-ligand binding in a single model. The pocket prediction subtask is modeled as a binary classification problem, and the docking is a regression problem regulated with distance map based loss. The model was trained and tested using PDBbind v2020 data. Comparison with force-field based tools and GNN-based methods show the advantages of FABind. Furthermore, FABind's inference runs much faster than most of the benchmarks. Case studies are also provided in the main text and supplementary file.

Strengths

The major contribution of this work is the combination of protein pocket prediction and protein-ligand binding in one single GNN model. paper is well written as well. I found it is very easy to follow.

Weaknesses

The protein-ligand complexes in training and test should be better explained. I am wondering whether negative protein-ligand pairs are used in both processes. Particularly, I am interested in seeing the performance for negative protein-ligand pairs. For protein pocket prediction, what would the model infer if a protein is undruggable, i.e., there is no native pocket on its smooth surface. In a different scenario, a protein could have multiple binding pocket; how would FABind behave in such cases?

Questions

Please see limitation above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors should discuss more about the use of negative protein-ligand pairs in model fitting and evaluation. Also, different scenarios of protein pockets should be considered in evaluations.

Summary

In this paper, a protein-ligand binding algorithm based on the E3NN framework, named FABind, is proposed for docking molecules and proteins. The algorithm firstly predicts the pockets on the interface of the proteins, and then utilizes the predicted pockets to dock with the molecule ligand, and the algorithm results show that FABind demonstrates superior performance to TANKBIND, E3BIND, and other methods on protein-molecule docking datasets.

Strengths

1. predict pocket in docking framework is reasonable. 2. The experimental results show that FABind outperforms TANKBIND, E3BIND and realizes a new sota.

Weaknesses

1. FABind used EGNN to update atom coordinates, although EGNN is equivariant neural network. however EGNN doesn't maintain rigid constraint for protein. It's the major weakness when using EGNN on rigid docking problem. 2. it's not clear why the neural network need a separate interfacial message passing module because you have already use cross attention neural network. you can use attention as a soft interface.

Questions

1. In the right of Figure 1, ligand is n*3, receptor is n*3, but pair representation is n*m*3, the dimension is a little bit confusion. 2. can FABind solve atom clash？ 3. what the performance on pocket prediction?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. No release code 2. no wet-lab experiments

Summary

**Overview**. The paper proposes a model for flexible blind docking, where, without knowledge of the protein pocket, the model must predict the configuration (translation, rotation, conformation) of a given ligand bound to a given protein. The authors categorise prior deep-learning methods as either regression- or sampling-based, with, e.g., TankBind, EquiBind, and E3Bind being regression-based and, e.g., DiffDock being sampling-based. The authors describe a trade-off: while sampling-based methods are accurate, they are slow in inference time; and while regression-based methods are fast in inference time, they are inaccurate. The authors propose a fast regression-based method that they claim rivals the accuracy of sampling-based methods. The method breaks the task of blind docking into two subtasks, **pocket prediction** and **docking**, and uses a learned equivariant **representation** of proteins and ligands to perform them. The model is reported to have comparable performance to DiffDock (SOTA) in blind docking, and overall better performance than DiffDock when unseen proteins are used. **Representation**. Their model represents proteins as amino acid residue graphs with $C_\alpha$ coordinates, edges between $8 Å$-away residues, and node features generated by ESM-2. Ligands are represented as atom graphs with edges for every bond a node features generated by TorchDrug. Pair embeddings between protein nodes and ligand nodes are computed with an outer product of the protein features and ligand features (following E3Bind). The authors propose a layer, called FABind, to update the coordinates and features of the protein and ligand graphs, which consists of three components. The first component, Independent message passing, updates the protein and ligand graphs' coordinates and features by performing Equivariant Graph Convolution (from EGNN) within the graphs separately. The second component, Cross Attention Update, updates the protein and ligand graphs' features (as well as the pair embeddings) by applying attention with message passing over all residue-atom pairs. The third and final component, Interfacial Message Passing, updates the coordinates and features of nodes in the protein and ligand graphs that are on a contact surface, using attention and message passing on the contact surface. **Pocket prediction and Docking**. Using the updated protein and ligand graphs, binary classification is done on the protein graph to determine which residues belong to the binding pocket using a binary cross-entropy loss. Then, using Gumbel-Softmax, a weighted mean of the decided pocket residues is computed as the predicted centroid of the pocket, and finally, the Huber loss between the predicted centroid and the true centroid is computed as a loss. The ligand coordinates and features are fed through the FABind layer several times ("iterative refinement"). One loss is computed by the Huber distance between predicted and true coordinates. Another loss is computed by the sum of three $L_2$ sub-losses. The first sub-loss is between the true distance matrix (a matrix of distances between pairs of atoms in the ligand) and the distance matrix computed from FABind's predicted ligand coordinates. The second sub-loss is between the true distance matrix and a distance matrix computed with an MLP from FABind's predicted pair embeddings. The third sub-loss is between these two predicted distance matrices. Training is done in a scheduled manner (following so-called teacher-forcing), where the model is first trained for docking with the known pocket; and then, once the pocket prediction is reasonably good, it is used for docking. **Results**. The model is evaluated on PDBBind 2020, with similar preprocessing to EquiBind, TankBind, and DiffDock. The model is reported to outperform all other models in the percent of predictions with ligand RMSD below 5Å, and all models except DiffDock with ligand RMSD below 2Å. The model is also reported to outperform DiffDock when presented apo structures of the proteins.

Strengths

1. The model is significantly faster than DiffDock (SOTA), while still achieving comparable performance in blind docking. 2. The model is reported to perform better than DiffDock in blind docking with unseen proteins. 3. The model does not require additional geometric constraints as in DiffDock (i.e. group transformation manifolds), making the formulation a bit easier to understand at times.

Weaknesses

1. The results are not significantly better than prior methods: the percent of predictions with a ligand RMSD below 5Å is higher than that of other models. However, the percent of predictions with a ligand RMSD below 2Å is not greater than that of DiffDock. Any ligand prediction with RMSD above 2Å is not very useful in itself (but may be useful for "warm-starting" and speeding up search-based methods). 2. The model seems to assume that there is one unique binding pocket that a given ligand will bind to. In reality, this is not true, as, in addition to an orthosteric binding site (the natural one), there might exist other allosteric binding sites (not the natural one). Therefore, sampling-based methods seem to have the upper hand in this case, as they are able to capture multiple different binding modes. 3. It seems that the model also assumes that there is one unique bound ligand configuration (since this is a regression-based model) which might not be true either. 4. The structure of the paper was a bit difficult follow and the related work section seemed a bit short. I would appreciate a more detailed comparison of this model and other prior models. A detailed explanation of how this model is different from previous models should be included. 5. I could not get the code to run because I could not find the module `metrics_to_tsb`. Please advise.

Questions

1. How does the model perform when independent message passing is removed? 2. How are the external contact surface edges defined? I did not see an explanation. 3. How does the model perform when used together with a searched-based method (i.e. "traditional docking software")? 4. In the apo structure docking experiments, where is most of the flexibility in the receptors (i.e. mostly in the backbone or mostly in the side chains)? Could you show how the model performs when presented with receptors that have most of their flexibility in the backbone (and when most of the flexibility is in side chains)? 5. How does the model handle the existence of multiple possible binding sites and/or multiple possible bound ligand configurations? Could you please show the models in these cases (and compare with DiffDock)? 6. Could you please elaborate on which pre-trained features were used in the protein and ligand graphs? 7. Why were edge features not included (e.g. single and double bonds in ligands)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

1. The model treats proteins as rigid, which is not true for a large portion of important proteins (e.g. GPCRs), though a reasonable assumption for many. 2. The model assumes a unique binding site and ligand configuration (there could be multiple possible sites and configurations).

Response to rebuttal

Thank you very much for the response that addresses many of my concerns. Regarding the first point on pocket finding: while getting rid of other chains is a sensible solution when one knows (approximately) the correct binding site, this does not apply to the blind docking/pocket finding scenario (where such chains are not known in advance). Moreover, often ligands bind with contacts to multiple chains rendering this procedure obsolete. Therefore to have a reliable pocket prediction method I suggest the authors to derive ways of overcoming this averaging problem. However, overall I believe that the paper deserves acceptance so I have raised my score to 6

Response

Thanks for your response. The pocket is extracted from native docking structure or not? If there are multiple pockets, does your method can solve this scenario. It's interesting to see evaluation on multiple-conformation scenarios.