Full-Atom Protein Pocket Design via Iterative Refinement

Summary

In this paper, the authors proposed a Full-Atom Iterative Refinement framework (FAIR) for protein pocket sequence and 3D structure co-design. Generally, FAIR has two refinement steps (backbone refinement and full-atom refinement) and follows a coarse-to-fine pipeline. The influence of side-chain atoms, the flexibility of binding ligands, and sequence-structure consistency are well considered and addressed. Extensive experiments on two datasets show the advantage of FAIR in generating high-quality protein pockets.

Strengths

1. The paper is well-written and easy to follow. Existing related works are well-discussed. 2. Figure 1 and 2 clearly illustrate the protein pocket design problem and the proposed method FAIR. 3. As far as I know, this is the first paper that studies protein pocket sequence-structure co-design with deep learning methods. The importance and background of the problem are well stated. The problem is well formulated in Sec. 3.1. 4. The coarse-to-fine architecture as well as the full-shot iterative refinement schemes are reasonable and effective. The modeling of side-chain atoms, sequence-structure consistency, and the flexibility of binding ligands are well considered. 5. Extensive experiments on CrossDocked and Binding MOAD dataset compared with 5 representative baselines show the effectiveness of FAIR. The code of FAIR is also provided for reproduction.

Weaknesses

1. In the experiments, the authors redesign all the residues that contain atoms within 3.5 Å of any binding ligand atoms. Can FAIR design larger regions of the protein pocket containing more residues? 2. The authors may explore the influence of initialization strategies on FAIR.

Questions

1. Can other protein design methods or structure-based drug design methods be adapted to the studied pocket design problem? 2. Why there is no hyperparameter weight to balance the two loss functions, Equation 9&10? 3. Can FAIR be leveraged for pocket optimization tasks?

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of FAIR is well discussed in Appendix D.

Summary

The paper introduces FAIR, the pipeline for protein pocket sequences and 3D structures co-design. It's important in drug design applications, since most of the small molecule drugs (ligands) bind their targets (proteins) inside the pockets. Curranty existing methods have disadvantages (inefficient generation, inability to generate side chains, etc.), that FAIR overcomes. FAIR demonstrates promising results, as proven through thorough and comprehensive experiments.

Strengths

Originality: The task of protein pocket design is not new and there are several deep learning methods that have already contributed to this field. However, the submission provides new insights and solve yet unsolved issues showing a better performance than previous methods. Related works are cited. Quality: The work is complete and technically sound. Authors support the claims and provide a baseline of FAIR performance. Authors compare FAIR with other methods and show the advantages of FAIR. The paper provides a thorough analysis of strong and weak sides of FAIR. It includes ablations studies and description of how to further improve the approach. Clarity: The text is written clearly. It contains all the necessary citations. The submission provides a comprehensive explanation and description of all methods and approaches including the technical details of experiments and equations. Significance: It's very important to for drug discovery to be able to design protein pockets since most of the small molecule drugs bind in the pockets. FAIR provides additional contribution by solving the issue with side chain and flexibility design.

Weaknesses

The paper lack of speed evaluation or discussion of the method (FAIR).

Questions

1. Can you please provide the information about FAIR speed (how much time does FAIR need to finish one protein co-design)? Is it possible to use it high-throughput (apply for thousands or millions of proteins)?

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The paper provides sophisticated discussion of limitations and future works.

Summary

The authors study the 3D protein-ligand interaction problem. They introduce a novel method for designing protein biding pockets conditioned on the ligand structure, termed FAIR. Unlike existing methods, FAIR co-designs sequence and structure of the pocket by iteratively modeling both backbone atoms and side chain atoms. The method also models refines ligands coordinates accounting for its flexibility.

Strengths

The method combines the ideas from many previous ML works dealing with sequence-structure co-design using graph representation. The novelty of this approach is in the iterative refinement that first ensures the stability of the backbone atoms before proceeding to the modeling of the side chain atoms. Thus, the method models the ligand flexibility and the effect of side chains on the residue types. The comparison study includes all the relevant baselines and other methods and the ablation study is extremely useful as it justifies the importance of each block in their architecture.

Weaknesses

Some clarifications in the sections 3.2.1 and 3.2.2 are needed (see questions). Also, the authors should be very careful when using the word "de novo". They need to show how their method can perform de novo design or explicitly say what metrics they use to show that their designed sequence are de novo.

Questions

Some minor concerns are listed below. Atom-level decoder: 1. The way section 3.2.1 is written, it seems that the encoder is using all atoms in the protein-ligand complex. The number of atoms in the protein-ligand complex can be very large resulting in a very large KNN graph. This could cause some memory restrictions. Why not focusing only on the pocket residues? 2. How is the variable number of atoms and atom types present in the ligand (as well as in different side chains) in different training samples handled ? 3. Using this notation, it’s not clear how the same atoms in different residues have different embeddings. Is this handled in the first MLP layer ? Residue-level encoder: It’s not clear what coarsening procedure is used for different types of ligands. If it’s treated as a “special” residue, then it’s not clear how a feature vector describing its biochemical properties is formed. How’s the 1-hot encoding containing ligand’s identity and distinguishing it from other ligand molecules formed ? Is there a vocabulary of all the ligands in the training set ? The authors should clarify this.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors claim that their method can do de novo design of the binding pocket. I wish this was shown in the paper. Most examples that are illustrated in the paper are just dealing with the recovery of the existing binding pockets. One suggestion to illustrate de novo design is by grafting a biding pocket in some existiing protein scaffold in a given location (that was not previously a ligand binding location).

Summary

This paper is the first to introduce a deep learning pipeline for the protein binding pocket re-design task. The architecture consists of a rotation invariant, hierarchical encoder at the residue and all atom level, followed by a hierarchical iterative refinement generative process at the residue and all atom level. The architecture aims to model key inductive biases of the problem at hand, and shows empirical improvements over baselines adopted from similar protein/molecule design tasks.

Strengths

- Overall sensible pipeline and inductive biases for all-atom protein pocket redesign. - The FAIR pipeline consists of a residue level, rotation-invariant encoder followed by all atom level encoder to encode the protein in a hierarchical manner, after which new residues are placed and iteratively refined in a hierarchical, rotation-equivariant manner. During iterative refinement, the atom positions of the ligand (small molecule) are also updated to incorporate ligand flexibility. - From a biochemical perspective, considering all-atom/atom level encoding and generation is probably very important for binding, as it is the sidechain orientations that determine ligand binding. - These are all sensible inductive biases -- rotation symmetry, hierarchy, iterative redesign, ligand flexibility -- and have been described clearly. - None of these are particularly novel components, because the technical problems have been solved in other work, but this is an effective application of existing ideas to a novel problem of all-atom pocket redesign. - Well written paper. - Clear description of each architectural components as well as experimental methodology. - I really loved the paragraph starting at line 294 onwards. This paragraph (and the overall paper) does at good job at showing how each architectural component of FAIR improves over existing papers/frameworks and is adapted to the specific problem at hand. - I do wish that the authors found a way to not make the reader jump to the appendix so many times. - Comparison to baselines. - Care has been taken to adapt existing papers/methods to the new pocket design task in order to provide baselines.

Weaknesses

- New task may not require major new technical innovations. - While the binding pocket re-design task is new in the deep learning literature to the best of my knowledge, it seems that adapting existing architectural ideas and putting them together in a smart pipeline (see strengths) works well. - From a machine learning perspective, I could not identify a new technical problem that previous works have not solved. (I may be missing something.) - Unsure whether pocket re-design is relevant to enzyme design. - I will caveat this weakness by saying I am not a domain expert at all. - I could immediately see this pipeline be applied for general binding pocket re-design and biosensor applications. One perhaps starts out with a structure of a protein-ligand complex and wants to improve binding affinity. However, perhaps this setup is a bit of a stretch for enzyme design. - For enzymes, we really would ideally like to keep the binding site conserved instead of re-designing it, based on my understanding of the field. In theory, it is of course nice if you can design a new binding pocket to discover new kinds of chemistry. However, as far as I am aware, basic scientists do not currently understand enough about catalytic mechanisms to be able to design a new one from scratch. - Thus, designing a new pocket and **claiming that it catalyzes the same reaction via the same/different mechanism** seems very hard to tackle from a basic science perspective, let alone computationally. Reasons: lack of data and proper annotation of enzymatic mechanisms. - Hard to evaluate binding pocket re-design. - I felt it was overall challenging to design evaluation metrics and setups for this task, because it is unclear whether recovering the RMSD and/or AA identities of the original ligand binding site is the best outcome for these models. - I have several questions about the Vina docking score evaluations.

Questions

- What is the Vina docking score of the training set? - It would be useful to have this as a baseline to compare all the methods to. - Figure 3 provided Vina score for the reference structures. I'd be keen to see the average Vina score across the train/validation/test sets as an entry in Table 1, too. - Can you also show Vina docking scores for all the models without re-docking/relaxation? - In the small molecule generation literature, it has been found that many generative models may generate very unrealistic ligand poses and that the Vina score without relaxation is poor. - Paragraph on line 302 onwards - Why are some residues used more and some lesser by the model? Is this observation related to binding, eg. hydrophobicity properties of residues? - If you are mentioning these observations, it may be worth explaining why... - How do we know the binding pose of the ligand in advance in real-world design scenarios? Do we always need to start with an existing (crystal) structure of the protein-ligand complex?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed technical limitations but not potential negative societal impact. Overall, I would encourage the authors to further discuss WHY binding pocket re-design is a meaningful real-world task in more detail that given at present, how it is relevant for designing bio-sensors and/or enzymes (probably I am wrong in my understanding re. enzyme design), and how we can evaluate these tasks in a meaningful manner in-silico. What are some limitations which we cannot address computationally? Perhaps this may not be very relevant if we are judging this paper only from the machine learning perspective, but better contextualizing this project beyond just ML could be useful to readers.

Summary

The Full-Atom Iterative Refinement framework (FAIR) is a novel approach for designing functional proteins that bind with specific ligand molecules. FAIR consists of two steps: full-atom generation and 3D structure co-design. It uses a coarse-to-fine pipeline, updating residue types and structures together in each round. FAIR outperforms baselines in efficiently designing high-quality pocket sequences and structures, with average improvements on AAR and RMSD exceeding 10%.

Strengths

1. This paper investigates protein pocket design, which determines the pocket structure and sequence based on the context of the protein scaffold and the binding ligand molecule. 2. It provides an end-to-end generative framework called FAIR that uses iterative refinement to co-design the pocket sequence and structure. FAIR solves the drawbacks of earlier research and effectively considers sidechains, ligand flexibility, and consistency of sequence structure for effective prediction. 3. FAIR outperforms baseline techniques in terms of various pocket design parameters. The gains on AAR and RMSD are often over 10%. FAIR generates data more than ten times quicker than conventional techniques.

Weaknesses

Figure 2 of the paper shows that the proposed FAIR adopts a two-stage learning process, so it is not an end-to-end deep learning framework. Therefore, a necessary concern is the training time of the proposed method.

Questions

1. Since Transformer with considerable parameters is used, please try to compare the parameters of the proposed method and other deep learning-based methods in the experimental section. 2. As stated in line 196 and 199, the authors use two feed-forward neural networks (g and g’) to encode internal and external interactions, respectively. Apart from the difference in the output channels, are they modeled with the same structure? If it is the same, then try to explain why two identical models (refer to Eqs. (5) and (6)) are used to model different interactions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The method proposed in the paper utilizes a two-stage iterative refinement strategy. However, the internal logic of these two refinement stages is not well correlated. Alternatively, why can't components such as the masking mechanism of the second stage be directly injected into the first stage?

Reply to the author

Thank the authors for your comprehensive and insightful rebuttal. After reading your responses, most of my concerns and confusions have been addressed. I would like to increase my score from 7 to 9. I kindly request that all the modifications, explanations, and discussions outlined in the rebuttal be fully integrated into the final version of the paper.

Follow up Qs

Thank you for the detailed response. Based on the discussions so far, I'm still not fully convinced by the evaluation. Re. Response 1: My point was regarding how hierarchical embedding of protein structure (https://arxiv.org/abs/2006.09275) as well as ligand flexibility/co-folding (https://arxiv.org/abs/2209.15171) have come up in other works. Re. Response 2: I understand, but neither of these citations seem to be about enzymes, correct? Having skimmed through them, they seem to reinforce the point that methods such as FAIR may be useful for biosensing applications where the goal is to bind with high specificity to a molecule. Re. Response 3 and 4: I understand that these metrics are widely used in the community, but I would like to push the authors to elaborate more about whether these metrics are useful for the pocket redesign task? - For instance, if we care about RMSD, perhaps consider reporting RMSD w/our re-docking, too. - If we care about Vina, perhaps elaborate more on how the test set Vina score of all the methods (whether with or without re-docking), including FAIR, is actually not improved over the test set Vina score of the original datasets themselves. This seems especially true without re-docking. - Please do consider adding a line on the test set Vina score is Table 1, 2, etc.

Continued response

> Our method FAIR can achieve the lowest Vina score on different settings and datasets. It can be attributed to FAIR’s strong capability of pocket sequence-structure co-design. Meanwhile, the coordinates of ligand molecules are updated during the refinement process. Therefore, FAIR relies less on redocking and relaxation. This is very interesting -- all ML method's Vina score gets worse without re-docking. However, FAIR seems less sensitive to re-docking than other methods. This is good. However, I am still cautious about the claim that "FAIR’s strong capability of pocket sequence-structure co-design". See my concerns around how FAIR does not improve over the Vina scores for the groundtruth protein-ligand complexes from the dataset.