Inverse protein folding is challenging due to its inherent one-to-many mapping characteristic, where numerous possible amino acid sequences can fold into a single, identical protein backbone. This task involves not only identifying viable sequences but also representing the sheer diversity of potential solutions. However, existing discriminative models, such as transformer-based auto-regressive models, struggle to encapsulate the diverse range of plausible solutions. In contrast, diffusion probabilistic models, as an emerging genre of generative approaches, offer the potential to generate a diverse set of sequence candidates for determined protein backbones. We propose a novel graph denoising diffusion model for inverse protein folding, where a given protein backbone guides the diffusion process on the corresponding amino acid residue types. The model infers the joint distribution of amino acids conditioned on the nodes' physiochemical properties and local environment. Moreover, we utilize amino acid replacement matrices for the diffusion forward process, encoding the biologically-meaningful prior knowledge of amino acids from their spatial and sequential neighbors as well as themselves, which reduces the sampling space of the generative process. Our model achieves state-of-the-art performance over a set of popular baseline methods in sequence recovery and exhibits great potential in generating diverse protein sequences for a determined protein backbone structure.
Paper
Similar papers
Peer review
Summary
The paper proposes discrete diffusion for the task of inverse protein folding (IPF). The many-to-one mapping of sequence to structure warrants a generative model of sampling multiple possible sequences that fold into a given structure. Nearly all prior works use autoregressive generative models so the exploration of non-autoregressive models is high interest for its improvements in accuracy and speed. The authors propose several IPF specific improvements to discrete diffusion that results in state-of-the-art performance on commonly used benchmarks. Additional analysis is performed on accuracy for different classes of residues such as surface exposed, diversity, and designability. The contributions can be summarized as follows: - First exploration of discrete diffusion for IPF. - Discrete diffusion improvements such as a BLOSUM weighted transition kernel, secondary structure auxiliary prediction, DDIM for faster sampling. - New SOTA performance on IPF benchmarks.
Strengths
I enjoy the direction and development of ideas in this paper. Protein sequences are largely driven by their local environment based on the graph construction. It makes a lot sense to free the decoding order to be learned rather than left-to-right. Though the method is not new, the application is novel and well motivated. I appreciate the authors exploring this direction for IPF. - Well-motivated by taking advantage of the graph structure inherent in the problem to improve sampling. - Well written. I could follow the math and figures to understand the method. I appreciate the authors shared code. - Removes limitations of left-to-right decoding order of previous works which results in higher diversity.
Weaknesses
- Comparison to alternative discrete diffusion methods. I hoped for more discussion on the merits and considerations of choosing D3PM over alternative methods of discrete diffusion such as analog bits and latent diffusion. - No in-depth analysis of foldability. There is a question of whether higher recovery rate is always better. For instance, foldingdiff [1] showed in Table S1 and S2 that foldability was much higher with ProteinMPNN than with ESM-1F -- the latter which has higher sequence recovery. Including a benchmark on foldability and not just a few examples would be important to understand foldability as a function of sequence recovery on a benchmark would be high importance to understand the relationship. I already believe the contributions are strong and this is merely a suggestion for higher impact. - No explanation of different benchmark numbers. I noticed the numbers for proteinmpnn don't match what is reported in this paper. [1] https://arxiv.org/abs/2209.15611 [2] https://www.science.org/doi/10.1126/science.add2187 [3] https://www.biorxiv.org/content/10.1101/2022.04.10.487779v1
Questions
- As mentioned above, what is the relationship between sequence recovery and designability/foldability? [note to self of checking their foldability results. - Why does the method only use C-alpha positions? ProteinMPNN found using all backbone atoms to improve results. I would be curious if all backbone atoms helps and if noise augmentation improves. - Have the authors tried not providing hand-crafted geometric features? I would’ve suspected these can be inferred from the raw coordinates like in ProteinMPNN. - Why do the benchmark numbers not match those reported in ProteinMPNN? - There is are some typos and weird language I noticed. - Line 78. “Meticulously curated” is a weird phrase. I would reword it. - Line 88. Needs capitalization. - Line 93. x_1^{pos} is bold face while the rest are not. - Line 134. Equivariant is misspelled. - Line 143. normalized is misspelled.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
4 excellent
Contribution
3 good
Limitations
- There no limitations discussed. It would be interesting to see where the method fails and initial preliminary thoughts on how it can be improved.
Summary
The paper highlights that "Existing discriminative models struggle to capture the diverse range of solutions, while generative diffusion probabilistic models offer the potential for generating diverse sequence candidates." They propose to use the denoise diffusion model together with the prior information from the secondary structure and BLOSUM matrix to train an inverse folding model. As shown in the experiments, their method achieves better results than the previous baselines, and they also show the folding results for the model predictions.
Strengths
[+] The proposed graph denoising diffusion model introduces a new perspective for addressing the challenging problem of inverse protein folding. It leverages the power of diffusion probabilistic models to generate a diverse set of sequence candidates for a given protein backbone. In protein design, diversity is a key point, since we want diverse candidates to build a library for wet-lab experiments. [+] The model achieves state-of-the-art performance compared to popular baseline methods in sequence recovery. [+] The method includes prior information during the model training, and the method is easy-to-follow. [+] The authors also check SASA and other properties, which is useful for protein design and engineering.
Weaknesses
[-] Although the paper mentions diversity, however, they mainly measure the recovery ratio (wildtype accuracy) and the PPL, which cannot tell the readers whether their model captures the "useful" diversity. The diversity is only shown in Figure 4, and I think it is not enough to demonstrate the model's advances in diversity. [-] The baseline PiFold or ESMFold algorithm can also introduce diversity by controlling the temperature and do sampling based on the probability score, a detailed comparison with the baselines could improve the experiments.
Questions
1. The authors show the trade-off between speed and recovery ratio. Here, I wonder if the authors use a standard Euler solver or any other sampling schedule to do the inference. 2. In Figure 6, the authors show the results of several PDB proteins, do the authors make sure that these proteins have < 30% sequence similarity as the proteins in the training dataset?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
n/a
Summary
The authors of the manuscript present GraDE-IF, a diffusion model based method for inverse protein folding given the backbone of the structure. The denoising network is a graph neural network that's equivariant to rotations and translations. Moreover, a biologically relevant inductive bias is incorporated into the discrete diffusion process by replacing the uniform amino acid transition probabilities with amino acid substitution scoring matrices. Finally, sampling is accelerated by deploying a variant of the Denoising Diffusion Implicit Model (DDIM).
Strengths
To the best of my knowledge, this is the first work that explores the use of a discrete diffusion model for inverse protein folding. The fact that prior biological knowledge is incorporated directly into the discrete diffusion process increases the appeal of this method. The presented results look promising.
Weaknesses
Even though I enjoyed reading the paper, there are still some concerns. First and foremost, in my opinion the reported metrics do not fully support the claim that the model is capable of generating diverse protein sequences. Figure 6 shows some qualitative examples of different sequences leading to plausible structures, but I'm missing quantitative results on diversity/uniqueness, and an average RMSD/pLDDT for a large batch of generated sequences folded by AlphaFold2 for different proteins. This would make the story much more convincing. Additionally, the paper is quite sloppy in some places, with confusing mathematical notation, wrong figure references, and occasionally poor sentence constructions. I appreciate that this is likely due to time constraints, but some time should be spent on fixing this for the final paper in case it gets accepted. I will address the more minor concerns in the "Questions" part.
Questions
1. As mentioned in "Weaknesses", I missed some kind of quantitative metric to measure diversity / uniqueness of generated sequences accross proteins. 2. Another valuable addition to the results, although it might not be possible due to time constraints, would be to show how this method could aid protein design, e.g. by providing a synthetic backbone and generating plausible corresponding sequences. 3. Figure 1: the text in the "prior" and "condition" blocks is quite small. For me this caused some confusion because at a quick glance it seemed like the bottom part of "condition" was an amino acid sequence rather than secondary structure annotation. 4. Table 1: Are the CATH versions correct? The first sentence of section 4.1 (line 233) mentions CATH v4.3.0. 5. Figure 4: The 45% threshold seems quite conveniently chosen such that only one sequence remains for the other two models. It would be informative to include this plot for some different recovery rate cut-offs, if not in the main text then at leats in the supplementary. Additionally, for the figure on the right, no unit is given for "speed", and there is no comparison to other models (PiFold, ProteinMPNN). 6. Missing references/proofs to support the connections made in line 45-51. 7. Missing defenitions: * FFN (Figure 1): mention feed forward network somewhere. * $\mathcal{E}$ (line 76) is never defined. * $d$ (line 126) is never defined (number of categories). * $\mathbf{A}$ (line 131) is never defined. 8. Confusing mathematical notation: * The graph notation seems inconsistent accross the paper. * I think there's a "pos" superscript missing in line 93 (i.e. $\mathbf{X} \rightarrow \mathbf{X^{pos}}$) * The expression in line 126 seems a bit confusing. Isn't the transpose of the identity matrix the same (i.e. $\mathbf{I}^T=\mathbf{I}$) and wouldn't $\mathbf{I} \ \mathbf{I}^T=\mathbf{I}$? It might be that this is the point you wanted to make, but it was not completely clear to me. 9. Wrong figure references line 260, line 289. 10. Not all appendices are referenced to in the main text. 11. A grammar check would improve the flow of the paper.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The authors adequately dicuss the limitations of their work in the appendix, even though I would prefer it if some of this discussion was transferred to the main paper.
Summary
This work presents a denoising diffusion model for protein inverse folding: predicting the amino acid sequences that fold into the given 3D protein structure. The proposed method leverages a discrete denoising diffusion model with respect to the graph structure representing the protein backbone. This work proposes to use Blocks Substitution Matrix for the transition matrix considering the different transition probabilities between amino acids, and further utilize the distinct types of secondary structure as a condition during sampling that guides the sampling process to appropriate 3d structures.
Strengths
- The paper is well-written and easy to follow. - The approach to using (discrete) diffusion models on protein inverse folding is novel to the best of my knowledge. - Two main contributions are novel and well-motivated: Employing BLOSUM for considering transitions between AAs and using the secondary structure information as a condition during sampling injects biological prior knowledge into the diffusion process which further reduces the sampling space and results in plausible AA sequences. - The proposed method shows superior performance for the inverse folding tasks in terms of perplexity and recovery rate. The diversity analysis shows that the generated sequences are diverse without losing recovery rate.
Weaknesses
- Using the distinct types of secondary structures as a condition for the sampling process is not clear. How is the information used? Is it an input to the model during each step of the sampling process? - The reason for not giving a higher rating is that although the approach to the task is novel, using BLOSUM and secondary structure conditions are novel, and the method shows superior performance, the components of the proposed methods are widely used in different domains: Diffusion models and E3 equivariant networks are widely used in the generation of protein structures given sequences (I understand that the task is not the same but in the similar domain) and DDIM is also used to reduce the sampling steps. - Minor: line 143 "orm.alized"
Questions
- Using DDIM for sampling, how long does it take to predict the sequence compared to the baselines, e.g. ProteinMPNN or ESM-IF1?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations are discussed in the Supplementary file.
Clarity on choice of 42 proteins
Can the authors clarify how these 42 proteins were chosen? How do we know these proteins were not cherry picked?
Re: Clarification on the reported 42 proteins
We fold the first 42 proteins in the test dataset in alphabetical order by their PDB ID. Due to time limitations, we were only able to fold such a small subset of proteins' structures for the generated protein sequences by the three models. We are currently working on conducting evaluations on the remaining proteins in the test set. We are committed to providing updated and comprehensive scores for a broader range of proteins.
Response
Thank you the response. I believe the method is technically sound. My score is from my concerns in the evaluation. > ProteinMPNN used random splitting by 80:%10%:10% (according to their experimental details), I am not sure this is true. Their experimental details state "we used a set of 19.7k high resolution single-chain structures from the PDB were split into train, validation and test sets (80/10/10) based on the CATH4.2 40% non-redundant set of proteins (1, 8)." where reference (1) is Ingraham et al. It seems to me ProteinMPNN and Ingraham follow the same splits. Can the authors clarify how they arrived at their conclusion? > we've extended foldability evaluation to a wider set of 42 protein I appreciate the addition of more proteins. I commented in the global comment asking how these 42 were selected. Are they in the test set?
Re-Reviewer 92pD: reults on ProteinMPNN and 42 proteins
We appreciate your meticulous attention to detail and the insightful comparisons you've made regarding the dataset split methodology. Regarding your concerns: 1. **Performance of ProteinMPNN**: (1) Indeed, Ingraham et al. [1] take the random split of the training, validation, and test sets by an 80/10/10 ratio, along with the careful consideration of protein CATH categories to remove redundancy, which has been a common practice in subsequent studies as well. However, we did not find explicit rules for dataset processing in ProteinMPNN's paper, specifically concerning the removal of overlap data. (2) Our further investigation into the ProteinMPNN's GitHub repository, including its discussion board, revealed an interesting detail. ProteinMPNN employed a random decoding technique to sample sequences, leading to an improvement in the recovery rate. In our initial implementation, we adhered to ProteinMPNN's default settings, which utilized the same input sequence order for decoding and yielded suboptimal results in terms of CATH recovery rate. By incorporating the random decoding strategy, we were able to enhance ProteinMPNN's result with a higher recovery rate of 49.9% and a perplexity of 4.576. We acknowledge the discrepancy in the methodologies employed and extend our gratitude for your keen attention to this crucial aspect. We will make clarifications and update this result in our revision. 2. **42 Proteins**: All the proteins are from the test dataset. As we have replied in the global comment, we fold the first 42 proteins in the test dataset in alphabetical order by their PDB ID. Due to time limitations, we were only able to fold this subset of proteins for the generated protein sequences by the three models compared. We are currently in the process of conducting evaluations on the remaining proteins in the test set. We are committed to providing updated and comprehensive scores for a broader range of proteins.
Response
Thank you for both the investigation and clarification. I appreciate the investigation and correction of ProteinMPNN's performance. I would have preferred to see foldability performance on a larger set of proteins but understand the time constraint. I am between a 5 and 6 due to the limited foldability but will put it as a 6. I would have put it higher if the initial foldability assessment was strong. To the AC, I am recommending weak acceptance on the condition a larger set of proteins is included in the final evaluation.
I would like to thank the authors for their elaborate response to my feedback and for investing time into generating new results. Here is my answer to the rebuttal: **Questions about new results:** a) Table 4: Can you comment on statistical significance? I think the claim “Our GraDe-IF constantly achieves the *highest TM score* with the smallest variance” might be a bit strong. And why were these 4 proteins selected? b) Great that most of the sequences seem foldable. I was just wondering, is there something the remaining three unfordable sequences have in common? For example, are these perhaps transmembrane, partly disordered, higher beta-sheet content, etc? This would give some insights into potential weaknesses and give some idea as to where you might run into more trouble when exploring the rest of the PDB IDs beyond the 42 you show here. **Weaknesses:** 1. Thank you for spending time and effort to generate new results. Did you come up with this “1-sequence_identity” score or do you have a reference? Moreover, are all generated sequences unique? 2. Much appreciated. **Questions:** 1. See above. In addition, are you planning on adding some more sample visualisations like in Figure 6 for the new proteins, either in the main paper or in the appendix? 2. Okay, fair enough. Maybe worth including in the conclusion / discussion. 3. \- 11. Thank you for making these changes.
Re-Reviewer j843: response to new questions
Thank you for the reply. Regarding your questions: **New Results**: a). The selection of the four proteins for our new visualizations remains consistent with our initial submission, as they were randomly chosen from the test set. We have now conducted testing on 100 proteins, and the corresponding results are presented in the table below. Notably, GraDe-IF demonstrates a significant performance improvement over PiFold and achieves comparable results to ProteinMPNN. In light of this, we would modify our claim to "achieve the best performance on TM score" as you have suggested. | Method | Success | TM score | |-------------|:---------:|:-----------------:| | PIFOLD | 85 | 0.80 +- 0.22 | | ProteinMPNN| 94 | 0.86 +- 0.16 | | GraDe-IF | 94 | 0.86 +- 0.17 | b). Our investigation into the three 'unfoldable' proteins revealed intriguing insights. These proteins faced challenges not only with our model but also with the baseline models. Notably, the structural determination of the three failed proteins, 1BCT (https://www.rcsb.org/structure/1BCT), 1BHA (https://www.rcsb.org/structure/1bha), and 1CYU (https://www.rcsb.org/structure/1CYU), were all based on NMR, which is an experimental technique that analyzes protein structure in a buffer solution. Due to the presence of multiple structures for a single protein in NMR studies, it is reasonable for folding tools to assign low foldability scores. In addition, we extended our investigation to another protein, 1H3L (https://www.rcsb.org/structure/1H3L), which our model successfully folded but PiFold failed to do so. This particular protein, a fragment of SigR, displays remarkable flexibility, leading to its low experimental resolution that causes a low structure prediction confidence. **Weaknesses**: 1. We followed [1] when defining the score for diversity. We will include this reference in the final version when discussing the related results. **Questions**: 1. Your suggestion of showcasing the performance of our model on a broader range of proteins is valuable. Given the substantial number of proteins in the test set, including them all might indeed dilute the main results. Instead, we intend to incorporate a selection of representative proteins in the revised appendix. These chosen proteins will exemplify various characteristics, such as those that are unfoldable, exhibit high pLDDT scores, manifest diversity, and more. 2. We appreciate your thoughtful suggestion and will certainly include this aspect in the revised version. **Reference**: [1] Zheng, Zaixiang, et al. "Structure-informed language models are protein designers." bioRxiv (2023): 2023-02.
Thank you for continuously updating your results, I appreciate how much work you are doing in this discussion period. a) Again, also for these new results, I disagree with “GraDe-IF demonstrates a *significant* performance improvement over PiFold”. The standard deviations for the TM scores are quite large, and all methods are actually “within reach” of one another. It’s fine to not be absolute state of the art, as long as the results are presented in a fair way. It’s still a very interesting method. b) Very interesting! I would definitely include a discussion about this in the paper / the appendix, to make your story stronger. All my other concerns were addressed, thank you. I think the exemplary proteins (appendix) will be very interesting to see.
Thank you for the detailed response. I agree with the authors that the novel combinations of some known approaches as in this case provide a simple yet effective method. I do not find other main concerns and would like to keep my score. I tend to accept this work if there is no main issue commented on by other reviewers in the experimental part.
Decision
Accept (poster)