Multi-Scale Representation Learning for Protein Fitness Prediction

Summary

This work targets at the protein fitness prediction and introduces a sequence-structure-surface multi-modality (aka. multi-scale) self-supervised learning scheme. The results show that S3F outperforms baseline algorithms and achieves SOTA in the ProteinGym benchmark. I like the inspiration and favor S3F's promising performance. However, the study is very close to some recently published papers at ICLR 2024 (i.e., ProteinINR and PPIFormer) but completely ignores them. A fair comparison and analysis of the difference is necessary and I am afraid this is a big reason for me to give a borderline score.

Strengths

(1) I advocate the research direction of pretraining on all feasible protein modalities, containing sequence, structure, and surfaces. Each representation form has its strengths and an incorporation is beneficial for the zero-shot protein fitness prediction. (2) The author conducts extensive ablation studies over function type, MSA depths, taxon, and mutant depth. This helps readers to better understand the impact of each component of their model. The OOD examination of unseen protein families is also encouraging to demonstrate the superiority. () In section 4.4, the author navigated the impact of structure quality and observed a clear performance drop when lower-quality structures were used. This phenomenon underscores the importance of accurate structures for fitness prediction. Notably, there have already been some studies that try to bridge the gap between representations in real and predicted structures [A], and I would recommend the author take a look if seeking further performance improvement. [A] Protein 3D Graph Structure Learning for Robust Structure-Based Protein Property Prediction. AAAI 2024. (3) The visualization and experimental analysis are elegant. I learned a lot from it.

Weaknesses

(1) Missing of some closely related baselines and relevant work, GearNet [A] is one of the earliest studies in structural pretraining. ProteinINR [B] also leverages sequence, structure, and surfaces in a self-supervised way. The only difference is that they are not specifically designed to solve zero-shot problems. Based on these facts, the so-called "multi-scale representation learning" of S3F is no longer novel to me. From my point of view, both GearNet and ProteinINR can be simply adjusted to realize zero-shot prediction. Thus, it would be more interesting to see whether S3F outpasses them in all categories of downstream tasks. Besides, there are also some appealing self-supervised methods to predict mutant effects. RDE [C] pretrain a structural encoder by masking and predicting side-chain angles. The author also did not discuss this line of research. Last but not least, PPIFormer [D] pretrained a structural encoder with simple MLM in complex structures and can conduct zero-shot fitness prediction. I suppose it should also be compared and mentioned. [A] Protein Representation Learning by Geometric Structure Pretraining. ICLR 2023. [B] Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning. ICLR 2024. [C] Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction. ICLR 2023. [D] Learning to design protein-protein interactions with enhanced generalization. ICLR 2024. (2) Some important relevant works are missing. For instance, the author claims that the weights of ESM-2 are frozen and only the structural encoder is tuned. This practice is very similar to the one in [A]. I would recommend the author at least cite it. [A] Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks. Communications Biology 2023. (3) The author used dMaSIF to generate the surface based on the backbone structures of proteins. This raises two doubts. Firstly, the side-chain atoms are ignored for surface generation, therefore the surface should be smaller than its standard one. Secondly, the fast-sampling algorithm developed by dMaSIF has unavoidable randomness. Different random seeds produce different surfaces. Has the author taken these two potential negative effects into consideration? Has the author adopted software like PyMol to acquire the surfaces?

Questions

(1) In line 256, the author said "... by ensembling them with EVE predictions through the summation of their z-scores". I am not familiar with this alignment part. Can you please explain the details more? (2) The title used "mutli-scale" reprsentations of proteins. However, from my personal point of view, multi-scale refers to atom-scale, residue-scale, and protein-scale, As this study proposed to leverage sequence, structure, and surfaces, "multi-modality" would be more appropriate. (3) The author claimed that the backbone structures remain unchanged post-mutation. This is fine in most settings. However, did the author consider a more challenging circumstance, what if the structure varies significantly due to the mutation? [A] proposes a co-learning framework to simultaneously forecast the fitness change and the structural update. Does the author consider this factor? [A] Thermodynamics-inspired Structure Hallucination for Protein-protein Interaction Modeling. ICLR submission. (4) If I understand it correctly, the objective is to develop an unsupervised model that can predict a score for each mutant to quantify the changes in fitness values relative to the wild-type. Thus, it is more related to the mutant effect prediction task rather than the pure fitness prediction. I would recommend the author use "mutant effect prediction" instead of "fitness prediction" to better depict the target problem.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper presents a multimodal framework that integrates protein sequence, structure, and surface information together to predict protein fitness. The task of protein fitness prediction is a critical quality assessment of protein embeddings. Protein language models (pLM) are used for sequence representation which is embedded in a graph representation of protein structure and processed with geometric vector perceptron (GVP). The dMaSIF, a protein surface representation model, is used to encode protein surface, together with pLM embeddings of nearest residues. Surface features and sequence-structure features are integrated by concatenation and linear layers to predict masked residue identities. The model is evaluated on ProteinGym, a golden protein fitness benchmark dataset, against various pLM models and multiple sequence alignment (MSA) -based models and achieves favorable results in terms of spearman correlation with mutational effects. Evaluations are further run on AlphaFold predicted structures and less representative proteins to prove generalization.

Strengths

**Originality** This is one of the first papers that integrates pLMs, protein structure and surface representations together to predict protein fitness and shows improvement on benchmarks. It is also novel to combine MSA information with it to further improve predictive power of the model. **Quality** The submission is technically sound with a detailed description of their framework and extensive experimental results on benchmark datasets with appropriate reasoning and explanation. **Clarity** The paper is clear to read and easy to follow. Presentation of results are simple yet effective. **Significance** This method achieves SOTA performance on protein fitness prediction benchmark and provides a standardized way to integrate three modalities of protein together for a unified representation.

Weaknesses

1. There is not adequate reference of prior work on similar topics, e.g. multimodal fusion and protein surface representation learning. The idea of integrating surface into protein representation is not novel, for example [1], but is neither acknowledged nor benchmarked in the paper. 2. Even though the performance of proposed method is better than competing methods, it's marginal. As the author writes in abstract '...these sequence-structure models have so far achieved only incremental improvements when compared to the leading sequence-only approaches.' However, SaProt (sequence+structure) provides 0.35 performance gain of correlation compared to ESM2 (seq) while S3F (proposed) provides 0.13 compared to SaProt. The advantage brought by surface representation is more incremental. 3. Statistical results should be provided for figure 2 when possible. How is the variation of performance on 217 assays? References: [1] https://arxiv.org/pdf/2309.16519

Questions

1. line 195. ESM features are integrated into surface representations. Why not just use surface feature itself because it contains chemistry and geometry information already? 2. line 218. 'we avoid information leakage from surfaces by removing the top 20 closest surface points for each selected residue' How to ensure 20 is sufficient? 3. line 230. What percentage of results are output from ESM? 4. Figure 3. Why do some residues have negative correlation to model scores? Is there any quantitative analysis of differences rather than just qualitative visualization? 5. line 347. 'ignoring side-chain information'. If side chain is ignored, how to get surface information?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The majority of experiments in the paper is accomplished with experimental structure rather than predicted structures. And the results show that ordinary prediction quality could harm performance of the method. Thus, it is not clear if the model can be finetuned on AFDB to achieve better performance because of the quality of side-chain conformations in predicted structures.

Summary

This paper proposes a new protein fitness prediction model, the Sequence-Structure-Surface (S3F) model, which integrates protein sequence information from a protein language model embedding, protein structure information processed through a Geometric Vector Perceptron (GVP) module, and protein surface information processed through the dMaSIF model and a GVP module. After pre-training S3F on CATH (to be specific, pre-training the GVP modules while freezing the protein language model weights), the model outperforms state of the art baselines on zero-shot protein function prediction. The paper also analyzes the breakdown of these results across different fitness prediction settings, finding that all settings benefit from adding structure, surface, or MSA information on top of sequence information, with the largest gains arising in binding and stability assays and in settings where protein language models have limited training data or biased training data.

Strengths

The paper is well written and presents a parameter-efficient, novel way to include protein surface information into a model for fitness prediction. The paper contains results that will be interesting to the protein modeling community, showing that modeling the surface explicitly improves function prediction beyond other baseline methods which include structure information. It seems very interesting to investigate why such surface information is not easily captured by other methods that do utilize structure information.

Weaknesses

1. The paper does not include results or analysis about the sensitivity of results for various hyperparameters of the model, such as the width, depth, and hidden dimension of GVP modules, the number of surface points to include in a neighborhood, and the choice of protein language model for embeddings. Since the paper focuses on the ProteinGym benchmark for its only evaluation, it would be nice to see more in-depth experimentation and/or ablation of different parts of the S3F model. 2. The central claim that surface information is important could also be strengthened with more experiments to understand how pre-processing structural information into surface features extracts useful information that is not as easily accessible with structure-based models that don’t do similar feature engineering/pre-processing.

Questions

1. In Table 1, the bottom half of the table compares alignment-based models with S2F and S3F ensembled with EVE scores. It would be interesting to see results for each of the baselines in the top half of the table (e.g. MIF-ST, ProtSSN, SaProt) also ensembled with EVE scores. 2. What is your current intuition about why structure-aware fitness prediction models are unable to leverage surface information to its full extent?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors assess limitations of their work, including the fact that their method is limited to considering substitutions, and cannot handle insertions or deletions. I think this is a salient point to keep in mind, which limits the applicability of their method to real protein design tasks. See also weaknesses above.

Summary

This paper augments a protein language model with two additional features, structure and surface information. The authors show that they can effectively use this information to predict protein function slightly better (though SOTA-level on relevant benchmarks) than sequence-only PLMs.

Strengths

Significance: Perhaps the strongest point of the paper is that it clearly shows they can leverage structure and surface data to improve the zero-shot performance over sequence-only methods. The improvement is relatively small compared to models that only use sequence however. Clarity: Paper is well written, has a good flow and logical steps. Soundness: For the most part the right experiments are being done, the data benchmarks are correct and relevant models are tested (see weaknesses for a notable exception).

Weaknesses

Novelty/Originality: I can’t tell how this paper is a real advance on (https://arxiv.org/pdf/2204.02337, ref 33 in this paper), with the title being nearly identical too, which is published 2 years ago. Yes the test set is different (evaluated on proteinGym), and some features of the architecture are also different, but the author choice of not benchmarking against Holoprot on the same dataset is problematic, if for nothing else because it’s the most clear way of showing the performance advance due to architectural improvement and no simply because of additional features (Because the “idea” of combining sequence, surface and structure is already executed.) From the abstract “Moreover, the function of certain 11 proteins is highly dependent on the granular aspects of their surface topology, 12 which have been overlooked by prior models. “ - Reference 33 (https://proceedings.neurips.cc/paper/2021/file/d494020ff8ec181ef98ed97ac3f25453-Paper.pdf) in your own citations is exactly about this, connecting sequence, structure and surface. Can you explain? - Relatedly I think the similarity (almost identical : "Multi-Scale Representation Learning on Proteins") in the paper title is a poor decision, if not for the appearance of plagiarism, for the fact that it makes the difference between the papers even less clear. It should be titled to embolden the difference. That paper is also doing fitness prediction.

Questions

I think the authors really need to contrast this work with reference 33 and argue why it’s a meaningful conceptual or performance advance. I think the paper would be better if the authors would do at least two ablation studies where at least - structure is dropped - both structure and surface are dropped (dropping sequence is more complex but it may also be interesting). It is surprising to me that in the non MSA case S2F basically does not improve on benchmarks, and it could be that the structure data is adding no performance on top of the surface. Comparing to the original sequence-only model in this architecture would also help clarify how much improvement is simply due to other factors than surface information.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Conceptually the paper is doing something that has been done before, with some small innovations. It is relevant however that it can indeed consistently improve on existing language models. ::::: Updating my previous score to 5

Update

Thanks for your response. I am satisfied with the additional experiments and the outstanding performance compared to those important baselines. However, I still believe the way to generate surface using dMaSIF is not a smart choice despite their effectiveness in GPU parallel. However, I appreciate the efforts in answering my question and would like to raise my score to 5. I hope the authors can incorporate new changes during the rebuttal period into the final revision.

Thank you

I thank the authors for their responses. Having read their answers, together with the other reviewers, I'm comfortable increasing my score. I do still feel like the conceptual advance on the previous paper and performance advance over the best benchmarks (given the auxiliary data it needs) is not at the level of a clear accept.

Thank you

Thank you to the authors for their additional experiments and careful response. I think the paper presents nice empirical results improving upon state-of-the-art ProteinGym performance, although the improvements are somewhat small (e.g. SaProt-MSA compared to S3F-MSA), and limited to that benchmark. Based on the responses and other reviewer comments, I will keep my score.

Reviewer response

Thank you for the response. If I understand correctly, the surface is calculated only from backbone structure? Then this definition is largely deviated from what people normally conceive as true protein surface which is mostly dictated by side chains. I believe the wording regarding surface is misleading and would cause confusion for audience interested in protein surface representation.