Summary
This work presents a model for a chat agent trained on protein question answering. The model encodes input protein sequence and structure information by leveraging powerful protein representation models, ESM-1b and ESM-IF1, respectively. The model employs a so-called “Protein-Language Pretraining” (PLP) module, analogous to Visual-Language Pretraining to align the modalities of protein sequence and natural language description of a protein. Structure and sequence embeddings are combined and projected into a shared space with a LLM decoder to produce natural language responses to prompts about the input protein.
Strengths
This model is able to combine multiple modalities of information related to proteins, sequence, structure, and descriptions. The use of the PLP-former is able to refine pre-trained sequence embedding features for the task of text description generation without losing the power of the original features.
Combines sequence and structure features via the use of the “multi-level projection adapter.” Since encoding models (such as the ESM models used here, or others) are now easily accessible and widely used for representing proteins, making use of both is the way to go. The empirical results in Table 1 support this, showing the performance degradation when structure is not incorporated.
Weaknesses
There are several points of ambiguity with the dataset and modeling (see the Questions section for details).
Although several NLP-based quantitative evaluation metrics are included in evaluation, these are likely insufficient for this task where it is not clear if the chat responses are correct. Some analysis, even if a case study of a few proteins, would strengthen the submission. E.g. Do responses have mutually exclusive functions listed, etc? For a work that aims to serve drug development or protein design, it is not sufficient to have the grammar and appearance of a correct response, but to also be factually correct. The SPICE and PubMedBERT scores move closer to such an evaluation, but they are not sufficient to “trust” the output of such a model yet. The authors rightfully list these concerns in their discussion. Some investigation could be done presently to highlight and quantify these limitations
Questions
Q1] The text description in the dataset are not clear. Some points are listed in the first paragraph of 3.2, but is not comprehensive. Where was the text taken from? For example, when subcellular location is mentioned, how is this represented? E.g. literature excerpts, GO terms, etc? More detail about the training descriptions would be helpful.
Q2] “which comprises 143,508 aligned pairs of protein 3D structures and the corresponding description” What is mean here by “aligned”? Does this refer to the structure-description pairing, an alignment of proteins, or something else?
Q3] The last component of the input data that is mentioned is “corresponding scientific literature.” What is meant by this? How is this literature chosen and used in the model?
Q4] How were training and test sets split? If random split, what was the overlap, sequence/description/family/function similarity across splits?
Minor comments and additional suggestions
For clarity, consider adding which of the VLP types your approach falls into (i.e. representation vs generative)
In the second paragraph of Section 1 RCSB-PDB is introduced without explaining what it is. The sentence containing this reference is also very unclear. I suggest re-framing it to make clear that you are making a connection between language and the evolutionary information contained in protein sequences. “Evolution” does not have a “spoken language” as such.
It is not clear how the learnable tokens are initialized/where they are coming from and their role. Why use 32 etc?
Notation slightly inconsistent, especially in equation (5) and accompanying description where matrices are vectors no longer in bold font.
Rating
6: marginally above the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.