Summary
This paper develops a method for training a supervised learning algorithm to carry out various protein classification tasks, including EC prediction, GO term prediction, and protein fold classification, among others. The main idea is to use a hierarchical representation of each protein, wherein spherical clusters are iteratively joined to identify key amino acids. These are then fed to a multilayer perceptron for classification. The approach is benchmarked on a several standard benchmarks and shows remarkably good performance relative to state-of-the-art methods. The paper also includes some visualizations of the learned key amino acids.
Strengths
The paper is clearly written and does a good job of explaining the importance of the task.
Many different benchmark datasets are analyzed, and many competing methods are included.
Weaknesses
I think this paper misrepresents what it's about. The paper claims to be about representation learning, but it's really about protein classification. The fact that you focus on one particular layer of their network (the one before the MLP) does not actually mean that this layer is any more fundamental than other layers. To my mind, representation learning should be about learning a representation in an unsupervised fashion that can be used to solve multiple downstream tasks. Instead, in this method, a new representation is learned for every supervised task. Indeed, I found it quite surprising, in Section 3.1, that representation learning was framed as a classification task. This just seems wrong to me.
The introduction also does a poor job of preparing us for what the method actually consists of. This is fundamentally a deep neural network method, but the intro talks about it as if it's doing multiple found of clustering.
The network itself is under-described. I don't think anyone reading this paper could even approximately reproduce this work. An example is that the text mentions self-loops but doesn't mention where they are.
I recognize that you are re-using public benchmarks, but I still find it surprising that the train/test splits allow such high sequence similarity. In the EC benchmark, for example, a train and test example can be up to 95% identical. The other benchmarks don't even discuss how train/test leakage is avoided.
It's distressing that you claim that protein fold recognition was introduced by a paper in 2018. This problem has been studied very extensively for at least 20 years prior to that. See, e.g., the Jaakkola-Haussler paper that won the best paper award at ISMB in 1999.
This observation leads me to my primary concern about this paper, which is that the results are simply too good to be true. If you really did improve across the board on all of these tasks, achieving such consistent and significant improvements, then this paper should be published in Nature rather than ICLR. To justify such remarkable results requires much more extensive validation and explanation, rather than just a single table with summary statistics. Taking, for example, the fold recognition problem, it would be necessary to examine in detail the predictions made by your method and the best-performing second best model. I'd like to know which proteins you got right that the other method failed at, and I'd like to understand how your method was able to achieve this gain. It's also not at all clear to me that the methods in Table 1 really do represent the state of the art. Some obvious missing contenders are AlphaFold and RosettaFold.
Minor comments:
The abstract talks about "performance" but fails to make clear what problem is being solved.
In the penultimate paragraph of the intro, I think you should elaborate a bit more on what the representation actually consists of. In particular, it's not clear whether the final MLP receives just the identities of the selected amino acids, or if it also retains some notion of the hierarchy.
The term "spherosome" doesn't seem to be appropriate here. I had never heard the term, but a web search suggests that it's the name of an organelle. Just say "spherical."
Questions
Are the sequential edges in your initial implementation actually important? As long as the sphere is larger than the connection between two adjacent amino acids, then you could just use a sphere.
I was surprised that the input layer of the model contains, among other features, the spatial coordinates of the amino acid. This seems counterintuitive, because the protein's representation should be invariant to scale, rotation and translation. Why does this even make sense?
You note that the fact that different proteins from the same family end up selecting the same amino acids as "representative" is only surprising if the proteins are quite diverged evolutionarily. How diverged is this family, and how well does this analysis when comparing proteins that share function but very little residual sequence identity?
Figure 5 shows that the selected residues differ by task. Is there any evidence that these residues are actually relevant to their respective tasks? Just selecting different residues is not, by itself, super surprising.
Figure 6 does not show that the proteins exhibit similar structural motifs. It simply shows that proteins from the same class have similar representations internally. Again, is there any evidence that this is not just a reflection of overall sequence similarity?
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.