Cell-ontology guided transcriptome foundation model

Summary

This paper introduces scCello, a Transcriptome Foundation Model (TFM) which incorporates prior information from cell ontology graphs denoting the relationships between different cell types in order to guide cell representations. The pretraining objective in the scCello framework encapsulates masked gene expression prediction, a supervised contrastive objective for aligning cell representations to cell ontology terms, and relational alignment of similar cell types, which helps guide representation learning during pretraining to reflect taxonomic relationships between cells. The foundation model, scCello, is pretrained on 22 million cells from the CellxGene public data repository, and demonstrates improved performance over other single-cell foundation models in downstream tasks, including cell type clustering, cell type identification, marker gene prediction, and batch integration.

Strengths

This work presents a novel foundation model for single-cell transcriptomics, scCello, that incorporates prior information about cell ontology in order to guide representation learning over a large pretraining corpus. The methodology and motivation for the paper is clear, and the work represents a significant step towards training foundational models that incorporate prior information, which subsequent works may build and improve upon. For pretraining, the authors introduce a multi-component loss function, encapsulating: (i) masked gene expression prediction, (ii) a supervised contrastive objective to align representations to of cells with similar cell ontology terms, and (iii) a relational alignment objective which enforces cell representations to follow the structural similarity of their cell types in the cell ontology graph. For quantifying cell type similarity, the Personalized PageRank (PPR) algorithm is used to derive structural similarity of pairwise cell types in the cell ontology graph, which is then used to guide representation learning. The single-cell RNA seuqencing data preprocessing methodology seems sound, and follows common techniques for data preprocessing used by other foundation models. The authors also include a hyperparameter and compute resource comparison of scCello against other established single-cell foundational models - Geneformer, scGPT, scTAB, UCE. Overall, the paper is well-written, motivated well, and clearly explained. Adequate details are provided for the loss objectives and methodology, and additional dataset acquisition and preprocessing details are provided in the appendix.

Weaknesses

The authors mention that cell type ontology identifiers were obtained for the 22 million cell pretraining dataset from the CellxGene database, to enable mapping between individual cells and cell ontology. While this allows for additional priors from cell ontology graphs to be used during pretraining of scCello, it also necessitates labeled data during pretraining, which may limit the scalability of scCello in terms of available pretraining data compared to other methods which do not require annotated pretraining single-cell data.

Questions

For the overall pretraining objective in equation 5, more details could be added about how the four loss terms are weighted relative to one another. Are their coefficients to weigh the different loss terms? If not, do the magnitudes of the different loss terms differ significantly? The authors quantify ontology relationships in the cell ontology graph by using structural similarity, and use the Personalized PageRank (PPR) algorithm to estimate pairwise node similarities for cell types. Examples of cell types associated with varying levels of structural similarity are also presented in Table 7. It would be helpful if the authors provided a motivation for going with a structural similarity measure such as PPR score, rather than a simpler metric such as the distance of two nodes in the cell ontology graph.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative impacts of their work, and agree to do a code implementation release upon acceptance of the paper.

Summary

The authors introduced a transcriptome foundation model (TFM) named scCello to resolve the current problem of most of the TFMs they treat cells as independent samples and ignore the taxonomic relationships between cell types. By integrating cell ontology information as well as incorporating three key objectives in the pretraining framework: masked gene prediction, cell type coherence loss, and ontology alignment loss, the model can learn gene coexpression patterns, cell type-specific representations, and structural relationships between cell types. The authors performed a series of experiments to show that the model can predict the unseen cell types, integrate datasets across diverse batches, cluster cell types, predict the cancer response to the drug, and predict the marker genes.

Strengths

1. The manuscript is well-written and well-structured, and it is easy to read. 2. The authors effectively incorporate cell type ontology information at the pertaining stage, demonstrating in the comprehensive experiment results that this ontology information enhances the model's ability to learn cell type taxonomic relationships, thereby producing more biologically meaningful cell representations.

Weaknesses

1. In the zero-shot cell type clustering experiment, it is not clear that the authors run the Louvain algorithm with default settings or the optimal setting on Seurat, Harmony, and scVI. 2. In the marker gene prediction part, what is the difference between predicted marker genes based on scCello and the traditional marker genes based on the differential expression level? 3. There is a paper (A Deep Dive into Single-Cell RNA Sequencing Foundation Models) indicating that the L1 logistic regression model achieves decent performance in cell type annotation task. Could the authors add the L1 logistic regression model for comparison in Table 2?

Questions

1. How did the method achieve a high score in the batch integration? In the loss function, there is no specific loss for batch correction. Could the authors give some explanations for this question? 2. Can the model predict the novel marker genes?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed all limitations.

Summary

This paper presents scCello, a single-cell, Cell-ontology guided Transcriptome Foundation Model (TFM) that leverages cell-type relationships from cell ontology graphs to enhance cell representation learning. scCello incorporates three levels of objectives during pre-training: masked gene prediction, cell-type coherence, and ontology alignment. This approach improves the model's generalization and transferability capabilities, leading to superior performance in tasks such as cell type identification, novel cell type classification, marker gene prediction, and cancer drug response prediction. scCello is also robust to batch effects and demonstrates high parameter efficiency compared to existing TFMs.

Strengths

1.Integrating cell ontology graphs into TFM pre-training: This is the most innovative aspect of the paper. scCello improves model understanding of biological relationships between cells by incorporating cell-type relationships from cell ontology graphs into TFM pre-training, enhancing the model's generalization capability and transferability. 2.Introducing a multi-level objective function: scCello employs a multi-level objective function that encompasses gene-level masked gene prediction, intra-cellular cell-type coherence, and inter-cellular ontology alignment. This multi-level approach enables the model to learn complex relationships between genes, cells, and cell types, leading to more precise and robust cell representations. 3.Validating the model's effectiveness and advantages: The paper presents a comprehensive set of experiments demonstrating the effectiveness and superiority of the scCello model in tasks such as cell type identification, novel cell type classification, marker gene prediction, and cancer drug response prediction. Additionally, the paper highlights scCello's robustness in handling batch effects and its parameter efficiency, suggesting the model's strong potential for real-world applications.

Weaknesses

1.Lack of Theoretical Justification: The paper heavily relies on empirical results to showcase scCello's effectiveness. However, it lacks a rigorous theoretical analysis of the proposed approach. A deeper theoretical understanding of the objective functions, their impact on learning, and how they contribute to generalization capability would significantly strengthen the paper. 2.Limited Comparative Analysis: The paper primarily focuses on comparing scCello with existing TFMs but doesn't thoroughly analyze its performance against methods specifically designed for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction. A more robust comparison with specialized methods would better illustrate scCello's strengths and highlight its relative advantages. 3.The paper's current model size might be insufficient to fully capture the complexities of single-cell transcriptomic data. This limitation could hinder the model's ability to achieve optimal performance on tasks requiring deep understanding of complex biological processes, especially when dealing with large and diverse datasets.

Questions

1.Can the authors elaborate on the theoretical foundations underlying the effectiveness of integrating cell ontology into TFM pre-training? Specifically, how do the proposed objective functions contribute to learning more informative and robust cell representations? 2.Can the authors provide a more comprehensive comparison of scCello's performance against specialized methods designed for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

1.The cell ontology is constantly evolving, and the paper acknowledges that scCello currently requires retraining the entire model for any updates. This limitation hinders the model's adaptability and its ability to keep up with the latest knowledge in cell biology. 2.The paper states that they aim to scale up the model size in future work. This indicates that the current model size might not be sufficient to fully capture the complexities of single-cell transcriptomic data, potentially limiting its performance on certain tasks or datasets.

Summary

This paper introduces a new transcriptome foundation model (scCello) to generate cellular representations from single-cell RNA-seq data. The key contribution is the integration of known cell type labels (previously annotated by CellxGene submitters) within two novel objectives. First, the authors introduce a cell-type coherence loss to minimize the distance between a cell's representation and the learned representation of its associated cell type. Second, the authors introduce an ontology relational alignment loss to ensure that the similarity between the representations of two cells matches the similarity of their corresponding cell types in an ontology graph. The authors benchmark their model against previously pre-trained transcriptome foundation models and traditional (non-foundation) models. They evaluate performance on tasks including zero-shot cell clustering, fine-tuned cell type classification, and marker gene discovery and find that scCello is state of the art on almost all tasks.

Strengths

- *Conceptually simple but effective*: Utilizing cell type annotations, which are almost always available in published scRNA-seq datasets, to enhance the model's representations is smart and relatively straightforward. Other methods have proposed incorporating cell type labels before, but a cell type classification loss that treats each cell type as independent has shortcomings. The ontology relational alignment loss presented here which uses pre-existing ontology graphs to determine the similarity between cell types seems intuitively effective and empirically turns out to be too. - *Meaningful task selection*: The authors do a good job benchmarking their model on meaningful tasks. For example, they evaluate clustering performance not just on heldout IID data, but also in unobserved cell types, tissues, and donors. The model's improved performance on novel cell type classification and marker gene prediction suggests that it has captured biologically meaningful information. - *Appropriate baselines*: The authors do not just compare to existing transcriptome foundation models, which have been heavily criticized for not being competitive with traditional methods. - *Algorithm for novel cell type classification*: Their method of comparing the similarity vector between a cell and prototype representations of each cell type to the similarity vector between a cell type and all other cell types, as derived from the ontology graph, is clever.

Weaknesses

- *Lacking analysis on ontology relational alignment loss*: While it seems intuitive that the ontology relational alignment loss is beneficial, the significant performance improvements are unexpected and probably warrant further analysis. For example, understanding which cell types show improved clustering performance could help provide some intuition. - *Missing some baselines*: Since the model assumes that cell type annotations are available, it should be compared against traditional methods that utilize these annotations (e.g. scANVI). - *Limited to 10x data*: The batch correction results, as is, are impressive, but it would be interesting to see if this method could handle data from different sequencing technologies.

Questions

1. The regularization loss (Eq. 2) was proposed to prevent class collapse. But it's unclear why this would help. In fact, it seems to be encouraging all samples of cell type c_{i} to have the same embedding: Linear(h_{c_{i}}). 2. How did you train scVI for Table 1? Did you train it on your pre-training dataset, in-distribution dataset, and out-of-distribution datasets jointly? If not, for a fair comparison, it seems like you should. 3. To avoid picking up housekeeping genes in the marker gene analysis, could you see if knocking out housekeeping genes leads to a representation change that differs from that induced by knocking out marker genes. I imagine knocking out housekeeping genes might shift the cells to a previously uncharted part of the latent space (dead cells), whereas knocking out marker genes might shift the cells to a different cell type.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors are forthcoming about the limitations of their model.

Summary

This paper introduces scCello, a new transcriptome foundation model (TFM), which learns cell representations over RNA gene expressions. Apart from using Masked language modeling (MLM), masking random gene expressions in cells, the work leverages structural knowledge from ontologies to improve the learned representations. It enforces cell type coherence, where cells of the same type should be close in representation space. Similarly, the representation of structurally close cells, measured over a PageRank measure, should be close. These structurally inferred objectives are encoded as separate contrastive losses. The pretaining objective tries to minimise the sum over the MLM loss and the two contrastive losses together with a regularisation term, which tackles class collapse. scCello is backed by a transformer-based encoder-only model with roughly 10.5 min parameters, which is rather small compared to competitors. The evaluation covers multiple downstream tasks and an ablation study over the optimisation objectives. The results show that scCello is either competitive with state of the art performance or improves it and that the aggregation of the losses is beneficial.

Strengths

- Data: It's great that the work leverages diverse data sources (structural and token-based) and tries to incorporate the strengths of both of them by using custom losses. - Experiments: The evaluation is extensive comparing the performance of the proposed model with multiple competitors and across multiple tasks. I also appreciated the ablation study and the analysis of overall performance wrt. #parameters. - Model: With a reasonable size the produced model can surpass many larger-sized competitors.

Weaknesses

**Incorporating structural knowledge** GNN have been used for incorporating structural knowledge into pretraining of transformer-based models. For instance, take a look at GraphFormers (https://arxiv.org/abs/2105.02605). Such methods can be more elegant than the proposed PPR metric and the contrastive loss, since everything can be captured inside the model and no custom metric is necessary. While the author show that their proposed approach is beneficial for overall performance, such methods should be discussed or compared to, since they overlap in their goal of fusing structural and textual data with a major contribution of the paper. **Limited Novelty** The paper does not advance any method or approach. It is an application of existing methods for creating a TFM model.

Questions

While you analysed the impact of the single training objectives, you didn't present numbers on different model sizes for scCello. Do you have any evidence how larger or smaller model impact downstream performances? If ontologies and knowledge base grow, how does the training approach scale with them? Since you use contrastive loss, you could run into bottlenecks, if the number of negatives grows?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

yes