Weaknesses
The following points are my major concerns regarding the highlighted contributions in the paper:
* As a method development paper, the algorithmic and engineering innovations presented are minimal. While the authors position this work as introducing a novel lightweight genomic foundation model, the primary architectural differences from DNABERT-2 are:
* Flash Attention 2 replacing Flash Attention Triton,
* ALiBi positional encoding instead of RoPE,
* and GEGLU activation in place of SwiGLU.
These modifications are sensible engineering adjustments for language model development but do not constitute genuine innovations.
* While Sequence Length Warmup (SLW) has shown advantages in training decoder-only models by introducing progressively longer, more complex sequences akin to curriculum learning, its relevance for encoder-only genomic models is uncertain. Given that dnaGrinder’s tokenized sequences vary only moderately in length (700-2300 tokens, as reported), simpler alternatives—such as truncating sequences to a max_token_length or using balanced sampling (e.g., LengthGroupedSampler) for length-consistent batches—could likely achieve similar benefits.
* The introduction of ME-BPE in dnaGrinder seems aimed at solving a memory efficiency problem that may have limited relevance in practical genomic modeling. While training the BPE tokenizer can indeed be memory intensive, applying it during pretraining and inference requires minimal memory overhead. Even if a system lacks the memory to load the entire corpus, cloud vendors provide high-memory instances at reasonable costs, making this less of a barrier. Moreover, in NLP, tokenizers are often trained on a representative subset of the corpus (using options like --input_sentence_size=<size> and --shuffle_input_sentence=true in sentencepiece), rather than the full dataset used for language model training. This subsampling approach effectively reduces memory requirements without compromising vocabulary quality or requiring the additional complexity of ME-BPE’s iterative adjustments. As a result, ME-BPE seems to address a problem that is not particularly impactful in real-world scenarios.
* The authors claim that this work is the first to apply repetitive sequence removal, showing a clear disregard for the existing literature. Numerous models have already addressed repetitive elements using various techniques: down-weighting (GPN [1], PlantCaduceus [2]), downsampling/removal (SpliceBERT [3], GPN-MSA [4], hgT5 [5]), and fragment deduplication (gLM2 [6]). Notably, hgT5 also utilized RepeatMasker to identify repeats. The proposed method aligns with what should be considered standard practice, rather than an innovation.
Minor issues:
* The repetitive element removal method is introduced in the Human Reference Genome Dataset section, but the authors should clarify whether it was applied only to the human reference genome or to the entire combined dataset. If the latter is the case, this removal method would be better placed in the Methods section for consistency and clarity.
* Encoder-only models like dnaGrinder are non-causal and do not require KV caching for inference or embedding extraction. The authors' claim that Flash Attention 2’s inference optimization is "especially beneficial for our model...with most not exceeding 1,000 bp" is factually incorrect. In BERT-style models, inference is achieved through a single forward pass without the need for KV caching, which is specifically beneficial for autoregressive generation—a context irrelevant to dnaGrinder. Thus, the supposed inference advantage of Flash Attention 2 does not apply here.
* Some suggested improvements to the paper:
- In Figure 1a, the plot displays two masked token embeddings, yet the tokens in “ME-BPE Tokenization” are not masked. Aligning these would improve clarity.
- In the last paragraph of Section 2.2.2, Further Pretraining, the phrase “our model performed 100,000 steps” should be corrected to “DNABERT-2 model performed 100,000 steps.”
- DNABERT-2 should be recognized as the state-of-the-art model in the benchmark alongside dnaGrinder, as it not only holds the highest combined number of top-1 and top-2 rankings but also has the highest average scores.
- In the results table, the “Params” header is marked with a ↓ symbol, explained as “↓ indicates that a lower value is better.” However, the question of whether smaller or larger language models perform better for genomic modeling remains unresolved. It would be beneficial for the authors to discuss this open question.
References:
1. Benegas, G., Batra, S. S., & Song, Y. S. (2023). DNA language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44), e2311219120.
2. Zhai, J., Gokaslan, A., Schiff, Y., Berthel, A., Liu, Z. Y., Miller, Z. R., ... & Kuleshov, V. (2024). Cross-species modeling of plant genomes at single nucleotide resolution using a pre-trained DNA language model. bioRxiv, 2024-06.
3. Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2024). Self-supervised learning on millions of primary RNA sequences from 72 vertebrates improves sequence-based RNA splicing prediction. Briefings in Bioinformatics, 25(3), bbae163.
4. Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2023). GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction. bioRxiv.
5. Ioannidis, N. (2024, March). GUANinE v1. 0: Benchmark Datasets for Genomic AI Sequence-to-Function Models. In Machine Learning in Computational Biology (pp. 250-266). PMLR.
6. Cornman, A., West-Roberts, J., Camargo, A. P., Roux, S., Beracochea, M., Mirdita, M., ... & Hwang, Y. (2024). The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling. bioRxiv, 2024-08.
7. Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023, November). Bend: Benchmarking dna language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations.