Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyenas new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level, an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 17 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on all 8 datasets on average by +9 accuracy points.
Paper
Similar papers
Peer review
Summary
In this work, the authors propose a new genomic foundational model that can be pretrained on the human reference genome. This model, called HyenaDNA, is built upon the previous model Hyena. The advantage of this model is that it can train on ultra-long sequences (up to 450k) at single nucleotide resolution with significantly fewer parameters. The authors introduce a warm-up technique to stabilize the training procedure, and they also employ a soft prompting method for downstream tasks. The authors show the efficacy of their framework through various experimental settings.
Strengths
1. This work tries to solve an important problem: developing new foundational models for DNA sequences. 2. Previous genomic foundational models are usually only able to be trained on relatively short sequences (<10kb). It is hard for them to deal with ultra-long sequences, which is necessary to capture long-range dependencies. This work provides a strategy to train the model on these long sequences. 3. The experimental results shown in the manuscript seem promising. 4. Overall, the manuscript is well written.
Weaknesses
1. It is a bit difficult to understand 3.1 especially for people who are not familiar with the previous Hyena manuscript. $x_1$, $x_2$, $v$, $W_{x_{1}}$, $W_{x_{2}}$, $W_{v}$ are not defined in the main text. Line 159, not sure “input time” refers to. Line 162, not sure “projections” refer to. Some minor points: 1. Section 2.1, no definition of D, is it equal to 4 representing {A,T,C,G}? 2. Line 320, A.5 should be A.4 3. Line 350, A.4 should be A.6 4. Appendix line 613, Fig. 4.2 should be Fig 4.1. 5. Line 206 equation, equation number missing
Questions
1. Line 206, it is not so clear what the embedding step refers to. Also, perhaps to double check the correctness of $x\in\mathbb{R}^{L\times(T+N)}$ 2. It seems that for different experiments, the authors train different HyenaDNA (different hyper-parameters, or using input sequences with different lengths as the pretraining data). Is there any specific reason that the authors use different models instead of e.g. using the biggest model that can handle up to 450k context, for all the experiments? 3. Table A.3, for different tasks the authors use different hyper-parameters, how these hyper-parameters be chosen? 4. The authors indicate in the main text their framework is more computationally efficient than Nucleotide Transformer since it has a significantly smaller number of parameters. However, in the appendix the authors mention that for each new task the entire model needs to be fine-tuned. How efficient this fine-tuning is compared to the fine-tuning procedure used in Nucleotide Transformer? How many computer resources are required for fine-tuning HyenaDNA? 5. The authors compare HyenaDNA with Nucleotide Transformer in 17 datasets. However, in the paper of Nucleotide Transformer there are 18 datasets in total. Why one dataset missing?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
The authors have pointed out some limitations of their work in the conclusion section. I don’t think there is any significant negative societal impact of this work.
Summary
This paper presents HyenaDNA, an advanced genomic foundation model leveraging the Hyena language model's capabilities, which are based on implicit convolutions. The authors highlight the limitations of prior Transformer-based genomic models, which have been constrained by token lengths and therefore impeded accurate modeling of long-range genetic interactions. These models also relied on tokenizers or fixed k-mers, which resulted in loss of single nucleotide resolution and crucial genetic variations. HyenaDNA addresses these issues. The model's unique abilities, including its sub-quadratic scaling in sequence length, usage of single nucleotide tokens, and full global context at each layer, mark a significant advancement in genomics.
Strengths
Overall, this is an excellent paper! I appreciate the application of the Hyena model to DNA sequences. The utilization of advanced sequence modeling to tackle significant challenges in biology is commendable. The concept of soft prompting for long-context models is exceptionally innovative and interesting!
Weaknesses
- To be frank, I believe the evaluation of short sequences may not be crucial for this paper, considering its primary focus on long-range modeling. I would suggest including a comparison with the Enformer model for a more comprehensive perspective. More benchmarks on the long-context task are necessary. - Moreover, additional technical details should be incorporated either in the main text or the supplementary materials to facilitate a thorough understanding of the methodology used. See more detail in the questions:
Questions
I would raise my rating if the authors resolve my concerns. - On line 132 , it would be beneficial to showcase $ T_{i, j} $. - Could you elaborate a bit more on Figure 3.1, specifically, what the arrows represent? - Could you provide detailed information about the baseline CNN model used in the GenomicsBenchmark? - Are there any results from training the model from scratch? The connection between the pre-training goal and the downstream task doesn't seem very apparent. - How well did the CNN model perform in predicting on the Nucleotide Transformer benchmark? - On lines 259 - 260, how would training from scratch for each task fare? If the advantages of pre-training are minimal, using $3200 x$ less pretraining data doesn't present a significant benefit. - Could you detail the fine-tuning process in section 4.2? - For Figure 4.1, it would be helpful to incorporate the data from table 4.1 to display the "upper bound" performance. - It would be advantageous to always include a training-from-scratch CNN or dilated CNN on all tasks. - The species classification result is quite impressive! Would it be possible to apply any Explainable $\mathrm{Al}$ (XAI) methods to discern the principles that the model has learned? - It would be beneficial to update the supplement to include more details about the Long Convolution. I had to refer to a previous paper to understand this concept.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
NA
Summary
This paper introduces HyenaDNA, a genome foundation model based on the Hyena architecture that replaces attention layers with implicit convolutions. Though being 2500 times smaller, it achieves better performance than the state-of-the-art model.
Strengths
- The paper is clearly written. - The proposed method achieves very strong performance with very few parameters and is able to process ultra-long dna sequences. - This paper investigates in-context learning and tunable prompting to the area of dna language model and show great performance improvements.
Weaknesses
- The comparison over baselines is unfair. - Since Hyena is fundamentally different from the Transformer-based architecture, simply comparing the number of parameters does not accurate reflects the model efficiency. I think it would be more accurate the compare the wall-clock time and GPU time used in both pre-training and downstream evaluation. Actually, according to the figure A.2, the model is slower than its attention-based variants when inputs are shorted than $10^4$, which is very common in genome analysis tasks. Therefore, it is very interested to see how efficient the model is on each downstream task compared to DNABERT and Nucleotide Transformers. - All the previous models use standard or parameter-efficient fine-tuning for downstream evaluation, I think it is important to also fine-tune HyenaDNA on each downstream tasks to show how good it is as a *backbone* model and for fair comparison with other models. Also, it is interesting to see how the baseline models performs with the tunable prompting. - This paper introduces a very strong model with a series of fancy techniques, however, as an academical publication, it fails to provide enough insights and explanations to the community about the superiority of its performance. - According to the Hyena paper, the Hyena architecture achieves a similar level of performance (slightly better) as the transformer-based model (e.g., GPT) with the same number of parameters in general language modeling. Therefore, the fact that it dominants transformer-based models in the dna language modeling indicates the existence of **fundamental errors** of all the existing works. If this is true, what are the errors? I can think of a few possibilities: - Masked language modeling vs casual language modeling - k-mer tokenization vs character-level tokenization - attention mechanism fails to capture the semantics of DNA. (however, the AttnDNA performs very well) - To be honest, I don't think any of the above can lead to such significant improvements. Thus, throughout ablation study and experiments under the same setting is necessary.
Questions
- How does AttnDNA perform on the datasets used by Nucleotide Transformers? - Can you implement Nucleotide Transformer and DNABERT on the GenomicBank datasets for comparison?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
In sum, the work is technically sound and empirically strong. However, based on the above thoughts, I tend to reject it for now. If the concerns are clearly solved, I would be happy to recommend a strong acceptance.
Summary
The authors train the Hyena operator model on the human genome and adapt it to downstream tasks in computational biology.
Strengths
Lots of clever things about this work that I really like: Very good use case of Hyena model with very long-range dependencies Curriculum learning is very clever and makes sense. Figure 3.2 is great to show this I think Table 4.4 with the use of XGBoost is very simple but effective in getting a statistic about the utility of an embedding. With XGBoost, you are going to get a pretty reasonable and comparable every time you use it. The model capacity shown by perplexity and sequence length is good. I really really like that it is able to work at single base pair resolution. I’m impressed and happy that section 4.4.3 uses held-out chromosomes for training and testing (rather than random sampling). Since you are working with multiple species, are there still some aspect of data leakage in train/test sets because of synteny? I think this is impressive being only trained on the human genome. I’m excited to see this model trained on all genomes!
Weaknesses
I think the GenomicsBenchmark claims about SotA could be improved. Definitions of SotA should be based on architecture, not the datasets. Since the dataset is newer than previously benchmarked models, I would recommend the authors evaluate the GenomicsBenchmark across other SotA models or architectures (like DNABERT and Nucleotide Transformer, which are mentioned in this paper). Alternatively the SotA claim is weak at best, and misleading at worst. I think section 4.4.3 could have a very simple baseline of a kmer count model. Can a simple kmer based model do just as well, or poorly? I’m sure it is very fast.
Questions
Figure 3.2 - Do longer sequences simply take longer to do a forward and backward pass? You show wall time. Are the number of steps relatively closer? How much is the result of your model’s result is its architecture versus using tunable prompting? What would happen if you used tunable prompting for the Nucleotide Transformer? How are the F1 and MCC scores for Table 4.2 generated? Which values are you comparing? How was the finetuning in section 4.4.1 done? Is it not soft-prompting?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
4 excellent
Limitations
Generally, I find the term “Foundation Model” to be ambiguous and unnecessarily buzzwordy. HyenaDNA is a large, pretrained model. Even though the authors have pretrained the model, it has to be finetuned and quite heavily engineered differently for each subtask. But, I guess this is what the field is wanting to say. While I do like that the important aspects of the paper are emphasized, some of the color, boldness, different font, and italicization can be toned down a bit. For example: “Therefore, having both long-range context and single nucleotide resolution simultaneously” doesn’t need to be blue and bolded. Also, what is the author’s intent in using blue text versus bold text?
Summary
This manuscript applies Hyena, a neural operator based on implicitly parametrized long convolutions and data-controlled gating, to the domain of DNA modelling. The subquadratic complexity of Hyena enables scaling to context lengths of up to 450,000 at single nucleotide resolution. This represents a significant improvement over previous methods based on dense attention, increasing pre-training sequence length by more than two orders of magnitude. Pre-training is done using a sequence length warm-up scheme, such that sequences grow longer as training progresses. The resulting model is benchmarked on a variety of downstream tasks using different adaptation methods.
Strengths
1. (**Originality**) HyenaDNA is the first work to apply Hyena, a novel breed of implicitly parametrized long convolutions, to DNA modeling. 2. (**Originality**) The authors experiment with two adaptation strategies commonly used in NLP to apply the pre-trained model to novel tasks. 3. (**Quality**) The method is evaluated on a large number of genomic datasets introduced as part of related work. This firmly situates HyenaDNA within the existing literature. 4. (**Significance**) The proposed model outperforms state-of-the-art attention-based methods on the majority of presented downstream tasks with considerably fewer parameters and scales to extremely large context sizes of 450,000 tokens at single nucleotide resolution. 5. (**Clarity**) Overall, the manuscript is written clearly, and the arguments are easy to follow. 6. (**Clarity**) Architecture and hyperparameter details are clearly presented. 7. (**Clarity**) Including background task descriptions for the Nucleotide Transformer, the Biotype Classification, and the Chromatin Profile Prediction benchmarks are helpful for understanding the evaluation protocol.
Weaknesses
1. (**Originality**) Beyond the application of Hyena to the DNA domain, the manuscript offers few new technical insights that could be of interest to the machine learning community. Sequence length warm-up is an established technique to stabilize training instabilities, and soft prompting, as well as few-shot fine-tuning, are commonly used adaptation strategies in the natural language processing community. 2. (**Quality**) Results are presented without any measure of deviation over multiple repetitions. This is true even for smaller scale fine-tuning experiments. 3. (**Quality**) For the GenomicsBenchmark, the authors should report more details about the CNN baseline (parameter count, number of layers, etc.), as well as training details and hyperparameters for AttnDNA. If the same hyperparameters are used for both HyenaDNA and AttnDNA, please state this explicitly. 4. (**Clarity**) Few-shot adaptation includes a tuning phase, making this adaptation method more similar to few-shot fine-tuning than in-context learning. Relevant literature should be cited. 5. (**Clarity**) The results reported in Table 4.1 for the Nucleotide Transformer do not seem to match with those reported in the original paper in Supplementary Table 6. Please explain this deviation.
Questions
- Why is HyenaDNA pre-trained autoregressively rather than non-autoregressively, as was done in Nucleotide Transformers? Both methods would be feasible since downstream adaptation of HyenaDNA always includes a fine-tuning phase. - Considering the limited vocabulary of just 4 nucleotides, it's somewhat unexpected that the reported perplexity score for a high-performing long-context model like HyenaDNA would be this high. Could you provide some insight on this? To underscore the significance of pre-training further, it would be valuable if you could supplement the evaluation of pre-trained embeddings using probing classifiers with an additional experiment where HyenaDNA is trained directly from scratch on the downstream datasets. - The manuscript mentions that both a long context and single nucleotide resolution are crucial. However, K-mer tokenization could help reduce the temporal dimension, and previous work has shown that a higher k leads to improved results (Ji et al., 2021). Therefore, it would be beneficial to include an experiment that compares HyenaDNA models pre-trained with different tokenization schemes. ### References - Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021). DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics, 37(15), 2112-2120. ### Minor comments - Possible Typo in Table 4.1 row 2 (Coding vs. Intergenomic): the improvement over baseline performance (+3.5) does not match with the reported base performance. Both Base and HyenaDNA are reported to have the same accuracy. - Typo on line 543 : sequence-level instead of sequence-leel - Typo on line 544: that instead of taht
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors address the limitations of their method and experiments in the manuscript. Possible malicious uses of HyenaDNA are not addressed.
Tables in markdown format (same as PDF)
### Rebuttal 2/3 ***We provide the tables in markdown format here for convenience as well. These are the same as those in the PDF.*** *** ### GenomicBenchmarks Ablations | Dataset | Baseline CNN (scratch) | AttnDNA (scratch) | AttnDNA (pretrain) | HyenaDNA (scratch) | HyenaDNA (pretrain) | HyenaDNA k-mer (scratch) | HyenaDNA Bidirectional (scratch) | DNABERT (pretrain) | | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | | Mouse Enhancers | 69 | 79.3 | 79.3 | 84.7 | **85.1** | 81.8 | 80.6 | 66.9 | | Coding vs Intergenomic | 87.6 | 89.3 | 91.2 | 90.9 | 91.3 | 86.7 | 90.3 | **92.5** | | Human vs Worm | 93.0 | 94.8 | **96.6** | 96.4 | **96.6** | 92.9| 95.9 | 96.5 | | Human Enhancer Cohn | 69.5 | 67.7 | 72.9 | 72.9 | **74.2**| 69.8 | 72.1 | 74.0 | | Human Enhancer Ensembl | 68.9 | 79.0 | 88.3 | 85.7| **89.2** | 88.0 | 85.9 | 85.7 | | Human Regulatory | 93.3 | 90.2 | 91.8| 90.4 | **93.8** | 90.2 | 89.1 | 88.1 | | Human Nontata Promoters | 84.6 | 85.2 | 90.1 | 93.3 | **96.6** | 83.5 | 88.5 | 85.6 | | Human OCR Ensembl | 68.9 | 68.3 | 79.9 | 78.8 | **80.9** | 70.2 | 75.3 | 75.1 | **Table R1. GenomicBenchmarks ablation results comparing HyenaDNA design components.** *** ### Nucleotide Transformer Ablation | Dataset | NT 2.5B | AttnDNA (pretrain) | HyenaDNA (pretrain) | HyenaDNA (scratch) | |------|------|------|------|------| | Enhancer | 58.0 | 59.3 | **59.7** | 58.6 | | Enhancer types | 47.4 | 51.9 | **56.7** | 48.4 | | H3 | 81.4 | 75.8 | **82.3** | 79.9 | | H3K4me1 | 55.9 | 38.7 | **56.7** | 43.4 | | H3K4me2 | 32.6 | 28.8 | **51.8** | 34.5 | | H3K4me3 | 42.1 | 28.3 | **61.2** | 40.2 | | H3K9ac | 57.5 | 49.2 | **63.6** | 52.6 | | H3K14ac | 55.0 | 41.6 | **65.6** | 48.0 | | H3K36me3 | 63.2 | 47.8 | **65.7** | 53.4 | | H3K79me3 | 64.2 | 58.9 | **71.4** | 59.7 | | H4 | **82.2** | 77.7 | 79.8 | 79.1 | | H4ac | 50.1 | 36.4 | **63.3** | 43.5 | | Promoter all | **97.4** | 96.3 | 96.5 | 96.1 | | Promoter non-tata | **97.7** | 96.6 | 96.7 | 96.5 | | Promoter tata | 96.4 | **96.6** | 96.4 | 96.1 | | Splice sites acceptor | **99.0** | 97.6 | 96.6 | 96.6 | | Splice sites donor | **98.4** | 98.1 | 96.8 | 96.5 | | Splice sites all | **98.3** | 98.0 | 97.9 | 97.3 | **Table R2. Nucleotide Transformer dataset results. Enhancers and histone mark tasks (names starting with “H”) use the MCC metric. Promoters and splice tasks use the F1 metric, except the “Splice sites all” which uses accuracy.** *** ### Species classification (scratch vs. pretrain) | Length | Scratch |Pretrained | |--------|------|------| 1k | 53.9 | 61.1 | 32k | 70.7 | 93.4 | 250k | 65.7 | 97.9 | 450k | 64.5 | 99.4 **Table R3. Species classification ablation, scratch vs. pretraining.** *** ### Pretraining GPU & Runtime Comparisons | Pretrain | DNABERT | Nucleotide Trx | HyenaDNA | HyenaDNA | | ----------- | ----------- | ----------- | ----------- |----------- | | Params | 110M | 2.5B | 436K | 1.6M | | GPUs | 8-2080 TI | 128-A100-80GB | 1-A100-40GB | 1-A100-40GB | | Wall clock | 25 days | 28 days | 80 mins | 80 mins | | GPU-hrs | 12,000 | 215,000 | 1.3 | 1.3 | **Table R4. Pretraining resources and runtime by model. The 436k HyenaDNA model was used on the GenomicBenchmarks while the 1.6M HyenaDNA model was used for the Nucleotide Transformer datasets.** *** ### Finetuning GPU & Runtime Comparison | Model | HyenaDNA | DNABERT | NT-2.5B | | ----------- | ----------- | ----------- | ----------- | | Parameters | 436 K | 110 M | 2.5 B (0.1% Trainable) | | GPUs | 1-A10 | 1-A10 | 1-A10 | | Mouse Enhancers | 2s | 32s | 9m 8s | | Human Enhancer Cohn | 14s | 9m 25s | 41m 55s | | Human Nontata Promoter | 10s | 5m 44s | 35m 36s | | Coding vs Intergenomic | 25s | 12m 32s | 1h 40m | | Human vs Worm | 25s | 12m 31s | 1h 46m | | Human Enhancer Ensembl | 1m 20s | 55m 30s | 2h 56m | | Human OCR Ensembl | 1m 32s | 1h 3m | 3h 30m | | Human Regulatory | 2m 37s | 1h 46m | 6h 39m | **Table R5. Finetuning resources and runtime for the GenomicBenchmarks by model.**
### Rebuttal 3/3 ### Continued: AttnDNA finetuning on Nucleotide Transformer datasets As suggested by reviewer **[nDFb]**, we finetune the AttnDNA model on the Nucleotide Transformer datasets, as shown in Table R2 above. We sweep a range of (different) hyperparameters for both models. Both AttnDNA and HyenaDNA are causal and use single nucleotide tokens, but AttnDNA significantly underperformed against its Hyena counterpart. This suggests the Hyena operator itself contributes significantly to the overall performance gains of HyenaDNA. This may be confounding, since as reviewer **[nDFb]** noted, Hyena performed similar to attention in language tasks in the original Hyena paper [Poli et. al, 2023]. One reason for the performance gap seen here may be the shallow Transformer size of AttnDNA (2 layers). Shallow Transformers can sometimes underform their convolutional counterparts on small datasets (eg, Vision Transformers vs CNNs on CIFAR-10) [Zhu et al., 2023], which may have more inductive bias spatially or temporally (Hyena does not require positional embeddings for the tokens). The Nucleotide Transformer model is far larger and uses a higher amount of pretraining data, and is able to perform more competitively with HyenaDNA. ### Citations Grešová, Katarína, et al. "Genomic benchmarks: a collection of datasets for genomic sequence classification." BMC Genomic Data 24.1 (2023): 25. Dalla-Torre, Hugo, et al. "The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics." bioRxiv (2023): 2023-01. Poli, Michael, et al. "Hyena hierarchy: Towards larger convolutional language models." arXiv preprint arXiv:2302.10866 (2023). Zhu, Haoran, Boyuan Chen, and Carter Yang. "Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective." arXiv preprint arXiv:2302.03751 (2023). Ji, Yanrong, et al. "DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome." Bioinformatics 37.15 (2021): 2112-2120.
Can you share more experiments details to reproduce the results?
Hey, Thank you for preparing the ablation study results! Since the results are too good to be true IMO (much better performance with 1000 times less parameters and much less data, which is extremely rare in the context of deep learning), I tried to reproduce your results on some of the `Nucleotide Transformer` datasets (the ones starting with H), but I am not able to reproduce the similar level of performance as shown here. For example, in the task of `H3K4me3`, the best test set performance I get is MCC: 0.4140 after training for 95 epochs, which is much lower than the 0.612 reported above. I also evaluated on a few randomly selected task (H3, H4, H3K4me2) but fail to reproduce either. It would be really helpful if you can share more experiments details. Or more ideally, it would be excellent if you can provide a python script to reproduce the experiments results. FYI: I download the datasets and randomly split it into train/dev/test (8:1:1). I train the model on 4 V-100-16G GPUs with a per GPU batch size of 64. And the average training time of a model contains 1 million paramaters per epoch is about 50s. I used the default hyperparamters in the appendix (lr 6e-4, bs 256, epoch 100, weight_decay 0.1, and no rc_aug). Also, as shown in the paper and based on my reproduction, this model often take about 100 epochs to converge, while other works (NT and DNABERT) generally takes 3-5 epochs, which is 20-30 times less. Simply comparing the per epoch training time as above is very unfair. Based on my experiments, in the task of `H3K4me3`, a HyenaDNA model with 1M parameters actually spends slightly more time to fine-tune than a NT with 500M parameters and a DNABERT with 100M parameters. This may differs a bit with different batch sizes and hardwares, but I don't think HyenaDNA is *more efficient* than the baselines in terms of training time and memory usage unless on ultra-long sequences.
Hi, thank you for your engagement, reviewer **[nDfb]**, and for bringing this up! We’d be happy to help you reproduce the results for the Nucleotide Transformer (NT) datasets. We’ll also be sure to provide more details on the exact finetuning procedure and settings in the manuscript for the datasets as well. ### To reproduce the HyenaDNA results on the Nucleotide Transformer datasets, you’ll need: - the correct model size, **2 layer, d_model=256, sequence_len=1k**, (not d_model=128 as the reviewer used) - the pretrained weights themselves (for this size), which previously had not yet been shared publicly - optimized hyperparameters per dataset Without having the correct model size (listed in the appendix Table A.3) and a pretrained model itself, there would be a large gap in performance, and likely what you are observing. We want to make sure that we account for any differences in code implementations that might be reimplemented online vs what we actually used. This includes accounting for other training settings that are important that may be left out in other demo implementations including: the cosine scheduler, optimizer parameter groups (ie handling hyperparameters per layer config), gradient clip, Pytorch Lightning automatic mixed precision handling, etc. As such, we have prepared a **docker image** with the following already on it: - an anonymized repo - the pretrained weights for NT finetuning - a Dockerfile (or instructions to install dependencies) - 5 sample NT datasets, preprocessed and split - a README with the exact commands (and hyperparameters as args) to launch and reproduce (or surpass) the reported results **Download repo/data instructions** You have 2 options: **1. (Easiest w/ Docker):** Pull the docker image with the data, dependencies and pretrained weights ready to go. ``` docker pull hyenadna/hyena-dna:latest ``` **Run the container** ``` # gives you an interactive shell (with the dependencies) docker run --gpus all -it -p80:3000 hyenadna/hyena-dna /bin/bash ``` You’ll end up inside the repo, and there’s a README.md on how to run the experiments (ie the exact launch commands for the 5 sample runs). **or,** **2. Manual setup:** Download this anonymous link using `gdown` to obtain the repo. This zip file has the data and pretrained weights inside the repo as well. After unzipping the repo, there’s the same README.md (in 1.) about how to reproduce the results. You’ll need to decide how you want to install the dependencies (manually or again with Docker). ``` gdown 1ufTP2zMAf1_KkxGxP15PJGNLzfreqpmd ``` ### Finetuning time There are many ways to look at efficiency, and we appreciate the reviewer providing feedback on this. Regarding finetuning time, we’ve provided HyenaDNA convergence results that aggregate the time per dataset. HyenaDNA averaged about 23 mins per dataset, ranging from 1 min to 45 mins, using a single A100 GPU. In the Nucleotide Transformer paper itself (section A.3.1), the authors note: ``` "On average, a fine-tuning run lasted 20 minutes for the 500M parameter models, and 50 minutes for the 2.5B parameter models.” ``` We compared our results to the best performing values from the **NT 2.5B** model, which used 8 A100s (ie, the same GPU type but 8 vs 1 GPU in HyenaDNA). Using this baseline, and the NT author reported description, HyenaDNA spends ~17x less GPU-hours finetuning on the same GPU type on the NT datasets. For pretraining, where most of the overall compute is spent, the gap between HyenaDNA and the NT 2.5B is even larger, as shown in the Table R4 of the common response. | Dataset | Pretrain (min) | Scratch (min) | |------|------|------| | Enhancer | 8 | 6 | | Enhancer types | 8 | 10 | | H3 | 10 | 5 | | H3K4me1 | 45 | 16 | | H3K4me2 | 40 | 50 | | H3K4me3 | 40 | 60 | | H3K9ac | 30 | 13 | | H3K14ac | 45 | 50 | | H3K36me3 | 20 | 45 | | H3K79me3 | 40 | 15 | | H4 | 1 | 5 | | H4ac | 30 | 55 | | Promoter all | 40 | 20 | | Promoter non-tata | 3 |10 | | Promoter tata | 2 | 2 | | Splice sites acceptor | 50 | 40 | | Splice sites donor | 20 | 35 | | Splice sites all | 30 | 30 | | Average | 23 | 26 | **Table R6. Time to convergence for HyenaDNA finetuning and training from scratch on the Nucleotide Transformer datasets.** Reviewer **[DWDg]** also suggested comparing the (pretrained) finetuning vs scratch training times on the NT datasets for HyenaDNA. Though they have similar training times for reaching their respective top metrics (23 vs 26 mins on average per dataset), the pretrained models generally reach the same performance metric (as the scratch models) faster, and then continue to eke out more gains, albeit more slowly. We appreciate the suggestion and will put these results in the appendix of the manuscript.
Able to reproduce the results with the provided code and model.
Thank you very much for preparing the docker image. Here I can confirm that with the provided codes and pre-trained weight, I am able to reproduce similar-level of results on H3K4me2 (44.7) and H3K14ac (64.8). Now I have no doubt with the evaluation results. Though the comparison with NT is a bit unfair (There is no validation set, so the hyper-parameters tuning and best model selection are directly based on test set), the result is still very impressive given the tiny size of the model. However, I am still confused about the great performance of the pre-trained model. Can you provide more details in the difference between the model you shared in the docker image v.s. those publicly available checkpoint? I am asking since my own experiments show that using those publicly available models and my own reproduction to replace the checkpoint you provided leads to very dramatically performance drop (e.g., 60 -> 40). If the main beneficial of the your comes from the Hyena operator, I do not understand why the this one is much more powerful than others given the fact that they have similar architecture and number of parameters. Is it because the pre-training is unstable and you are not able to consistently get a good model with different size or you apply different strategy and tricks to this specific model? Is is possible that there's some data leakage issue on this checkpoint? (I do not suspect you at all but just provide a possibility here.) I think it would be really beneficial to the community if you can discuss these in more details in your paper. If you can provide the pre-training code for us to reproduce the pre-training stage, it would be the best. I understand this needs a lot of work, but I think we are responsible to make sure everything is correct before recommending a paper with such exceptional performance.
Thank you! We very much appreciate the level of engagement from the reviewer, it’s helping us improve the manuscript and communicate the work better. ### NT comparison Indeed, the NT uses a validation set, however, there are 2 sets of results in their paper. One where they use the validation set hyperparameters, and one where they do not. In the first set of results in **Figure 2**, the validation set is used to select the hyperparameters for the test set. However, in the NT **Supplementary Table 6**, these results reflect the **best test results overall across all hyperparameters - regardless of validation set hyperparameters**. We compare with the best NT scores overall, from Supplementary Table 6, which are the highest test scores regardless of validation hyperparameters, and a more fair comparison with our results. ### Pretrained model difference From what the reviewer described about using an outside checkpoint (outside the Docker image we provided), the main reason is the ***different model size***, ie, they do not have the same number of parameters. Specifically, the reviewer appeared to use a model with **2 layers, and d_model=128, which has 436K parameters**. The model (in the Docker) that we actually finetuned on the NT datasets is **4x** the size, using **2 layers, d_model=256, and has 1.6M parameters**. This model size is reported in section A.2.2. in the appendix. We see that there may be confusion on model size for the NT datasets, and so we will be sure to highlight this more clearly in the manuscript. Note: the model size scales quadratically in model dimension / width (ie, 2x the d_model, means 4x the number of parameters). Also, as mentioned in the previous common response on Aug. 13, this model was previously not released publicly, and likely different than what the reviewer used. ### Pretraining code Thank you for the suggestion. We agree with the reviewer that providing more details about the pretraining and releasing the code publicly will help the community. Fortunately, the Docker image we shared does already have the exact pretraining code we used. This code will be part of our public release plan, as well as the hyperparameters. We’ve also provided instructions for the reviewer if they’d like to reproduce the pretrained weights (for the NT finetuning) and see for themselves. **(Using the previous Docker image)** **1. Download** the data via `gdown` into the `/wdr/data/hg38` directory of the Docker image repo. (For convenience, we’ve uploaded this anonymously for the reviewer) ``` # from the repo root, download and unzip mkdir /wdr/data/hg38 && cd /wdr/data/hg38 gdown 1MDYoSU7mb8ReS7trL36x03b5RDo9KJhc && gdown 1U3U8UZWLy37ycb0DublSbcsyZby1Rvt4 gunzip hg38.ml.fa.gz && cd /wdr ``` **2. Launch** (from `/wdr`) ``` python train.py wandb=null experiment=hg38/hg38_hyena model.n_layer=2 model.d_model=256 dataset.bed_file=/wdr/data/hg38/human-sequences.bed dataset.fasta_file=/wdr/data/hg38/hg38.ml.fa model.fused_dropout_add_ln=True dataset.batch_size=256 dataset.max_length=1024 dataset.max_length_val=1024 trainer.precision=bf16 ``` This should reproduce the same pretrained weights we used for the NT. It reaches a test loss ~1.16 in about 80 mins on a single A100. ## Explicitly, our plans after the review period, we will: - publicly release pretrained weights (including the exact model for the NT finetuning) - release all code for pretraining and finetuning (our full implementation) - discuss further details for pretraining and finetuning in order to reproduce results in the appendix - provide the Dockerfile and image to make it even easier for the community to test our code ### Differences in code In accordance with the double blind review process - we can only speculate how our code (in the Docker image) may differ from other implementations of HyenaDNA online, or your own reimplementation. To reiterate our comment on this in our previous common response on Aug 13th, differences with other **simplified demonstration implementations online** may include: - differences in model size settings - lack of cosine decay learning rate scheduler - lack of optimizer parameter groups, that allow allow for different hyperparameters per layer (eg, the Hyena layer requires weight decay to be 0) - using gradient clip (with value of 1.0) - automatic mixed precision handling in Pytorch Lighting (eg, not available on colab) Demonstration code online is best suited for understanding the core architecture components. For reaching SotA, more sophisticated training code is required to reach a model's full potential. For HyenaDNA, we'll be sure to share these tips in greater detail of the manuscript. We thank the reviewer for their valuable feedback, and as always, we’re happy to provide any further clarifications the reviewer might have.
Able to reproduce the results by pre-trained a model from scratch.
Thanks for providing the instruction for model pre-training! Here I can confirm that will the provided code, I can pre-train a model from scratch and achieve similar-level performance on the selected downstream tasks. Though we still do not fully understand where how this model is able to be so effective given its size, I think the empirical results suggest that HyenaDNA is a very solid milestone work in the area of genome foundation model. I have raised my score from 4 to 7.
We're glad the reviewer was able to reproduce the pretrained model using thing the provided code, and reach similar downstream performance. We very much thank the reviewer for their lively discussion and sharp questioning, and appreciate the upgraded score. We're super optimistic about this line of work, and will certainly continue driving its applications. This includes more empirical support as well as diving deeper into understanding the mechanisms that enable its effectiveness. Let us know if there are any other concerns you might have, and thanks again!
Continuation of rebuttal response
### Continuation of question responses **4. K-mer tokenization:** We thank the reviewer for this suggestion. In the ablation presented in the common response in Table R1, we observed consistent (and significant) degradation in performance on the GenomicBenchmarks when using 6-mer tokenization on HyenaDNA and training from scratch, from 2 to 10 accuracy points. **Typos** We thank the reviewer noting the typos, and we will make the necessary changes. **Limitations** We will address the potential for malicious uses of HyenaDNA in the camera-ready. We believe it is an important conversation to have for any potentially powerful and widespread technology. ### Citations Li, Conglong, Minjia Zhang, and Yuxiong He. "The stability-efficiency dilemma: Investigating sequence length warmup for training GPT models." Advances in Neural Information Processing Systems 35 (2022): 26736-26750. Press, Ofir, Noah A. Smith, and Mike Lewis. "Shortformer: Better language modeling using shorter inputs." arXiv preprint arXiv:2012.15832 (2020). Benegas, G., S. S. Batra, and Y. S. Song. "DNA language models are powerful zero-shot predictors of non-coding variant effects." (2022). Rajarajeswari, Pothuraju, and Allam Apparao. "DNABIT compress–genome compression algorithm." Bioinformation 5.8 (2011): 350. Zhang, Zhe, et al. "Analyzing effects of naturally occurring missense mutations." Computational and mathematical methods in medicine 2012 (2012). Ohno, Susumu. "So much" junk" DNA in our genome. In" Evolution of Genetic Systems"." Brookhaven Symposium in Biology. Vol. 23. 1972. Tay, Yi, et al. "Charformer: Fast character transformers via gradient-based subword tokenization." arXiv preprint arXiv:2106.12672 (2021). Kalchbrenner, Nal, et al. "Neural machine translation in linear time." arXiv preprint arXiv:1610.10099 (2016). Yu, Lili, et al. "Megabyte: Predicting million-byte sequences with multiscale transformers." arXiv preprint arXiv:2305.07185 (2023).
We thank the reviewer for their time and their comments! We wanted to share a friendly reminder that the discussion period is about to end tomorrow morning at 1PM EDT. We hope the reviewer is able to take into account our response, including additional ablation experiments and clarifications provided, into the final evaluation of the work. If there is no update, we certainly respect the reviewer’s decision. Thank you!
First, I'd like to thank the authors for their comprehensive response and the additional ablations. A context of 1 million tokens is indeed impressive and offers significant potential for genomic applications. However, as other reviewers have noted, the degree to which such an extended context is exploited needs to be explored further on long-range benchmarks. Given the efficiency of the proposed method, I am confident that we will see more in-depth analyses in the future. Thank you for addressing the discrepancies between your results and those reported in the Nucleotide Transformer paper. It remains surprising that a compact model like HyenaDNA can consistently outperform billion-parameter networks. This is counterintuitive to the general trends observed across the deep learning literature. Additionally, I appreciate the authors providing access to model weights and code, ensuring successful reproduction of the stated results. Exploring the inductive biases of Hyena or identifying the weaknesses of existing methods will definitely be interesting. Similar to the small size, using character-level tokenization contradicts findings from the natural language processing community and DNABERT. However, as your newly added results suggest, this appears to be a crucial design choice worth considering in genomic applications. Regarding DNA perplexity, I appreciate the detailed discussion and the added references. Reports of similar perplexities in the compression and genomic sequence modeling literature, combined with your experiments comparing pre-trained vs. from-scratch Hyenas, bolster my confidence that pre-training is essential. This reinforces the foundation model argument presented in the paper. Lastly, I'm grateful to the authors for addressing the dual-use nature of this work in the camera-ready version. Your thorough response and robust results on additional ablations have addressed many of my initial concerns. Therefore, I have decided to increase my original score from 6 to 7.
***See comments 3/3 of common response for the analysis of AttnDNA finetuning on the Nucleotide Transformer datasets.***
Thank you for your response
I would like to thank the authors for their response. I have one follow-up question regarding the computational efficiency of fine-tuning. The authors mention in their comments that "the Nucleotide Transformer uses a parameter-efficient finetuning: only 0.5% parameters are updated.". However, the original paper says that they only need 0.1% parameters for fine-tuning. And if I understand correctly, fine-tuning HyenaDNA requires all the parameters. Therefore I am a little bit surprised that in table R5, HyenaDNA can be much faster than NT-2.5B. In addition, the number of fine-tuned parameters of NT-2.5B should be significantly smaller than the number of parameters of DNABERT. However, as shown in Table R5, DNABERT is still significantly faster than NT-2.5B. Can the authors do some more elaborations on that?
Thank you for the comments, and for catching the mistake. It is indeed 0.1% parameters that are updated for the Nucleotide Transformer (NT) model. We will update that number for the NT in Table R5 of the common response. Yes, it is also correct that HyenaDNA updates all its parameters during finetuning. **In regards to how HyenaDNA is much faster (in overall GPU-hours) than the NT-2.5B even though it only updates 0.1% of its parameters (requiring forward and backward passes):** - the NT model still needs to compute a very large forward pass on all its parameters, which consequently is still quite time consuming. As a rough benchmark, Li et al (2020) showed that a neural network (including BERT, like the NT model) can have a backward pass that is 3x longer than a forward pass (in latency). So although the NT model saves a lot of compute time by freezing most of its parameters (saving backward pass), its sheer size still requires a lot of compute (forward pass) compared to HyenaDNA (as well as DNABERT). - Another reason for the large difference between the NT model in Table R5 is that HyenaDNA is pretrained on far less data while achieving competitive (or better) results than the NT model. We provided Table R5 to compare total GPU-hours (at the request of a reviewer) to provide a different perspective on efficiency, which can be viewed in multiple dimensions. - Another efficiency dimension, in the original manuscript (Figure 4.1), is to control for the same model and data size, and compare the runtime for a small, 2 layer model with attention vs HyenaDNA. In that case, the main efficiency benefit of HyenaDNA is primarily at longer sequences, which grow near linearly ($N log N$) vs $N^2$ with attention. Thanks very much for the engaging and thoughtful questions. ### Citation Li, Shen, et al. "Pytorch distributed: Experiences on accelerating data parallel training." arXiv preprint arXiv:2006.15704 (2020).
We thank the reviewer for their time and the positive comments about the importance of the work! We wanted to share a friendly reminder that the discussion period is about to end tomorrow morning at 1PM EDT. We hope the reviewer is able to take into account our response, including additional ablation experiments and clarifications provided, into the final evaluation of the work. If there is no update, we certainly respect the reviewer’s decision. Thank you!
I would like to thank the authors for their detailed responses. After reading the authors' as well as other reviewers' comments, I decide to increase my score by 1.
concern about the short sequence benchmark
- The key concern is that I don't see much reason why the short sequence prediction is also improved, which is probably just because the CNN baseline is not so good (also, reviewer nDFb cannot reproduce this result). And based on the DeepSEA benchmark, I would say the difference is relatively small. The motivation should be the long context task from the design motivation view. - For training from scratch, do you use the same hyperparameters to train or fine-tune the hyperparameters for training from scratch, such as the possibility that training from scratch may need longer training, a smaller learning rate? Also in the cross-specials benchmark, do you also use the warm-up? - Could you please provide a time comparison between training from scratch and the pretrain+fine-tune setting? This could be a great comparison to know if pre-training is necessary.
Thanks for your engagement, we appreciate your feedback and interesting questions. **1. Short range performance:** We agree with the reviewer, the focus should be on the long range design and capabilities of HyenaDNA. The short range benchmarks are primarily meant as a comparison for common benchmarks, and to show HyenaDNA is at least as good as existing models. They are also readily available, and use low compute resources to enable most in academia to test it out. That being said, the short range models do allow us to test out our other design choices on DNA sequences. From our ablations in the common response Table R2, it appears other design choices also improve the short range tasks, including the single nucleotide tokenization, which is useful for any range. We also agree that the CNN is a weaker baseline. Another reviewer did suggest using a stronger baseline on the GenomicBenchmarks with DNABERT, which is now provided in Table R1, and which achieves SotA on 1 of 8 GenomicBenchmarks datasets. The Nucleotide Transformer is arguably a much stronger baseline model (previous SotA) and evaluated on harder tasks across 18 datasets, for which HyenaDNA performs competitively as well. For the DeepSEA benchmark, HyenaDNA’s main benefit is the much smaller model size while performing similarly. Notably, this dataset is nearly saturated, and so reducing the error rate becomes exponentially more difficult compared, for example, the NT datasets. We look forward to continue applying HyenaDNA to longer range benchmarks. **2. Training from scratch:** For training from scratch, we sweep hyperparameters as well, though sometimes the best hyperparameters are the same for both scratch and finetuning. It did not appear that the best learning rates were significantly changed because of the finetuning vs scratch. What we did notice was that the training becomes much more stable with the pretrained models, i.e. the loss curves are smoother and less “jumpy”. This was especially true for the long range species classification task, where the ultralong sequences cause severe instability. **2.a Species warmup:** Indeed, we use sequence length warmup on species classification for the ultralong-range (250k+). **2.b Reproducing results:** Please see the common response **[dated Aug 13]** to reviewer **[nDfb]**, in which we provide a Dockerfile that contains the exact settings and launch commands to reproduce 5 sample datasets from the Nucleotide Transformer. It appears the reviewer was missing the correct code, hyperparameters, and pretrained weights. We're confident the correct code (found in the Docker image) will reproduce the reported scores. **3. Pretrain vs scratch time:** Thank you for the suggestion. We have now added this information in Table R6 of the common response - comparing convergence times between scratch and pretrained finetuning on the Nucleotide Transformer datasets. Though they have similar training times for their respective top metrics (23 vs 26 mins on average per dataset), the pretrained models generally reach the same performance metric (as the scratch models) faster, and then continue to eke out more gains, albeit more slowly. We appreciate the suggestions and will put these results in the appendix of the manuscript. Thanks very much for the great follow-up questions.
Reply
Thanks a lot for the response! But considering the lack of other long sequence benchmarks, I would not change the score (especially since it's already weak Accept)
We respect the reviewer's decision. We thank the reviewer for their time and the engaging discussion. We appreciate the suggestions on clarifying the methods, suggesting stronger baselines on GenomicBenchmark, and the suggested ablations on pretraining vs. scratch, and autoregressive vs bidirectional design choices. They've helped us identify and quantify how these designs led to improved results over previous methods.
Update the score
After considering more results from the training from scratch, it has resolved most of my concerns now, except for the lack of a long-sequence benchmark. So, as promised in my review, I am raising my score to 7.
Thank you! We definitely appreciate the reviewer updating their score.
Follow up
Thank you for your comments and continued discussion. I will keep my score the same, so long as the claims about the GenomicsBenchmark in the abstract and text are updated.
We appreciate your time! We will update the GenomicBenchmarks claim as you have suggested. Thank you.
Decision
Accept (spotlight)