Self-supervised learning for DNA sequences with circular dilated convolutional networks

Due to their intrinsic properties, DNA molecules commonly exhibit long-range interactions along a linear sequence representation. Taking this information into account when modeling DNA sequences is therefore important for obtaining more accurate sequence-based inference. Many deep learning methods have recently been developed for this purpose, but they still suffer from two major issues. First, the existing methods can only handle short DNA fragments, thereby losing longerrange interactions. Second, the current methods require massive supervised labeling while missing most order information within the sequences. Consequently, there is a need to develop an efficient deep neural network modeling framework to extract wide contextual information for more accurate sequence-based inference tasks. Our new framework, named Revolution, takes full DNA sequences as input, without any condensation, and can give accurate predictions for DNA sequences up to 10kbp. In variant effect prediction, our method increases the Area Under the Receiver Operating Characteristics (AUROC) by 19.61% on 49 human tissues on average. Revolution is also demonstrated to work on the plant sequences by improving 2.36% AUROC on average for predicting open chromatin regions (OCRs). The data, models, and code can be freely accessed at https://github.com/wiedersehne/Revolution-DNAPretraining.