GraviBERT: Transformer-based inference for gravitational-wave time series

We introduce GraviBERT, a novel deep learning framework for gravitational wave inference, built on a multi-scale feature extractor with a transformer encoder and a suitable regression head. A key novelty of GraviBERT is its staged training: a BERT-style self-supervised pretraining phase to learn transferable representations, followed by supervised fine-tuning on labelled data. GraviBERT demonstrates consistent transfer learning across detector configurations and waveform models. On in-domain data, pretraining reduces the mean absolute error (MAE) by up to 31% and accelerates convergence by ∼6.6×, with mean relative precision for point estimates reaching the few-percent level and MAE in effective spin of ∼10−3 at low signal-to-noise ratio (SNR). For domain adaptation to new detector noise profiles, the pretrained model converges up to 15× faster on small target datasets and reduces estimation errors by up to ∼47%, demonstrating detector-agnostic learning. Cross-waveform approximant transfer achieves up to 44% MAE reductions and up to 15× training speedups, with R2 scores consistently exceeding 0.9 for mass parameters at SNR = 10 compared to 0.74–0.87 when training from scratch. GraviBERT works directly with noisy waveforms, and in its current form quantifies predictive uncertainty through MC dropouts. After pretraining, the regression head can be adapted to multiple downstream inference tasks in gravitational-wave astronomy.