Two Views, One Truth: Spectral and Self-Supervised Features Fusion for Robust Speech Deepfake Detection

Recent advances in synthetic speech have made audio deepfakes increasingly realistic, posing significant security risks. Existing detection methods that rely on a single modality, either raw waveform embeddings or spectral-based features, are vulnerable to non-spoof disturbances and often overfit to known forgery algorithms, resulting in poor generalization to unseen attacks. To address these shortcomings, we investigate hybrid fusion frameworks that integrate self-supervised learning (SSL)-based representations with handcrafted spectral descriptors (e.g., MFCC, LFCC, CQCC). By aligning and combining complementary information across modalities, these fusion approaches capture subtle artifacts that single-feature approaches typically overlook. We explore several fusion strategies, including simple concatenation, cross-attention, mutual cross-attention, and a learnable gating mechanism, to optimally blend SSL features with fine-grained spectral cues. We evaluate our approach on four challenging public benchmarks (LA19, DF21, ITW, ASV5) and report generalization performance. All fusion variants consistently outperform an SSL-only baseline, with the cross-attention strategy achieving the best generalization with a 38% relative reduction in equal error rate (EER). These results confirm that joint modeling of waveform and spectral views produces robust, domain-agnostic representations for audio deepfake detection.