Brain Image Reconstruction with Retrieval-Augmented Diffusion

Reconstructing visual images from brain signals is a rapidly evolving research with promising applications in brain-computer interfaces, cognitive neuroscience, and assistive technologies. While visual reconstruction based on functional Magnetic Resonance Imaging (fMRI) has previously achieved notable success, this paper explores cost-effective brain signals, i.e., electroencephalography (EEG) and magnetoencephalography (MEG). These signals are less precise than fMRI, which presents greater challenges for reconstruction. To address this problem, we propose BReAD (Brain Image Reconstruction with Retrieval-Augmented Diffusion), a novel framework that combines EEG/MEG signals with retrieval-augmented diffusion models to improve image reconstruction quality. BReAD utilizes the semantics decoded from brain signals for (1) retrieving semantic priors from a large-scale image database and (2) serving as a conditional constraint during the diffusion process. Extensive experiments demonstrate that BReAD significantly outperforms existing approaches in both qualitative and quantitative evaluations, paving the way for more robust and practical brain-to-image reconstruction systems. Our codes are available at https://github.com/Promise-Z5Q2SQ/BReAD.