Response to Reviewer 4xL4
Thank you for your comments and feedback!
**Q1: ”Its motivation is based on the signal differences between EEG and fMRI, concluding that EEG's performance limitations are due to constraints in decoding and reconstruction frameworks. ”**
Although fMRI plays an important role in neuroimaging research, fMRI has low temporal resolution, bulky and non-portable equipment, high cost, and is non-invasive but limited by the magnetic field environment. So fMRI-based work is almost impossible to apply in practice, and this have hindered the development of the brain-computer interface (BCI). This motivates us to propose a zero-shot visual decoding and reconstruction framework that can be proven effective on Image-EEG datasets. Our manuscript provides an empirical guidance on practical BCI applications. We hope that the BCI and neuroscience communities pay more attention to the implementation of similar technologies and real data rather than overfitting only on existing fMRI datasets.
**Q2: “However, the proposed EEG encoder merely adds a Channel Attention layer compared to NICE [1], with no detailed explanation provided. Additionally, the loss function in Section 2.4 is taken directly from [2], demonstrating a lack of originality. Furthermore, the visual reconstruction framework shows no substantial difference from existing fMRI methods [3,4], as it also utilizes pre-trained stable diffusion models and their variants, with minor differences, such as the incorporation of Sdedit [5], being tricks to enhance generation quality. Thus, the manuscript fails to substantiate its claims and contributions convincingly.”**
Neural network model structures, including image reconstruction strategies, are often inductive biased. On the one hand, researchers in the BCI community have been using spatial-temporal convolutional models to process EEG signals since EEGNet. On the other hand, the open source of StableDiffusion allows us to leverage models like CLIP that were trained with massive datasets as a teacher to guide the training of our brain models where we have a relative scarcity of data.
Although it utilizes existing machine learning techniques , we demonstrate for the first time that EEG-based zero-shot visual decoding and reconstruction can be competitive with fMRI. Previous published work either focused on the EEG image decoding, or reconstructed images of known categories on small-scale and controversial datasets; or focused on the fMRI based image decoding and reconstruction, to achieve advanced performance on the fMRI datasets.
Our work has introduced joint subject training for cross-subject evaluation, which is expected to solve the problem of decreased decoding performance due to subject differences when the amount of training data is sufficient. To the best of our knowledge, this is the first work to simultaneously achieve state-of-the-art performance on downstream zero-shot retrieval, classification, and reconstruction tasks on a dataset of the size of THINGS-EEG.
**Q3: “The processing of visual information in the brain involves multiple stages and typically takes around 100-150 milliseconds to reach higher visual areas where complex processing occurs. EEG signals at 50 milliseconds are likely still within the retina and optic nerve stages. Thus, the generation of images from 50ms EEG signals, as shown in Figure 7, contradicts established neuroscience principles. This suggests that the visual stimulus reconstruction framework heavily relies on the image generation model, which may have limited significance for the field of neuroscience.”**
We have updated codes in the anonymous code link (https://anonymous.4open.science/r/Visual_Reconstruction-AC56). And we provided examples of reconstructing images with different random seeds for growing windows across time scales (README.md). From the two example stimulus images provided, the reconstructed images show uncertainty between 0 to 50 ms and 0 to 250 ms because the stimulus has just arrived at the primary visual cortex and it takes time for the brain to react and be captured, the reconstructed image during this period shows uncertainty (probably due to noise unrelated to the stimulus). Here, since the prior of natural images is added to the two-stage framework, high quality images can be reconstructed even from 0 to 50 ms and 0 to 250 ms. Then, with the accumulation of visual stimulus information, the semantics of the reconstructed images gradually become clear. After 500 ms, the images decoded by EEG tend to be stable, which means that there may be no new information added.
Combined with Figure 7, the random seed image reconstruction example figure linked to the anonymous code we provided, and Figure 29 in Appendix H.1, we can see that these results just confirm the previous research and are consistent with the priors of neuroscience.