NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models

1. About the novelty and mathematical feasibility of the NERV encoder, and concerns about statistical analysis: The NERV encoder is designed with a multi-layer attention mechanism, specifically to handle noisy and multi-channel time-series data like EEG signals. In Section 3.6, we detail its architecture, including spatial-temporal convolutions and cross-attention modules, which address the challenges of EEG’s low resolution and high noise. Our experiments show strong performance across multiple zero-shot classification tasks (e.g., in Tables 1 and 2), and we used the CAT score to evaluate semantic consistency of generated images. These results are based on multiple trials and include hyperparameter robustness tests, so they’re not just coincidental. 2. About the lack of analysis on generated images in Figure 1, and their resemblance to the originals: In Figure 1, we showcase images generated purely from EEG embeddings, which often result in abstract landscapes rather than specific objects. This reflects the nature of EEG data, which captures high-level semantic concepts rather than detailed visual features. We’ve discussed this phenomenon in Section 4.5, highlighting that the generated images demonstrate the model’s ability to grasp abstract meanings. Improving the visual details will be a focus of our future work. 3. About insufficient analysis of regularization parameters and fully connected layers in the diffusion model: The parameter choices for the diffusion model were carefully tuned through extensive experimentation. Section 3.5 explains the regularization process and the denoising steps in detail, with specific formulas (e.g., Equations 3 to 13). We’ve also added supplementary experiments in the appendix, showing how different parameter settings affect results, confirming the robustness of our approach. 4. About the novelty of our work compared to Song et al. and Li et al.: Our work differs fundamentally from Song et al. and Li et al. Song et al. focused on EEG-based image classification, not generation. Li et al. relied on visual signals to assist diffusion-based reconstruction. In contrast, our model is the first to use pure EEG signals as input for image generation. Additionally, we introduced a two-stage diffusion process and the novel CAT score, which are not addressed in prior work. These innovations establish clear distinctions between our approach and theirs. 5. About the lack of discussion on improving generated image quality: Improving image quality remains a major challenge in EEG-to-image tasks. As discussed in Section 4.5, existing methods struggle to reconstruct specific objects due to the noisy and abstract nature of EEG signals, as well as limitations in training data. We’ve identified potential future directions, such as better semantic alignment mechanisms and higher-quality EEG datasets, to enhance visual output quality.

1.While we acknowledge that high N-WAY classification results (above 10-way) are slightly lower than ATM-S, our primary goal is not solely classification. Instead, our focus lies in EEG-to-image generation, which is fundamentally different and more challenging. 2.While we draw inspiration from ATM-S, there are key distinctions in our approach: (a) Encoder Design Differences: Our proposed NERV encoder utilizes a multi-attention mechanism specifically tailored for noisy, multi-channel time-series data like EEG. It employs both spatial-temporal convolution (ST-Conv) and temporal-spatial convolution (TS-Conv) in tandem to extract complementary features. ATM-S does not explicitly employ this dual convolution strategy, which is crucial for processing EEG signals with spatiotemporal dependencies. (b) Two-Stage Diffusion Model and generated image analysis findings: We adopted a two-stage diffusion process to generate intermediate embeddings before final image synthesis. This approach balances abstract semantic alignment and high-quality image generation by pure EEG, whereas ATM-S primarily uses a dual-stage generation with the image embeddings. Our findings demonstrate that the first stage generates embeddings non-closely aligned with semantic concepts, while the second stage refines these embeddings to produce more visually diverse and photorealistic images. This stepwise process addresses the limitations of single-stage methods, particularly in abstract semantic scenarios. (c) Generated Image Analysis Findings: We observed that images generated directly from EEG embeddings tend to represent abstract or generalized visual concepts, such as landscapes or background scenes. This outcome reflects the inherent limitations of EEG data in encoding detailed, object-specific information. The two-stage approach improves upon this by progressively refining embeddings, leading to more diverse outputs and better alignment with broad semantic categories. However, generating highly detailed or object-specific images remains a challenge due to the high noise levels and low spatial resolution of EEG signals. (d) New Evaluation Metric (CAT Score): We introduced the CAT Score to assess semantic alignment in EEG-to-image generation, focusing on broader semantic categories rather than pixel-based comparisons. This is a novel contribution compared to existing metrics.

1. I acknowledge that there have been similar attempts, such as in [1]. However, our innovation lies in proposing an entirely new framework that adapts models previously used solely for EEG-image classification to image generation tasks. Additionally, we introduced a two-stage method to improve image generation quality and developed a new state-of-the-art encoder, NERV. 2. Regarding the dataset in [1], we encountered common issues with their open-source code (as seen in https://github.com/bbaaii/DreamDiffusion/issues/34). Once their team resolves these issues, we will include our reproduction results and evaluate the performance of our framework on their dataset and other benchmarks. 3. As you mentioned, EEG recordings cannot capture detailed features of the original images. In this work, we found that focusing on extracting "concepts" rather than the "fine details" of the original images may be a feasible approach. 4. Due to the lack of standard waveforms in EEG and the inherent variability in responses to the same image stimulus across subjects, combined with the noisy nature of EEG signals, quantification is highly challenging. To address this, we took the following measures in our work: (a) For training, we averaged the EEG signals of the same subject responding to the same image four times to reduce variability. (b) To demonstrate the generalization capability of the trained EEG diffusion model on unseen data, we used zero-shot EEG prompts for image generation. This means the input EEG signals used for image generation were not seen by the model during training. Despite this strict setup, our results, such as those in Figure 1, show that our framework can capture the broader semantic concepts of EEG signals. 5. The specific definition of the CAT score and how the five tags are created are detailed in Appendix A.2 (lines 790–793). 6. The mention of top-1 and top-5 in Table 1 is a typo—thank you for pointing that out! Since 2-way and 4-way classifications involve only 2 and 4 categories, respectively, all results in Table 1 represent top-1 accuracy. 7. We will address the arxiv.org references and revise the sentence in line 47 in the camera-ready version. We sincerely appreciate your valuable feedback. Thank you!

In general, the contribution of the paper is quite limited. You state in your comments that "... introduced a two-stage method to improve image generation quality ..." and "... focusing on extracting 'concepts' rather than the 'fine details' ..." - However, the paper neither provides a comprehensive analysis of image generation quality nor presents a clear methodology that effectively focuses on extracting "concepts" instead of "fine details."

1. Response to Innovation Concerns: (a)Two-Stage Diffusion Framework: Unlike prior works, our model directly generates images from pure EEG data without relying on image features as guidance, marking a significant shift in EEG-to-image generation. (b)NERV EEG Encoder: NERV is tailored for noisy, multi-channel EEG signals, using advanced attention mechanisms to extract spatial and temporal features, achieving SoTA performance in classification and generation tasks. (c)CAT Score Metric: We propose a new evaluation metric designed specifically for EEG-to-image generation, addressing limitations of traditional metrics by focusing on semantic alignment. (d)Benchmarking on ThingsEEG: We establish a benchmark for EEG-to-image generation, providing a robust foundation for future comparisons. These contributions advance the field by bridging neural activity and visual representation, moving beyond traditional classification tasks. We will emphasize these innovations more clearly in the revised manuscript. 2. As we mentioned in line 791-793, regarding the CAT Score Methodology: (a) Five Tags per Image: Each image in the ThingsEEG test dataset is labeled using ChatGPT-4o with the prompt: "Provide 5 one-word descriptions of the image, ranging from high-level to low-level." This method ensures consistency in generating semantic descriptions for evaluation. (b) No Manual Refinement: All labels for the CAT Score are directly generated by ChatGPT-4o without any manual intervention or refinement. This approach maintains objectivity and reproducibility in the evaluation process. 3. In Figures 32-40 of the Appendix (line2862-3023) , it is clearly indicated that these results were generated using the open-source code (https://github.com/dongyangli-del/EEG_Image_decode) provided by ATM-S, trained under the same conditions as other encoders. The results demonstrate that, under identical experimental conditions, the claimed image quality from the ATM-S paper could not be reproduced. From the results, it is evident that the images generated by ATM-S under the same experimental conditions are not significantly better than those produced by NERV. This suggests that NERV achieves state-of-the-art (SoTA) performance not only in classification accuracy but also in overall semantic relevance and quality of generated images, matching or even surpassing ATM-S.

4. EEG-to-image generation is still an emerging field, and due to the inherently low signal-to-noise ratio of EEG data, the generation process across all current methods remains highly unstable. This instability is reminiscent of the early days of GAN development, where outputs often lacked consistency and robustness. In this paper, the results shown in Figure 1 were curated to highlight the best-case performance of our model, demonstrating the potential of our approach. We believe the same curation process was likely employed in the ATM-S paper to showcase their most successful results. For transparency, Figures 32-40 in the Appendix present results generated using the open-source code provided by ATM-S, under identical experimental conditions. These results indicate that ATM-S does not consistently outperform our method, supporting the conclusion that instability is a shared challenge across EEG-to-image models. Despite this, our NERV encoder achieves state-of-the-art (SoTA) performance in base tasks, and it offers the potential to generate high-quality images, as evidenced by Figures 1 and 4. This demonstrates that our framework is capable of producing meaningful outputs under optimal conditions, showcasing its promise in advancing the field. We aim to address the broader challenges of EEG-to-image generation in future work to further improve the stability and reliability of results. 5.Response Regarding the Use of the MLP Projector in Image Generation: You are correct; the diagram mistakenly omitted the MLP projector in the image generation stage. We appreciate your careful observation and bringing this to our attention. The revised manuscript will correct this error in Figure 2 to accurately reflect the role of the MLP projector in our framework. Thank you for pointing this out! 6. The final results reported in the paper are based on zero-shot performance evaluated on the test set. This evaluation method follows the standard practice established in recent EEG-to-image classification works, such as [1], [2], and [3]. Specifically, we trained the model five times with different random seeds, and the best results from each run (evaluated on the test set) were averaged to ensure robustness and reproducibility of the reported performance. 7. The optimal configuration of using five multi-head attention layers in the NERV model was determined based on experiments conducted during training and validated on the test set in a zero-shot setting. This approach aligns with the evaluation methodology used in recent EEG-to-image classification studies. The configuration was chosen to maximize performance on zero-shot tasks, ensuring the model generalizes well to unseen data. We will clarify this in the revised manuscript to avoid any ambiguity and to highlight that the configuration was derived through systematic experimentation across both training and testing phases. Thank you for bringing this to our attention! [1] Song, Yonghao, et al. "Decoding Natural Images from EEG for Object Recognition." arXiv preprint arXiv:2308.13234, ICLR2024. [2] Chen, Chi-Sheng, and Chun-Shu Wei. "Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning." arXiv preprint arXiv:2406.16910 (2024). [3] Fei, Teng, and Virginia de Sa. "Image Reconstruction from Electroencephalography Using Latent Diffusion." arXiv preprint arXiv:2404.01250 (2024).