Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning

Thank you for your valuable feedback and constructive suggestions regarding our submission. We appreciate your insights and provide detailed responses to each of your comments below, outlining the corresponding improvements we have made to the paper. 1. We focused on the ThingsEEG dataset, which is currently the largest publicly available EEG-image dataset, encompassing 1,650 training categories and 200 testing categories. While this dataset poses significant challenges, we recognize the importance of validating our method on additional datasets, and we plan to enhance the generalizability of our findings in future work. 2. Thank you for your suggestion regarding including more baseline comparisons. We agree that a broader baseline evaluation strengthens the study. However, we emphasize that EEG-based image decoding remains a relatively nascent field, with a limited number of existing works. The most relevant and established baselines that we have selected include: BraVL (Du et al., 2023): A multimodal learning approach using multilayer perceptrons (MLPs) to align EEG, image, and text representations. NICE (Song et al., 2024): A contrastive learning framework based on CLIP for aligning EEG and image embeddings. EEGClip (Singh et al., 2024): A supervised learning method for the joint representation of EEG and image signals. We specifically selected NICE and its variants as baselines because they represent the state-of-the-art in EEG-image contrastive learning. Our contributions extend these works by: Introducing the Similarity-Keeping (SK) Loss, which captures intra-batch relationships and enhances representation learning. Developing the NervFormer EEG encoder, which integrates graph attention mechanisms for more effective signal processing. 3. Our experiments were primarily designed around a 200-way zero-shot classification setup to evaluate the model's generalization to unseen image categories. In this setup, the model achieved a top-1 accuracy of 19.3% and a top-5 accuracy of 48.9% on a test set comprising image classes that were entirely unforeseen during training. This highlights the model's capacity to decode EEG signals corresponding to new image categories, demonstrating robust generalization. The zero-shot learning design itself is an inherently rigorous test of generalization, as the testing images and EEG samples belong to categories not present in the training data. We believe this evaluation provides substantial evidence of the model's capacity to generalize to new visual concepts. 4. We conducted detailed analyses to understand the interpretability and behavior of our model, and we highlighted these aspects more prominently in the revised version. Here are the key findings related to our model's interpretability: Spatial-Temporal Dynamics Analysis: Using Grad-CAM visualizations, we observed that the model consistently focuses on the occipital and parietal regions of the brain, particularly within the 100-500 ms window post-stimulus onset. These regions and time periods align with well-established neuroscience findings regarding visual processing dynamics, such as the hierarchical processing from V1, V2, and V4 in the occipital cortex to object recognition in the inferotemporal cortex. This analysis validates the neuroscientific relevance of the model and provides a biologically grounded explanation of its decision-making process. Frequency Band Analysis: We analyzed time-frequency dynamics, focusing on the alpha and gamma bands, which are known to be associated with visual attention and high-level visual recognition in neuroscience. The model's emphasis on these bands during decoding indicates its effectiveness in capturing the neural correlates of attention and recognition, further validating its interpretability. Model-Specific Insights: The Similarity-Keeping (SK) Loss enhances the alignment between EEG and image embeddings while preserving intra-batch relationships. This mechanism not only improves performance but also contributes to interpretability by ensuring that the learned representations reflect meaningful neural patterns rather than noise. Comparisons between variants of our model (e.g., MUSE vs. MUSE-SK-GA) showcased unique strengths across different subjects, providing insights into how specific architectural components, such as Graph Attention (GA) modules, influence performance and focus areas in the brain.

Thank you for your valuable feedback and constructive suggestions regarding our submission. We appreciate the insights you provided, and below we outline our detailed responses to your comments, as well as the corresponding improvements we have made in the paper. 1. We acknowledge this limitation and agree that validating our method on additional datasets would significantly strengthen the generalizability of our findings. Unfortunately, publicly available EEG datasets that offer sufficient size and diversity in image categories are limited. Moving forward, we aim to collaborate with other researchers to benchmark our framework across a variety of experimental protocols, ensuring a broader applicability of our results. 2. We appreciate your suggestion regarding the EEG scalp maps. In the revised manuscript, we have selected key temporal and spatial patterns, specifically focusing on the 100–500 ms time window and the occipital regions. We have enlarged these maps for clarity, providing a more detailed visualization of the neural dynamics associated with our findings. 3. Thank you for this valuable suggestion. In the revised manuscript, we have included a discussion that compares EEG with recent fMRI studies, such as those utilizing contrastive learning for image classification [1]. We highlight the fundamental differences in their principles, noting that while EEG offers superior temporal resolution and portability, fMRI provides valuable spatial information. This comparative analysis aims to underscore the unique advantages of EEG for visual information decoding applications. 4. We have increased the text size in Figure 4 for enhanced readability, ensuring that all presented information is easily accessible to the reader. 5. We have expanded the description of the EEG encoder within the revised manuscript. This expanded content includes details regarding its spatial-temporal filtering and attention mechanisms. Furthermore, we conducted additional analyses focusing on specific frequency bands, such as alpha and gamma, along with temporal windows (e.g., early vs. late responses) [2]. We relate these findings to established visual processing mechanisms in neuroscience literature to provide a deeper understanding of the underlying dynamics. References: [1] Scotti, P., et al. "Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors." Advances in Neural Information Processing Systems 36 (2024). [2] She, L., Benna, M.K., Shi, Y. et al. "Temporal multiplexing of perception and memory codes in IT cortex." Nature 629, 861–868 (2024). Thank you again for your thorough review and constructive comments. We believe that the revisions made in response to your feedback will significantly enhance the clarity and rigor of our manuscript. We look forward to your continued evaluation of our work.

1. We agree that the roles of our proposed loss functions and EEG encoder need more clarity. In the revised manuscript, we will expand on the following aspects: (a) Similarity-Keeping (SK) Loss: We have provided a thorough explanation of how this loss enhances inter-sample similarity preservation, particularly in EEG embeddings, and elaborate on its design, inspired by principles of neuroscience. (b) NervFormer Architecture: We have clarified the advantages of the NervFormer architecture in extracting both spatial and temporal features from EEG data. While its superiority may not always be evident in specific metrics (e.g., Table 1), we believe this limitation arises because transformer-based architectures typically require larger datasets for optimal training. As the sequence length increases, the receptive field grows, making these models more prone to overfitting. This observation aligns with previous findings, such as those reported in Phan et al. (2023) [1]. 2. (a) Impact of Similarity-Keeping (SK) Loss: As shown in Table 2, when SK loss is integrated into the original MUSE framework, changing the InfoNCE loss to SK-InfoNCE, the model achieves a slight but consistent improvement in both top-1 and top-5 accuracies across subjects. For example: Top-1 accuracy improves from 19.2% to 19.3%. Top-5 accuracy remains approximately stable. While this performance gain may seem incremental, the SK loss plays a crucial role in ensuring more stable training by maintaining inter-sample similarity, which benefits generalization in zero-shot settings. This supports our hypothesis that preserving batch-level relationships enhances EEG-image alignment. (b) Effect of the Graph Attention (GA) Module: The addition of the GA module (STConv-GA in Table 2) enhances the model’s capacity to capture inter-electrode relationships. For instance: MUSE-SK-GA achieves 19.3% top-1 and 48.1% top-5 accuracy, a noticeable improvement over MUSE-SK without the GA module. The improvement is particularly marked in individual subject-specific performance, such as Subject 10, where the top-1 accuracy rises from 20.1% to 22.7%. 3. In Table 3, we evaluate the NervFormer encoder to understand its impact on zero-shot classification: Replacing STConv with NervFormer (MUSE-Nerv vs. MUSE) results in a drop in average performance (14.7% vs. 19.2% top-1 accuracy). However, when combined with SK and GA (MUSE-SK-Nerv-GA), the NervFormer demonstrates subject-specific improvements, as seen for Subject 10, where it attains a 21.9% top-1 accuracy, outperforming all STConv-based variants. This highlights a key observation: Transformer-based encoders like NervFormer can be more effective when enhanced by mechanisms such as SK and GA, though they may struggle with smaller datasets due to susceptibility to overfitting. This finding aligns with previous works emphasizing the necessity of careful regularization when applying Transformers to EEG data. 4. We agree with your observation and have adjusted all figures to enhance readability. 5. We confirm that no label information is utilized during training, adhering strictly to the self-supervised paradigm. The contrastive loss updates the EEG encoder by aligning embeddings based on cross-modality similarity (EEG-image). In the revised manuscript, we have clarified this training process and explicitly describe how the image encoder remains frozen during training. 6. The time-frequency maps reveal that our model focuses on alpha and gamma bands, which are consistent with known processes of visual attention and recognition. We have elaborated further on this observation in the revised version. 7. The variable 'S' represents the similarity matrix calculated between EEG and image embeddings. We have revised the manuscript to ensure that all equations are accompanied by clear definitions and intuitive descriptions. Reference: [1] Phan, H., Lorenzen K. P., Heremans, E., Chén, O. Y., Tran, M. C., Koch, P., ... & De Vos, M. (2023). L-SeqSleepNet: Whole-cycle long sequence modelling for automatic sleep staging. IEEE Journal of Biomedical and Health Informatics.

Thank you for your valuable feedback and constructive suggestions on our submission. Below, we provide detailed responses to your comments and outline the corresponding improvements we will make to the paper. 1. We acknowledge that Song et al. (2) shares similarities with our approach, as both involve contrastive learning for EEG-image alignment. However, our work introduces several significant innovations that distinguish it from prior research: (a) Similarity-Keeping Contrastive Loss (SK-InfoNCE): We are the first to incorporate a similarity-keeping loss term into the contrastive learning framework. This term regularizes the learning process by preserving intra-batch sample relationships, drawing inspiration from the cortical mapping of object relationships in the visual cortex. (b) Key Differences: While Song et al. use standard contrastive loss within their CLIP-based framework, they do not consider these inter-sample relationships, which our method explicitly models. (c) Enhanced EEG Encoders: We propose novel EEG encoders, including Spatial-Temporal Convolution (STConv) and NervFormer, which integrate spatial filtering, temporal modeling, and graph attention mechanisms. These architectures are specifically tailored to EEG data and leverage its spatial and temporal dynamics more effectively than the simpler temporal convolution utilized in Song et al. 2. Our results in Table 1 demonstrate that models incorporating similarity-keeping loss (e.g., MUSE-SK) consistently achieve improved performance in top-1 and top-5 accuracy compared to their counterparts lacking this component. We acknowledge that these improvements may appear incremental, which could be attributed to the dataset complexity. The zero-shot classification task on the ThingsEEG dataset is highly challenging, inherently limiting the scope for substantial performance gains. Nonetheless, our model exhibits stable improvements in embedding alignment under these conditions. 3. We appreciate your insights on the NICE model's performance. We have expanded our discussion in the revised manuscript to provide a more comprehensive comparison between our MUSE framework and the NICE model. This includes elaborating on the differences in architecture, hyperparameters, and training methodologies that contribute to the variations in performance outcome. 4. Thank you for your feedback on table presentation. We have revised Tables 2 and 3 to eliminate redundancies and enhance clarity. Additionally, efforts have been made to ensure that all figures are clearly readable. References: 1. P. Singh et al., "Learning Robust Deep Visual Representations from EEG Brain Recordings," In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2024, pp. 7538-7547. 2. Y. Song, et al., “Decoding Natural Images from EEG for Object Recognition.” In The Twelfth International Conference on Learning Representations (ICLR), 2024.

1. Thank you for raising this important concern. While previous studies, such as Kilgallen et al. (2024), highlight limitations in EEG-based image classification, it is important to note that our work builds upon successful implementations that have used EEG to learn image patterns through deep learning. Although some papers, like those by Bharadwaj, H. M. (2023), Li, R. et al. (2020), and Ahmed, H. et al. (2021), suggest that using EEG to decode and classify images is ineffective, these studies were grounded in simpler models such as SVM, LSTM, and EEGNet, leading to expected poor performance. This phenomenon was also observed in Song's paper, which emphasizes that EEG-image contrastive learning is a unique domain that necessitates specialized architectural design, which is precisely the contribution and exploration presented in our MUSE framework. We discuss this in detail and include references mentioned in your comment in Section 2. 2. We appreciate your request for clarification. Our MUSE framework employs a Spatial-Temporal Convolution (STConv) architecture that differs significantly from the Temporal-Spatial Convolution (TSConv) used in Song et al. (2023) (see Table 1; https://arxiv.org/abs/2308.13234). Specifically, we process the spatial dimension corresponding to the positioning of EEG electrodes first, followed by convolution along the temporal dimension. This design allows for more efficient denoising and extraction of relevant EEG patterns, enhancing our model's capability to capture useful brainwave information rather than noise, setting it apart from Song's method. 3. We acknowledge your concerns regarding the interpretability methods employed in our study. While the work by Sujatha Ravindran and Contreras-Vidal (2023) utilized simulated data to evaluate other models, it remains unclear how these approaches perform with real, noisy, and non-stationary EEG data. Our findings indicate that Grad-CAM has demonstrated effective interpretability, performing comparably to methods like LIFT, as noted by Cui et al. (2023). Therefore, employing Grad-CAM as the interpretability tool in our research is entirely justified and provides substantive insights into our model's decision-making process. 4. Thank you for your feedback on figure clarity. We recognize that the layout may appear crowded due to the dense presentation of experiments and results. We have rearranged the visual presentation in the revised version, ensuring that all figures are adequately sized for improved readability. 5. We appreciate this observation. To address your concern, we have provided detailed machine specifications, including hardware configurations and relevant details, in line 323 to ensure proper interpretation of our training duration.