Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Summary

This paper proposes a framework that integrates 3D brain structures with visual semantics using a Vision Transformer 3D. By aligning fMRI features with multiple levels of visual embeddings, it eliminates the need for subject-specific models and allows extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with LLMs. The topic is intriguing, and the proposed method offers practical applications. Major concerns and minor comments include: 1. Benchmarking with Real-World Datasets: As a novel machine learning approach, the performance of the proposed method should be benchmarked with more real-world fMRI datasets to evaluate the generalizability of the results. The current study may be limited in scope or sample size, and using a wider variety of datasets will help demonstrate the robustness and scalability of the approach across different subjects and conditions. It would be beneficial to include datasets with diverse characteristics, such as different brain regions, tasks, and populations, to ensure comprehensive evaluation. 2. Discussion on Advantages and Disadvantages: The author(s) should discuss the advantages and disadvantages of the proposed method in the field of neuroscience and brain decoding. This discussion should include a comparison with existing approaches, highlighting the unique contributions and potential limitations of the new method. Additionally, insights into the interpretability and explainability of the proposed method would be valuable. For instance, how does this method enhance our understanding of brain activity patterns? Are there any trade-offs between model complexity and interpretability? Addressing these questions will provide a clearer picture of the method’s potential impact and areas for improvement. 3. Details on Cross-Validation: Please provide more details about the cross-validation used in the empirical studies, perhaps in Section 4. It is essential to specify the type of cross-validation technique employed (e.g., k-fold, leave-one-out) and the rationale behind its selection. Detailed information on the partitioning of the data, the number of folds, and any stratification strategies used will help in understanding the robustness of the validation process. Additionally, discussing the metrics used for evaluation and how they were computed across different folds will add clarity to the reported results. 4. Pseudocode for the Proposed Method: The proposed method can be summarized in the form of pseudocode (algorithm). Providing a step-by-step algorithmic representation will make the methodology more transparent and easier to reproduce. The pseudocode should outline the key steps involved in data preprocessing, feature extraction, model training, and inference. Including comments within the pseudocode to explain the purpose of each step and any critical hyperparameters or configurations will further enhance understanding. 5. Minor Typos and Grammar Mistakes: There are some minor typos and grammatical mistakes throughout the paper. A thorough proofreading and editing process is recommended to improve the overall readability and professionalism of the manuscript.

Strengths

Please see the Summary section

Weaknesses

Please see the Summary section

Questions

Please see the Summary section

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please see the Summary section

Summary

This paper presents an innovative framework that leverages Vision Transformer 3D (ViT3D) to integrate 3D brain structures with visual semantics, enhancing visual reconstruction and language interaction from fMRI data. By aligning fMRI features with visual embeddings through a unified feature extractor and integrating with LLMs, the framework enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. The framework demonstrates exceptional performance and holds significant potential for applications in neuroscience and human-computer interaction.

Strengths

Provides a multi-task visual neural decoding framework that is simpler and more elegant compared to previous methods, offering a robust example for integrating visual and neural signals with unified modeling. Introduces a method for precise localization of language-based concepts within brain signals simply using the GradCAM method.

Weaknesses

The Dual-Stream fMRI Feature Extractor is already a common method in the field, but the combination of the two features appears suboptimal. The authors thoroughly discuss the trade-off between low-level and high-level features. However, based on the results in fig 4, the reconstructed images’ low-level features are inferior to previous methods.

Questions

I am curious if the method of combining low-level and high-level features in eq4 is the most effective. While other works also use this approach, it seems to force the model to trade off between the two types of features. Are there potentially better methods? In visual reconstruction, the method with LLM appears effective only when the beta value is low. Does it overlap in function with the CLIP embedding?

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors discuss the limitations well in the appendix.

Summary

The paper introduces a new, subject-agnostic visual reconstruction pipeline. They introduce a way to integrate across fMRI readings from different subjects and enhance their integration using LLMs. Through this integration they see a consistent improvement of high level semantic feature baselines in their reconstruction.

Strengths

* An interesting new approach to align fMRI feature extraction with ViTs and LLMs * Brain captioning with LLMs is highly novel. The results are promising in Table 1. * Promising results on NSD reconstruction. * Interesting set of analyses using GradCAM.

Weaknesses

* Concerns about preprocessing and claims about cross-subject analysis * The authors propose a trilinear interpolation to combine BOLD signals across patches across subjects. To me, this makes many assumptions that may not be true or might miss several problems. The authors need to address these questions to substantiate the claim of removing subject-specific modeling or using some kind of mixed subject pretraining to combine subjects. * For example, this method ignores inter-subject variability such as anatomical differences or functional differences, which can vary among individual subjects. * The interpolation makes a linear assumption on BOLD responses which may not be true. I realize this will follow with alignment to a ViT later but this imposes strong assumptions about BOLD signal space. * My main concern here is whether this kind of preprocessing will generalize to modeling over new subjects. Is this approach applicable to general subject populations? Could the authors add some clarification and expand on this point? * LLM Interaction details * My understanding is that there is a finetuning objective when using the LLM interaction. I think section 3.4 was cut off somehow and this wasn’t very clear to me. * Some details were missing making Table 1 a bit difficult to understand. See questions

Questions

* What does fine-tuned model refer to in line 204? Is it the LLM (just making sure) * When you use BLIP2 generated captions, are you passing the original NSD image and generating a caption? Is this in comparison to your LLM that uses an fMRI embedding in place of [image]? I think this was confusing. * Although GradCAM is easily interpretable, I do wonder if this would benefit from something more recent for interpretability like [1]. * Could MindEye2 (https://medarc-ai.github.io/mindeye2/) be used as a baseline? [1] Materzynska et. al. Disentangling visual and written concepts in CLIP. CVPR 2022.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

* Limitations are addressed but may want to consider toning down some of the claims about subject variability (see weaknesses).

Summary

- This model focuses on the task of reconstructing image stimuli from fMRI readings - Instead of training a subject specific model, a subject-generic model is trained - The model is built around a pre-trained LLM core, which is finetuned to take the fMRI as input and then: - engage in dialogue about the image stimulus - reconstruct the image stimuli by generating a language prompt which is then passed to UnCLIP - create a heatmap of activations in the brain associated with a given concept

Strengths

- The multi-subject aspect of the training appears novel - The use of an LLM for image reconstruction and dialogue about the stimulus also appears novel - The image reconstructions are of comparable or better quality than existing approaches - Ultimately, I think the evaluation could be improved (see Weaknesses), but given the novelty of the approach (using an LLM to do decoding and reconstruction), I argue for acceptance.

Weaknesses

- It's unclear how much the image reconstructions reflect visual processing in the brain. It would be good to do an ablation to see how much the reconstruction depends on $\hat{z}_c$. What if $\hat{z}_c$ were replaced with random noise in the same way that an ablation was done on $\hat{z}_v$? Basically, I am curious about how much decoding can be done from the prompt $a_r$ only. - In the same vein, it would be interesting to do an experiment where the generated prompt $a_r$ is given to a generative model that does not require $z_c$ and $z_v$, i.e., a model that can take a text prompt and generate an image. - It's claimed that the model can engage in complex dialogue about the semantic content of the stimuli, but from the example prompts, it seems that many of the answers to the dialogue questions may be available in the prompt. For example, figure 11 seems to suggest that many of the complex reasoning prompts contain enough information to answer the question in the prompt. E.g. ``What is the distribution of the cows in the field based on the description and their positions in the image.'' A good ablation test would be to perform the complex reasoning task without the fMRI inputs. - Some technical details are hard to follow, especially with regards to concept localization (see questions).

Questions

- line 293 --- how are concepts "extracted"? - line "Finally, patches containing non-task-relevant information are filtered out" -- how are the masked portions chosen? - line 256-258: I think there is a typo somewhere. I'm not sure what the sentence is supposed to say. - Fig 1: From the figure, it seems that one can query the LM for concept localization in the brain? But where is that described? What do annotations for that dataset look like? - A related question: line 533: what does it mean to localize the results using GradCAM? Can you describe how that was done? - Fig 5 and 6: Equation (4) says that $\beta=1$ corresponds with all noise input. But increasing noise seems to correspond with better image quality? - Line 484: Typo: "Language Extension" - Table 2: what is `C_Subj`?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations and potential negative societal impact are discussed in the appendix.

Request for Feedback on Response Clarifications

Dear Reviewers, Could you please provide feedback on our responses to your questions? We are eager to know if our answers have addressed your concerns. Additionally, are there any further comments or insights you might have regarding our work? Your input is crucial for us to enhance the quality of our submission, and we greatly appreciate your guidance and time.

Response to authors

I thank the authors for their detailed response to my questions. I will keep my score and continue to recommend acceptance. I found the author's ablation studies with randomized inputs to be convincing, both for reconstruction as well as dialogue. One interesting point is raised by the finding that the intermediate text prompt $a_r$ is enough to achieve good reconstruction quality. This calls into question how much of the reconstruction actually reflects _visual_ processing in the brain, as opposed to some other higher-level concept processing. But this is not really a weakness of the paper, and more something to explore further.