Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities

Summary

The paper focuses on decoding visual stimuli from recorded fMRI activity. Proposed method initially pre-trains an fMRI feature learner (and essentially a signal denoiser) on unlabeled data using a contrastive training scheme resembling to masked autoencoders. Subsequently this feature encoder is finetuned by using guidance from an image auto-encoder to attend to signal patterns that are informative for reconstruction. Output of this fMRI feature learner is then used to condition a latent diffusion model to be able to visually reconstruct the stimuli in the image domain with high resolution.

Strengths

The paper tackles a very interesting problem with nice, visually illustrated results. Proposed idea of performing neural signal guidance in the latent space such that a generative latent diffusion model can be used for high-resolution image synthesis is quite a neat approach to solve this problem.

Weaknesses

There is no higher level justification or discussion present in the paper on why one would/should use fMRI for this problem (as opposed to other alternatives) towards building a non-invasive BMI. Moreover from a technical perspective, the paper only puts together several existing methods but some technical details are appearing to be mixed up in the descriptions. These should be corrected and some clarifications are still necessary in terms of the presentation of results.

Questions

- Motivation of the study from a non-invasive brain-machine interface perspective is somewhat disputable, since the neurophysiological source signal considered here is fMRI. Can the authors extensively discuss and justify to the reader why their choice of fMRI (with low temporal resolution and longer temporal scales of responses, noisy nature, and hard-to-integrate experimental recording/design) would be a better candidate than other alternatives (e.g., EEG, or (partly-)invasive methods) to tackle this problem? - Figure 2 with the "VQGAN" does not make sense. Starting from the last paragraph of page 5 and then in Figure 2, the paper talks about a VQGAN, whereas the LDM that the authors use should certainly be a pre-trained VQ-VAE with an encoder-decoder pair? Also based on the notation in Eq 9 and line 193, I presume the VQ-VAE encoder should be E_g rather than E_G on line 191? - Description of Phase 1 has some ambiguities. Beginning of Sec 3.2 was understandable. Then line 140 starts to talk about an unmasked original "image" at some point, whereas the fMRI inputs were not yet denoted as "images" so far? The narrative can be made more consistent. Perhaps in Section 4.1 or 4.2, authors should also briefly describe the image-like fMRI input data representation scheme in a few words. - Presentation of quantitative results are also ambigious. Figure 3 is not discussed in text, and achieved accuracies are not readable from the figure. Ablation studies from Section 5.2 are completely separated from the main quantitative result of the paper that is written in the Abstract: 39.34% in 50-way-top-1 semantic classification accuracy. In Section 5.1 or 5.2, it is not clear where this number really comes from? In general, the paper should revise its presentation of quantitative evaluations. - Methodological comparisons to DC-LDM should be a bit more elaborated. Does this method only differ in the use of a contrastive pre-training phase, or are the encoder network architectures of the 24-layer FRL also different? Can the authors clarify the differences a bit more in depth on how significant their contributions are? - Authors do not explicitly state this, but is the general idea of contrastively using a masked autoencoder scheme for denoising fMRI data a novel contribution of the paper? If not, can the authors refer to similar works and discuss that it is a powerful approach that is adopted for pre-training in their model? Minor comments: - It appears like in no occasion an fMRI conditioned reconstruction comes out directly identical to the ground truth, and hence it would be probably better/more realistic in Figure 2c to use a different airplane image as generated. - What is the "CMA" block indicating in Figure 1? Is it the cross-attention module? - In Figure 1, the dotted background pattern prevents the equation symbols/text to be easily read. Perhaps this figure text/equations can be re-generated in a better visible way.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Sufficiently addressed.

Summary

The authors aim to decode the visual stimuli from neural responses by reverse-mapping the signals from functional MRI (fMRI) to the images the participants see while being scanned. The authors claimed to achieve this through a two-phase framework. In the first phase, they pre-train an fMRI feature learner inspired by mask auto-encoder (MAE) from unlabeled fMRI data. This phase tries to discern the common patterns and features in fMRI across participants responding the same stimuli. In the second phase, they further tune the fMRI feature learner with cross-modality correspondence from the visual stimuli, i.e., the images. This process is similar to the cross-modality guidance commonly used in vision language models. Finally, after the two phases of training, the fMRI feature learner is used to generate signals that serve as conditions for a latent diffusion model (LDM) for image generation. The authors demonstrated that the final model is able to produce images from the participants’ fMRI signals such that the generated images well correspond — in category — to what the participants see. On the quantitative benchmarks, the proposed method outperform the baselines by a significant margin. As much as I appreciate this work, I would like to demystify it a bit. In my opinion, it’s effectively performing two tasks sequentially: (1) fMRI signal classification into 50 or 100 classes, and (2) class-conditional image generation. The holistic framework and storytelling might have made it more impressive than it actually is — though admittedly it is still impressive.

Strengths

1. “Mind reading” is a very bold topic. Generating the image in a participant’s mind from the fMRI recordings is arguably a reasonable way. 2. The design with three stages: contrastive pre-training, cross-modality fine-tuning and latent diffusion model is very reasonable and well conveyed. 3. The ablation studies are presented neatly.

Weaknesses

1. The experimental results, though impressive, do not seem very comprehensive. For example, in Figure 3, qualitative results are shown for two datasets while bar plots are only displayed for one. 2. Also, I would recommend showing some intermediate results — for example, directly evaluating the fMRI representation accuracy by training a simple model to classify the learned fMRI representations after phase 2.

Questions

1. Do you think it is reasonable to present the intermediate fMRI representation accuracy as described in Weaknesses 2? If I understand it correctly, you have $image_{real} \rightarrow (human) \rightarrow fMRI \rightarrow representation \rightarrow image_{gen}$, and you are showing $acc(classifier(image_{gen}), label(image_{real}))$. I wonder if you can also show $acc(classifier'(representation), label(image_{real}))$ and even $acc(classifier'(representation), classifier(image_{gen}))$.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Nothing came to my mind.

Summary

This paper proposed to decode visual stimuli from neural responses recorded by fMRI. First, it pretrains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. Second, it tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder.

Strengths

(1) The method is straightforward and easy to understand. (2) The related work discussion seems comprehensive.

Weaknesses

(1) The selection of baseline methods for comparison appears to be insufficient, thus hindering the ability to effectively demonstrate the effectiveness of the proposed method. To enhance the evaluation, it would be beneficial for the authors to include a comparison with the methods mentioned in the related work section. (2) Another crucial factor that warrants consideration is the quality of the fMRI data, as it can significantly impact the performance of the model. It would be valuable if the authors could provide some ablation studies pertaining to this aspect as well. (3) The evaluation metric solely employed is n-way top-k accuracy, which could be supplemented with additional evaluation metrics to provide a more comprehensive assessment. Metrics such as FID (Fréchet Inception Distance), SSIM (Structural Similarity Index), MSE (Mean Squared Error), among others, would be advantageous for a more qualitative evaluation. (4) Since human data is used in the study, the authors can discuss some potential ethical concerns.

Questions

Please see the comments above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation is discussed in Section 6, however, since human data is used in the study, the authors can discuss some potential ethical concerns.

Summary

The paper proposes an approach for decoding visual stimuli from neural responses (fMRI images). The rationale behind the proposed approach lies in the difficulty of learning the complex relationship between a stimuli and the neural responses to it, and the noisy nature of fMRI images. The authors propose a multiple phase approach comprising a contrastive learnign based method for learning denoised representations of fMRI brain activities (a new DC-MAE method is proposed), a feature learner that combines previsous representation space with image representation through cross attention and a diffusiuon model to reconstruct image stimuli from brain activities which is conditioned on the opuput of the feature learner.

Strengths

The approcah looks like a smart agregation of state of the art components that allows reaching high top-k accuracy (the chosen metric). The paper is easy to read and the methodological choices are well motivated. The method is validated by top k accuracy and achieves significantly better results than the baselinesit is compared to. A thourough ablation study shows the impact of the various components and of the hyperparameters of the method.

Weaknesses

Few aspects of the method are not enough detailed (see questions)

Questions

To which space does v_i belongs, is is a time series ? I don’t understand the use of 1D conclutional models that maps v_i^m_1 and v_i^m_2 into embeddings. In Phase2 are all models (E_F, E_I, D_F, D_I) retrained or are only the cross attention layers trained ? The way the latent diffusion model is conditionned seems new. Hwo does this compare to previous attempts of designing conditional latent diffusion models ? Concerning metrics it is said in section 4.3 « We employ the pre-trained ImageNet-1K classifier [50] as a semantic correctness evaluator. » wht does this mean ? Concerning the top-k accuracy experiments did the authors use a statistical test to attest the significativity of the result ?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

limitations are addressed

Summary

The paper propose a novel two-phase fMRI representation learning method to encode the visual stimuli from neural responses. It is a significant challenging task due to noisy fMRI signals and complex intricate pattern of brain visual representation. The proposed two-phase can reconstruct image stimuli from brain activities.

Strengths

1.The overall framework is novel and interesting. The use of double-contrastive mask auto-encoder and image-guided auto-encoder are novelty. 2.The introduction is clear. 3.The superiority experimental results are obtained.

Weaknesses

1.The motivation of two phase is confused. Why two phase representation learning is needed? And what is the difference between the two phase? 2. How the contrastive MAE works? It is not clear 3.The model may be too big? How the efficiency? The experiments are weak. How to set the mask? The model’s size? More quantification is needed.

Questions

see above

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

the proposed method is too complex to follow.

response to the rebuttal

Thanks to the authors for their responses. I have examined all reviews and responses carefully. Several appreciated clarifications are made, and I'm more comfortable with this submission now. Justifications on choosing fMRI over alternatives is fair enough [A1]. But I still think that the motivating point-of-view should not be directly from the perspective of "building a non-invasive BMI", since I somehow can not see this application to yield a realistic BMI with an fMRI in the loop. All of the mentioned clarifications should be included in the revised manuscript. Some of the points that will be fixed in the revisions are quite important for the reader to follow the paper (e.g., sudden changes in terminology preferences). Figure 3a should be revised such that these bars are readable quantitatively. Numerical conclusions referring to residuals should be explicitly stated somewhere in the paper too [A4]. Overall, I'd be happy to increase my score to the accept region, assuming presence of these revisions to higher clarity.

Response to the rebuttal

Thanks to the authors for their responses. I read all reviews and the answers carefully. Many clarifications and additional results have been obtained that strengthen the paper. Yet I am not fully satisfied with the answer about the nature of v_i, and more generally I still feel the description of the method lacks details. I still don’t understand why a 1D convolutional layer is used, why are the data averaged across the time dimension, and if so what is the axis along which a 1D convolutional layer is used? Also, what do you mean by 1D spatial pattern? I am not an expert in neuroscience and i likely lack some background knowledge to understand the processing you describe which is maybe standard in the neuroscience field. Yet I believe providing more details would help any machine learning reader to understand the paper better even if he/she is a non expert in the field of neuroscience.

Thank you for the rebuttal

Thanks the authors for the rebuttal. My concerns have been addressed and I have changed my score to weak accept.

Response to Rebuttal

I have read the rebuttal by the authors. They well explained the challenge in training a separate classifier for classifying the intermediate representations. I do not have any further concerns, and decide to maintain my rating at 7.

Response to rebuttal

Dear Authors, Thank you for answering the points that were raised, your response will be taken into account. Best, Your Area Chair