Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors

Summary

This paper presents a novel fMRI-to-image approach (MindEye) that can achieve excellent reconstructions of natural scenes. The main idea is to retrieve and reconstruct viewed images from fMRI with brain activity information. The model consists of two parallel submodules. The retrieval submodule uses contrastive loss to retrieve images and generate features with high cosine similarity with the corresponding image embeddings. In the reconstruction submodule, the diffusion prior and CLIP image embedding are used to generate aligned semantic features. In order to better reconstruct the image, a separate encoder is trained to generate low-level features, and jointly output the reconstructed images with the previous semantic features. The experimental results also demonstrate the superior performance of the method.

Strengths

Interesting idea. This paper provides a novel method to achieve fMRI-to-image task, and explores this task by introducing retrieval, diffusion prior and low-level information. Aside from the framework, the experimental results also fully illustrate its performance.

Weaknesses

1. In this paper, most of the modules are existing models. The novelty of the paper requires further elaboration. 2. Lack of theoretical proof, most of them are descriptions. For example, the aligned embeddings mentioned in the paper, whether there is a theoretical proof of the rationality of the aligned embeddings for image reconstruction. 3. The network structure uses multiple large-scale models, such as Clip and Diffusion, which are relatively time-consuming, while the paper lacks an explanation of the computational cost. 4. Limited datasets. Only NSD dataset is used, and the generalization of the model is not considered. 5. The experiment is valid, only the performance of each sub-module is considered. The performance after removing a certain submodule is not clear.

Questions

1. How about the comparisons with other methods in terms of computational cost and model parameters? 2. For data augmentation method, is using generative methods to generate more samples an alternative? 3. Is the diffusion prior module pre-trained? Also, I wonder what is the benefit of using a diffusion prior?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Please refer to the weakness.

Summary

The authors present MindEye, an innovative fMRI-to-image approach that utilizes contrastive learning and a diffusion prior for retrieval and reconstruction tasks. They conduct thorough comparisons, establishing MindEye's superiority over existing methods in terms of both qualitative and quantitative evaluations. The authors attribute the success of MindEye to its specialized submodules, improved training techniques, and models with increased parameterization. Additionally, they showcase MindEye's ability to preserve low-level image features in reconstructions through the use of img2img. Overall, MindEye represents a significant advancement in the field, pushing the boundaries of fMRI-based image retrieval and reconstruction.

Strengths

The authors conducted a comprehensive comparison of MindEye with existing methods, employing both qualitative side-by-side comparisons and quantitative evaluations. The results demonstrate that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. Notably, MindEye excels at retrieving the exact original image even when faced with highly similar candidates, indicating that its brain embeddings retain fine-grained, image-specific information. This remarkable capability enables accurate image retrieval even from extensive databases such as LAION-5B. The paper is well written and the presentation is to the point.

Weaknesses

The methodology employed in this study has some limitations in terms of originality, as a majority of the approach relies on external state-of-the-art models. The authors primarily train simple MLPs and utilize a pre-trained diffusion prior.

Questions

1. Exploring the applicability of the model to different patients would yield valuable insights. It raises the question of whether the model can provide meaningful results beyond the specific patient it was trained on. 2. A sensitivity analysis investigating the relationship between image output and fMRI input would be highly intriguing. It would shed light on the crucial components of the input that contribute to generating the CLIP embedding and ultimately influence the quality of the reconstructed image. 3. The authors suggest that increasing the number of parameters improves the results. It would be informative to know if they experimented with deeper networks to explore the potential benefits of a more complex architecture.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

1. Lack of Methodological Originality: The study relies heavily on external state-of-the-art models, which diminishes the originality of the methodology. The authors predominantly employ simple MLPs and a pre-trained diffusion prior, which limits the novelty of their approach. 2. Applicability to Different Patients: Exploring the generalizability of the model to diverse patient populations would be valuable. It is essential to understand if the model can yield meaningful results beyond the specific patient it was initially trained on. 3. Sensitivity Analysis: Conducting a sensitivity analysis would provide valuable insights into the relationship between image output and fMRI input. Understanding which specific components of the input are crucial for generating the CLIP embedding and influencing the quality of the reconstructed image would enhance the understanding of the model's behavior. 4. Deeper Networks and Parameterization: The authors suggest that increasing the number of parameters improves the results. It would be beneficial to know if they explored the use of deeper networks, as a more complex architecture may have potential benefits. Investigating the effects of different network depths could shed light on the impact of model complexity on performance.

Summary

This paper introduces a method for fMRI-to-image conversion that facilitates realistic image reconstruction from brain recordings. The proposed method comprises two stages: semantic (high-level) and perceptual (low-level) pipelines. In the semantic stage, fMRI voxels are projected into the image space using a Multilayer Perceptron (MLP) and a diffusion prior. Notably, the paper employs the contrastive learning of the CLIP model to map fMRI data into a pre-trained CLIP model. Subsequently, the diffusion prior is used to map CLIP text embedding to image embedding. On the other hand, in the perceptual pipeline, image embeddings are mapped using an MLP projector and a CNN decoder. The method's experimental results are derived using the NSD dataset.

Strengths

The method presented in the paper is clear, and the validity of the claims is compelling. I also appreciate the experimental evaluation conducted, which yields promising results both from a quantitative and qualitative standpoint. The paper presents a commendable ablation analysis of various loss functions and data augmentation strategies.

Weaknesses

I think the main weakness of the method is the use of pre-trained models and evaluations with large datasets only. I was wondering what will happen when we train a model without having a pretraining model. Having employed contrastive learning in the form of CLIP, with other proposed modules, I think the overall novelty of the paper might not be very high. However, I think the main contribution of the paper is clear and robust. The presented method is quite large which makes it hard for building it with restricted resources.

Questions

I wonder how the proposed model manages the domain shift when the structure of the training functional magnetic resonance imaging (fMRI) data significantly differs from that of the downstream task. Regarding the architectural improvement ablation, what about potential implications of training the overarching concept using a more compact architecture with fewer parameters. It would be interesting to explore if a streamlined model could yield similar or improved results while offering more efficient computational processing and potentially mitigating the risk of overfitting. What about the computational complexity of the proposed method, specifically concerning image generation? How does this model compare to other methods in terms of computational demands? The computation complexity would be crucial for understanding its practical applicability, particularly in settings where computational resources may be limited. Finally, I am curious about the choice of dataset. The paper utilizes NSD dataset for model training and evaluation. Could the model generalize well when applied to different datasets, specifically smaller ones? Moreover, when the dataset's size is reduced, can the model maintain its performance and possibly outperform comparative methods?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the paper discusses the possible limitation of the current work.

Summary

The paper improves the fMRI reconstruction method using contrastive learning strategy and diffusion prior model. The concept is relative simple but the details of the proposed method, which is the key to make difference, is well-implemented. First, the BiMixCo implements a contrastive loss between the fMRI voxel representation and the CLIP image representation with the utilization of mixup augmentation of the fMRI vectors. Second, the Diffusion prior is implemented on the fMRI representation after the MLP backbone (which is the input to the MLP projector of which output is the input the contrastive loss) for the generating the representation for reconstructing the image. Moreover, the paper implements additional low-level pipeline based on VAE and Stable Diffusion so that an initialization of reconstruction can be attained. The overall reconstruction and retrieval performance significantly outperforms the baselines with large margin, particularly for the retrieval performance.

Strengths

- While the problem is not new and the proposed methods are clever combinations of existing techniques rather than brand new ones, the overall fMRI-based reconstruction and retrieval performances are impressive. - Both retrieval and reconstruction results are presented with thorough ablation studies to justify the modeling choice. - The visualization results are compelling and convincing.

Weaknesses

- The performance gaps between MindEye and other baselines are vast while those for reconstructions are not as much. It seems like the reconstruction results for the baselines are copied from the original papers, while the retrieval results are reproduces..Is that correct? What is the main reason for such huge difference? The representations used for reconstruction and retrieval should not be that different, so such a wide gap is little mysterious. In case the retrieval results for the baselines are reproduces, why not also reproduce the results for the reconstructions? - The effect of BiMixCo is not very convincing since as shown in Table 4, it seems to be helpful only for the retrieval task while not as much for the reconstruction task. Would there be some additional justification?

Questions

Please see my comments in Weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

- The exact process for the evaluation for the retrieval task is not very clearly described. - The performance gap between the reconstruction and retrieval is not clearly explained.