NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping

Summary

This paper introduces NeuroBOLT, a transformer-based model. NeuroBOLT utilizes multi-dimensional representation learning across temporal, spatial, and spectral domains to translate raw EEG data into comprehensive fMRI activity signals across the entire brain. Experimental results showcase NeuroBOLT's ability to effectively reconstruct resting-state fMRI signals across primary sensory, high-level cognitive areas, and deep subcortical regions.

Strengths

1. The paper tackles one of the most challenging and competitive topics in neuroscience. 2. The motivation behind the paper is quite clear, and the experimental section is logically sound. 3. The figures and tables in the article are clear and well-organized, making it highly readable.

Weaknesses

1. The method abbreviation and the title are not closely related. It is unclear where 'BOLT' comes from in the title, and even after reading the abstract, it remains confusing. 2. In fact, there has been a lot of work on fMRI-EEG in recent years, especially in 2023 and 2024, but the author's related work lacks a significant amount of relevant literature. 3. In the abstract and introduction, the author's description of the method is inconsistent with the organization in the methodology section, resulting in a need for improved readability. 4. The writing of the article needs to be further standardized. For example, 'FMRI data' is sometimes written with a capital 'F' and other times as 'fMRI data'. 5. Although the layout and presentation of the tables are aesthetically pleasing, the font size is too small, making them difficult to read even when enlarged. 6. The paper does not provide code or data to support the reproducibility of results. 7. This paper lacks details on the parameter selection for the baseline methods. Although the authors state, 'The baseline models are from [44] and [16], where we choose the models with the best downstream classification task performance,' the datasets and tasks in references [16] and [44] are not entirely consistent with those in this paper. Therefore, the authors should specify the exact process of parameter selection. 8. The equations and symbols in the article are not very standardized. The authors should provide notation to help readers understand. 9. The authors should conduct statistical tests to validate the significance of their methods. 10. For readers in the NeurIPS community, the theoretical contribution of this paper appears to be weak. 11. I don't quite understand what the author means by the third point of contribution: 'Successful resting-state fMRI reconstruction To our knowledge, this is the first study to successfully reconstruct the resting-state fMRI signal from raw EEG data, with only 26 electrodes.' What is the significance of 26 electrodes? 12. Equation 2 does not appear to be a complete equation.

Questions

Please see the twelve weaknesses above.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Limitations are discussed in the section of Discussion and Conclusion.

Summary

The manuscript proposes an EGG-to-fMRI synthesis model. The framework implements a transformer architecture and uses a multi-channel feature combination expanded across the temporal axis. To evaluate the proposed model, EGG and fMRI data from 22 participants were recorded while they were in the resting state with eyes closed.

Strengths

The manuscript addresses an interesting problem and can open opportunities for multimodal neuroimaging analysis. Overall, this line of investigation is little explored, therefore, the present manuscript is novel and of interest to the community. The present manuscript is quite complete, (1) the model and rationale behind are sound; (2) a dataset is collected which allows a faithful evaluation of the proposed translation (from EEG to fMRI); (3) it's rather easy to read and follow the manuscript, (4) the reported results are promising.

Weaknesses

The biggest weakness is that the framework and the dataset are only addressing the resting state. While this is an important baseline to investigate, it would have been great to explore the fidelity of the proposed framework when participants are presented with some stimuli. It is unclear whether the source code and dataset will be released publicly. The stability of the results is fully guaranteed given no statistical analyses are performed.

Questions

What is "In-scan" in Table 1? This is not explained in the manuscript. It is unclear to me why all methods are not evaluated for both inter-subject and in-scan. For example, isn't it possible to evaluate BIOT [44] for inter-subject data?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The manuscript sufficiently discusses its current limitations. I think the limitations section should tap into the particular scenario that the model has been evaluated on (resting state) and whether the results will be generalisable to other scenarios remains to be shown.

Summary

In this work, the authors present a deep learning architecture for inferring functional magnetic resonance imaging (fMRI) signal from electroencephalography (EEG) data. The proposed model, named NeuroBOLT, utilizes transformer backbones to provide spatial, temporal, and frequency-based features from the EEG are utilized for reconstruction. The authors demonstrate the performance of their architecture on a small (N=22) data set using a propriatary data set of simultaneously measured EEG-fMRI.

Strengths

fMRI reconstruction from simultaneous EEG is a fascinating topic, and a difficult problem to tackle. The approach taken by the authors in this work is novel for the task at hand, i.e. using a multi-scale spectral feature embedding. Although the decision to use multi-scale spectral embeddings is not new in MRI analysis, as far as I could find the approach has not been utilized for this particular problem and the authors address novel problems for their application to simultaneous EEG-fMRI data in a deep learning architecture. At best this paper is a novel methodological tweak applied with state of the art architectures to see improvements over other deep learning baselines. The breadth of the experiments attempted by the authors is promising; however, see my discussion below for more of a discussion of the limitations of the experiments performed. The authors also perform an ablation study to explore how the inclusion of Multi-Scale Spectral features improves model performance, thus demonstrating the benefit of combining the multi-scale spectral features with the spatiotemporal. This is well appreciated.

Weaknesses

The major weaknesses of this work come down to weaknesses in the empirical evaluation. I am afraid that in its current state, the evaluation does not lead to a convincing demonstration of this method for fMRI reconstruciton, and the claims in the introduction about novelty coming from the application to multiple brain regions and resting-state fMRI seem somewhat overemphasized. Currently this brings it to a full reject as the paper is otherwise sound but the limitations in the evaluation are significant enough to bring it well below the threshold, and cannot be easily addressed in the rebuttle I believe. First, I will highlight the lack of reported standard deviations or error bars in any of the results. No standard deviations are provided in tables 1 or 2, or in any of the figures providing results. In the checklist, the authors state "Error bars are not reported at the current stage because it would be too computationally expensive to compute over all the brain regions and for all participants,also there is limited space in the paper to put all the statistics. But we could always add this information if reviewers think it’s important to know." While I appreciate the authors' acknowledgement of this exclusion, I do think error bars are absolutely necessary to demonstrate the efficacy of the proposed method. The demonstrated improvements are often quite small (e.g. improvement from 0.540 to 0.588 in table 1), and it is not clear if the purported improvements can be explained away from model noise. I could not find any information about controlling model initialization or seeds as well to ensure that random initializations played a less significant role between experiments even with the same architecture on different regions. I absolutely think error bars are necessary for this work, and the reasoning provided by the authors is not mitigated elsewhere or behind a more significant barriers other than training and evaluation time. Additionally, the authors could have mentioned this omission in the limitation section of their main paper since I had to go to the checklist to be sure the authors were aware of the issue. Second, in the abstract the authors highlight the ability to "generalize to other brain areas" and "other conditions (such as resting state)". Unless I am missing something, I cannot find any experiments by the authors that address these particular gaps. The authors do provide inter- and intra-subject predictions which is interesting; however, their model is still only trained on individual ROIs, and they don't include any experiments demonstrating transfer learning between models trained on other regions, and they do not include any experiments studying other tasks BEYOND resting-state fMRI. Thus, the paper falls into the same limitation as past works which were only focused on task, just in the other direction. This work would have been much more compelling if they could demonstrate a model which trained well both on task and rest related data, or even better, which could reconstruct task-related data despite only being trained on resting-state fMRI. The acknowledgement of the limitations in the literature is thus misleading as the proposed method still suffers from these same limitations. The choice to only demonstrate the results for several ROIs highlights this limitation - it would be okay if the authors did not seem to imply elsewhere that their model gets around the single-ROI training approach from past methods. Clearly, the N in this study is quite small. This is to be expected as simultaneous EEG-fMRI is still quite rare as a sequence to collect; however, the authors seem to gloss over all of the myriad issues which will come with training their data over such a small data set. I am not penalizing this work for the small N in and of itself, but as I cannot find any mention of common obstacles such as overfitting, bias towards particular kinds of reconstruction errors, and other limitations that would inevitably arise. I am extremely surprised I could find no mention of pretraining anywhere in this work, which I almost imagine would be hugely necessary for these kinds of studies with very small data sets. Again, it's not necessarily a limitation in and of itself to not do these things, with how the paper currently reads these seem to be touted as benefits of the new approach which are not backed up by evidence.

Questions

How well does the model trained on other ROIs transfer to reconstruction of completely different ROIs? How might scanner model and particular parameters of the resting state sequence affect reconstruction? Why do you not compare anywhere with Source Localization? Source Localization is only mentioned once offhand, and its limitations and efficacy as a reconstruction technique are not gone into in detail. I am surprised it was not included as a baseline method in fact.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do provide some discussion of the limitations of their work; however, as I have noted above, there are some limitations which are ommitted from the main body of text which at least should have been acknowledged in this section. The authors state they have IRB approval in section 4.1 of their paper. I see no reason for additional review.