Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI

Summary

This manuscript introduces the Large Brain Model (LaBraM), an innovative approach designed to enhance the capabilities and scalability of EEG-based deep learning models in brain-computer interaction (BCI). The authors explore the potential of Large EEG Models (LEMs) for universal perceptual capabilities through unsupervised pre-training, followed by fine-tuning for various downstream tasks. However, EEG data presents unique challenges like mismatched electrodes, diverse data lengths, and low signal-to-noise ratios. To address these, LaBraM implements cross-dataset learning by segmenting EEG signals into channel patches and employs vector-quantized neural spectrum prediction for rich neural tokenization. This is further enhanced by pre-training neural Transformers to predict original neural codes for masked EEG channel patches. LaBraM is presented in three sizes, Base (5.8M), large (46M), and Huge (369M), with the "Huge" variant being the biggest EEG model to date. Pre-trained on approximately 2,500 hours of EEG signals from around 20 datasets, LaBraM showcases superior performance across several downstream tasks, outshining existing state-of-the-art methods. The authors also delve into the data requirements for training various sizes of the model.

Strengths

1. The manuscript represents a significant contribution to the field, pioneering the development of the largest Large EEG Model (LEM) for EEG decoding. By effectively addressing two pivotal challenges in this field — the utilization of large-scale EEG data and the required data volume — this work lays a solid foundation for future researchers in this field 2. The clarity and coherence of the presentation are commendable, facilitating an in-depth understanding of the proposed methodologies. 3. A notable strength of this work is the comprehensive experimental evaluation. The authors have conducted a plethora of experiments, providing supplementary results, detailed ablation studies, and a thorough discussion on hyperparameter settings in the appendix. 4. The Figures, complemented by lucid annotations, further enhance the comprehensibility and accessibility of the work to the audience.

Weaknesses

1. While the authors acknowledge the challenge of varying configurations in EEG data collection, particularly concerning electrode variations across different datasets, the manuscript does not provide a comprehensive solution to this challenge. Specifically, the Temporal & Spatial embedding section offers an ambiguous explanation regarding spatial embedding (SE). The method appears to encode channels based merely on their sequential order, which may not accurately represent the channel's functional significance or location in the brain. Given the heterogeneity in electrode configurations and positions across various datasets, it's imperative for the authors to elucidate how their approach effectively manages this inconsistency, and use qualitative results to demonstrate the model has learned different electrode configurations. 2. The paper draws parallels to another LEM, BIOT, developed by Yang et al., and even follows some experimental settings from the same. Given the apparent similarities, it remains unclear as to what drives the performance improvements of the proposed model — is it solely attributed to the increased model size, enhanced training data volume, or specific architectural designs? A more in-depth comparative discussion and analysis between the two models would be beneficial to ascertain the genuine contributions and innovations of the current work.

Questions

Will the pre-trained checkpoints be released for open-source development as well?

Rating

8: accept, good paper

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Summary

The authors introduce a Transformer-based model and a pretraining methodology to learn representations from EEG data in a self-supervised manner on a large collection of datasets. The approach combines two parts. First, a vector-quantized tokenizer (similar to a VQ-VAE) is trained to quantize 1-s single-channel EEG patches in order to minimize a regression objective in the spectral domain (i.e. on the amplitude and phase of the input patch). Second, a Transformer encoder is pretrained on a self-supervised task in which the model must predict the tokens that correspond to masked input patches. Three variants of the proposed model (with 5.8 up to 360M parameters) are pretrained on a diverse dataset of more than 2,500 h of EEG recordings and then finetuned on one of four supervised classification or regression downstream tasks. The proposed models outperform existing baselines on these tasks. Ablations on the amount of pretraining data, tokenization approach, masked prediction loss and masking support the proposed pretraining task and model configuration.

Strengths

Originality: The proposed approach follows logically from existing work on pretraining Transformers on a corpus of combined EEG datasets, however the combination of a vector quantization tokenizer, a regression objective in the spectral domain and the pretraining on a very large set of EEG recordings appears novel. Quality: The paper is of good quality, with strong core results showing the performance of the proposed approach, and multiple supporting analyses and ablation studies that support the methodological choices that were made. A few claims might not be completely supported by the results (see Weaknesses). Clarity: The paper is overall clearly written, with mostly clear descriptions of the proposed approach and of the results. Some methodological points require clarification (see Weaknesses, first point). Significance: Overall, this study is an important step towards bridging the gap between the approaches used in large language modeling and EEG processing. The results on “EEG scaling laws” (Section 3.6) are a first attempt at answering an important question in the field of deep learning and EEG.

Weaknesses

- Some methodological points require clarification, e.g. the impact and choice of windowing/tokenization hyperparameters (Q1), the learning and reuse of spatial embeddings (Q3), the potential overlap between pretraining and downstream datasets (Q5) and the sampling of examples during pretraining (Q8). - I don’t think the results are clear enough to deduce what is claimed in the analysis of Figure 5, i.e. that the performance saturates for the base model at 500 h and for the large model at 2000 h. For instance, the Large models seem to continue learning over 2000 h on TUEV. Similarly, the Base model might not be saturated yet; the performance curve seems pretty noisy. Maybe repeating this analysis with a log-scale would be clearer (1h, 10h, 100h, 1000h, 2500h). - The use of the term “BCI” (e.g. in the title) is confusing as this typically refers to a subset of the tasks/datasets considered in this work. For instance, the term BCI is usually used to describe cases where there is an interface between brain activity and a computer that bypasses normal communication pathways. Under this definition, tasks such as pathology detection (TUAB) or event detection (TUEV) are not BCI tasks. I would recommend adapting the language of the manuscript to make this clearer.

Questions

1. Some of the hyperparameter choices for the windowing and tokenization steps are not clear to me. First, what were the window strides ($s$ in first paragraph of Section 2.1) used for the different datasets? Tables 3 and 4 report a “Data stride” value but I’m not sure whether that’s the same thing. Second, what is the impact of the selected patch size and patch stride on performance, i.e. are these choices important? Related to the point about how symmetric masking might be providing regularization that is useful for larger models, would a smaller window and/or stride help create more pretraining examples? 2. It is not clear to me whether the weights of the temporal encoder from the tokenization step are reused in the pretraining step, or if only the architecture is the same. From the dimensions of the large and huge models I assume the weights could not be reused as the sizes are not the same. 3. My understanding is that new spatial embeddings must be learned from each montage that is seen (i.e. the $i$ in Equation 2). How many different spatial embeddings were learned during pretraining? Also, what spatial embeddings were used in the different ablations of Table 10 if the model didn’t have a chance to learn a spatial embedding for TUAB and TUEV (or were examples from this EEG montage already seen in the pretraining data)? 4. Section 2.3: What is the self-distillation loss term? It doesn’t seem to be in the final pretraining objective of Equation 12. 5. Is there an overlap between TUAB/TUEV and the different training sets taken from the Temple University EEG Corpus (TUAR, TUEP, etc.)? If so, could this explain why including TUAB and TUEV in the pretraining set didn’t change the results much (Section 3.5)? I believe this shouldn’t impact the comparison with BIOT as the reported results from the BIOT paper appear to be from the model also pretrained on TUAB and TUEV. 6. Related to the previous question, looking at Figure 4 it looks like adding TUEV to the pretraining dataset actually negatively impacts downstream performance. Is this effect significant and if so, what could be driving this decrease in performance? 7. Section 3.2 on preprocessing: the notch filtering at 50 Hz will not adequately remove power line interference in datasets collected in North America, such as the TUEG datasets. I expect results to improve if the authors correctly notch filter those datasets at 60 Hz instead. 8. How are the pretraining examples sampled? Since this is not described in the manuscript I would assume sequences were sampled uniformly across the entire training corpus, however I was wondering if the authors have considered more balanced sampling schemes, e.g. taking datasets, recordings and/or experimental paradigm-related information into account when sampling. 9. Where did the baseline results for Table 6 come from? They don’t seem to be in the BIOT paper. 10. A few typos: - Figure 2: “Fuorier spectrum” - Appendix J: “PRE-TRAINGING” - Equation 14: Missing closing parenthesis

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Summary

The paper proposes a method and model for self-supervised training on large-scale EEG data coming from various datasets with different electrode configurations. First they train a neural tokenizer which learns to compress the EEG signal into vector-quantized encodings and reconstruct the amplitude and phase of the EEG signal from those. Then, given the trained neural tokenizer, a model is trained to reconstruct the vector quantized encodings of an EEG signal decoded from a masked version of the same encodings. After these pretraining phases, the model is finetuned and evaluated on a downstream task. The authors report improved decoding accuracy on pathology diagnosis and clinical EEG event type classification compared to published work.

Strengths

* Interesting method for self-supervised learning from EEG data * Large collection of publicly available datasets * Improved results over other self-supervised methods * Analysis of scaling behavior

Weaknesses

1) Some papers have not been mentioned that work on heterogeneous datasets: * [Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling](https://openreview.net/forum?id=hiOUySN0ub) * [EEG Decoding for Datasets with Heterogenous Electrode Configurations using Transfer Learning Graph Neural Networks](https://arxiv.org/abs/2306.13109v1) * [Generalizable Movement Intention Recognition with Multiple Heterogeneous EEG Datasets](https://ieeexplore.ieee.org/document/10160462) 2) The choice to use mean squared error on phase values is strange to me. Due to their cyclical nature, very nearby phases, e.g., -pi+eps,pi-eps would get a large squared error that would also depend on whether one uses phases from 0 to 2pi or -pi to pi etc. So would make more sense tome to either always only use the minimum distance, so if one would put the phases on a unit circle the minimum distance on the circle, or maybe even regress fourier coefficients instead of amplitude/phase. I even wonder what would happen if one just does not put any loss on the phase prediction at all, only on the amplitudes that would be interesting to check as well. 3) Why bold lowest std in table 1 and 2, better remove that, is rather confusing 4) Font could be a bit bigger in Figure 1 and also parts of Girue 2 (e.g., channel names on bottom) Fig 2 the amplitude phase plot on top right is confusing to me. Symmetric in Figure 2 not symmetric

Questions

1) I assume Table 1/2 only compares to other self-supervised models? That should be written a bit more explicitly otherwise there are other papers worth citing: https://www.springerprofessional.de/en/chrononet-a-deep-recurrent-neural-network-for-abnormal-eeg-ident/16824220 https://www.sciencedirect.com/science/article/pii/S1053811920305073 Might be in any case good to also show a purely supervised baseline. 2) Are TUAB and TUEV disjoint recordingwise from TUAR TUEP TUSZ and TUSL?

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good