Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals

Summary

The authors proposed Du-IN, a speech decoding framework that incorporates a self-supervised pre-training stage and a classification stage. Experiments are conducted on 12 subjects with a task of word classification. Comparisons are conducted to demonstrate effectiveness.

Strengths

* (a) The paper is clearly written and easy to follow. * (b) The task of speech decoding using intracranial recordings is interesting.

Weaknesses

* (a) Although the task of decoding speech from intracranial recordings is an interesting topic, the algorithmic novelty of this paper is limited. The pretraining framework the authors proposed is very similar to BEIT other than the underlying data modality being images and sEEG. Simply applying an existing model BEIT to sEEG data with minor modifications to the backbone architecture does not meet the high standards required for acceptance at a top conference like NeurIPS. * (b) The comparisons conducted are insufficient. In particular, besides BrainBERT and Brant, the authors compared with, there are also many pre-training approaches for neurological signals. I list some more up-to-date approaches from a contrastive learning perspective and a masked autoencoding perspective. For contrastive-based approaches, there are TS-TCC [1] and [2]. For masked autoencoding based approaches, there are Biot [3] and Neuro-BERT [4]. As a matter of fact, the authors could at least follow the comparison setup in Brant for a fair comparison. * (c) Reproduction details are missing. The reported results on BrainBERT and Brant are ~7.5% and ~12.5%, as compared to the reported performance of ~63% of the proposed Du-IN. It is astonishing that the proposed Du-IN outperforms BrainBERT and Brant by almost 50%. Both BrainBERT and Brant are masked autoregressive based pertaining approaches. Although BrainBERT and Brant vary from Du-IN, it does not make sense that pretraining could lead to such a tremendous difference. The authors should report the hyperparameters used to reproduce Brant and BrainBERT. Otherwise, It is highly possible that the authors did not tune BrainBERT and Brant properly. * (d) Although the decoding of speech is an interesting question, it seems rather well-established in open vocabulary scenarios [5][6], decoding words of limited vocabulary traces back to at least ten years ago. **References** [1] Woo, Gerald, et al. "Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting." arXiv preprint arXiv:2202.01575 (2022). [2] Baevski, Alexei, et al. "Data2vec: A general framework for self-supervised learning in speech, vision, and language." International Conference on Machine Learning. PMLR, 2022. [3] Yang, Chaoqi, M. Westover, and Jimeng Sun. "Biot: Biosignal transformer for cross-data learning in the wild." Advances in Neural Information Processing Systems 36 (2024). [4] Wu, Di, et al. "Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining." IEEE Journal of Biomedical and Health Informatics (2024). [5] Willett, Francis R., et al. "A high-performance speech neuroprosthesis." Nature 620.7976 (2023): 1031-1036. [6] Metzger, Sean L., et al. "A high-performance neuroprosthesis for speech decoding and avatar control." Nature 620.7976 (2023): 1037-1046.

Questions

* Are decoding words the same as decoding pronunciations in [1][2]? Is it possible that the sentiment is associated with words that the decoding task is not speech but sentiment? **References** [1] Willett, Francis R., et al. "A high-performance speech neuroprosthesis." Nature 620.7976 (2023): 1031-1036. [2] Metzger, Sean L., et al. "A high-performance neuroprosthesis for speech decoding and avatar control." Nature 620.7976 (2023): 1037-1046.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors discussed the drawbacks of the proposed framework which is essentially close-set prediction rather than open-set. Potential negative societal impact was not mentioned.

Summary

SETTING: Decoding spoken speech from invasive neural recordings, to wit, sEEG. In particular, the MS aims to classify 61 independently produced (Chinese) words. APPROACH: The focus of this study is on unsupervised pretraining via autoencoding, with either (a) vector-quantization or (b) masking. Masks are 100-ms long and cover all channels. For both VQ and masking, the core model is a "spatial encoder" (linear + conv) followed by a (temporal) transformer. There are ~15 hours of data per subject. After pre-training, this model is trained end-to-end (on ~3 hours of labeled data) on the classification task. RESULTS: The architecture outperforms other architectures proposed for this task, and masking does appear to improve performance.

Strengths

This is a nice study and the results are compelling. Temporal masking looks promising as a generic technique for sEEG data. The dataset is extremely useful and is likely to become the basis of benchmarks for future research in this area.

Weaknesses

The paper does not address some important questions (admittedly there is not much space), which I list below. In addition to these, I list here some minor issues: --For comparing with other results in the decoding literature, it would be very useful to report cross entropy in addition to (or instead of) accuracy, since the latter can only be interpreted in light of the fact that there are 61 classes. --The reader shouldn't have to go to Appendix C.3 just to find out what the names in Table 2 mean. Also, what does "CLS" stand for? --The (vqvae+vq) and (vavae) rows in Table 2 should be marked as "pre-trained." --The MS calls the token's temporal width the "perception field." Is this a standard term? Perhaps the authors mean "receptive field"? --The authors use "codex" where they mean to use "code."

Questions

(1) The spatial encoder: The name suggests that its convolutions are across channels, but this is inappropriate across sEEG electrodes (and within electrodes should take into account the spatial layout of channels). Can the authors clarify this? (I see in Appendix H that the *linear layer* combines channels.) (2) The authors give some reasons for the superiority of their model over BrainBERT and Brant, but it would be helpful to have some insight on the performance relative to other models as well. E.g., (a) The MS focuses on pretraining, but in fact it appears that the biggest gain over the baselines comes from the architecture itself (Du-IN trained fully supervised)---an improvement of ~25 percentage points over the CNN-BiGRU! Can the authors shed light on this difference? (b) The EEG-conformer likewise uses a "temporal transformer," but performance is ~10 percentage points lower than the supervised-trained Du-IN. What do the authors see as the key difference here? (c) The authors mention LaBraM but don't compare Du-IN to it. It would be nice to see this comparison as well.... (3) Intracranial EEG data is generally (but not always!) filtered into the range of ~75-150, because otherwise the low frequencies, which have higher power, tend to dominate. (The low frequencies can be provided as separate channels.) This is a problem because low frequenceies generally contain much less information about speech. In this study, frequencies below 70-Hz are not filtered out. Indeed, it appears that reconstructions under vector quantizations retain mostly low-frequency information (Fig. 7). Can the authors explain these results? (4) Channel selection: (a) "To streamline, we utilize these top 10 channels for both pre-training and downstream evaluation." Does that mean that, for each subject, the best 10 electrodes (for that subject) were used? (b) If the choice to use just 10 channels was made on the basis of decoding results on the test set, then the results are biased upwards. Can the authors confirm that this choice was made based on validation data? (Is Fig. 4b on validation data?) (c) How does channel selection interact with model choice? Were all models in Table 2 evaluated with just 10 channels? Does their performance as a function of #channels also resemble Fig. 4b? (5) It seems quite possible from Fig. 4a that the authors are decoding speech perception as much or more than speech production, since many of the most highly contributing electrodes are below the Sylvian fissure. This would limit the use of this decoder as a speech prosthesis for people who cannot speak. Can the authors comment on this? On this head, although Fig. 4 is a nice summary, but it would be very useful to have a list/table of the electrodes used for decoding for each subject, labeled by cortical area.

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

N/A

Summary

The authors introduce Du-IN, a novel decoding technique for speech production the leverages sEEG data. To achieve their goals, the authors gathers a novel dataset. The Du-IN model learns multi-variate (across channel) representations from sEEG signal using a discrete cookbook and a masked modeling based self-supervised training. The proposed algorithm achieves SoTA performances on the speech 61-words classification task. Furthermore, analysis of optimal model configuration and ablation studies reveal that best performances are achieved with electrodes in brain regions known to be involved in speech production.

Strengths

The paper presents strong results of high significance for the field of speech decoding. In particular: - The authors have collected a novel sEEG dataset, which will be made publicly available, hence fostering further progress in the field of brain signal decoding. - The authors propose a novel framework for sEEG signal decoding - the Du-IN algorithm, which achieves SoTA performances, thus significantly advancing the field of brain decoding. - Authors provide additional ablations studies to validate design choices and recover known results from Neuroscience, strengthening the overall soundness of their technique. - The paper is well written and clearly organized, with a detailed Appendix providing further details.

Weaknesses

- In Table 2 both the Du-IN (vqvae+vq) and Du-IN (vqvae) are marked as not pre-trained. However in the Appendix C.3 it is stated that their Neural Encoder weights are loaded from the pre-trained Du-IN VQ-VAE model. - Minor spelling: Line 156 DuIR $\to$ DuIN

Questions

- There seems to exist substantial variation of decoding accuracy across subjects. Is this difference due to different electrode locations, or number of electrodes (even if only 10 channels seem to be optimal) or are other concerns with data quality relevant? - Could the manual selection of channel be partially subsumed by a sparsity regularization term on the channel-to-embedding projection matrix?

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have adequately addressed the limitation of their work with a dedicated section in the paper.

Summary

The authors introduce a neural network architecture, a self-supervised learning (SSL) method for pre-training models for sEEG decoding and a novel dataset. - The architecture starts with segmenting the sEEG signal along the time dimension, followed by a linear module combining information across the different sEEG channels into embedding vectors and ending with a transformer with attention along the time dimension. - The SSL method has two stages: - 1. The architecture followed by an additional decoder network is trained to reconstruct the original signals (autoencoder). However, the output of the transformer is not directly passed to the decoder, instead, each token is replaced with the closest embedding vector from a trainable codebook (vector quantisation). - 2. Time windows of the input are masked and the architecture is trained to predict which codebook embedding represents best each masked window. - The dataset introduced contains sEEG recordings of 12 subjects while they were speaking aloud Chinese words from a vocabulary of size 61. In the experiments, the models were pre-trained using the SSL tasks using unlabelled data from a single subject and fine-tuned on labelled data from the same subject for classifying the 61 words. The authors conducted different ablation studies regarding pre-training (no pre-training, and only step 1 of pre-training using quantized features for fine-tuning or not), dataset, codex size and dimensionality, and length of the time windows. They also compare with pre-existing methods, two including a pre-trining phase and two without. The two with pre-training are pre-trained on all 12 subjects.

Strengths

*Significance*: As pointed out by the authors, sEEG speech datasets are rare. Their promise of publishing the dataset they introduce here is good news for the community as it will lower the entry threshold for future research. Additionally, they demonstrate how SSL-based pre-training allows improving performance compared to only supervised training. *Clarity*: The text has a good structure and is well-written. The figures also help in understanding the method. *Originality*: SSL techniques are still rather novel to the BCI community. This work sheds light on some of them.

Weaknesses

**Major** 1. The authors pre-train their method on the test subject (Lines 201-202) whereas the baseline methods with pre-training (BrainBERT and Brant) were pre-trained on all 12 subjects (lines 511-518). This does not allow for a fair comparison. Additionally, this choice is not properly justified. 2. The authors mention in section 4.5 Ablation study, that they found optimal hyperparameters for their architecture (codex of size 2048 and dimension 64, and perceptual field of 100ms). They use these hyper parameters to compare to the other methods (see Table 5). However, the manuscript does not mention the use of a holdout set for conducting this ablation study. This is problematic for two reasons: 1. This suggests the method could have a low capacity to generalise be overfitting to their dataset 2. This is not fair for the other methods as no hyperparameter optimisation is mentioned for them. 3. The authors mention they resampled their signals to 1000Hz (line 210). However, some architecture compared have been introduced with different sampling rates (Brant used 250Hz and EEG-Conformer used 200 and 250Hz). The text does not mention any adaptation of the architectures compared or of the sampling rate. This is problematic as a convolution kernel supposed to cover 1s of signal at 250Hz will only cover 0.25s at 1000Hz. 4. Some aspects of the method are undefined or unclear. Please see the Questions section below. These aspects need to be clarified in the manuscript. **Minor** 1. Because of the flaws in experimental design described above, the statements about the superiority of their method compared to others (lines 14, 71, 248 and 318) are unjustified. 2. All figures should be in a vectorized format. 3. The text in the figures should be approximately constant and the same as the article’s text. It is too small in figures 5 and 7. It is on the limit in figures 1,2 and 3. 4. The term “fusing channels” is used lines 16, 39, 108 and 117 but is rather unclear. It is only explained line 134. 5. Line 29: provide examples of non-invasive recording methods (EEG, MEG…) 6. Section 2.3, additional publication the authors should be aware: 1. A preprint of Brant-2 is now available https://arxiv.org/abs/2402.10251 2. SSL methods trained on EEG with attention mechanism for combining information across channels: EEG2Rep (https://arxiv.org/abs/2402.17772) and S-JEPA (https://arxiv.org/abs/2403.11772). 7. Line 54: Indeed, there is a lack of open sEEG recordings but some still exist. In reference [1], they mention they use a public dataset, in [16] they mention their data is available upon request and Brant-2 (not cited here), they use a 26-subject public dataset. Existing sEEG datasets should be cited. 8. Line 98: typo “Metzger1” 9. As I understand, the terms “Du-IN Encoder”, “neural transformer”, “neural encoder”, and “neural tokeniser” all refer to the architecture in figure 2. If this is indeed the case, it would improve readability to chose one of them and stick to it throughout the manuscript. 10. Line 166: missing “to” in “$z_c$ to get” 11. Figures 2 and 3: It would improve readability to place the different variables ($e^p_i$, $e_i$, $e^t_i$, $c_j$,…) on these figures. 12. Line 162: for consistency, you could use $j$ as index for the codex instead of $i$. 13. Lines 180-193: the objective function of the masked training is unclear (see questions). 14. Line 247: Please mention that the baseline descriptions can be found in appendix C.3. 15. Line 301: The research line of Feng et al. was already explored by Christian Herff in 2015 (https://doi.org/10.3389/fnins.2015.00217) and is still today (https://www.sciencedirect.com/science/article/pii/S1053811923000617). It could be interesting to draw a parallel. **Overall comment** The introduction of a novel sEEG dataset is valuable but the evaluation of the machine learning algorithms that are introduced is poor. Therefore, I believe NeurIPS might not be the best fit for this publication.

Questions

1. Lines 134-140: The description of the spatial encoder is ambiguous: 1. Does “linear” mean fully-connected layer? Appendix C.1 mentions “Linear Projection: 10->16” which would mean that an operation is applied to each channel independently and that the layer is actually not a fully-connected one. 2. Convolutions are mentioned, but over which dimensions? Over time? Channels? Both? 2. Figure 3: What are the numbers below “l2-norm” and bellow “neural regressor” corresponding to? 3. Why did you use a different signal length for pre-training and for downstream classification? 4. Line 210: Why did you use a sampling rate of 1000Hz? With a low-pass filtering at 200Hz, a sampling rate of 400Hz would be sufficient 5. Line 211: Did you apply the normalisation on each channel independently (i.e., mean and std computed for one channel only) or using all channels together? 6. Line 222: How is the training time split between VQ-VAE and MAE training? 7. Line 228, “selected from the validation set”: does this mean early stopping? Was it performed using the validation loss, validation accuracy or another metric? 8. Table 2, “Du-IN (mae)”: this row corresponds to the scores obtained after a VQ-VAE training followed by a MAE training. Is the first step exactly “vqvae” or “vqvae+vq”? To lift the ambiguity, you could rename the row to “Du-IN (mae+vqvae)” or “Du-IN (mae+vqvae+vq)”. 9. Why refer to your method as a “VAE”? As I understand, for a given input, the “reconstructed sEEG signals” are obtained in a deterministic manner (please, correct me if I’m wrong am wrong). The term “variational” in VAE suggests there is a stochastic operation involved. 10. Lines 180-193: the objective function of the masked training is unclear: 1. Line 188: “neural tokens” is ambiguous. What is being predicted? 2. Equation (7) and (8): how are the $z_i$ obtained? 3. Equation (8): $m_i$ is undefined. What is the meaning of $m_i$?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors included a Limitations section in their manuscript in which they already mention other approaches to speech decoding, at the phonemes level and at a semantic level. It could be relevant to remind the reader that sEEG recordings are invasive. Therefore, they are only done with patients who would require them anyway for medical reasons (epilepsy monitoring, etc…). And this presents a major obstacle to making these methods widely accessible and usable.

Summary

Due to the locality and specificity of brain computation, existing models that learn representations of a single channel yield unsatisfactory performances on more difficult tasks like speech decoding. This paper tries to build a stereoElectroEncephaloGraphy (sEEG) foundation model with Vector Quantization (VQ) pre-training. We hypothesize that building multi-variate representations within certain brain regions can better capture the specific neural processing. To this end, we collect a well-annotated Chinese word-reading sEEG dataset, targeting language-related brain networks over 12 subjects, and we developed the Du-IN model that can extract contextual embeddings from specific brain regions through discrete codebook-guided mask modeling. Extensive comparison and ablation experiments verify that our model achieves SOTA performance on the downstream 61-word classification task.

Strengths

**(S1)** This paper provides an interesting solution for the sEEG learning framework that models multiple sEEG channels by a spatial encoder and quantizes sEEG signals from the temporal aspect. To evaluate the proposed Du-IN framework, the authors collect and contribute a Chinese word-reading sEEG dataset for speech decoding (as word classification). **(S2)** The manuscript is clearly structured and easy to follow. Empirical analysis and ablations of proposed modules provide comprehensive understandings.

Weaknesses

**(W1) The employed techniques in the proposed framework lack novelty to some extent.** Firstly, except for the spatiotemporal modeling architecture for sEEG, the VQVAE tokenization and masked code modeling pre-training framework is adopted from popular BeiT frameworks [1, 2], which has been adopted to various scenarios like [3, 4, 5]. Some specially designed modules are necessary for introducing the BeiT-like framework, and the current version is not effective and well-designed from my aspect (view W2 for concerns). Secondly, the authors regard the speech decoding tasks as word classification, which might be too coarse to decode the actual meanings of human speeches. Therefore, the conducted decoding task might not be sufficient to verify the effectiveness of the Du-IN framework. **(W2) Weak comparison experiments.** Firstly, the authors only provide a few supervised learning and self-supervised pre-training baselines in Table 2 with a single-word classification task (as speech decoding). More relevant contrastive learning [6] and masked modeling pre-training methods [7] for sEEG should be considered, which should also be referred to in the background section. Meanwhile, some supervised baselines with more parameters can be added for a fair comparison as Du-IN models. Secondly, the results of some baseline methods are very bad (e.g., BrainBERT and Brant). It’s hard to believe that pre-training models will yield worse results than random initialized supervised learning models. I suggest the authors conduct a fair and comprehensive comparison experiment. **(W3) Not explicitly modeling multiple subjects.** Since the invasive sEEG signals for different subjects have different channels, I couldn’t find an explicit modeling strategy in the proposed encoders and pre-training framework that considers fusing or selecting information from multiple subjects. There might be some challenges, like representation conflicts and missing channels of different subjects, which will decrease the overall cross-subject performances. I suggest the authors conduct such analysis of multiple subjects, e.g., cross-validation of several subjects and transfer learning of pre-trained representation across datasets of different subjects. ### Reference [1] MAGE: Masked Generative Encoder to Unify Representation Learning and Image Synthesis. CVPR, 2023. [2] BEIT: BERT Pre-Training of Image Transformers. ICLR, 2021. [3] Human Pose as Compositional Tokens. CVPR, 2023. [4] MAPE-PPI_Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding. ICLR, 2024. [5] VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling. ICML, 2024. [6] Time-series representation learning via temporal and contextual contrasting. IJCAI, 2021. [7] Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining. IEEE Journal of Biomedical and Health Informatics, 2024. ==================== Post-rebuttal Feedback ==================== After considering the authors' rebuttal and other reviewers' comments, I decided to keep my score because some of my concerns were not solved directly (e.g., W1 and W3). Meanwhile, the authors have provided some useful results (e.g., more comparison experiments) to support the effectiveness of the proposed methods. Since the revision could not be uploaded, I encourage the authors to improve the manuscript further for submission to the next conference or journal.

Questions

**(Q1)** For different subjects with different channels, does the spatial encoder need different parameters for each subject? And does the output embedding of each subject in the same shape? **(Q2)** Does the authors conduct ablation studies of masked code modeling, e.g., the masking ratio and pre-training epochs? Since the proposed Du-IN is a framework with several networks and the pre-traning & fine-tuning pipeline, it’s better to analyze and clearify each module for better understanding and practical usage.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Although the authors discussed some limitations in Sec. 5, potential negative societal impact of the proposed dataset and the Du-IN framework were not clearly discussed. More details of the implementations and the collected dataset can be provided for the sake of the neuroscience community.

Response to Ethics Reviewer i4Vv

Thank you for the valuable comments from the perspective of ethics! We agree with the statement, "considering how to make this method applicable to a wider population, as sEEG is an invasive procedure." And we think that our method improves the feasibility of invasive BCI technology to some extent. Since our method is the first to demonstrate the feasibility of decoding speech using about one sEEG electrode, which has greater clinical application value than before. For patients, who have lost their ability to communicate and take care of themselves in daily life due to neurological diseases (locked-in syndrome, amyotrophic lateral sclerosis, etc.), our method offers a solution that causes less damage to the brain than ECoG or microelectrode array technology. We have added these to the Broader Impact Section in our revised paper, which is available in the Anonymous Github Repo at line 328 in the initial manuscript. Additionally, developing a high-performance speech decoder using non-invasive technologies like EEG or MEG is highly appealing, as it significantly reduces the risks for subjects. Although previous studies[R1] have attempted speech decoding using EEG, the performance has been limited by the low signal-to-noise ratio. Our study aims to explore the feasibility of high-performance speech decoding using sEEG technology, which could serve as a reference for future work with non-invasive methods like EEG and MEG. Moreover, since our study found that only specific brain regions are involved in speech decoding, we hope this neuro-inspired channel selection can be applied to non-invasive technologies like EEG or MEG. This would enable our Du-IN VQ-VAE to extract fine-grained brain region states. Another approach could be using product quantization[R2] to enhance Du-IN VQ-VAE, allowing for explicit decoupling of states from different brain regions, though its effectiveness is still under investigation. **References:** [R1]Jo H, Yang Y, Han J, et al. Are EEG-to-Text Models Working?[J]. arXiv preprint arXiv:2405.06459, 2024. [R2]Jegou H, Douze M, Schmid C. Product quantization for nearest neighbor search[J]. IEEE transactions on pattern analysis and machine intelligence, 2010, 33(1): 117-128.

Response to Ethics Reviewer cWxa

Thank you for the careful Ethics Review and we believe these reviews are helpful for our submission to avoid ethical issues. We have extended the discussion accordingly at Ethics Statement Section in our revised paper, which is available at Anonymous Github Repo at line 328 in the initial manuscript. We agree with the statements, “The paper should include a detailed description of the informed consent process” and “The paper should provide a clear explanation of the steps taken to anonymize the data and protect participant privacy.” Our informed consent signing process is as follows: 1. If the experimental participants are adults and have full civil capacity, we will ask them to sign a written informed consent after the participants have fully informed consent; 2. If the experimental participants are minors or do not have full civil capacity, we will ask the participant's legal guardian to sign a written informed consent after the participants and their legal guardians have fully informed consent. And our informed consent form includes the following points: 1. Contact information of research institutions and researchers; 2. Research direction and purpose; 3. Risks involved in the research; 4. Personal information, data, and usage methods to be used in the research; 5. Privacy protection statement (all personal identification information (PII) will not be disclosed); 6. Data storage statement (retained after deleting all personal identification information (PII)); 7. Voluntary statement of participants; 8. Statement that participants can withdraw unconditionally at any time. Our data storage and protection procedures include the following processes: 1. Our data collection, transfer, and analysis tasks are only completed by researchers who have signed relevant confidentiality agreements; 2. The collected raw data will be copied twice as soon as possible, one copy to a storage computer that is not connected to the Internet and encrypted, and the other copy to a mobile hard disk and encrypted and stored offline; 3. The use of the data is only authorized to the research leader and the main researchers (less than 5 people), among which the main researchers can only access data that does not contain personal identification information (PII); 4. After the study is completed, all personal identification information (PII) on both nodes (storage computer, mobile hard disk) will be deleted immediately. To prevent unauthorized access or possible data leakage, we use double encryption on the storage computer, that is, a static password and a dynamic password (received by mobile phone or email); physical isolation is used on the mobile hard disk, that is, it is locked in a filing cabinet, and the key is only kept by the research leader and the main researchers. We have added the above content to the Ethics Statement Section in our revised paper, which is available at Anonymous Github Repo at line 328 in the initial manuscript.

Feedback to Authors' Rebuttal

Thanks for the comprehensive responses to my concerns! I acknowledged and agreed with some points, e.g., I appreciate the additional cooperation experiments for W2 (it is better to add them to comparison tables). However, I found several essential issues not well solved and decided to keep my score at the current stage. * (W1) The concerns of the novelty of the proposed method and whether it could called decoding speech still existed. Despite previous works in neuroscience applications being accepted with a naive design of VQ-based methods, it does not mean more recently proposed research should follow their standards. The NeurIPS community requires innovation of the proposed methods or the research problem to some extend. Meanwhile, this work regards speech decoding as a coarse word classification task, which could be tackled well with a more complex formulation. Therefore, I suggest the authors to improve their method and resubmit to the next conference or a journal. * (W3) The issue of multiple subjects has been a challenging problem of sEEG-related tasks for a long time. I think the authors should take it into consideration or employ a more complex decoding formulation, as I mentioned in W1. * Regarding responses to W2, Q1, and Q2, I have no further concerns and suggest the authors revise the manuscript with these discussions.