Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signals

Response to Reviewer hHPV Part I

Thank you for your helpful feedback! We appreciate that you find our method novel and recognize the clarity of our communication. We are also grateful for your acknowledgment of the transparency in our work, including the use of public datasets and the sharing of our codebase, which we hope will facilitate further exploration and analysis. Below, we briefly respond to each of your concerns. If, after reading our responses, you still feel that any aspects of our work could be improved, we would be grateful for your further suggestions to help enhance the paper. We also really appreciated that the reviewer clearly differentiated the issues into sections, so we can address them precisely, one by one. Our response has five sections. # Section 1: Response to Issue 1 We sincerely thank the reviewer for their detailed feedback and for conducting their own analyses to benchmark our work. We greatly appreciate the effort invested in this review, as it highlights important areas for clarification and improvement. Below, we address the concerns raised: ### New Baselines We have incorporated statistical baselines, CLAP, and TF-C in our evaluation to provide a more comprehensive comparison. These additional baselines offer clearer context for understanding our model's performance. Results, including comparisons with these baselines, have been updated in the manuscript as well. To further clarify our choices: modeling multivariate wearable signals, which include a wide range of modalities such as heart, brain, and physical motion signals, poses unique challenges, as no universally recognized open-source baseline or state-of-the-art model exists in this domain. To address this, we selected baselines from three representative paradigms: 1. **Statistical Features** Handcrafted statistical features are a traditional yet widely used method in signal processing literature. We included this baseline as a sanity check to benchmark our model against simpler approaches. 2. **SSL Time-Series Frameworks** Wearable sensory data is inherently sequential, making time-series modeling strategies relevant. We compared our approach with Chronos, a state-of-the-art time-series forecasting framework, and TF-C, a commonly used self-supervised learning method for time-series data. 3. **Spectrum-Based Methods** Spectrogram-based modeling has demonstrated state-of-the-art performance in tasks such as music classification and physiological signal analysis. CLAP was selected as a representative baseline for this paradigm. ### Linear Probing Algorithm To ensure better reproducibility, numerical stability, and fairness in performance comparison, we have updated our linear probe implementation. Previously, our results were based on a custom linear layer optimized with the Adam optimizer. To standardize the evaluation pipeline, we now utilize well-established methods for solving the linear probe tasks. Specifically: - **Classification tasks** are addressed using Newton's method with conjugate gradient. - **Regression tasks** (e.g., vital signs prediction) are solved using Cholesky decomposition with a closed-form solution. These updates provide a more robust and interpretable framework for evaluating our model's representations across various tasks. The updated results is attached in the next part of response, and we've also updated it in the Experiment section.

Response to Reviewer hHPV Part III

# Section 2: Response to Issue 2 We truly appreciate the reviewer’s time and effort in evaluating our model, especially for actively testing our dataset, which speaks to their commitment to a thorough review. We understand the challenges inherent in small datasets, and while we are grateful for the feedback, we do feel that drawing strong conclusions based solely on these limited examples may not offer a complete picture of our model’s capabilities. On the topic of demographic features like age and BMI, we respectfully acknowledge that these variables are important for application-specific modeling. However, as our work focuses on advancing representation learning for wearable sensor data—without relying on demographic inputs—this falls outside the scope of our study. As pioneers in developing a foundation model for this type of data, our primary goal is to establish a versatile, scalable approach. Application-specific improvements are certainly worthwhile, but they are more appropriate for future work that dives deeper into individual tasks. It is important to note that demographic and wearable data capture distinct types of information, and their combination is often addressed in more specialized research, which, as we noted, is not the focus of our study. We acknowledge that our zero-shot performance is not yet promising at this stage, as discussed in the limitation section. However, we hope this initial attempt provides valuable insights for the community and practitioners on how to leverage the representations from our signal encoder. It is clear that it will not perform on par with state-of-the-art signal-text alignment models, as those models are trained from scratch with paired input, which is difficult to obtain in wearable sensory data. Therefore, the lightweight connector module we proposed is a preliminary attempt, given these practical constraints. We look forward to further research and improvements in this direction and hope that our work can inspire future advancements. # Section 3: Response to Issue 3 This is not our primary focus. The use of `[CLS]` is simply a way to highlight that the model is agnostic to the type of sensor input. The feature relevance and waveform visualizations are meant to set up the subsequent analysis with nonlinear dynamic metrics (NLD), which provides insights into how the model behaves across different layers. This approach allows us to quantify and examine the features learned by the model—for example, shallow layers capturing more periodic features, and deeper layers capturing more unpredictable patterns. We are not claiming superiority in terms of embeddings but using this method to analyze the model’s behavior. We acknowledge that the figures could have been misleading, and we have relocated them to the appendix to clarify their purpose. This ensures that the main text now focuses on key aspects like model performance, technical details, and a discussion of limitations.

Response to Reviewer hHPV Part IV

# Section 4: Response to Issue 4 We appreciate the reviewer’s insightful comment regarding the size of the pre-training dataset. To address this concern, we have significantly expanded both the pretraining and downstream evaluation datasets: ### Pretraining Data Originally, for computational efficiency, we had not utilized all available samples in our pre-training dataset. To address this, we have now included all available samples and supplemented the dataset with several additional collections, including datasets containing both ECG and PPG signals, as well as a large multi-channel EEG dataset with over 50,000 samples. As a result: - The pre-training dataset has grown from **37k samples (62 hours)** to **241k samples (402 hours)**. - Leveraging time series augmentation techniques detailed in the appendix, we further expanded the dataset to **2.5 million samples (approximately 4000 hours)**. All experiments were re-run with the model retrained on this significantly larger pre-training dataset. ### Downstream Data We incorporated several new datasets to further validate the model's performance. These include: - An EEG dataset with 500 subjects and five tasks. - A dataset focused on muscular disorders. - A new ECG abnormality detection dataset. This expanded evaluation framework ensures a more comprehensive assessment of the model across diverse health-related tasks. ### k-Fold Cross-Validation on Datasets with small number of subjects. In response to the reviewer’s suggestion of using k-fold cross-validation stratified by subject ID for tasks involving small number of subjects (such as Driver Fatigue Detection with N=12), we have conducted additional experiments. For these experiments, we selected all downstream tasks with a subject count of 30 or fewer. Importantly, we note that a small number of subjects does not necessarily imply a small number of samples, as some datasets (e.g., Wesad and UCI HAR) have a large number of samples despite fewer subjects. The results from these experiments consistently show that **NormWear outperforms all baselines**, and these findings are reflected in **Table 12 of Appendix C** in the updated manuscript. | Downstream Tasks | Statistical | Chronos | CLAP | TF-C | NormWear-L (Ours) | |-------------------|--------------------|--------------------|-------------------|-------------------|-------------------| | WESAD | 79.992 ± 0.707 | 83.332 ± 0.841 | 87.824 ± 0.463 | 82.701 ± 0.536 | **89.585 ± 0.683** | | UCI-HAR | 95.602 ± 0.148 | 91.956 ± 0.256 | 96.864 ± 0.175 | 97.382 ± 0.138 | **98.179 ± 0.06** | | DriverFatigue | 69.614 ± 1.138 | **72.48 ± 2.848** | 66.251 ± 0.471 | 65.026 ± 1.198 | 68.971 ± 1.32 | | GAMEEMO | 64.281 ± 1.292 | 56.694 ± 0.878 | 64.119 ± 0.543 | 62.925 ± 0.999 | **67.863 ± 0.72** | | Noninvasive | 92.83 ± 0.386 | 92.223 ± 0.356 | 92.612 ± 0.272 | 88.707 ± 0.622 | **93.381 ± 0.516** | | **Avg.** | 80.464 ± 0.734| 79.337 ± 1.036 | 81.534 ± 0.385 | 79.348 ± 0.699 | **83.596 ± 0.660** | These additions and experiments provide strong empirical evidence supporting the robustness and scalability of NormWear, even when applied to smaller datasets or tasks with limited subjects. We have updated the relevant tables in the manuscript to reflect these expanded datasets and results, ensuring a more comprehensive evaluation. # Section 5: Response to the Questions **Q: Discuss the limitations of resampling to 65Hz for ECG signals (this loses some meaningful physiological information)** A: We appreciate the reviewer’s comment on the resampling of ECG signals to 65Hz. The decision to resample at this rate was primarily motivated by engineering considerations, as it strikes a balance between computational efficiency and maintaining sufficient information for the intended tasks (e.g., heart rate and heart rate variability). For most standard clinical and physiological analyses, such as detecting heart rate or HRV, 65Hz is generally adequate, as these signals do not require the extremely high frequency resolution provided by higher sampling rates. While higher sampling rates may capture more granular details in the ECG waveform, such as the morphology of certain smaller features, our focus in this work is on broad physiological trends that can be effectively detected at this resolution. We do acknowledge that in future work, when higher resolution or more detailed waveform analysis becomes relevant, resampling at a higher frequency will be explored. Thank you again for your constructive feedback!

Response to Reviewer 9HJ7 Part I

Thank you for your helpful feedback! We appreciate that you found our idea of a common foundation model for various physiological signals to be interesting, and that you recognized the depth of our investigations into NormWear’s components, feature visualizations, and fusion techniques. Below, we briefly respond to each of your concerns. If you have further suggestions after reading our responses, we would be grateful to hear them and work to improve our submission accordingly. Our response has three sections. # Section 1: Response to Questions ### Question 1: What is the justification for two hyperparameters in equation (1)? Is this loss used in conjunction with another loss, and if not, it appears that one hyperparameter is enough? Thank you for your question. The two hyperparameters in Equation (1) correspond to the weighted average of two complementary objectives: cosine similarity to align the direction of the two representations, and Manhattan distance to bring their magnitudes closer. Although using one or two hyperparameters is theoretically equivalent, we choose the latter for its simpler implementation. By intuition, the two loss components are equivalently important, and the purpose of optimizing these two losses does not contradict each other, thus, we choose `[0.5, 0.5]` for the loss weights. ### Question 2: It appears that Table 10 was not referred to in the text, can authors provide more information about it? Thank you for pointing this out. We apologize for the confusion. Table 10 contains results from a simple test of the model's ability to run on edge devices and its hardware requirements. This was not a key focus of the paper but was included as a reference, as we plan future deployment and wanted to provide some initial insights for developers who may consider deploying the model. We acknowledge that this aspect is peripheral to the main focus of our work, and we will clarify this point further in the manuscript to avoid any misunderstanding. # Section 2: Response to the Concerns Raised in Weakness ## Weakness1: The number of subjects in the pre-training and evaluation datasets makes the conclusions intransferrable to large datasets for claims of “NormWear as a foundation model”. A foundation model is really a generalist model that can perform well on a variety of corner cases and downstream applications. Some modalities (e.g., EEG) have less than 50 pre-training/evaluation subjects, for example, their evaluation of “Driver Fatigue detection” has only 12 × 20% = [2–3] subjects in the test set, which is very low to conclude generalizable performance and conclusions for health applications. I believe this weakens the conclusion of NormWear being “[the first] foundation model specifically designed for wearable sensing data, capable of processing any number of multivariate signals from sources such as the heart, skin, brain, and physical body.” I recommend the author revise the language or provide additional empirical backup for NormWear being a foundation model. **Response1:** We sincerely thank the reviewer for highlighting the critical importance of dataset size and diversity in supporting the claim of NormWear as a foundation model. We deeply appreciate this thoughtful suggestion, as it has guided us to further strengthen the empirical foundation of our work. To address this concern, we have significantly expanded both the pretraining and downstream evaluation datasets to ensures a more comprehensive evaluation across diverse health-related tasks.: ***(a) Pretraining Data*** Originally, for computational efficiency, we had not utilized all available samples in our pre-training dataset. To address this, we have now included all available samples and supplemented the dataset with several additional collections, including datasets containing both ECG and PPG signals, as well as a large multi-channel EEG dataset with over 50,000 samples. As a result: - The pre-training dataset has grown from **37k samples (62 hours)** to **241k samples (402 hours)**. - Leveraging time series augmentation techniques detailed in the appendix, we further expanded the dataset to **2.5 million samples (approximately 4000 hours)**. ***(b) Downstream Data*** We incorporated several new datasets to further validate the model's performance. These include: - An EEG dataset with **500 subjects and five tasks**. - A dataset focused on muscular disorders. - A new ECG abnormality detection dataset. After running all experiments with these expanded datasets and additional tasks, our results confirm that **NormWear continues to achieve peak performance across a wide range of scenarios**. We have updated the relevant tables in the manuscript to reflect these findings, ensuring they provide a clear and robust empirical basis for our claims. We hope these updates satisfactorily address the reviewer's concerns and further substantiate NormWear's position as a foundation model for wearable sensing data.

Response to Reviewer 9HJ7 Part II

## Weakness 2: Several major claims in the paper seem overstated. For example, the delta between NormWear and Chronos in Table 1 seems very small considering that Chronos is not even a proper foundation model on physiological signals (Chronos is just a model trained on some time-series datasets, and to the best of my knowledge, there’s no prior work showing that Chronos is even close to SOTA for physiological signals such as PPG/ECG/EEG). Despite this shortcoming for Chronos, its difference between NormWear in the first 8 evaluations is very small, and in some cases it is even better. Similarly, authors make several big claims about processing frequency domain and CWT (see examples above), however, in Table 9, they show that the difference between processing with CWT vs. raw input is not that much (76.25 vs. 78.27). I recommend authors provide further explanation/discussion regarding these claims. **Response 2:** We sincerely thank the reviewer for highlighting the need for further discussion and context regarding our claims and evaluation. This feedback has been instrumental in refining our manuscript. **1) New baselines:** We have incorporated statistical baselines, CLAP, and TF-C in our evaluation to provide a more comprehensive comparison. These additional baselines offer clearer context for understanding our model's performance. Results, including comparisons with these baselines, have been updated in the manuscript as well. To further clarify our choices: modeling multivariate wearable signals, which include a wide range of modalities such as heart, brain, and physical motion signals, poses unique challenges, as no universally recognized open-source baseline or state-of-the-art model exists in this domain. To address this, we selected baselines from three representative paradigms: - **Statistical features:** Handcrafted statistical features are a traditional yet widely used method in signal processing literature. We included this baseline as a sanity check to benchmark our model against simpler approaches. - **SSL Time-series frameworks:** Wearable sensory data is inherently sequential, making time-series modeling strategies relevant. We compared our approach with Chronos, a state-of-the-art time-series forecasting framework, and TF-C, a commonly used self-supervised learning method for time-series data. - **Spectrogram-based methods:** Spectrogram-based modeling has demonstrated state-of-the-art performance in tasks such as music classification and physiological signal analysis. CLAP was selected as a representative baseline for this paradigm. **2) Linear Probing Algorithm:** To ensure better reproducibility, numerical stability, and fairness in performance comparison, we have updated our linear probe implementation. Previously, our results were based on a custom linear layer optimized with the Adam optimizer. To standardize the evaluation pipeline, we now utilize well-established methods for solving the linear probe tasks. Specifically, classification tasks are addressed using Newton's method with conjugate gradient, while regression tasks (e.g., vital signs prediction) are solved using Cholesky decomposition with a closed-form solution. These updates provide a more robust and interpretable framework for evaluating our model's representations across various tasks. **3) CWT vs. raw input:** We recognize the importance of providing a deeper discussion regarding the differences observed. While the performance gap between CWT and raw time-series input might appear modest (76.25 vs. 78.27), this outcome underscores the robustness of our model. Specifically: - The results demonstrate that our model is capable of effectively extracting meaningful representations regardless of input type, which is a key strength of NormWear. - The ablation study confirms that using CWT does indeed yield better results, validating its utility in certain scenarios. We have revised the manuscript to emphasize these points, ensuring that our claims are well-supported and balanced. The updated main result table is attached below, which is also updated under Experiment section. Thank you again for this valuable feedback, which has helped us provide a clearer and more nuanced discussion. The new result table is updated in the next part of the response, and we've also updated it in the manuscript under Experiment section.

Response to Reviewer 9HJ7 Part IV

# Section 3: Response to the suggestions on writing improvements: Many important details of technical implementation is missing from the paper, I recommend the authors incorporate all necessary information to aid the reader. I provide few examples below: - **Q1:** Information about how patches are selected and how many patches are there for each segment, appear to be missing. - **Q2:** Architectural hyperparameters regarding the tokenizer, the reconstruction module (de-tokenization), the details of the encoder/decoder transformer (token dimension, number of attention heads, positional encoding, dimension of MLP hidden layer, normalization, ...) appear to be missing. **A1&2:** We've updated these technical details in appendix A table 8. - **Q3:** The details regarding the downstream evaluations (linear probing) appear to missing **A3:** We've updated the linear probing pipeline, the details are stated in previous response section. - **Q4:** The details about how sentences are chosen in Section 3.2, what language model (or encoder) was used to get the “question semantic” embeddings appear to be missing. Hyperparameters of equation 1/2 and L295-311 appear to be missing from the paper. **A4:** We've add the citation for the text encoder we used, as well as sentence template example under appendix B. For the hyperparmeter of the loss function, we've updated it in appendix A table 6. - **Q5:** Details of masking strategies in Table 8 are missing. We did explain with text and figures. It would be great if the authors provide details about how confidence bounds are selected in Tables, e.g., Table 1. It is surprising that they get such narrow confidence bounds with such small N (e.g., 2/3 for Driver Fatigue detection if I understand correctly)? **A5:** Thank you for pointing these out. The reported score was calculated from repeated running sessions without setting a fixed random seed. Though in the updated table we update the linear prob algorithm to be more deterministic and reproducible with Newton’s method for classification and Cholesky’s method for regression. Regarding the presentation improvement, we truly appreciate the time and effort you have taken to provide insights on how to improve the manuscript. We understand that clarity and flow are essential. To address the concerns you raised, we have made significant revisions to the paper, not only improving the writing but also incorporating updates to the dataset size, new baselines, and results. These changes have naturally resolved many of the earlier writing issues, while others were directly addressed in our revisions. We believe these updates enhance both the clarity of our claims and the overall robustness of our findings. As some of the specific sentences you mentioned have already been reworded or removed in the revised version, and given that the modifications are substantial, we are unable to list each change individually here. However, we sincerely appreciate the points you raised, as they have significantly contributed to improving the overall quality and clarity of the presentation. If you have any additional concerns or areas where clarity could be further improved, we would be grateful for your further suggestion. Thank you again for your constructive feedback!

Response to Reviewer 9HJ7

Dear reviewer, Thank you for your timely reply and your constructive feedback. **Foundation model for multimodal wearable-based physiological signals:** Modern wearables come in diverse form factors and capture a wide range of modalities, from brain activity and cardiac or respiratory function to physical movements. While recent advances in AI/ML have leveraged wearable signals, no single foundation model exists that is agnostic to input modality, effectively represents diverse combinations of wearable modalities, and addresses a variety of health sensing tasks. There exists closed source models or data, or focus entirely on designing a specific deep learning based method on one single application. This issue in the field has to be acknowledged and it requires a more generalized modeling methodology. In the current work, **instead of focusing on training our model on a single large dataset we focus our attention on diversifying datasets containing different wearable input modalities and sensing tasks.** We conduct all of our experiments on **entirely publicly available datasets,** and we do ensure to release all the **code, cleaned preprocessed data, and model checkpoints** generated during this study for the core benefit of the community for future research. **Problem addressed by our method:** In our contribution, we address most of the core problems, where NormWear can take any wearable sensor signals (for heart, brain, IMU, muscle, PCG, etc.) with **any number of channels** (e.g. PPG only, PPG+ECG, IMU+PPG, 1/4/12/64 channels of EEG, etc.), with **any input channel order,** and shows its informative representation can be easily applied on **diverse health related tasks: for mental health, activities recognition, brain tumor area detection, seizure detection, muscle dystrophies and neuropathies detection, and several other blood related disease risk evaluation with common vital sign estimation tasks.** We are glad to see reviewers’ acknowledgement of our core contributions. The applicability of our model on **both large and small downstream datasets perfectly matches the reviewer’s belief that “A foundation model is really a generalist model that can perform well on a variety of corner cases and downstream applications.”** **Role of pretraining size:** We acknowledge that pretraining size is undeniably important to ensure the foundation model’s generalizability on various downstream tasks. Nevertheless, this single aspect should not overshadow the importance of addressing core modeling challenges, engineering considerations, and domain-specific issues. In our work, we emphasize the need to prioritize these foundational strategies rather than solely focusing on pretraining dataset size. Our approach not only demonstrates the potential of scaling with 2.5 million signal samples with varied channel configurations but also underscores the value of tackling fundamental modeling challenges to ensure meaningful progress in the field. To this end, we sincerely thank the reviewer once again for their valuable feedback and guidance. The insights have not only significantly helped to improve our work but also will leave valuable guidance on OpenReview, serving as a valuable reference for future researchers. We would like to thank you for the opportunities to discuss and contribute to the continued advancement of the field. Best, Authors

Response to Reviewer 9HJ7

## Regarding the pretrain data size: We appreciate the reviewer's feedback and would like to clarify some details regarding the size and composition of our pretraining dataset. Specifically, our dataset consists of **2,576,418 segment samples, totaling approximately 4,294 hours**. This figure differs from the 400 hours mentioned in the review. Additionally, these segments encompass a variety of sensor channels, which we believe challenges the statement that “this is arguably by orders of magnitude smaller than the standards for even a single waveform modality.” When treating the sensors individually (e.g., EEG, PPG, IMU, ECG, GSR, etc.), our dataset contains **8,965,538 sensor signal series, totaling 14,943 hours**. To further clarify and enhance transparency, we have included two pie charts in the overview figure, illustrating the proportions of different sensor types and physiological signals represented in our dataset. These visualizations aim to provide a clearer picture of the data's diversity and breadth. That said, while we acknowledge the importance of dataset size, we maintain the perspective stated in our previous response: the pretraining dataset size is one aspect of our work, but it should not overshadow our core contributions. These include, which are also gratefully acknowledged by the reviewers, the **novel modeling strategy accommodating flexible numbers and types of sensor channels, the peak performance achieved across diverse health applications, the first to achieve zero-shot inference on multi-variate wearable signals, and the fact that our entire work is open-sourced for the community**. We hope this provides a more comprehensive view of our contributions and addresses the concerns raised. ## Regarding the ablation study alignment: We are pleased to share the up-to-date version of our ablation study, aligned with the pipeline update as described in our 1st round of response. We've included all of the core ablations including the data input format, different masking scheme and different inter-channel fusion logic. All the experiments are now completed with same amount of data (both pretrain and downstream), and all observations and arguments remain consistent, and the detailed result are presented in the following table (in next response) which we will also update in the Appendix as well.

Response to Reviewer ZArW Part I

Thank you for your helpful feedback! We appreciate that you found our fusion and tokenization approaches to be well-studied and justified, and that you recognized the strength of our ablations and the impressive generalization performance of NORMWEAR across diverse tasks. Below, we briefly respond to each of your concerns. If, after reading our responses, you still have suggestions for improvement, we would love to hear them and further refine our work. ## Weakness 1: Semantic alignment training procedure is unclear - After pre-training the backbone encoder, the embedding space is aligned with the semantic text. However, I cannot seem to understand how this is done nor what datasets are used (i.e. are they the same as the pre-training datasets?). This seems to be a critical aspect of the model in order to enable zero-shot performance, but little detail is given. Two papers are cited, Zhang et al., 2024; Liu et al., 2024, but they do not seem to imply one specific approach. **Response** Thank you for your valuable comment and for pointing out the need for clarification on the semantic alignment training procedure. As described in Section 2.5 of the manuscript, we propose a novel temporal fusion mechanism that aligns the backbone encoder's output with a semantic space, enabling zero-shot inference. Specifically, we use the semantic embedding of a query sentence and the latent output from the backbone model to create a fused representation. The alignment is supervised using a loss function that incorporates both Manhattan distance and cosine similarity, ensuring that the physiological representation is aligned with the semantic representation in both direction and magnitude. The pre-training is still conducted on the pre-training datasets, and the semantic alignment step occurs after the backbone encoder has been trained. The pre-training and downstream datasets are always disjoint throughout all experiments in this paper, meaning that the downstream evaluation is conducted on data that the model has not seen during pre-training. We've updated the above clarification in the paper under section 2.4. We hope this clarifies the training procedure. Thank you for your insightful feedback, which helped improve the explanation. ## Weakness 2: Insufficient baselines - It is argued that chronos is a SOTA method, however, as noted, chronos was designed for time-series forecasting, which none of the downstream tasks are. - The ViT baseline is not well explained in how it is set up to do zero-shot downstream, including it's learning objective. - Ideally, the baselines should encompass the SOTA method for a given task, so as to understand how this model compares against each task specifically. The scope of the paper is "towards" a foundation model so this isn't a hard requirement, but would be nice. **Response** Thank you for your thoughtful feedback regarding the baselines used in our work. We appreciate your insight into the need for more targeted baselines, particularly for zero-shot performance and task-specific comparisons. We have now added additional baselines to better address this concern. Regarding the ViT baseline, we would like to clarify that it is not used for zero-shot downstream tasks, but rather as a "vision-based" approach for comparison. We understand that a spectrum-based model would provide a more relevant baseline for our tasks, and we have now included this in the updated baselines. The following are the details of our updates: **1) New baselines:** We have incorporated statistical baselines, CLAP, and TF-C in our evaluation to provide a more comprehensive comparison. These additional baselines offer clearer context for understanding our model's performance. Results, including comparisons with these baselines, have been updated in the manuscript as well. To further clarify our choices: modeling multivariate wearable signals, which include a wide range of modalities such as heart, brain, and physical motion signals, poses unique challenges, as no universally recognized open-source baseline or state-of-the-art model exists in this domain. To address this, we selected baselines from three representative paradigms: - **Statistical features:** Handcrafted statistical features are a traditional yet widely used method in signal processing literature. We included this baseline as a sanity check to benchmark our model against simpler approaches. - **SSL Time-series frameworks:** Wearable sensory data is inherently sequential, making time-series modeling strategies relevant. We compared our approach with Chronos, a state-of-the-art time-series forecasting framework, and TF-C, a commonly used self-supervised learning method for time-series data. - **Spectrogram-based methods:** Spectrogram-based modeling has demonstrated state-of-the-art performance in tasks such as music classification and physiological signal analysis. CLAP was selected as a representative baseline for this paradigm.

Response to Reviewer ZArW Part II

## Weakness 2: Insufficient Baseline - Continued **2) Linear Probing Algorithm:** To ensure better reproducibility, numerical stability, and fairness in performance comparison, we have updated our linear probe implementation. Previously, our results were based on a custom linear layer optimized with the Adam optimizer. To standardize the evaluation pipeline, we now utilize well-established methods for solving the linear probe tasks. Specifically, classification tasks are addressed using Newton's method with conjugate gradient, while regression tasks (e.g., vital signs prediction) are solved using Cholesky decomposition with a closed-form solution. These updates provide a more robust and interpretable framework for evaluating our model's representations across various tasks. New results, including comparisons with these baselines is attached below, and we've also updated it in the manuscript as well. | **Downstream Tasks** | **Statistical** | **Chronos** | **CLAP** | **TF-C** | **NormWear (Ours)** | |---------------------------------------------------|-----------------|-------------|----------|----------|---------------------| | WESAD | 66.213 | 71.489 | 72.383 | 69.865 | **76.060** | | UCI-HAR | 95.784 | 91.593 | 96.420 | 96.892 | **98.954** | | DriverFatigue | 63.249 | **76.722** | 61.889 | 66.882 | 74.292 | | **Activity Recognition Avg.** | 75.082 | 79.935 | 76.897 | 77.880 | **83.102** | |---------------------------------------------------|-----------------|-------------|----------|----------|---------------------| | Epilepsy (eye open) | 82.489 | 82.41 | 85.094 | 89.153 | **92.743** | | Epilepsy (eye close) | 87.457 | 88.218 | 89.867 | 94.416 | 94.828 | | Epilepsy (health area) | 86.274 | 81.08 | 83.711 | 85.619 | **88.541** | | Epilepsy (tumor area) | 82.816 | 81.034 | 83.644 | 86.348 | 87.197 | | Epilepsy (seizure) | 88.272 | 97.572 | **97.734**| 93.998 | 97.053 | | GAMEEMO | 51.009 | 53.747 | 52.551 | **56.275**| 54.937 | | **EEG Main Tasks Avg.** | 79.720 | 80.677 | 82.100 | 84.302 | **85.883** | |---------------------------------------------------|-----------------|-------------|----------|----------|---------------------| | PhysioNet Challenge | 42.592 | 48.166 | 43.058 | **50.934**| 46.904 | | ECG-Abnormal | 97.092 | 98.585 | 97.23 | 98.275 | **99.140** | | PPG-BP (HTN) | 59.499 | 52.425 | 56.757 | **65.229**| 62.341 | | PPG-BP (DM) | 47.823 | 51.164 | 42.455 | **57.883**| 55.893 | | PPG-BP (CVA) | **71.25** | 50.278 | 51.667 | 58.125 | 70.625 | | PPG-BP (CVD) | 51.219 | 58.31 | 50.91 | **58.674**| 51.773 | | PhysioNet EMG | **99.309** | 61.6 | 98.627 | 78.308 | 99.216 | | **Risk Evaluation Avg.** | 66.969 | 60.075 | 62.958 | 66.775 | **69.413** | |---------------------------------------------------|-----------------|-------------|----------|----------|---------------------| | Noninvasive-BP | 92.31 | 91.79 | 91.922 | 87.481 | **92.420** | | PPG-Hgb | 94.219 | **95.005** | 94.291 | 93.408 | 94.632 | | Fetal-fPCG | 98.929 | 99.048 | **99.195**| 99.077 | 99.072 | | **Vital Signs Avg.** | 95.153 | 95.281 | 95.136 | 93.322 | **95.375** | |---------------------------------------------------|-----------------|-------------|----------|----------|---------------------| | **Micro Avg.** | 76.727 | 75.276 | 76.284 | 78.255 | **80.875** | | **Macro Avg.** | 79.231 | 78.992 | 79.273 | 80.570 | **83.443** |

Response to Reviewer ZArW Part III

## Weakness 3: Many model components in the Memory Stream Inspired Mechanism in Sec 2.4 are not clearly explained - In MSiTF, it is argued that the representations are optimized for human sensing, how does this occur, specifically? This is not justified clearly. In Fig. 3, standard deviation and mean are used, but this does not seem to be explained in this section. - How is recency score important? It is stated that the further the time-step to the most recent time step, the lower te score, what time steps are being considered here? It is not clear, especially because the query is text and key/values are embedding time-series, and thus on different time scales. - Importancy score seems to act as a gate for the inputs time stamps, but it is not explained how the gate determined to be on or off. - The text states that final score is a summation, but how are scores used? - According to Fig. 3., it looks like the scores operate independently from each other in different model components, rather than being summed together. - In Eq. 1, it seems like there are two loss components, a l1 loss and cosine distance. Why are they both used together when they both work towards increasing similarity between Y and \hat{Y}? There is no text nor empirical results justifying this. **Response** Thank you for your detailed feedback regarding the Memory Stream Inspired Mechanism (MSiTF) and the components involved in our temporal fusion process. We've updated the text in section 2.5 for better clarification. We appreciate your suggestions, and we put a summerized clarifications below to address the points you've raised. **1) Optimization for Human Sensing:** We understand the importance of clearly explaining how the model is optimized for human sensing tasks. In our approach, we leverage a temporal fusion mechanism that incorporates recency scores, relevance scores, and importance scores to prioritize specific time steps based on their relevance to the task at hand. The recency score is derived from the temporal distance of each patch to the most recent time step, with closer time steps receiving higher weights due to their stronger correlation with immediate human actions or conditions. This is particularly important in human sensing tasks, where recent data points are often more critical for accurate prediction. **2) Use of Mean and Standard Deviation:** To address the concern about the use of mean and standard deviation, we clarify that this is part of our variational-inspired approach. By computing the mean and standard deviation of patch embeddings, we inject stochasticity into the representation process, encouraging the model to explore variations in the data. This mechanism allows the model to capture the inherent variability in human sensing data, which is essential for robust zero-shot inference. And from the result table we observe that the performance boost up with this variational mechanism. **3) Importance Score as a Gate:** The importance score acts as a binary gate that determines whether a specific representation at a given time step should be retained or discarded. This gate is learned during pretraining via a trainable linear transformation, which optimizes the model's ability to select the most relevant information at each time step. The importance score is derived using a Gumbel-softmax function, which allows the model to make hard decisions while remaining differentiable during training. **4) Summation of Scores:** Regarding the final score, it is indeed a summation of the relevance, recency, and importance scores. These scores are used to weight the contributions of each time step when aggregating the patch representations. This weighted sum allows the model to prioritize more relevant, recent, and important patches in the fusion process, leading to a more informative fixed-length representation. **5) Loss Components in Eq. 1:** The use of both L1 loss and cosine distance in our objective is to balance two complementary aspects of alignment: L1 loss focuses on minimizing the absolute difference between the target and predicted representations, while cosine distance encourages similarity in directionality between the two. We find that both losses contribute to a more effective alignment of the representations, enhancing the model's performance in zero-shot inference tasks. We hope these explanations address your concerns and provide more clarity on the functioning of the MSiTF mechanism. Thank you again for your constructive feedback, which has helped us refine our approach.

Response to Reviewer ZArW Part IV

## Weakness 4: Experimental results are somewhat lacking - No experimental results showing the strength of their tokenization method compared to a simple Conv1d tokenizer. - Only having one metric reported makes it difficult for us to understand whether the performance gain is consistent. - In Fig. 4a), t-SNE clusters of the different classes being different is not too surprising, as each signal is quite different, so it would be nice to understand visualize and understand how the model is able to capture differences among specific classes or specific risk levels. - In Fig. 4b), it is unclear what the visualizations extracted by the intermediate layer imply and how they show that the model has learned meaningful information. **Response** Thank you for your valuable feedback. We appreciate the opportunity to clarify and improve our presentation of the experimental results. Regarding the comparison with the Conv1D tokenizer, we did have an explicit evaluation in Appendix Table 11, where we compare our approach to Conv1D tokenization on raw time series data. This comparison highlights the advantages of our tokenization method in terms of performance. To address the concern about the limited metric reporting, we have expanded the metrics section to include additional performance measures, such as AUC ROC, AUC PR, accuracy, precision, recall, and F1 score. These metrics provide a more comprehensive view of the model's performance and demonstrate that NormWear generally achieves peak performance across the tasks. The result table is updated in the next part of response, and we've also added it to the manuscript under Experiment section. For the t-SNE visualizations, we would like to clarify that they are not intended to emphasize the superiority of the clusters but rather to demonstrate that the model is capable of recognizing patterns without needing to be explicitly told which sensor is involved. The model's ability to distinguish these patterns and assign appropriate attention scores is evident in the visualizations. While we understand that this may seem intuitive, we wanted to showcase this as a part of our model's capabilities, without overemphasizing the significance of the t-SNE visualization. Regarding the feature analysis of the intermediate layers, we appreciate your point. Our aim with these visualizations is not to make claims about the model's learned features, but rather to offer insights into what the model captures at each layer. The nonlinear dynamics is intended to help us better understand the behavior of the model. Such as Lyapunov exponent (sensitivity to initial conditions), Hurst exponent (self-correlation/seasonality), and persistence entropy (unpredictability in system states). Once again, thank you for your feedback. We have updated the paper to include these clarifications and hope that these changes address your concerns effectively.

Response to Reviewer p9eE Part I

Thank you for your detailed and insightful feedback! We appreciate that you found our work to address an important and under-explored area and that you recognized the thoughtful design of our NORMWEAR model, as well as its strong performance across diverse healthcare applications. Below, we briefly respond to each of your concerns. If, after reading our responses, you feel there are areas where our work can be further improved, we would greatly value your additional feedback to help us refine and strengthen our contributions. Our response has two sections. # Section 1: Response to the Questions ****Questions:**** ****Q1**** Impact of dataset size: Could the authors elaborate on the potential impact of the limited pre-training dataset size on the model's generalization performance? Are there plans to expand the pre-training dataset in future work? ****A1**** The dataset size was indeed a critical consideration in our work. We have already expanded the pre-training dataset by 80 times with the number of samples from 37k to 2.5 million (~ 4000 hours), significantly improving its diversity and scale. A detailed explanation of this expansion and its impact on model performance is provided in the next sections of our response. ---------------------------------------------------------------------------- ****Q2**** Model Release: Given the lack of openly released models in this domain, I strongly encourage the authors to publicly release their model and code to facilitate future research and comparisons. ****A2**** Thank you for your valuable suggestion and for highlighting the importance of model and code release in advancing research within this domain. We fully agree with your perspective, and we are committed to publicly releasing both our model and code upon publication. We believe this will help facilitate further research, foster transparency, and enable meaningful comparisons in future work. ---------------------------------------------------------------------------- ****Q2**** Tokenization terminology: The use of "tokenization" might be confusing, given its association with language models. "Pre-processing" or "feature extraction" might be more suitable terms in this context. Could the authors clarify their usage of this term? ****A2**** Thank you for raising this concern. We acknowledge the potential for confusion with the term "tokenization" due to its association with language models. As clarified at the beginning of the tokenization section, in the context of wearable sensing, we use this term to describe the signal processing stage prior to inputting the processed data into the deep learning-based encoder. This includes the patching step, where the CWT-based scalograms are divided into smaller patches, which we refer to as “tokens”. We hope this explanation clarifies our intended usage of the term. ---------------------------------------------------------------------------- ****Q3**** Insufficient information on dataset selection: The rationale behind the selection of pre-training and downstream datasets is not clearly articulated. Why were certain larger datasets reserved for downstream evaluation only? The paper mentions limited information on one of the datasets (BP with 1000 users) and should clarify its origin (self-collected or public). The provenance of all datasets should be explicitly provided, ideally with inline references within Tables 3 and 4. ****A3**** Thank you for your feedback regarding dataset selection and the need for more detailed information. We have updated the manuscript to include comprehensive tables for both pre-training and downstream datasets, providing all necessary details such as citations, number of samples, types of signals, and corresponding downstream tasks where applicable. These updates tables (Table 1 and 2) are added in the main section of the paper under Method section. As for the rationale behind dataset selection: - For pre-training, we prioritized datasets with the largest number of samples, diverse channels, and a variety of sensors to effectively train the channel fusion module. - For downstream tasks, we selected datasets that cover a broad range of health-related applications, including mental health, human activity recognition (HAR), EEG-based tasks, disease risk prediction, and vital sign monitoring. This diversity ensures a comprehensive evaluation of our model across different domains. We hope these updates and clarifications address your concerns. Thank you again for highlighting these important points, which helped improve the clarity and completeness of our work.

Response to Reviewer p9eE Part II

# Section 2: Response to the concerns raised in weakness ## Weakness 1 & 2 - Limited size of the pre-training dataset - Lack of simple statistical baselines ****Response 1 & 2**** Thank you for your detailed and insightful feedback regarding the dataset size and inclusion of simpler baselines. For the pre-training dataset size, our primary focus was to develop a modeling strategy capable of handling wearable signals with arbitrary channel numbers and sensor types (e.g., heart, brain, physical activity signals). In response to your concern and to enhance the robustness of our approach, we have significantly expanded our pre-training and downstream datasets. This expansion addresses the generalization concern, and we have re-run all experiments to validate the effectiveness of our model. Updated results are provided in the revised manuscript, and our model consistently achieves peak performance across all scenarios. Our detailed updates are stated below: **1) Pretraining data:** Originally, for computational efficiency, we had not utilized all available samples in our pre-training dataset. To address this, we have now included all available samples and supplemented the dataset with several additional collections, including datasets containing both ECG and PPG signals, as well as a large multi-channel EEG dataset with over 50,000 samples. As a result, the pre-training dataset has grown from 37k samples (62 hours) to 241k samples (402 hours). Since there are very few publicly available multivariate sensor datasets, we further leverage time series augmentation techniques detailed in the appendix, we expanded the dataset to 2.5 million samples (approximately 4000 hours). All experiments were re-run with the model retrained on this significantly larger pre-training dataset. **2) Downstream data:** We incorporated several new datasets to further validate the model's performance. These include an EEG dataset with 500 subjects and five tasks, a dataset focused on muscular disorders, and a new ECG abnormality detection dataset. This expanded evaluation framework ensures a more comprehensive assessment of the model across diverse health-related tasks. **3) New baselines:** We have incorporated statistical baselines, CLAP, and TF-C in our evaluation to provide a more comprehensive comparison. These additional baselines offer clearer context for understanding our model's performance. Results, including comparisons with these baselines, have been updated in the manuscript as well. To further clarify our choices: modeling multivariate wearable signals, which include a wide range of modalities such as heart, brain, and physical motion signals, poses unique challenges, as no universally recognized open-source baseline or state-of-the-art model exists in this domain. To address this, we selected baselines from three representative paradigms: - **Statistical features:** Handcrafted statistical features are a traditional yet widely used method in signal processing literature. We included this baseline as a sanity check to benchmark our model against simpler approaches. - **SSL Time-series frameworks:** Wearable sensory data is inherently sequential, making time-series modeling strategies relevant. We compared our approach with Chronos, a state-of-the-art time-series forecasting framework, and TF-C, a commonly used self-supervised learning method for time-series data. - **Spectrogram-based methods:** Spectrogram-based modeling has demonstrated state-of-the-art performance in tasks such as music classification and physiological signal analysis. CLAP was selected as a representative baseline for this paradigm. **4) Linear Probing Algorithm:** To ensure better reproducibility, numerical stability, and fairness in performance comparison, we have updated our linear probe implementation. Previously, our results were based on a custom linear layer optimized with the Adam optimizer. To standardize the evaluation pipeline, we now utilize well-established methods for solving the linear probe tasks. Specifically, classification tasks are addressed using Newton's method with conjugate gradient, while regression tasks (e.g., vital signs prediction) are solved using Cholesky decomposition with a closed-form solution. These updates provide a more robust and interpretable framework for evaluating our model's representations across various tasks. Results, including comparisons with these baselines is attached in the next part of the response, and we've also updated it in the manuscript as well.

Response to Reviewer p9eE

We sincerely thank the reviewer for their continued constructive feedback and for recognizing the clarified motivation and contributions of our work. ## Clarification on the "linear probing methods": We apologize for the confusion in our initial response. To clarify, the methods we used are standard logistic regression (solved using Newton's method) and linear regression (solved via Cholesky's method). While evaluation results can naturally vary depending on the chosen methodology, the updated framework offers a more consistent basis for comparison with robustness to hyperparameters (e.g. learning rate, batch size, etc.), ensuring a fairer comparison. Notably, our model continues to perform strongly across the board, maintaining its effectiveness under this revised evaluation approach. We hope this clarification addresses the concerns about the linear prob method, and we will correct the terminology used in manuscript as well. ## Regarding statistical test: We acknowledge the importance of having statistical validation. Following the suggestion, we first run the downstream evaluation 100 times for each model on all the tasks, without fixing random seed. We observed that the outcomes stay consistent due to the stability of the optimization process. We then conduct a permutation test, across the results from these 100 runs, to assess whether our method significantly outperforms the baselines. We declare the alternative hypothesis as whether the score (AUC ROC) of NormWear is greater than the baselines in comparison. The reported P value represents the probability of observing a test statistic as extreme as, or more extreme than, the observed difference under the null hypothesis, assuming that the AUC ROC score of NormWear is not greater than the baseline. The results indicate that in nearly all cases, the statistical significance (p-value) is less than 0.01, providing statistical significance evidence of the robustness and superiority of our approach. In the following table, we include the results from conducting the statistical test across different task groups (the groups were highlighted with different colors in the tables in main sections) and the total average scores. | Ours\Baselines | Stats | Chronos | CLAP | TFC | |-----------------------|-------|---------|--------|--------| |NormWear - activity | P<.01 | P<.01 | P<.01 | P<.01 | |NormWear - eeg | P<.01 | P<.01 | P<.01 | P<.01 | |NormWear - risk | P<.01 | P<.01 | P<.01 | P<.01 | |NormWear - vital | P<.01 | P<.01 | P<.01 | P<.01 | |NormWear - micro avg. | P<.01 | P<.01 | P<.01 | P<.01 | |NormWear - macro avg. | P<.01 | P<.01 | P<.01 | P<.01 | |-----------------------|-------|---------|--------|--------| |Conover post hoc | <.001 | <.001 | <.001 | <.05 | We also present the critical difference diagram (CD) to visually compare the performances of multiple models across datasets, highlighting whether their performance differences are statistically significant. In order to achieve CD diagram, we first conduct Friedman Chi square test on the scores achieved by the models across all the downstream tasks, and observe P value of P < .001, making sure all the models' performance are coming from different distribution. Then we conduct Conover post hoc test to check the pair-wise model performance difference, where the P values corresponding to NormWear vs. baselines are presented in the last row of the above table. Finally, we create CD diagram based on these results, and result in the following diagram (described by text below). Our proposed model, NormWear is far apart from the bar, indicating its statistical significance against competitive baselines. ### Average score rank: | Stats |Chronos | CLAP | TFC | NormWear | |-----------------------|-------|---------|--------|--------| |0.48 | 0.50 | 0.53 | 0.65 | **0.83** | ### Cross bars: | 0 |1 | 2 |3 | |-----------------------|-------|---------| ---------| |(Stats, Chronos) | (Chronos, CLAP) | (CLAP, TFC) | **(NormWear (ours))**| We will include the above results in the manuscript as well. We deeply appreciate your thoughtful suggestions, which have significantly contributed to enhancing this work. We remain fully open to any additional feedback that could further improve the study and better address your concerns.