FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space

Summary

This paper proposes a self-supervised contrastive learning framework for extracting comprehensive features from multimodal time-series sensing signals. The framework refines contrastive learning by modeling shared and private features and exploiting orthogonality constraints. Moreover, a Temporal Structural Constraint is proposed to make up for the locality of processing the temporal information locality appropriately in contrastive learning. Finally, to demonstrate the effectiveness of the proposed FOCAL, the authors conduct extensive experiments on multiple multimodal sensing datasets.

Strengths

1. Extensive experiments are conducted on four benchmarks: MOD, ACIDS, RealWorld-HAR and PAMAP. 2. This work is relatively novel and technically sound. 3. The description and explanation of motivation are clear. 4. It seems to be a pioneer work to consider and solve temporal information locality. 5. The results are promising.

Weaknesses

1. In the experiments, what is the intuition of the hyperparameter setting of Eq.5? Sensitivity analysis appears to be lacking. 2. Whether the proposed temporal structural constraint is also applicable to other comparative learning frameworks to solve the problem of temporal information locality. 3. From the ablation results, the private space seems to be important for contrastive learning, somewhat similar to intra-modal contrastive loss. This work is based on strong orthogonality constraints, but in fact, the performance of the FOCAL-noOrth setting has not dropped too much. I am worried whether the orthogonality assumption is too strong. 4. Eq.4 is essentially a triplet contrastive loss, are there possible alternatives? Moreover, the value of the “margin ” does not seem to be mentioned in the paper. 5. I think Figure 6 should be compared with the visualization results of the best baseline (or some variants of FOCAL), so as to reflect the differences or superiority. like Supervised vs. MoCo vs. FOCAL? Otherwise, it feels like visualization for visualization's sake.

Questions

See Weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors provide a comprehensive discussion about limitations and potential extensions in the appendix.

Summary

This paper proposes a multimodal contrastive learning framework (FOCAL) for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. In which, FOCAL first decouples each modality into two subspaces, i.e., shared and private spaces, and uses simple soft orthogonal loss as an objective. Then, the temporal structural constraint for modality features is proposed to enhance feature temporally. The authors use four multimodal sensing datasets with two backbone encoders and two classifiers to conduct many experiments.

Strengths

1. The motivation and contribution of the paper are clearly described. 2. Many experiments are presented in the manuscript.

Weaknesses

1. The biggest concern is the effectiveness of multimodal feature factorization. From the line 161, this work uses the simple soft orthogonal constraint to decouple or factorize each modality feature. However, the single orthogonal constraint is too simple and may not decouple multimodal features well. In particular, the information is easily leaked between modalities, and many prior works have tried to solve this problem such as [1-3]. In addition, there is no visualization result for both subspaces in the experimental part, so it is difficult to determine whether the factorization is effectiveness. [1] Li Y, Wang Y, Cui Z. Decoupled Multimodal Distilling for Emotion Recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6631-6640. [2] Ouyang J, Adeli E, Pohl K M, et al. Representation disentanglement for multi-modal brain MRI analysis[C]//Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27. Springer International Publishing, 2021: 321-333. [3] Yang D, Huang S, Kuang H, et al. Disentangled representation learning for multimodal emotion recognition[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 1642-1651. 2. In the line 29, the word “ignore” may not be as appropriate. For example, the CLIP contrastive frameworks does not emphasize shared or private spaces, but we cannot assume that this work ignores heterogeneity, only that they do not explicitly consider. Therefore, it might be better to replace "ignore" with "explicitly consider". 3. In the experiments, why not consider some more common multimodal scenarios, such as the MOSI [4] and MOSEI [5] datasets, which contain the three most common modalities in the real world, i.e., Language, Visual and Acoustics, and whose features are packaged as time-series-based, in line with the settings in this paper. [4] Zadeh A, Zellers R, Pincus E, et al. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[J]. arXiv preprint arXiv:1606.06259, 2016. [5] Zadeh A A B, Liang P P, Poria S, et al. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018: 2236-2246. 4. The public datasets lack the necessary references, please check carefully. (Line 199-206)

Questions

Will the MOD dataset be open-released?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

No limitations are provided in this paper.

Summary

The paper proposes FOCAL, a contrastive learning method for multimodal time-series signals. The main idea is to first encode each modality into a factorized orthogonal space, and design four pretraining objectives that enforce modality consistency, transformation consistency, orthogonality constraint and temporal locality constraint. Experimental results demonstrate the efficacy of FOCAL.

Strengths

+ The two motivations for designing the contrastive objectives make sense to me --- modality-private features can contribute in contrastive learning, and the temporal structural constraint also seems valid. + The authors conduct an extensive evaluation of FOCAL --- it is compared with many contrastive learning baselines and evaluated on four multimodal datasets. The improvement achieved by FOCAL is large. + The ablation study is great and provides a good analysis on the four pretraining objectives. + The paper is clearly presented and easy to follow.

Weaknesses

While I generally agree with the motivations and the proposed four objectives, Eq (5) is a combination of four objectives, and it seems hard to balance the four terms. I wonder if some of the training objectives may compete during training, and how do the authors set the three hyper-parameters $\lambda_p$, $\lambda_o$ and $\lambda_t$? Does it require some manual hyper-parameter tuning? Jointly optimizing the four objectives does not look like an easy task.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes the limitations have been well discussed in Appendix.

Summary

The paper introduces FOCAL, a new contrastive learning framework for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Unlike previous frameworks that focused solely on shared information between sensory modalities, FOCAL also factors in exclusive modality information crucial for understanding physical semantics. It also properly handles temporal information locality, which previous time series contrastive frameworks have not done. FOCAL creates a factorized latent space of shared and private (modality-exclusive) features from each modality and applies a temporal structural constraint on these features. Through extensive testing on four multimodal sensing datasets, FOCAL consistently outperforms the state-of-the-art baselines in downstream tasks, demonstrating its superior performance.

Strengths

- This paper is excellently written, with clear logic and motivation that is easy to understand. - The results presented are impressive and Figure 1 does a fantastic job in aiding the understanding of the paper. - The idea of modeling modality-shared and specific information is very great. Great work!

Weaknesses

I couldn't find any significant weaknesses in this paper. However, I should note that I'm not an expert in this topic. While I am familiar with multimodality, and contrastive, my understanding of time series analysis is limited

Questions

- Is there any hyperparameter search performed for λp, λo, and λt which control the weights of each loss component? Additionally, was there any hyperparameter search carried out for other methods?" - "Why does the FOCAL-wDistInd perform worse than the noOrth model? Does the replacement of orthogonality with distributional independence contribute to this difference?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

NA

Summary

The proposed FOCAL is a constrasitive learning method or time-series signals. The major contributions are: 1. For multi-modality samples, FOCAL learns shared feature that are similar between modalities, and a private features that are similar intra-modality but different across modalities. 2. The shared and private features are orthoigonal, so that they focus on different aspects. 3. The Temporal Locality Constraint which restricts the average distance within a short sequence to be less than the average distance between two random sequences. The "average" operation allows some samples to be similar even though they are from distant samples in time-domain. Extensive experiments are conducted with 4 datasets and 12 baselines, which gives consistent SOTA performance on classification and clustering tasks.

Strengths

1. The idea of private feature is novel, and effective according to the ablation study. 2. Although the temporal structural constraint is a simple modification, it provides noticable performance increase. 3. The proposed method is simple and easy to follow. Detailed network design, data processing, training configurations are provided in Appendix.

Weaknesses

1. The proposed contribution is mostly the loss functions, e.g., private and shared feature constrative loss, temporal locality loss, etc. I believe these loss can be applied to existing methods with simple modification. It would be very strong to show that the proposed loss serves as a plugin that enhance other contrastive learning methods.

Questions

See weakness.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discussed little on the techinical limitation and future directions. It would be great the discuss more on failure cases.

Additional experiment on private and shared feature constrative loss

The benefit of temporal constraint is proven by Table 15 and 16. However, the major contribution of the paper is the shared and private loss, and the Orthogonality loss. I would suggest similar experiments to be done, i.e., enhance baselines with the proposed private and shared feature constrative loss, Orthogonality loss.

Friendly Reminder: Author-Reviewer discussion phase

Dear Reviewers, We hope this message finds you well. As we approach the final days of the author-reviewer discussion phase, we would like to kindly remind the reviewers to take a moment to review the rebuttals and provide any further feedback you might have. We genuinely appreciate your comments, and should you have any questions or concerns, please don't hesitate to let us know. Best, Submission #3273 Authors

Thanks for the rebuttal!

The provided responses have addressed my concerns.