Summary
The authors propose SimMTM, a novel representation learning method for time-series data based on random masking. They experimentally verify the effectiveness of the proposed method from both in-domain and cross-domain perspectives. Comprehensive comparative experiments with existing methods are conducted, demonstrating the effectiveness of the proposed approach.
Strengths
The authors have conducted extensive comparative experiments with existing methods and quantitatively demonstrated the effectiveness of the proposed method. In addition to in-domain generalization, they also focus on cross-domain generalization. This verification is indispensable for validating the effectiveness of deep representation learning assuming large-scale open data.
Weaknesses
**Presentation quality**
The current presentation style of this paper is confusing. Particularly, there is a lack of consistency in the representation of formulas and symbols, making it very difficult to grasp the overall logic of the paper. Some specific examples include:
1. In Equation 3, the symbol $\times$ is used to denote both the Cartesian product and scalar multiplication when defining the domain of $R$.
2. The transposition notation in Equation 3 is counterintuitive as $\mathbf{s}$ is defined as a row vector.
3. Equation 7 is overly verbose and exacerbates the difficulty in reading, as it does not contribute to the subsequent. It would be sufficient to simply declare that only the situation where $i = k$ is treated as positive pairs.
4. While $\mathbf{s}$ is defined as a single vector, $\mathbf{s}^+$ is defined as a set of vectors.
Some other points that I could not make from the paper are discussed below in the Questions section.
**Comparison with prior literature**
The comparison with prior literature is insufficient. For example, references [1, 2] can be cited as representative examples of representation learning methods for deep neural networks under time-series data. In particular, a model that generalized the method in [2] as a generalized contrastive learning method not limited to the time-series domain was proposed later. Both [2] and [3] have superior theoretical guarantees for identifiability from the perspective of nonlinear ICA. The authors should clearly indicate the proposed model's advantages over these approaches.
**References**
1. Hyvarinen, A., & Morioka, H. (2016). Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. *Advances in neural information processing systems*, *29*.
2. Oord, A. V. D., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. *arXiv preprint arXiv:1807.03748*.
3. Hyvarinen, A., Sasaki, H., & Turner, R. (2019, April). Nonlinear ICA using auxiliary variables and generalized contrastive learning. In *The 22nd International Conference on Artificial Intelligence and Statistics* (pp. 859-868). PMLR.
Questions
- What is the definition of $\operatorname{Encoder}(\mathcal{X})$? Typically, a function $f \colon X \rightarrow Y$ can use all the information in $X$ to calculate $Y$, but it seems that the Encoder function defined here applies the same function in parallel to some dimension. However, it is not clear from the text which dimension this parallel processing applies to.
For example, there would be at least two possibilities: $\operatorname{Encoder}(\mathcal{X})$ is kind of a “syntax suger” for either $\bigcup\_i \operatorname{Encoder}( \\{x\_i\\} \cup \\{\bar{x}\_i^j \\}\_{j} )$ or $\bigcup\_i ( \\{ \operatorname{Encoder}(x\_i) \\} \cup \\{ \operatorname{Encoder}(\bar{x}\_i^j) \\}\_{j} )$. The same can be said about the definitions of the Projector and Decoder.
- Although I understood the training pipeline of the proposed method, I had questions regarding its behavior at inference time. Does the proposed method compute internal representations from the true sequence and $M$ randomly masked sequences at inference time, similar to training time? Or does it only use the true sequence without random masking at inference time? Additionally, how is training conducted during the fine-tuning stage in the pipeline?
- The authors mention the analysis of a quantity called the CKA value in line 262 of Section 4.4, but it is unclear what this quantity actually is. While there is an explanation of Pre-training/Fine-tuning CKA in the caption of Table 5, if the CKA value mentioned in the main text refers to these, its meaning should be clearly explained in the text as well. Moreover, I did not understand why a small $| \Delta_{\mathrm{CKA}} |$ leads to “acquiring adaptive representations for different tasks” (l.264). In principle, it should be possible to drastically change the internal representation by fine-tuning while maintaining the degree of change in the internal representation along the evolution of the layers. Suppose $x_\mathrm{first}$ and $x_\mathrm{last}$ be the representation in the first and the last layer at the pre-training phase, respectively. We can consider one idealistic situation that the representations after fine-tuning are given as $y_{\mathrm{first}} = x_{\mathrm{first}} + \delta_{\mathrm{task}}$ and $y_{\mathrm{last}} = x_{\mathrm{last}} + \delta_{\mathrm{task}}$ with a task-dependent fixed vector $\delta$. In such a situation, the values of Pre-training/Fine-tuning CKA would be nearly the same, and the difference in CKA values would go to zero by construction. However, the internal representations drastically change depending on the task choice. In this context, I believe there is a logical gap in the authors' argument.
- In masked modeling-based representation learning, the proposed method uses the true sequence without masking, along with the input. This setup is generally considered to make it more challenging to avoid trivial local solutions that merely output the input as is. Can it be said that such a phenomenon does not occur? Moreover, if a trivial local solution can be avoided, what factors of the proposed method enable this?
- As a naive control condition for the idea of using multiple different random masks, it is possible to consider the variant of using only one random mask that reduces the masked portion $r$, or takes the position of the mask to have a specific rules that is not completely iid. Can the proposed method be claimed to be effective against such variants? In particular, the model structure of using multiple random masks is expected to have at least some overhead in terms of training/inference computation time compared to such simple control conditions.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The limitations of this paper are adequately discussed in the appendix. Furthermore, the effectiveness of the proposed method can be confirmed from various perspectives through exhaustive comparative experiments. However, as mentioned earlier, there are major concerns primarily about the presentation of the paper. These are points that affect the reproducibility of the results and the authors' claims, so it would be necessary to resolve all problems of unclear logical progression for this paper to be accepted.