Continuous Temporal Domain Generalization

Summary

The paper introduces the problem of Continuous Temporal Domain Generalization. It extends the Temporal Domain Generalization problem, that aims at developing models under temporally varying data, to handle data collected at arbitrary and continuous time points. The paper proposes a framework based on Koopman theory for this problem. It models the underlying dynamics and proposes an optimization strategy. The paper includes several classification experiments to demonstrate the effectiveness of the proposed approach.

Strengths

* The paper introduces a new problem continuous temporal domain generalization where data is continuous and irregularly observed. This is relevant and challenging setting. * The paper is supported by proofs and the relevant assumptions under which the model is well-defined are clearly stated. * The paper includes extensive experiments on various datasets, both synthetic and real-world, to demonstrate the effectiveness of the proposed method. The proposed model improves over existing methods.

Weaknesses

* the paper could discuss more the limitations of the proposed approach, in particular Assumption 1 that states the conditional probability distributions follow an ODE. While reasonable to make, this assumption is quite restrictive; I understand that the considered problem is challenging so relevant assumptions should be made. However, the paper would need to discuss the validity of this assumption. Also, another limitation can come from abrupt changes throughout time that might not be well captured by a continuous ODE formulation (mentioned in A.1.7.); this assumption is likely to be violated in real-world settings. * the paper does not seem to follow the checklist; there is no justification for each points and if I am not mistaken, some "Yes" answer are not correct e.g. a Broader Impact section is missing.

Questions

* How does it compare to domain invariant approaches (e.g. IRM, V-REx etc.)? A domain-invariant approaches might be better at handling abrupt changes.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

* see limitations points in "Weaknesses".

Summary

The article introduces Continuous Temporal Domain Generalization (CTDG), which extends traditional Temporal Domain Generalization (TDG) methods by addressing the challenges posed by continuous and irregularly spaced temporal domains. The authors propose a Koopman operator-driven framework (Koodos) to handle data evolving continuously over time and collected at arbitrary intervals. The framework aims to capture continuous dynamics in data and models, optimize generalization across continuous temporal domains, and leverage prior knowledge of dynamic patterns. The experiment results demonstrate the efficacy of this approach compared to some existing methods.

Strengths

The primary contribution is the introduction of the CTDG problem and the Koodos framework, which extends TDG to handle continuous and irregular temporal domains. The paper also provides a comprehensive optimization strategy and theoretical analysis to support the proposed approach.

Weaknesses

The method assumes data dynamics can be well-characterized by the Koopman operator. While powerful for linearizing nonlinear dynamical systems, its effectiveness across various domains (e.g., finance, healthcare, social media) requires more extensive empirical studies, especially for highly noisy or chaotic systems. The proposed hybrid loss function's complexity may present challenges in training and convergence. A detailed analysis of its behavior and optimization strategies would be beneficial. The method introduces additional complexity and hyperparameters. A comprehensive discussion on hyperparameter sensitivity and guidelines for setting loss weights would enhance the method's practical applicability. Theorem 2, which compares continuous-time and discrete-time approaches in approximating temporal distribution drift, does not directly address generalization bounds or risks in domain generalization. While it provides insights into error accumulation over time, it lacks analysis of the model's performance on unseen temporal domains. The experimental comparisons are somewhat limited. Only one continuous baseline is used, and it appears that key comparison metrics are missing. Methods like AdaRNN[1] and Diversify[2], which aim to address the same issue, are not discussed or compared. More comparisons to state-of-the-art discrete TDG methods adapted to the continuous case would strengthen the evaluation. [1] AdaRNN: Adaptive Learning and Forecasting of Time Series [2] Diversify to Generalize: Learning Generalized Representations For Time Series Classification

Questions

1. In typical domain generalization challenges, the objective is to align the joint distribution \(P(X, Y)\) across different domains. For Temporal Domain Generalization (TDG), this would extend to aligning \(P(X(t), Y(t))\) at different time points. How does Koopman Theory address this issue in aligning these temporally varying distributions? 2. Authors focuses on the existence of continuous and relatively stable dynamic changes across continuous time, how effective is this method when dealing with continuous but non-periodic or irregular data variations or if a new dataset significantly differs in its statistical properties from the training data? is there potential to integrate other mechanisms or models to enhance the handling of such substantial data changes? 3. In the Methods section, you emphasize addressing changes in the conditional probability distribution over time. However, the datasets used, such as Rotated MNIST, Rotated 2-Moons, Yearbook, and Cyclone, are commonly associated with covariate shifts rather than shifts in the conditional probability distribution. These datasets appear to maintain a stable relationship between the labels and the input data or a latent feature space. Could you clarify how the changes in the conditional probability distributions are represented or accounted for in these datasets?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

There is no potential negative societal impact is found in this work.

Summary

The paper presents a novel approach called Continuous Temporal Domain Generalization (CTDG), addressing the challenge of training predictive models under continuously evolving and irregularly observed temporal domains. Unlike traditional TDG methods that rely on discrete time intervals, CTDG captures continuous dynamics of both data and models. The proposed Koopman operator-driven continuous temporal domain generalization (Koodos) framework leverages Koopman theory to learn underlying dynamics and enhances it with a comprehensive optimization strategy. Extensive experiments demonstrate the effectiveness and efficiency of the proposed approach.

Strengths

The paper introduces a novel problem (CTDG) and proposes an innovative solution (Koodos) leveraging Koopman theory. The methodology is sound and well-executed. And it is well-supported by robust theoretical foundations and demonstrates excellent performance in the experimental results.

Weaknesses

The explanation of how the article addresses domain changes with arbitrary temporal sampling is not very clear. There is some overuse of formula characters, and the use of L_pred2 seems rather arbitrary. Additionally, the numerous hyperparameters in the loss function increase the cost of tuning for different datasets.

Questions

The article repeatedly emphasizes the importance of "C" in CTDG and the arbitrary selection of time t. However, does the proposed method specifically address this aspect, or does it primarily rely on Koopman theory? In the methodology section, the article mentions that domain conditional probability distributions will not experience abrupt changes. However, this situation holds true only when the sampling time intervals are equal. The method addresses arbitrary sampling; does this imply it also deals with abrupt changes? Notably, in equation (15), the weights of L_pred and L_pred2 are both α, the weights of L_recon and L_consis are both β, and the weight of L_dyna is γ. Could you explain the rationale behind this? How are these hyperparameters set for different datasets, and what is the basis for their selection? Do different weights significantly affect the results?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper introduces a new task: Continuous Temporal Domain Generalization (CTDG) to address the limitations of traditional TDG in handling continuously evolving and irregularly observed temporal data. By proposing the Koopman operator-driven framework (Koodos), this work leverages Koopman theory and optimization strategies to learn and control the continuous dynamics of data and models. Experiments demonstrate the effectiveness and efficiency of Koodos in managing complex high-dimensional nonlinear dynamics and generalizing across continuous temporal domains.

Strengths

1. This paper introduces a valuable new task, Continuous Temporal Domain Generalization (CTDG), which moves beyond the discrete nature of traditional TDG tasks, enhancing their relevance to real-world applications. 2. The paper proposes the Koopman operator-driven framework (Koodos), effectively leveraging Koopman theory and optimization strategies to understand and control the continuous dynamics of both data and models. 3. It expands the evaluation setting of traditional TDG tasks by introducing datasets with truly continuous data and temporal distribution shifts. This setting encompasses the previous discrete TDG evaluation as a special case, providing a more comprehensive evaluation framework for all TDG methodologies. 4. The authors further evaluate their method and other TDG baselines under this new CTDG evaluation setting, achieving significantly improved performance. 5. The paper is well-written and well-organized.

Weaknesses

1. Evaluation is limited to small datasets and models, lacking assessment on higher-dimensional datasets, which limits the practical applicability of proposed benchmarks and obscures performance in scenarios with large models and high-dimensional data. 2. The continuous nature of CTDG proposed here appears similar to task settings in CIDA and AdaGraph, which focus on domain adaptation. This reduces the novelty of the task setting in this paper, necessitating further discussion and experimental comparisons for clarification. 3. TKNets, another Koopman theory-based TDG method, is relevant but lacks clear differentiation through discussion and experimental comparisons. 4. Including information on the training and inference costs of each method would enhance the completeness of the study.

Questions

1. Does CTDG still adhere to the assumption of smooth distribution shifts? 2. Is the quantitative evaluation in the paper still performed on the last domain? 3. Regarding the Yearbook dataset under CTDG evaluation settings, is it essentially still discrete but can be approximated as continuous due to its large number of domains? If so, which other datasets in CTDG use dense discrete domains to approximate true continuous data?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See my weakness part.

Response to Authors' Rebuttal

I appreciate the authors' rebuttal, which has addressed some of my major concerns. As a result, I'll slightly increase my rating. However, I'm not entirely confident about this evaluation, particularly regarding the assessment of the methodology. I may further reassess after considering the discussions between the authors and other reviewers. I would be grateful if the authors could address the following concerns: ## 1. Necessity and Urgency of Improvements While I acknowledge that the improvements in task setting and problem modeling for TDG are necessary, I question their urgency. Given that most current TDG datasets are relatively toy-like, with low dimensionality and limited to specific scenarios with relatively simple temporal distribution shifts: - Is the main bottleneck in applying TDG to real-world applications still on the temporal aspect? - Will the proposed modeling and methods hold in more complex scenarios? ## 2. Dataset Realism I appreciate the inclusion of Yearbook and Cyclone datasets, which are indeed more realistic than previous TDG datasets. However, I'd like to point out that these datasets are still quite simple in terms of: - Data dimensionality - Dataset size - Distribution shift complexity ## 3. Validation of Improvements While the design intuitions for the improvements seem reasonable and meaningful, I feel the authors haven't used appropriate datasets to motivate and validate the value of these improvements. Using Yearbook as an example: - High school graduation photos are typically taken at a fixed time each year - Is there a significant difference between continuous-time and fixed-interval discrete-time in this context? ## 4. Theoretical Contributions The authors state, "Our primary contributions are theoretical proof that the model for continuous temporal domains is also continuous." However: - Didn't we already have such insights or proofs from DRAIN or GI? - I acknowledge that formally proving this point is a major contribution if previous work hasn't done so. ## 5. Comprehensiveness of Comparisons Regarding comparisons with other TDG methods: - Are the comparisons comprehensive? - Could we improve the performance of these methods in the CTDG setting by dividing them into denser discrete temporal domains?

Follow-Up on Rebuttal Response

Dear Reviewer Drn2, We greatly appreciate your time to review our paper and your valuable comments. We noticed that we haven’t yet received your response and wanted to kindly inquire if there’s anything further we can do. We have made extensive clarifications, added detailed discussions, and provided additional experimental results as you suggested. We hope we have effectively addressed your concerns and clarified any potential misunderstandings. We are eager to hear your thoughts on the efforts we have made during the rebuttal period. Thank you once again for your invaluable review.