Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization

Summary

The paper presents a novel approach for time series out-of-distribution generalization via pre-trained large language models. The authors introduce a Tri-level learning framework that combines sample-level and group-level uncertainties accompanied by a theoretic perspective. Furthermore, a stratified localization algorithm is proposed for the tri-level optimization problem followed by a theoretical convergence guarantee. The paper demonstrates performance gain in six real-world time series datasets, demonstrating the effectiveness of the method.

Strengths

1. The method proposed by the authors is sensible and seems to perform well compared to other out-of-distribution generalization methods. 2. Authors provide an in-depth analysis of their method and provide a convergence analysis. 3. The paper conducts a fair experiment on six real-world time series datasets, showing superior performance in out-of-distribution generalization.

Weaknesses

I do not see any major weaknesses.

Questions

Can this method be adapted for OOD time series regression and forecasting?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors adequately addressed the limitations.

Summary

The paper explores the challenge of OOD generalization in time series data using pre-trained LLMs. The authors propose a novel framework TTSO that integrates both sample-level and group-level uncertainties. They also introduce a stratified localization algorithm tailored to this tri-level optimization problem, theoretically demonstrating its convergence. The paper presents extensive experiments to validate the method’s effectiveness.

Strengths

1. The tri-level learning framework uniquely combines sample-level and group-level uncertainties, providing a comprehensive approach to OOD generalization. 2. The paper includes a solid theoretical foundation, with analyses that justify the proposed method and its iteration complexity. 3. The stratified localization algorithm offers a novel solution to the tri-level optimization problem, enhancing scalability and computational efficiency. 4. Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed method, showing significant performance improvements. 5. The study leverages the advanced capabilities of LLMs in time series analysis, contributing to the emerging field of using foundational models for non-linguistic data.

Weaknesses

1. The tri-level learning framework might be overly complex for practical applications, potentially limiting its adoption. 2. The proposed method, especially the stratified localization algorithm, may incur high computational costs, which could be a barrier for large-scale applications. 3. The paper could benefit from a more comprehensive comparison with other state-of-the-art methods in time series OOD generalization. 4. There is a need for more discussion on the real-world applicability and potential limitations of the proposed method in various domains. 5. Some sections of the paper are dense and challenging to follow, particularly the theoretical analyses, which might be difficult for a broader audience to understand.

Questions

1. How does the tri-level learning framework handle scenarios with highly imbalanced time series data? 2. Can the proposed stratified localization algorithm be adapted for other types of data beyond time series? 3. What are the specific computational requirements for implementing the proposed method on large-scale datasets?

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

The paper studies the problem of OOD generalization in time series tasks, building on recent observations that use data level uncertainties and group level uncertainties. This has been a successful way to build robust, transferrable representation. The paper additionally also includes an additional maximization for data augmentation that makes it the "tri-level" learning framework. The paper also theoretically analyzes generalization properties of the proposed algorithm. The TTSO method is also used to fine-tune LLMs which is used for time series classification in OOD scenarios. In this part, it builds on some recent works that utilize pre-trained or fine-tuned LLMs in other domains, leveraging the superior feature learning capacities of these large frontier/foundation models.

Strengths

* The paper addresses an important problem to build robust techniques for time series data, which is under studied and often quite challenging due to the inherent noise in time varying datasets. * Using an LLM within a framework for generalization in time series appears to be a novel framing * The paper does extensive theoretical and empirical analysis to demonstrate the properties and performance of TTSO.

Weaknesses

On the surface of it, the paper shows a good improvement over several commonly used methods in OOD generalization, improving on different benchmarks. However, many aspects of the paper are unclear: * Two of the main aspects of the paper are not sufficiently well motivated in my opinion -- * (a) _OOD generalization in time series_: Why is this particular method effective for time series? as far as the assumptions made in the paper go, this is a general technique that can be applied to any arbitrary dataset/modality. While it is true that time series problems do not receive as much attention, that alone is not motivation enough. What assumptions in this paper restrict the use of TTSO for a broader set of modalities? If none, then how does this compare on other benchmarks? * (b) _LLMs for augmentation_: Why is the LLM necessary? The idea that LLMs can be used to produce different views of the sample is interesting, but the empirical performance appears to be too be marginal at best (+1.4% gain vs not using it from table 1). It doesn't seem to justify the additional computational burden for the small gain in performance. It would be helpful if the paper clearly articulates what the hypothesis is exactly here with regard to LLMs -- the ablation study in Fig 2 is a good start -- what kinds of LLMs benefit when fine-tuning? Is there a bias within the GPT-2 model used that is beneficial for these datasets? What about more advanced architectures/or LLMs trained on larger datasets, are they expected to give a bigger boost in performance or does it plateau? * I think the paper is interesting and has a lot to say, but i would recommend making the core hypothesis and idea cleaner as I have outlined above.

Questions

Please see above

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes, it appears so.

Summary

Out-of-Distribution（OOD）generalization in ML emphasizes on improving model adaptability and robustness aginst unseen and potentially adversarial data. This paper explores OOD generalization for time series data with pre-trained Large Language Models and proposes a novel tri-level learning framework to handle the data distribution uncertainties. It goal is to address not only sample-level but also group-level uncertainties in the new dataset. The paper offers a theoretical analysis to justify the method and develops a cutting plane strategy for the tri-level optimization problem. It demonstrates guaranteed convergence. Extensive experiments on real-world datasets confirm the effectiveness and efficiency of the proposed method.

Strengths

1) OOD generation is an important problem in ensuring the robustness and reliability of machine learning models. It becomes increasingly important in AI as machine learning models and systems are expected to be deployed in numerous real-world applications in the near future. 2)The tri-level learning framework is grounded in robust theoretical principles and offers a fresh perspective for modeling and studying the OOD problem. This framework opens new avenues for investigating OOD challenges in time series data. 3)The TTSO algorithm is proven to converge, with the authors also determining its convergence speed. 4)Extensive experimental studies have been carried out to validate the effectiveness of the proposed methods.

Weaknesses

While the paper is well organized overall, the clarity of this paper can be further enhanced via providing examples to illustrate the concepts of sample-level and group-level uncertainties in time series.

Questions

Can the TSSO framework be employed to manage data uncertainties in other type of data, for example natural language data?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

N/A

Acknowledgement of Rebuttal

Thank you for your clarifications and additional analysis. (a) Regarding time series, I agree versatility is a strength, but the reason for studying time-series OOD generalization needs to be motivated better other than it is under studied. There is no clear reasoning for why this method is suitable for time series, which have their own sets of challenges and problems from noisy data, rate variation, length variation etc. different from images/text problems. (b) The fact that LLMs are optional cannot be an argument when it is in the title of the paper -- it is indeed a strength that LLMs can be incorporated, but this is not motivated sufficiently enough. The performance does not show a clear benefit, and the new results seem to further make this a bit muddled since we see larger models doing poorer than smaller ones, which is pretty much counter to current intuition on scaling LLMs. Given these issues, I will maintain my score as it is.

Summary of Contributions and Modifications

We sincerely thank the Area Chairs for managing the review process and the reviewers for their thoughtful consideration and evaluation of our work. To facilitate further discussions, we have summarized our contributions and modifications as follows: **Contributions** * We propose a novel **tri-level learning** framework, TTSO, to tackle time series OOD generalization with pre-trained LLMs. By integrating both sample-level and group-level uncertainties, TTSO advances beyond traditional single or bi-level methods in OOD generalization. We provide a theoretical analysis based on the Vapnik-Chervonenkis dimension to elucidate **generalization properties** of TTSO, and demonstrate its effectiveness through extensive real-world datasets. * We develop a stratified localization method using cutting planes, which is **more computationally** efficient compared to hypergradient-based tri-level optimization methods. And the decomposable nature of cutting planes provides a promising pathway for distributed implementations of TTSO, allowing the algorithm to be effectively applied to large-scale applications. * We conducted a **thorough theoretical analysis** of the proposed algorithm. The proposed stratified localization algorithm for tri-level optimization shows guaranteed convergence and we theoretically derive that the iteration complexity of the proposed algorithm for achieving an $\epsilon$-stationary point is bounded by $\mathcal{O}(\frac{1}{\epsilon^2})$. **Modifications** * (Clarity Enhancements) We have included examples using the HHAR dataset to better illustrate the concepts of sample-level and group-level uncertainties in time series data (Appendix H). * (Discussion on Applicability and Limitations) We have added more discussion of TTSO framework on the real-world applicability and potential limitations (Appendix G & F). * (Ablation Experiments) To clarify the efficacy of LLMs in our framework, we conducted additional ablation experiments, the results of which are now included in the revised manuscript (Appendix I). * (More Baselines) We have expanded our experimental analysis to include further comparisons with more time series OOD generalization methods in the revised manuscript (Section 5). * (Explanations to Theorem) We have added more explanations to make theoretical sections more accessible to a broader audience (Appendix A). We hope these revisions will address the concerns raised and better demonstrate the contributions of our work.