Summary
This paper presents a methodology for in-context fine-tuning of a time series foundation model (TimesFM) to enhance forecasting capabilities. The authors did begin with a base TimesFM checkpoint and adapted it to leverage not only the target time series' historical data but also related series examples at inference time. Their findings demonstrate that in-context fine-tuning can significantly boost zero-shot performance on popular forecasting benchmarks, surpassing both the base model that was fine-tuned directly on the target domain. Notably, it even outperforms a version of the base model that was fine-tuned directly on the target domain. Although this study focuses on one specific foundation model for time series forecasting, exploring in-context fine-tuning with other foundation models could be a promising area for future research. Additionally, investigating improved relative positional encoding, the authors mentioned NoPE for continued pre-training, particularly tailored for handling in-context examples and length generalization, would be another valuable direction!
Strengths
(1) The authors were motivated by the strong zero-shot performance of the GPT model in time series forecasting and proposed an adaptation of the TimesFM foundation model to leverage additional information through in-context examples. They initially pre-trained TimesFM in its original form to establish a base checkpoint, then modified the model’s architecture and continued pre-training with in-context examples to develop a new pre-trained foundation model, TimesFM-ICF!
(2) Regarding originality, the paper introduces the concept of in-context fine-tuning specifically adapted for time-series data. While leveraging context in machine learning is not new, the authors skillfully adapt this idea from NLP to address the unique challenges of time-series forecasting. This approach is especially compelling, as it combines established principles from LLMs with the nuances of temporal data, filling a gap in current literature. The authors’ methodology redefines the possibilities for time-series forecasting and opens new research avenues, suggesting that similar examples in time-series data can enhance model performance beyond what traditional methods achieve.
(3) The research quality is evident in the rigorous methodology applied throughout the study. The authors support their claims with extensive empirical evaluations on prominent forecasting benchmarks, showing clear performance improvements over both supervised deep learning methods and existing time-series foundation models. This empirical evidence reinforces the credibility of their approach, and the thorough experimental design—such as careful dataset selection and detailed result analysis—reflects a high standard of quality!
(5) Clarity is another strong point, as the authors clearly communicate their ideas and findings. The paper’s structure is logical and well-organized, leading the reader smoothly through the introduction of the problem, methodology, and results. Clear language and precise terminology help demystify complex concepts, making the paper accessible even to those less familiar with time-series forecasting. Illustrative examples and figures further aid understanding, showing the practical implications of in-context fine-tuning!
(6) The paper’s significance is substantial. By demonstrating that in-context fine-tuning significantly improves forecasting accuracy, the authors offer insights that could influence both research and practical applications. The findings suggest that this approach could be a game-changer for practitioners, enabling improved forecasting accuracy without the need for extensive task-specific training data. Moreover, the potential applications of in-context learning extend beyond time-series forecasting, expanding the relevance of this research to other domains
Weaknesses
(1) The novelty of this work could be made clearer with a deeper comparative analysis against established methods. The authors claim their in-context fine-tuning approach outperforms traditional supervised learning and other foundation models. However, it would be helpful if they could further clarify what sets this approach apart, especially concerning few-shot learning techniques commonly applied in language models. In particular, it would be insightful for the authors to explain how their method handles the unique temporal dependencies in time series data, a challenge that typical language model techniques might not address! It would also be useful to compare the approach to well-known techniques, such as those used in [1] and [2]. It could provide a stronger context for understanding the distinctiveness and value of the method within the broader landscape of in-context learning for time series data!
(2) The experimental design would benefit from testing on a wider variety of datasets and forecasting scenarios. Currently, the benchmarks don’t seem to fully reflect the range of time-series data seen in real-world applications. It might help to include datasets that capture different characteristics—like high seasonality, irregular sampling, or multivariate setups—since these are common in practice. For instance, datasets like LOTSA (as used in [4]) show different levels of noise, seasonality, and trend, which would provide a more comprehensive picture of the model’s performance. It would also give insight into how well the model generalizes across different types of time series and adapts to diverse patterns!
(3) To strengthen the paper, I would suggest a more detailed analysis of the criteria used to select in-context examples during inference. The authors mention that using related time series examples improves model performance, but it would be helpful to explain how they determine which examples are " related " enough to be effective. An ablation study comparing different selection approaches, such as the use of domain knowledge, could clarify the impact of these choices on model performance and provide useful insights for future applications!
(4) To add balance to the discussion, it would be valuable to include a deeper look at the approach's limitations. Specifically, it could be helpful to examine situations where the method might struggle, such as when it encounters new or unusual time series patterns not represented in the training set or in the in-context examples. I think it would also be beneficial if the authors considered possible strategies for addressing these limitations or outlined some directions for future work. For example, they might explore ways to detect when the model encounters out-of-distribution patterns or consider methods to adapt the model to new data patterns without extensive retraining!
[1] Language Models are Few-shot Learners, Brown et al. (2020)
[2] Improving In-Context Few-Shot Learning via Self-Supervised Training, Chen et al. (2022)
[3] What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, Garg et al. (2022)
[4] Unified Training of Universal Time Series Forecasting Transformers., Woo et al. (2024)
Questions
(1) While the authors indicate that related time-series examples can improve forecasting accuracy, they do not clarify the criteria for selecting these examples! A more detailed explanation of the selection process, whether based on similarity metrics, temporal proximity, or other factors, would provide valuable insight into the model's adaptability. Additionally, understanding how the model performs when the in-context examples are less relevant or of lower quality would be useful! This could open a discussion on the model's robustness in real-world scenarios where data may be noisy or incomplete!
(2) Another potential area for improvement is the discussion of limitations regarding scalability and computational efficiency. As the number of in-context examples increases, so may the computational load, which could affect the model's feasibility for large-scale applications. Insight into how the model manages larger context windows, as well as any diminishing return in performance with an increasing number of examples, would be valuable. This consideration could also connect to the findings from the [1], which investigates context efficiency in LLMs. A comparison with the proposed approach might highlight areas for adaptation or further improvement!
(3) A more comprehensive look at the model's performance across different forecasting horizons would strengthen the evaluation. While the authors mention that the model adapts to various contexts, empirical results on how it performs across short and long-term forecasting horizons would be valuable. This could reveal whether in-context fine-tuning is consistently effective or better suited for specific forecasting tasks!
(4) The authors suggest that their approach rivals traditional fine-tuning on the target domain, but a deeper analysis of the trade-offs between in-context and conventional fine-tuning would be beneficial. Specifically, in which scenarios might one approach be preferable? Clarifying these nuances would aid practitioners in choosing the appropriate method for their specific forecasting needs!
(5) The paper could broaden its discussion on the implications for future research. While the authors suggest potential adaptations for other foundation models, a more detailed exploration of how these adaptations might extend to different domains, such as image or text data, would be valuable. Drawing connections to the broader landscape of foundation models, including references like [1], would offer a richer context for the implications of their work!
After addressing these suggestions and the weakness part, it would be possible to improve the score!
[1] Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting, Rasul et al. (2024)
,