In-context Time Series Predictor

Thank you very much for recognizing the contribution of our paper. We are delighted to receive your acknowledgment. We are pleased that you found our work to have a "category theory" flavor, as this aligns with what we aimed to convey in the paper. Using the ICL perspective, we analyzed previous Transformer-based time series forecasting structures, identified their issues, categorized them into Temporal-wise and Series-wise Transformers, and explained how past strategies, such as channel independence and patching operations, enhance Temporal-wise Transformers. We are happy to provide a summary of the core discussions in the paper as follows: | **Methods** | **Tokenization** | **ICL (x, y) Pair** | **Token Representation for** | **Attention to Learn** | **Suited For** | **Not Suited For** | **ICL Context Examples Describing** | **Channel Structure** | **Problem** | |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | **Temporal-wise Transformer** | Timestep embedding | (Timestep index, Multi-channel values) | Single-step inter-series dependencies | Temporal-wise dynamics of series dependencies | Strong and consistent inter-series dependencies | Weak inter-series dependencies in real-world datasets | Underlying inter-series effects vary with the time indices in the context lookback (likely nonexistent or weak) | Fixed | Only series relationships in tokenization; risks overfitting to series-wise effects from the input context | | **Temporal-wise Transformer (with Channel Independence)** | Scalar embedding | (Timestep index, Single-channel value) | Scalar value at a single timestep | Temporal-wise dependencies | Strong and fast-changing temporal dependencies | Strong inter-series dependencies | Temporal relationships within the context for one series | Fixed | Unable to model series-wise relationships | | **Temporal-wise Transformer (with Patching)** | Patch embedding | (Timestep index, Multi-channel values in a temporal interval) | Temporal-wise and series-wise relationships in a local interval | Temporal-wise relationships between local intervals | Temporal and series relationships can be automatically balanced by this structure | Few-shot/zero-shot cases with different series relationship characteristics | Temporal-wise relationships between local intervals | Fixed | Fixed channel structure in tokenization prevents generalization to datasets with different structures | | **Series-wise Transformer** | Series embedding | (Input series, Output series) | Temporal-wise relationships within a single series input | Aligned inter-series relationships | Real-world datasets with strong temporal-wise and aligned series-wise relationships (e.g., traffic) | Unaligned series-wise relationships, such as shifting effects between series | No context examples | Flexible but requires additional training to adapt to datasets with new channel structure | Restricted to specific dataset channel structures; unable to learn unaligned series-wise effects | | **ICTSP** | Embedding of forecasting tasks | (Input series, Output series) | Temporal-wise relationships in a forecasting task | Forecasting task predictors, considering both aligned and unaligned inter-series relationships in shifted context examples | Datasets with fast-changing temporal relationships, aligned or unaligned series relationships; transfering to new datasets with different channel structures | / | Historical ground truth forecasting tasks as references | Flexible; additional training is not necessary since context examples exist | / | In the revised paper, we have added more discussions on the effectiveness of forecasting task tokenization's capturing of different relationships, and included additional ablation studies to demonstrate the superior and consistent performance of ICTSP compared to prior models across full data, few-shot, and zero-shot tasks. Thank you again for your recognition and valuable suggestions. If you have any further questions, we would be more than happy to address them.

``` 1. Could you provide results with ICTSP’s input length set to 512 for a more equitable comparison with other baselines in Table 1 and 6? ``` Thank you for this meaningful question. First, the choice of input length \(L_I\) is widely discussed in the literature, with different models often requiring different optimal lengths depending on the dataset. For example: - **GPT4TS** [1] (Section H.10) suggests using the optimal \(L_I\) reported in original papers rather than a single \(L_I\) for all baselines, as longer lookbacks often cause overfitting in some models, resulting in unfair comparisons. - Studies like **TiDE** [2] (Section 5.1) selects the best \(L_I\) from multiple tested lengths to ensure fair comparisons. - **DLinear** [3] (Figure 6), **PatchTST** [4] (Table 9), and **iTransformer** [5] (Figure 6) show that for many real-world datasets, shorter \(L_I\) often performs better as longer lookbacks can lead to overfitting. Thus, effectively utilizing longer \(L_I\) while avoiding overfitting remains a common challenge. ICTSP addresses this by treating segments of length \(L_b + L_P\) as forecasting task tokens and creating \(N = L_I - (L_b + L_P)\) task tokens to incorporate lookback information. With \(L_b = 512\), each token represents a forecasting task used by models like PatchTST or GPT4TS. ICTSP can be viewed as leveraging past forecasting task datapoints to optimize the predictor for the current forecasting task, enhancing generalization performance. While we believe this rationale supports our approach, **we are happy to conduct an ablation study using only 512-step input information**. However, this configuration disrupts ICTSP's **forecasting tasks as context** structure and is **not recommended for practical use**. ICTSP requires \(L_I > L_b + L_P\) to split the input into multiple ground truth forecasting tasks. Given \(L_b = 512\), when \(L_P = 720\), \(L_I > 512 + 720 = 1232\) is necessary to construct at least one ground truth forecasting task context token. ### Ablation Study: ICTSP with limited lookback information To fully align with the baseline models using \(L_I = 512\), we **relax ICTSP's design**, which originally required all context tokens to be ground truth forecasting tasks. Instead, we **allow some forecasting tasks to include incomplete information**. Specifically, we retain \(L_I = 1440\) but **mask the first \(1440 - 512 = 928\) timesteps in the input**, ensuring **only the last 512 steps contain true information**. The masked portion is **filled with the mean value** of each series token **from the last 512 steps**. To maintain the presence of some true forecasting task tokens for shorter \(L_P\), we shorten \(L_b\) to 336. All other hyperparameters remain unchanged. This adjusted structure is denoted as **ICTSP (Limited)**. We repeated experiments on the ETTm2 and Weather datasets, covering full data, few-shot 10\% (FS10\%), few-shot 5\% (FS05\%), and zero-shot (ZS) settings for the ETT datasets. Full results are available in Table 11 of the revised paper. Below is the summary of average MSE and average rank compared to six strong baselines. | Methods | ICTSP | ICTSP (Limited) | Time-LLM | GPT4TS | PatchTST | TimesNet | DLinear | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | Full ETTm2 (Avg) | **0.247** | 0.250 | 0.251 | 0.267 | 0.255 | 0.291 | 0.267 | | Full Weather (Avg) | **0.218** | 0.220 | 0.226 | 0.237 | 0.226 | 0.259 | 0.249 | | FS10% ETTm2 (Avg) | 0.275 | **0.274** | 0.277 | 0.293 | 0.296 | 0.320 | 0.316 | | FS10% Weather (Avg) | 0.237 | 0.238 | **0.234** | 0.238 | 0.242 | 0.279 | 0.241 | | FS05% ETTm2 (Avg) | 0.285 | 0.281 | **0.274** | 0.308 | 0.315 | 0.399 | 0.345 | | FS05% Weather (Avg) | **0.256** | **0.256** | 0.261 | 0.264 | 0.270 | 0.264 | 0.298 | | ZS m1→m2 (Avg) | **0.255** | 0.258 | 0.265 | 0.314 | 0.297 | 0.322 | 0.335 | | ZS h1→h2 (Avg) | **0.337** | 0.339 | 0.352 | 0.407 | 0.381 | 0.421 | 0.494 | | AvgRank | **1.66** | 1.97 | 2.66 | 4.56 | 4.34 | 6.31 | 6.19 | From the average rankings, we observe that even with masking applied to information outside the 512-step window, **ICTSP (Limited) still significantly outperforms all baselines**. In full data and zero-shot experiments, ICTSP (Limited) performs slightly weaker than the original ICTSP but remains superior to all baselines. However, in few-shot training experiments, ICTSP (Limited) surpasses the original ICTSP. It is important to note that this improvement is **due to an additional constraint in the original ICTSP**, which, despite using $L_I=1440$, imposes **zero-filling** restrictions outside of $L_b=512$ to ensure fair comparison with baselines using $L_I=512$ (see Section A.3.3 Discussion of fair comparison for details on **few-shot experiments**).

In the full data and zero-shot experiments, all data can be used with any $L_I$ without causing data leakage. However, in few-shot experiments, due to skipping in data sampling, a longer \(L_I\) accesses more lookback data during validation/testing, giving an unfair advantage. To address this, the original ICTSP applied **zero-masking** to restrict access to only 512 valid timesteps at the start of validation/testing, gradually expanding to \(L_I = 1440\) as more valid data became available. Nonetheless, zero-filling in forecasting task tokens caused issues because of the **last-value demeaning operation** used on the forecasting task tokens, resulting in meaningless negative last-value placeholders and reduced model performance. **ICTSP (Limited)** resolves this by using **mean-value filling**, ensuring more reasonable values in forecasting task tokens. This adjustment allows ICTSP (Limited) to outperform the original ICTSP in few-shot experiments. To clarify this, we **replaced the zero-filling used for few-shot fair comparison in the original ICTSP with mean-filling** and present the results below. ICTSP with mean-filling **significantly outperforms** all models, including Time-LLM, across all experiments. This highlights that ICTSP’s few-shot capability is inherently **stronger** than original results presented in the paper when proper mean-filling strategies are applied. Full results can be found in Table 12 of the revised paper. | Methods | ICTSP (Zero-filling) | ICTSP (Mean-filling) | ICTSP (Limited) | Time-LLM | GPT4TS | PatchTST | TimesNet | DLinear | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | FS10% ETTm2 (Avg) | 0.275 | **0.264** | 0.274 | 0.277 | 0.293 | 0.296 | 0.320 | 0.316 | | FS10% Weather (Avg) | 0.237 | **0.234** | 0.238 | **0.234** | 0.238 | 0.242 | 0.279 | 0.241 | | FS05% ETTm2 (Avg) | 0.285 | **0.272** | 0.281 | 0.274 | 0.308 | 0.315 | 0.399 | 0.345 | | FS05% Weather (Avg) | 0.256 | **0.248** | 0.256 | 0.261 | 0.264 | 0.270 | 0.264 | 0.298 | ``` 2. Could you conduct an ablation study on in-context examples, possibly using randomly selected examples (without look-ahead bias) instead of adjacent history to clarify the influence of this design choice? 3. Could you provide more detailed analysis or visualizations related to the Adaptive Model Reduction to support its impact on performance? ``` Thank you for your valuable questions. We will address both issues together in our response. Regarding the **Adaptive Model Reduction** of ICTSP, because we use a pre-LayerNorm Transformer as described in Eq. 1, it inherently maintains a direct residual shortcut from the input token to the output token. a) With small-scale weight initialization, if Transformer layers introduce no residual effects (bypassed), the model simplifies to a linear projection, akin to a **NLinear model**(as we apply last-value demeaning for tokens). b) If attention layers are bypassed during training, the model acts as a pure **univariate MLP predictor**, which many recent TSF models [5-7] use as their core temporal predictor. Note that neither of above scenarios utilizes context tokens. c) When context examples provide no useful information for the predictor of the target token, ICTSP reduces to a **Series-wise Transformer (like iTransformer)**. Here, the target token can be directly interpreted as the series embedding since the future portion of the target token is not trained in-context. d) When **context forecasting task tokens** aid in determining the target token's predictor, ICTSP adjusts predictions based on the them. Context examples act as training samples in a supervised learning setup. Even with limited subset of context examples, ICTSP can still update its predictor in-context, outperforming models that ignore context entirely. However, its performance remains below that of the full ICTSP, which uses more context samples. ### Ablation Study: Adaptive Model Reduction and Random Selection of Context Examples Based on the above discussion, we designed the following progressive experiments to compare the performance differences resulting from using only a subset of the model structure. The experiments were conducted on the Weather and ETTm2 datasets, as shown below. The original results can be found in Table 13 of the revised paper.

- **Model Variants** A. We **remove all Transformer blocks** from ICTSP. In this case, the model only includes the input and output linear layers for the target token, effectively **reducing it to an NLinear model**. B. We **retain only the MLP feedforward layers** within the Transformer blocks of ICTSP. The model can then be considered a **univariate MLP predictor**. C. We **retain the Transformer blocks** in ICTSP but **drop all context tokens**. The model then reduces to a **Series-wise Transformer (iTransformer)**. D. We **retain the full ICTSP structure** but **randomly keep only 25% of the context tokens** in the context examples. The model sees randomly incomplete in-context forecasting task samples. E. We increase the random retention rate of context tokens to **50%**. F. We further increase the random retention rate of context tokens to **75%**. G. We **use all context tokens**, representing the **full ICTSP model**. | Methods | A. w/o Transformer (Reduced to NLinear) | B. w/o Attention (Reduced to Univariate MLP) | C. w/o Context Tokens (Reduced to Series-wise Transformer) | D. w/ 25% Random Selected Context Tokens | E. w/ 50% Random Selected Context Tokens | F. w/ 75% Random Selected Context Tokens | G. Full ICTSP | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | Weather (96) | 0.168 | 0.148 | 0.145 | 0.146 | 0.140 | 0.140 | 0.139 | | Weather (192) | 0.213 | 0.191 | 0.193 | 0.190 | 0.188 | 0.185 | 0.186 | | Weather (336) | 0.259 | 0.245 | 0.249 | 0.244 | 0.241 | 0.240 | 0.239 | | Weather (720) | 0.325 | 0.319 | 0.314 | 0.311 | 0.312 | 0.309 | 0.306 | | **Weather (Avg)** | **0.241** | **0.226** | **0.225** | **0.223** | **0.220** | **0.219** | **0.218** | | ETTm2 (96) | 0.163 | 0.163 | 0.165 | 0.162 | 0.161 | 0.161 | 0.159 | | ETTm2 (192) | 0.219 | 0.219 | 0.220 | 0.216 | 0.214 | 0.215 | 0.212 | | ETTm2 (336) | 0.274 | 0.272 | 0.271 | 0.268 | 0.268 | 0.268 | 0.268 | | ETTm2 (720) | 0.361 | 0.354 | 0.361 | 0.358 | 0.355 | 0.349 | 0.347 | | **ETTm2 (Avg)** | **0.254** | **0.252** | **0.254** | **0.251** | **0.250** | **0.248** | **0.247** | The transition from **A. NLinear** to **B. Univariate MLP** shows a **significant performance improvement**, as the MLP introduces more weights, enabling better handling of temporal relationships within tokenization. The improvement from **B. Univariate MLP** to **C. Series-wise Transformer** is **less significant**. This indicates that on weakly correlated real-world datasets, the primary improvement of the Series-wise Transformer compared to a Temporal-wise Transformer comes from the use of series tokenization for better handling of temporal relationship, rather than using the attention layers to capture aligned series-wise relationships. The MLP structure in **B.** alone is sufficient for this architecture to achieve performance comparable to the Series-wise Transformer. As the model expands from **C.** to **D., E., F., and G.**, the number of visible context examples increases. Using **only 25%-50% of randomly selected context tokens allows ICTSP to significantly outperform the Series-wise Transformer** without any context examples. This demonstrates that the attention layers in the model effectively perform the task of in-context predictor determination. **Increasing the proportion of context tokens to 75% provides further improvements**, with performance **nearly matching the full ICTSP** using all context tokens. ``` 4. How does ICTSP handle cross-channel dependencies, especially in cases with specific inter-channel relationships in the data (like traffic)? ``` Thank you for raising this important question, which is key to understanding the significance of our paper. In the main paper, we discussed that **Temporal-wise Transformers** take the values of different channels at a given timestep as input tokens. This approach results in token vectors that primarily encode potentially weak inter-series relationships from real-world datasets, while leaving the more critical **temporal-wise relationships** in TSF tasks to the attention layers with dynamic weights to construct algorithms. This leads to significant overfitting to inter-series effects in the lookback window. Recent models with **patch embeddings** mitigate this by introducing local temporal relationships inside tokenization, reducing overfitting. Also, models like **PatchTST** enforce channel independence, cutting off inter-series relationships entirely to focus solely on temporal relationships. **Series-wise Transformers**, like **iTransformer**, push this further by embedding entire series as tokens, emphasizing temporal relationships even more in tokenization while leaving series-wise relationships to attention layers. However, these models mainly capture **temporally aligned effects** but not unaligned effects, making them less effective for datasets with stable **shifted inter-series dependencies**.

**ICTSP** addresses this by incorporating historical forecasting tasks as context tokens, which naturally prompts the model to account for shifted inter-series dependencies. These task tokens inherently exhibit shifted relationships with one another, guiding the model to explore these dependencies. **Figure 4** illustrates this clearly. On a synthetic dataset (`Multi`) with weak temporal relationships but strong shifted inter-series dependencies, **Temporal-wise Transformer** and **ICTSP** significantly outperform **Series-wise Transformer**. On the real-world dataset (`ETTm2`), where temporal relationships are stronger and inter-series dependencies weaker, **ICTSP** and **Series-wise Transformer** outperform the overfitted Temporal-wise Transformer. To further aid your understanding, we propose a **feature space perspective** to explain why ICTSP effectively models both temporal-wise and series-wise relationships. - **Feature Space Perspective** - From the ICL perspective, the attention layers in Transformers construct algorithms (e.g., linear regression, shallow MLPs) between the input (x, y) pairs. These layers can be seen as dynamically solving a supervised learning problem based on the input datapoint tokens. - **Temporal-wise Transformers** can be viewed as solving a regression problem where the input is temporal indices (x), and the output is multi-channel values (y). - **Series-wise Transformers** and **ICTSP**, on the other hand, can be seen as solving a regression problem where the input is a time series segment (x), and the output is another time series segment (y). - In the default assumptions of regression problems, it is relatively straightforward to model relationships between input features within the same datapoint and relationships between the same input features across different datapoints (e.g., through linear combinations). However, it is much harder to model relationships between **different input features across different datapoints**. - For **Series-wise Transformers** and **ICTSP**, we want the input time series segment (x) to have **shifting lag effects** between its features (similar to the classic Box–Jenkins method). However, without explicitly indicating this, the default assumptions of regression problems make it easier for ICL to model: - Relationships between features within the same datapoint (temporal dependencies). - Relationships between the same features across different datapoints (temporal-aligned inter-series dependencies). - However, it is much harder for ICL to recognize that relationships may exist between **different features across different datapoints** (unaligned inter-series dependencies). This is a key limitation of Series-wise Transformers when modeling cross-channel dependencies. - **ICTSP** addresses this limitation by explicitly providing historical forecasting tasks as **context example tokens**. These historical forecasting tasks inherently have a **shifted structure**, which directly prompts the model to consider the necessity of **unaligned inter-series relationships**. - As a result, ICTSP achieves: - The ability to model **temporal relationships** through its tokenization approach. - The ability to model **aligned inter-series dependencies** through its output token formulation. - The ability to model **unaligned inter-series dependencies** through the provision of shifted context forecasting examples. The underlying cause of these challenges lies in the **shifting time-dependent structure** unique to time series data. In other supervised learning problems, we typically do not consider shifting certain features to analyze their relationships with others. Therefore, in ICL problems where each datapoint is treated as a token, ICTSP outperforms Series-wise Transformers by effectively teaching the model to learn these shifting dependencies. We greatly appreciate your insightful feedback and are open to any further questions or suggestions. We look forward to engaging in more detailed discussions and thank you for your consideration.

### References [1] Zhou, Tian, et al. "One fits all: Power general time series analysis by pretrained lm." Advances in neural information processing systems 36 (2023): 43322-43355. [2] Das, Abhimanyu, et al. "Long-term Forecasting with TiDE: Time-series Dense Encoder." Transactions on Machine Learning Research. [3] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023. [4] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations. [5] Liu, Yong, et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." The Twelfth International Conference on Learning Representations. [6] Ekambaram, Vijay, et al. "Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023. [7] Wang, Shiyu, et al. "Timemixer: Decomposable multiscale mixing for time series forecasting." arXiv preprint arXiv:2405.14616 (2024).

I sincerely appreciate the authors for their detailed and thoughtful response, as well as for conducting additional experiments. I still have some follow-up questions: 1.Regarding your response to Q1, the results are only on ETTm2 and Weather rather than the whole dataset. Seeing the experiment results for all the data would be more convincing. It does not need to include FS 10% / 5 % or ZS. Just additional results like Table 1 in paper would be fine. 2.Regarding your response to Q2, Q3, unfortunately I may not be entirely clear of what I am proposing about the “randomly selected examples”. My primary doubt lies in whether the purported “in-context learning” mechanism genuinely contributes unique forecasting capabilities. Specifically, I am interested in understanding whether the model’s performance gains stem from effectively having a longer input length, or from reformulating the original forecasting problem into the proposed ICTSP framework. (See weakness 2) The experiment described in your response, involving x% randomly selected context tokens, does not fully address this question. It still could have a longer input window. Instead, I propose testing with random examples that are not from adjacent history, perhaps drawn from much older (less relevant) history or even other datasets. In this scenario, the benefit of a longer input window would be eliminated. If “in-context learning” is truly effective, the model should demonstrate improved performance even with these random examples, indicating that it can derive meaningful insights from the provided context. That said, the above is just a suggested experiment to address my concerns. If you can propose alternative experiments that more directly address this question, I would welcome those as well.

| Methods | A. w/o Transformer (Reduced to NLinear) | B. w/o Attention (Reduced to Univariate MLP) | C. w/o Context Tokens (Reduced to Series-wise Transformer) | H. 10% Distant Context Examples | I. 25% Distant Context Examples | J. 75% Distant Context Examples | G. Full ICTSP | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | Weather (96) | 0.168 | 0.148 | 0.145 | 0.146 | 0.145 | 0.142 | 0.139 | | Weather (192) | 0.213 | 0.191 | 0.193 | 0.192 | 0.191 | 0.189 | 0.186 | | Weather (336) | 0.259 | 0.245 | 0.249 | 0.246 | 0.243 | 0.242 | 0.239 | | Weather (720) | 0.325 | 0.319 | 0.314 | 0.314 | 0.311 | 0.310 | 0.306 | | **Weather (Avg)** | **0.241** | **0.226** | **0.225** | **0.225** | **0.223** | **0.221** | **0.218** | | ETTm2 (96) | 0.163 | 0.163 | 0.165 | 0.164 | 0.164 | 0.161 | 0.159 | | ETTm2 (192) | 0.219 | 0.219 | 0.220 | 0.218 | 0.219 | 0.215 | 0.212 | | ETTm2 (336) | 0.274 | 0.272 | 0.271 | 0.270 | 0.270 | 0.268 | 0.268 | | ETTm2 (720) | 0.361 | 0.354 | 0.361 | 0.355 | 0.352 | 0.351 | 0.347 | | **ETTm2 (Avg)** | **0.254** | **0.252** | **0.254** | **0.252** | **0.251** | **0.249** | **0.247** | It can be observed that even when introducing relatively distant context examples, they still provide better performance compared to the Series-wise Transformer, which does not use any context tokens. Additionally, when only a small portion of context examples is introduced, the performance improvement compared to having no context is relatively modest. However, as the number of context examples increases, their contribution to performance also grows. This aligns with previous observations in ICL literature regarding the impact of the number of context examples used. ### Ablation Study: Understanding the Specific Contributions of Context Examples We designed the following experiments to demonstrate how **introducing forecasting tasks as prompt tokens** specifically contributes to performance. These experiments aim to validate that ICTSP relies on context examples to work **via ICL**. Particularly in the zero-shot setting, where **no related training samples' information is in model parameters**, ICTSP’s superior performance is primarily **driven by its ability to update the forecasting predictor using context examples**. Different types of context forecasting examples provide **specific improvements** to the corresponding forecasting task targets. We use ETTm2 as the target dataset and train ICTSP under three different training set configurations: ETTm2, ETTm1, and a combined set of (ETTh1, ETTh2, ETTm1). For these configurations, we conduct the corresponding full data training and zero-shot transfer learning. **During evaluation, we independently generate context examples from series $X_1, \dots, X_7$ in ETTm2 and feed them to the context part of ICTSP for test set MSE evaluation**. We use $L_P=96$, with all other settings matching those in the main experiments. We construct the following variants: 1. **w/o Context**: No context examples are used from any series. 2. **Context ($X_j$)**: Only the context examples corresponding to series $X_j$ are used. 3. **Context (Full)**: The full ICTSP using all context examples. For each variant, we report the MSE ($X_j$) corresponding to each series $j$ in the test set forecasting results. Below are the results for the three experimental settings. For each MSE ($X_j$), the **best value** is **bolded**, and the *second best* is *italicized* for the **w/o Context** and **Context ($X_j$)** parts of experiments. ### ETTm2 -> ETTm2 (full data) From the results below, we can observe that using the context examples from a specific series $X_j$ improves the model's performance on the corresponding MSE ($X_j$). However, **we cannot fully attribute this improvement to the benefits of ICL. In the full data setting**, the model's parameters already contain information about the relationships between series within the dataset. As a result, the **model may still leverage the information from context examples through these fixed relationships encoded in the weights**. We can see that introducing $X_j$ context not only **improves the performance on $X_j$** but also significantly enhances the performance of **other series that are highly correlated with $X_j$**. Therefore, we **cannot really distinguish** whether the improvement arises from fixed series relationships in the parameters or from ICL.

Dear Reviewer H6bM, Thank you for your valuable follow-up questions earlier. We have provided additional responses to your concerns above. With approximately two days remaining until the end of the ICLR rebuttal period, we would like to know if the information provided resolves your concerns or if you have any further questions. If so, we would be happy to provide a quick answer. If you still have questions about why our model can demonstrate ICL capabilities without relying on very large-scale LLM parameters, we have addressed this in our responses to other reviewers. You may refer to the following reply for more details: [https://openreview.net/forum?id=dCcY2pyNIO&noteId=FORo1hmDUB](https://openreview.net/forum?id=dCcY2pyNIO&noteId=FORo1hmDUB). Once again, we deeply appreciate your thoughtful questions, which have significantly helped us better present our research. Thank you for the time and effort you’ve devoted to the review process. If you are currently on holiday, we also wish you a wonderful break. Best regards, Authors of Submission #12786

Thanks for the authors' response. Though I am still not fully convinced that the additional experiments are sufficient to claim the benefits of ICL in time series, they indeed provide more interesting observations. I am changing my score accordingly.

``` 1. Can you add more discussions for these methods based on patching embedding from an ICL perspective? ``` Thank you for highlighting the core content of our paper. From the ICL perspective, explaining how patch embedding improves Temporal-wise Transformers is a key part of our work, elaborated in Section 2.3. We view patch embedding as an enhancement to Temporal-wise tokenization: when applied across all series, it corresponds to Figure 3(d); when applied independently to each series, it combines the methods in Figures 3(b) and 3(d). We argue that the performance improvement of the **patching method** in Figure 3(d) stems from its ability to **introduce more learnable temporal-wise relationships directly within the tokenization**. This avoids relying solely on attention layers to construct these relationships algorithmically. #### Patch embedding enhances learnable temporal relationships in timestep embedding while preserving autoregressive token structure unlike series embedding In real-world multivariate time series forecasting, temporal-wise dependencies are generally more reliable, while series-wise relationships are often weak or even absent. Vanilla temporal-wise timestep tokenization fixes potentially non-existent series relationships in token representations and leaves critical temporal relationships to be dynamically constructed by attention layers, which is suboptimal. Patching addresses this by incorporating both local temporal-wise and series-wise relationships into tokens, naturally emphasizing stronger temporal relationships inside the token and reducing overfitting to non-existent series-wise effects in the input context. Similarly, PatchTST applies **channel-independent tokenization** alongside patching, explicitly dropping inter-series connections. This forces the model to focus only on temporal relationships, further reducing overfitting on real-world datasets. Figure 4 illustrates this with two examples. On a synthetic dataset (`Multi`) with weak intra-series temporal relationships but strong inter-series (unaligned shifted) dependencies, the vanilla Temporal-wise Transformer performs unusually well, contrasting with its behavior on real-world datasets. However, on a real-world dataset (`ETTm2`), where temporal relationships dominate and series-wise dependencies are weaker and more volatile, Temporal-wise Transformers perform poorly compared to models like Series-wise Transformer and ICTSP, which prioritize temporal relationships in tokenization. Although patch embeddings blend characteristics of Temporal-wise and Series-wise Transformers, they are categorized as Temporal-wise Transformers in the paper because their tokenization segments along the temporal dimension, preserving the **next-token prediction (autoregressive) structure**, where the **next token is the "answer" to the previous token**. **Adding patch embeddings does not disrupt this relationship between tokens, whereas series embedding methods completely break this structure**. ``` 2. For the Full-data TSF setting, can you rerun the baselines under the same experimental settings? Alternatively, you could articulate the rationale behind your experimental settings, which would make the comparisons more convincing. ``` Thank you for this meaningful question. First, the choice of input length \(L_I\) is widely discussed in the literature, with different models often requiring different optimal lengths depending on the dataset. For example: - **GPT4TS** [1] (Section H.10) suggests using the optimal \(L_I\) reported in original papers rather than a single \(L_I\) for all baselines, as longer lookbacks often cause overfitting in some models, resulting in unfair comparisons. - Studies like **TiDE** [2] (Section 5.1) selects the best \(L_I\) from multiple tested lengths to ensure fair comparisons. - **DLinear** [3] (Figure 6), **PatchTST** [4] (Table 9), and **iTransformer** [5] (Figure 6) show that for many real-world datasets, shorter \(L_I\) often performs better as longer lookbacks can lead to overfitting. Thus, effectively utilizing longer \(L_I\) while avoiding overfitting remains a common challenge. ICTSP addresses this by treating segments of length \(L_b + L_P\) as forecasting task tokens and creating \(N = L_I - (L_b + L_P)\) task tokens to incorporate lookback information. With \(L_b = 512\), each token represents a forecasting task used by models like PatchTST or GPT4TS. ICTSP can be viewed as leveraging past forecasting task datapoints to optimize the predictor for the current forecasting task, enhancing generalization performance.

While we believe this rationale supports our approach, **we are happy to conduct an ablation study using only 512-step input information**. However, this configuration disrupts ICTSP's **forecasting tasks as context** structure and is **not recommended for practical use**. ICTSP requires \(L_I > L_b + L_P\) to split the input into multiple ground truth forecasting tasks. Given \(L_b = 512\), when \(L_P = 720\), \(L_I > 512 + 720 = 1232\) is necessary to construct at least one ground truth forecasting task context token. ### Ablation Study: ICTSP with limited lookback information To fully align with the baseline models using \(L_I = 512\), we **relax ICTSP's design**, which originally required all context tokens to be ground truth forecasting tasks. Instead, we **allow some forecasting tasks to include incomplete information**. Specifically, we retain \(L_I = 1440\) but **mask the first \(1440 - 512 = 928\) timesteps in the input**, ensuring **only the last 512 steps contain true information**. The masked portion is **filled with the mean value** of each series token **from the last 512 steps**. To maintain the presence of some true forecasting task tokens for shorter \(L_P\), we shorten \(L_b\) to 336. All other hyperparameters remain unchanged. This adjusted structure is denoted as **ICTSP (Limited)**. We repeated experiments on the ETTm2 and Weather datasets, covering full data, few-shot 10\% (FS10\%), few-shot 5\% (FS05\%), and zero-shot (ZS) settings for the ETT datasets. Full results are available in Table 11 of the revised paper. Below is the summary of average MSE and average rank compared to six strong baselines. | Methods | ICTSP | ICTSP (Limited) | Time-LLM | GPT4TS | PatchTST | TimesNet | DLinear | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | Full ETTm2 (Avg) | **0.247** | 0.250 | 0.251 | 0.267 | 0.255 | 0.291 | 0.267 | | Full Weather (Avg) | **0.218** | 0.220 | 0.226 | 0.237 | 0.226 | 0.259 | 0.249 | | FS10% ETTm2 (Avg) | 0.275 | **0.274** | 0.277 | 0.293 | 0.296 | 0.320 | 0.316 | | FS10% Weather (Avg) | 0.237 | 0.238 | **0.234** | 0.238 | 0.242 | 0.279 | 0.241 | | FS05% ETTm2 (Avg) | 0.285 | 0.281 | **0.274** | 0.308 | 0.315 | 0.399 | 0.345 | | FS05% Weather (Avg) | **0.256** | **0.256** | 0.261 | 0.264 | 0.270 | 0.264 | 0.298 | | ZS m1→m2 (Avg) | **0.255** | 0.258 | 0.265 | 0.314 | 0.297 | 0.322 | 0.335 | | ZS h1→h2 (Avg) | **0.337** | 0.339 | 0.352 | 0.407 | 0.381 | 0.421 | 0.494 | | AvgRank | **1.66** | 1.97 | 2.66 | 4.56 | 4.34 | 6.31 | 6.19 | From the average rankings, we observe that even with masking applied to information outside the 512-step window, **ICTSP (Limited) still significantly outperforms all baselines**. In full data and zero-shot experiments, ICTSP (Limited) performs slightly weaker than the original ICTSP but remains superior to all baselines. However, in few-shot training experiments, ICTSP (Limited) surpasses the original ICTSP. It is important to note that this improvement is **due to an additional constraint in the original ICTSP**, which, despite using $L_I=1440$, imposes **zero-filling** restrictions outside of $L_b=512$ to ensure fair comparison with baselines using $L_I=512$ (see Section A.3.3 Discussion of fair comparison for details on **few-shot experiments**). In the full data and zero-shot experiments, all data can be used with any $L_I$ without causing data leakage. However, in few-shot experiments, due to skipping in data sampling, a longer \(L_I\) accesses more lookback data during validation/testing, giving an unfair advantage. To address this, the original ICTSP applied **zero-masking** to restrict access to only 512 valid timesteps at the start of validation/testing, gradually expanding to \(L_I = 1440\) as more valid data became available. Nonetheless, zero-filling in forecasting task tokens caused issues because of the **last-value demeaning operation** used on the forecasting task tokens, resulting in meaningless negative last-value placeholders and reduced model performance. **ICTSP (Limited)** resolves this by using **mean-value filling**, ensuring more reasonable values in forecasting task tokens. This adjustment allows ICTSP (Limited) to outperform the original ICTSP in few-shot experiments.

To clarify this, we **replaced the zero-filling used for few-shot fair comparison in the original ICTSP with mean-filling** and present the results below. ICTSP with mean-filling **significantly outperforms** all models, including Time-LLM, across all experiments. This highlights that ICTSP’s few-shot capability is inherently **stronger** than original results presented in the paper when proper mean-filling strategies are applied. Full results can be found in Table 12 of the revised paper. | Methods | ICTSP (Zero-filling) | ICTSP (Mean-filling) | ICTSP (Limited) | Time-LLM | GPT4TS | PatchTST | TimesNet | DLinear | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | FS10% ETTm2 (Avg) | 0.275 | **0.264** | 0.274 | 0.277 | 0.293 | 0.296 | 0.320 | 0.316 | | FS10% Weather (Avg) | 0.237 | **0.234** | 0.238 | **0.234** | 0.238 | 0.242 | 0.279 | 0.241 | | FS05% ETTm2 (Avg) | 0.285 | **0.272** | 0.281 | 0.274 | 0.308 | 0.315 | 0.399 | 0.345 | | FS05% Weather (Avg) | 0.256 | **0.248** | 0.256 | 0.261 | 0.264 | 0.270 | 0.264 | 0.298 | 3. Can you provide more details for Computational Costs? What are the number of layers and hidden dimensions of different models? Thank you for your attention to the details of our paper. In Table 5, we used the reported default settings from previous studies for the baselines. For Autoformer, Informer, PatchTST, and iTransformer, we set \(L_I = 512\) with 3 Transformer layers (or two encoder layers and one decoder layer). We used the number of series \(C = 7\) and dimension \(d = 128\). For ICTSP, we followed the settings described in the first paragraph of Section 3, which include the same number of layers and hidden dimensions with \(L_I=1440\). We calculated the computational costs using the Python `ptflops` package. Lastly, we greatly appreciate your feedback on our paper. If you have any more questions, we are more than happy to provide further explanations and look forward to engaging in more discussions with you. Thank you very much! ### References [1] Zhou, Tian, et al. "One fits all: Power general time series analysis by pretrained lm." Advances in neural information processing systems 36 (2023): 43322-43355. [2] Das, Abhimanyu, et al. "Long-term Forecasting with TiDE: Time-series Dense Encoder." Transactions on Machine Learning Research. [3] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023. [4] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations. [5] Liu, Yong, et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." The Twelfth International Conference on Learning Representations.

The provided link to the code repository is inaccessible, which hinders the reproducibility of the results. Could you please update or fix the link to the source code? We sincerely hope the reviewer can confirm again whether the code repository we provided is accessible. We have contacted several individuals for testing, and the feedback we received consistently indicates that the link [https://anonymous.4open.science/r/ICTSP-C995](https://anonymous.4open.science/r/ICTSP-C995) is working properly. Since we uploaded the repository to this link on August 7, 2024, we have not made any changes to it. Therefore, if there were any instances where it was inaccessible, it may have been due to temporary website maintenance or network fluctuations. We would like to highlight that `anonymous.4open.science` offers a highly useful modification tracking feature. When you open the link and check the bottom left corner of the website, you will see `Last Update: Aug 7, 2024`. This indicates that we have not made any modifications to the repository after this date (e.g., temporarily closing the repository). If any changes were made to the link, such as rewriting code, adding, or deleting files, the `Last Update` date would reflect the most recent change. Moreover, if the anonymized link is closed and reopened under the same name, the `Last Update` would display the new link’s creation date (not the last commit date from the original GitHub repository). This platform is widely used for anonymizing repositories, and we encourage you to verify these features yourself. Therefore, from August 7, 2024, before we submitted the paper, to the discussion period, we have not made any modifications to the anonymized repository, and it should have remained accessible. If you still find the link inaccessible, we have provided an alternative link [https://anonymous.4open.science/r/ICTSP-ALT-6BE4](https://anonymous.4open.science/r/ICTSP-ALT-6BE4), which contains the same code as the original repository for your reference. Additionally, we have provided a repository for your questions regarding Weakness 2, which demonstrates ICTSP’s application to other time series tasks: [https://anonymous.4open.science/r/ICTSP-MultiTask-53B3](https://anonymous.4open.science/r/ICTSP-MultiTask-53B3). We will explain its contents in our response to Weakness 2 below.

``` 1. The innovation appears limited. While the paper extensively discusses the approach from an in-context learning perspective, the methodology itself seems simplistic and more akin to a heuristic trick due to excessive textual explanation without solid theoretical support. ``` We appreciate the reviewer’s valuable feedback. While the reviewer noted a lack of solid theoretical support in our paper, we want to clarify that our work and experiments are grounded in prior ICL studies with well-established theoretical foundations. Here are key proven conclusions from the cited ICL research that directly support applying ICL to time series forecasting in our paper: 1. Transformers can learn linear predictors ([1,2,3,4]) This enables Transformers to perform fundamental linear time series forecasting predictors through ICL, such as NLinear [5] and RLinear [8]. 2. A single Transformer layer can implement simple operations, including shifting input features, computing differences between input features, and transferring values from one token to a target token ([2,4,6]) This allows Transformers to process the autocorrelated structure of multivariate time series inputs appropriately. 3. Transformers can learn function classes, including shallow MLPs and decision trees ([3,4,6]) This enables Transformers to perform the core MLP predictor used like recent MLP-based time series forecasting models [7,9,10] through ICL. 4. Transformers can select the best-performing algorithm based on input context pairs ([3,4,6]) This allows Transformers to dynamically choose the most optimal predictor type based on the properties of the input data, such as the strength of temporal relationships, aligned series-wise relationships, and unaligned series-wise relationships. These proven theoretical results underpin all discussions in our paper. In the revision, we highlight more about the links of these findings to statements in our work. ``` References 1 [1] Zhang, Ruiqi, Spencer Frei, and Peter Bartlett. "Trained Transformers Learn Linear Models In-Context." R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models. [2] Akyürek, Ekin, et al. "​​ What learning algorithm is in-context learning? Investigations with linear models." The Eleventh International Conference on Learning Representations. 2022. [3] Li, Yingcong, et al. "Transformers as algorithms: Generalization and stability in in-context learning." International Conference on Machine Learning. PMLR, 2023. [4] Bai, Yu, et al. "Transformers as statisticians: Provable in-context learning with in-context algorithm selection." Advances in neural information processing systems 36 (2024). [5] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023. [6] Guo, Tianyu, et al. "How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations." The Twelfth International Conference on Learning Representations. [7] Ekambaram, Vijay, et al. "Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023. [8] Li, Zhe, et al. "Revisiting long-term time series forecasting: An investigation on linear mapping." arXiv preprint arXiv:2305.10721 (2023). [9] Das, Abhimanyu, et al. "Long-term Forecasting with TiDE: Time-series Dense Encoder." Transactions on Machine Learning Research. [10] Wang, Shiyu, et al. "Timemixer: Decomposable multiscale mixing for time series forecasting." arXiv preprint arXiv:2405.14616 (2024). ```

However, we respectfully disagree with the reviewer’s comment that *"the methodology itself seems simplistic and more akin to a heuristic trick due to excessive textual explanation without solid theoretical support."* We believe this view may not reflect a consensus in the AI/ML community. Below, we provide examples from AI research, including time series and ICL domains, to illustrate this point. While the ICL field includes studies with rigorous theoretical proofs, as we highlighted earlier, key papers we cite, such as [1,2,3], do not include theoretical derivations or proofs. Instead, they rely on textual explanations and discussions of empirical results. Despite this, they have had significant impact, offering valuable insights for further theoretical and empirical research. AI researchers often prioritize simple, intuitive, and effective methods that provide meaningful insights. In time series forecasting, works like DLinear [4], PatchTST [5], GPT4TS [6], and iTransformer [7] show how existing structures can achieve broad impact by being transfered and applied to time series tasks. If these works are considered to lack novelty and solid support, even the methods they build upon, like ViT [8] and GPT-2 [9], could be similarly critiqued. Yet, these foundational contributions are widely recognized. In the following section, we aim to help identify the novelty of our research more clearly. ``` References 2 [1] Min, Sewon, et al. "Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?." EMNLP. 2022. [2] Garg, Shivam, et al. "What can transformers learn in-context? a case study of simple function classes." Advances in Neural Information Processing Systems 35 (2022): 30583-30598. [3] Shi, Zhenmei, et al. "Why Larger Language Models Do In-context Learning Differently?." R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models. [4] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023. [5] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations. [6] Zhou, Tian, et al. "One fits all: Power general time series analysis by pretrained lm." Advances in neural information processing systems 36 (2023): 43322-43355. [7] Liu, Yong, et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." The Twelfth International Conference on Learning Representations. [8] Alexey Dosovitskiy, et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." International Conference on Learning Representations. 2021. [9] Radford, Alec, et al. "Language models are unsupervised multitask learners." OpenAI blog 1.8 (2019): 9. ``` ### About the novelty of this research First, we would like to mention that the other 3 reviewers identified the novel ideas in our paper as key strengths. To clarify the innovations, we summarize our contributions below: - **Novel Purpose** We explore activating the ICL capabilities of Transformers for TSF solely through model design, without relying on large-scale LLM pretrained weights. This approach enhances generalization and improves few-shot and zero-shot performance, a direction not previously explored for TSF Transformers. - **Novel Findings** Using the ICL perspective, we systematically analyze Transformer-based TSF models, categorize them, identify key issues, and provides a series of findings and insights: a) **Temporal-wise Transformers** underperform due to overemphasizing channel-wise dynamics in the input context while neglecting temporal-wise relationships, leading to poor generalization on real-world datasets with weak series dependencies. b) **Series-wise Transformers** can model both temporal-wise and aligned series-wise relationships. However, they lack context inputs with shifting structures, making them less effective at handling unaligned series-wise relationships. c) Components like **channel independence**, **random dropping**, and **patching** enhance Temporal-wise Transformers by encourage tokenization focusing more on temporal-wise relationships. These findings, supported by evidence from prior work and our experiments, have not been described in prior literature and offer valuable directions for future research.

- **Solving Unresolved Problems** Building on the issues identified in previous models, we developed ICTSP, a structure capable of addressing all these problems, which are crucial for constructing a multivariate TSF model with strong generalization capabilities. ICTSP effectively resolves the following challenges: 1. Modeling **temporal-wise dependencies** (tokenization along the temporal dimension). 2. Modeling **aligned series-wise dependencies** (input series in tokenization). 3. Modeling **unaligned series-wise dependencies** (introducing a shifted forecasting task tokenization structure in the context examples). (These first three points are supported by the main experiments and Figure 4.) 4. Avoiding **overfitting to unstable channel-wise effects** caused by context timestep tokens (segment tokenization ensures temporal relationships outweigh weak series relationships in the representation for real-world data). 5. Offering a **flexible input channel structure** that accommodates any number of input series without requiring modifications to the model structure (forecasting tasks as tokens). 6. Enabling seamless transfer to **datasets from different domains** without additional training (using historical forecasting tasks as context examples, the model can automatically and implicitly fit the most suitable predictor). Previous Transformer-based TSF models generally fail to address at least two of these challenges: iTransformer: (3, 6), Time-LLM: (2, 3), GPT4TS: (2, 3), ForecastPFN: (2, 3), Crossformer: (5, 6), PatchTST: (2, 3, 6), FEDFormer: (4, 5, 6), Autoformer: (1, 4, 5, 6), Informer: (1, 4, 5, 6). - **Insightful Presentation** Reviewer uZY9 noted that our research offers a novel perspective for categorizing and summarizing prior models while showcasing ICTSP's innovations. We fully agree. To clarify further, we propose summarizing the content of Section 2 in the following table, which clearly shows how the ICL perspective identifies issues in previous models and addresses them with appropriate ICTSP structure. | **Methods** | **Tokenization** | **ICL (x, y) Pair** | **Token Representation for** | **Attention to Learn** | **Suited For** | **Not Suited For** | **ICL Context Examples Describing** | **Channel Structure** | **Problem** | |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | **Temporal-wise Transformer** | Timestep embedding | (Timestep index, Multi-channel values) | Single-step inter-series dependencies | Temporal-wise dynamics of series dependencies | Strong and consistent inter-series dependencies | Weak inter-series dependencies in real-world datasets | Underlying inter-series effects vary with the time indices in the context lookback (likely nonexistent or weak) | Fixed | Only series relationships in tokenization; risks overfitting to series-wise effects from the input context | | **Temporal-wise Transformer (with Channel Independence)** | Scalar embedding | (Timestep index, Single-channel value) | Scalar value at a single timestep | Temporal-wise dependencies | Strong and fast-changing temporal dependencies | Strong inter-series dependencies | Temporal relationships within the context for one series | Fixed | Unable to model series-wise relationships | | **Temporal-wise Transformer (with Patching)** | Patch embedding | (Timestep index, Multi-channel values in a temporal interval) | Temporal-wise and series-wise relationships in a local interval | Temporal-wise relationships between local intervals | Temporal and series relationships can be automatically balanced by this structure | Few-shot/zero-shot cases with different series relationship characteristics | Temporal-wise relationships between local intervals | Fixed | Fixed channel structure in tokenization prevents generalization to datasets with different structures | | **Series-wise Transformer** | Series embedding | (Input series, Output series) | Temporal-wise relationships within a single series input | Aligned inter-series relationships | Real-world datasets with strong temporal-wise and aligned series-wise relationships (e.g., traffic) | Unaligned series-wise relationships, such as shifting effects between series | No context examples | Flexible but requires additional training to adapt to datasets with new channel structure | Restricted to specific dataset channel structures; unable to learn unaligned series-wise effects | | **ICTSP** | Embedding of forecasting tasks | (Input series, Output series) | Temporal-wise relationships in a forecasting task | Forecasting task predictors, considering both aligned and unaligned inter-series relationships in shifted context examples | Datasets with fast-changing temporal relationships, aligned or unaligned series relationships; transfering to new datasets with different channel structures | / | Historical ground truth forecasting tasks as references | Flexible; additional training is not necessary since context examples exist | / |

``` 2. The paper does not address whether this model could be applied to other time series tasks, such as classification, interpolation, or anomaly detection. The utility of the model appears confined to the forecasting tasks, necessitating stronger performance to justify its specialized use. ``` Thank you for raising this point. The ICTSP structure is highly transferable to multi-task time series analysis. By simply replacing the forecasting task’s `(lookback, future)` structure with `(lookback, new_target)`, it can adapt to various tasks. In our follow-up work, we pretrained an ICL model capable of solving multiple tasks by dynamically identifying the task type from context examples. For your reference, we provide the following link: [https://anonymous.4open.science/r/ICTSP-MultiTask-53B3](https://anonymous.4open.science/r/ICTSP-MultiTask-53B3). This repository demonstrates ICTSP’s application to tasks like forecasting, classification, imputation, cropping, shifting, hyper-resolution (up-sampling), moving average, differencing, and exponential smoothing. It includes data generation scripts (`ts_generation/`) and in-context tokenization for multi-task learning (`data_provider/`). However, we must clarify that discussing multi-task learning is beyond the main focus of this paper. ### Why doesn’t this paper discuss time series tasks beyond TSF? As outlined above, the primary goal of this paper is to diagnose existing TSF Transformers from an in-context perspective, identify their limitations, and demonstrate how ICTSP addresses these issues to improve generalization ability. A key distinction between the forecasting task and other tasks (like classification and imputation) lies in the **autoregressive structure**, where the next step is the prediction answer of the previous step. This creates a unique **shifting structure** between `(lookback, future)`, unlike classification `(lookback, y)`, which is more akin to a traditional supervised learning task, which has been widely studied in prior ICL research [1,2]. In this paper, we focus on how previous Transformer structures handle temporal, aligned series, and unaligned series dependencies, which are tightly tied to the **causal autoregressive nature** of forecasting. For example, ICTSP uses **context examples** to introduce a **shifted structure**, helping the model consider unaligned series-wise relationships, which Series-wise Transformers (like iTransformer) lack. In standard supervised learning settings of general ICL, features are treated independently before training, but in time series, **tokens inherently exhibit shifted autocorrelation**, making shifted structure of context examples essential. This necessity is supported by our experiments (Figure 4). Thus, the key contributions of this paper is to **highlight these unique considerations for using Transformers in TSF tasks** to the TSF community. Therefore, given space constraints, the paper focuses its core content on **ICL for TSF**, as summarized above. ``` References 3 [1] Müller, Samuel, et al. "Transformers Can Do Bayesian Inference." International Conference on Learning Representations. [2] Hollmann, Noah, et al. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." The Eleventh International Conference on Learning Representations. ``` We respectfully disagree with the reviewer’s comment that our results are not strong enough. Our work activates the in-context learning abilities of Transformers **without relying on LLM pretrained weights**, using a model with fewer than **10M parameters**, compared to the **7B parameters** in LLaMA used by Time-LLM. Despite the drastically reduced computational cost, our model achieves bettern performance, which we believe demonstrates **stronger results**. Below, we summarize the results for full-data, few-shot (10%, 5%), and zero-shot experiments: | Methods | ICTSP | Time-LLM | GPT4TS | PatchTST | DLinear | |-|-|-|-|-|-| | AvgRank (Full) | **1.36** | 1.93 | 4.29 | 2.79 | 5.00 | | #Rank1 (Full) | **18** | 8 | 0 | 2 | 0 | | AvgRank (10%) | **1.57** | 1.89 | 3.21 | 4.07 | 4.93 | | #Rank1 (10%) | **15** | 10 | 1 | 1 | 1 | | AvgRank (5%) | **1.56** | 2.36 | 3.48 | 3.96 | 6.04 | | #Rank1 (5%) | **12** | 7 | 1 | 2 | 0 | | AvgRank (0%) | **1.13** | 1.88 | 4.54 | 3.29 | 5.33 | | #Rank1 (0%) | **21** | 3 | 0 | 0 | 0 |

``` 3. The significant improvements in zero-shot tasks over full-data tasks lack detailed explanations. The paper does not sufficiently clarify why the ICTSP demonstrates a pronounced advantage in few-shot and zero-shot learning scenarios than in full-data tasks ``` Thank you for recognizing the significant improvements ICTSP achieves in few-shot and zero-shot scenarios. These results are critical to our paper, showcasing ICTSP’s key advantage over previous structures. By introducing forecasting tasks as context tokens, ICTSP activates in-context learning abilities that dynamically selects the most suitable in-context predictor \(f^*\) based on the provided examples. Unlike **Temporal-wise Transformers**, ICTSP does not use a fixed channel structure in token representation. Instead, it dynamically infers series relationships from input tokens, allowing it to adapt to new series structure when transferred to new dataset. This contrasts with Temporal-wise Transformers, which often encode series relationships from training data as fixed latent structures in the token representation. Compared to channel-independent Temporal-wise Transformers like Time-LLM or PatchTST that do not consider cross-channel information, ICTSP excels at dynamically inferring series relationships. Without the requirement of additional training, ICTSP adapts to multivariate TSF data with varying distributions through context forecasting examples. This adaptability is a unique capability, enabling ICTSP’s superior performance in **few-shot scenarios (handling temporal distribution shifts)** and **zero-shot scenarios (handling different channel structures)**. ### Appreciation Thank you once again for your highly valuable feedback on our paper. Your suggestions have been instrumental in improving the presentation of our work. If you have any further concerns, we would be more than happy to address them.

Thank you for your thoughtful response. The provided link is accessible, though I encountered multiple access failures on November 2-3, 2024. To be clear, I fully believe that no changes were made to the code during the submission period. I raised this issue solely to bring it to your attention. However, the reasons I find the novelty of this paper insufficient are as follows: 1. I fully understand and acknowledge that a well-trained Transformer has ICL capabilities. However, as stated in line 212, the authors’ motivation for proposing ICTSP is "to fully exploit the Transformer's ICL capabilities." We must recognize that utilizing such capabilities assumes the model is already well-trained and inherently possesses them (line 24-27). Only then can these capabilities be further exploited. Since ICTSP is trained from scratch (as indicated in line 352), how can you claim to be leveraging the Transformer’s ICL capabilities? Moreover, the analysis in Section 2.2, which uses an ICL perspective to evaluate other methods, also seems to misrepresent causality. For models trained from scratch, what they learn aligns with their training paradigm. Applying an ICL perspective to interpret these training outcomes is unreasonable. This further highlights what I consider to be a flawed motivation. 2. While I acknowledge the substantial theoretical support for ICL in prior research, this is not directly relevant to the contributions of this paper. The value of ICL does not automatically validate all work claiming to utilize it. The most important thing is the authors need to demonstrate their utilization of LIC capabilities from the perspective of the model itself rather than relying solely on experimental results. Moreover, there an another question: According to Equation 2, whether the final output sequence length should match the label length in the example.

Dear Reviewer hkWk, Thank you very much for raising a series of insightful questions during the review process. These questions have been incredibly helpful in improving the presentation quality of the core content of our work. With only about one day left until the end of the ICLR rebuttal period, we kindly wish to know if the responses we provided have addressed your concerns. If you have any additional questions, we would be more than happy to provide a quick response. Once again, we deeply appreciate your thoughtful questions, which have greatly enhanced the quality of our presentation. Thank you for the time and effort you have dedicated to reviewing our paper.