Rebuttal by Authors (Cont')
Based on your comments, we have identified and categorized the weaknesses [W#] you highlighted into three key areas: Structural Similarity with PatchTST, Query-Adaptive Masking in PatchTST and CATS, and Lack of Emphasis on Broader Comparisons.
---
**[W1 - Structural Similarity with PatchTST]**
**Comment:** *"The backbone structure of this paper is very similar to PatchTST, with the core difference being the introduction of the cross-attention module and the use of the shared projection layer."*
While both CATS and PatchTST utilize patching, it’s important to clarify that beyond this similarity, the two models diverge significantly in their overall architecture and approach to time series forecasting. PatchTST emphasizes univariate forecasting and patching while largely retaining the standard Transformer structure by keeping the vanilla Transformer encoder intact and simply substituting the decoder with a linear layer. In contrast, CATS takes a more radical departure from traditional Transformer models by **not only eliminating the self-attention mechanism but also redesigning the decoder**. Unlike PatchTST, which maintains the vanilla Transformer encoder, CATS removes the masked self-attention from the decoder as well, effectively addressing the challenges that arise from this change.
Eliminating self-attention introduces significant challenges, particularly in time series forecasting. These challenges can be summarized as follows:
1. **Inability to Generate Necessary Queries**
- **Problem:** Without self-attention, the fixed input tokens used in traditional decoders are insufficient to generate the necessary queries for cross-attention.
- **Solution:** We introduced learnable queries, establishing horizon-dependent parameters as learnable queries. This ensures that each forecasting query independently generates the necessary input for cross-attention.
2. **Temporal Information Loss Due to Concatenated Linear Layers**
- **Problem:** Concatenated linear layers at the end of the decoder could lead to temporal information loss.
- **Solution:** We designed the decoder to process each horizon's query independently using parameter sharing across layers. This approach minimizes temporal information loss by ensuring that each query's processing remains independent, thus preserving temporal information.
Given that PatchTST closely aligns with the vanilla Transformer structure—using an identical encoder and only replacing the decoder with a linear layer—such reasoning could be extended to suggest that any model based on the vanilla Transformer architecture is "very similar" to PatchTST. However, **the introduction of novel mechanisms in CATS results in a distinct decoder architecture from traditional Transformers, designed to be more efficient and effective for time series forecasting, setting CATS apart from both the vanilla Transformer and PatchTST.**
---
**[W2 - Query-Adaptive Masking in PatchTST and CATS]**
**Comment:** *"It is quite unusual that PatchTST performs significantly worse than CATS after introducing Query-Adaptive Masking."*
Regarding Query-Adaptive Masking, as mentioned in our rebuttal, applying this technique to PatchTST leads to significant performance degradation because masking a single query impacts the entire output. However, as detailed in Appendix B.3, CATS is designed to handle "**multiple inputs with different forecasting horizons sharing a single model** (in line 493)". This structure allows for the masking of individual queries, which only affects the specific horizon they pertain to. This probabilistic masking enables the model to focus more effectively on learning from the unmasked queries, thereby avoiding the performance pitfalls observed in PatchTST.