Reply to Official Comment by Reviewer 6VCF (1/2)
We really appreciate that you have both helped improve the paper, and are now re-evaluating the manuscript in light of our work together.
> the main contribution of this work lies in the long-sequence decoding strategy ... which aims to reduce error accumulation
Based on reviewer feedback, we will definitely revise the abstract to emphasize the novel benefit of SutraNets is the ability to simultaneously: (1) capture long-term dependencies by reducing signal path, (2) reduce error accumulation in autoregression, (3) enable a K-fold improvement in training parallelism for RNNs, and (4) generate coherent outputs.
> in Figure 1, the approach uses (y0, y3, y6, y9) to generate (y12, y15, y18) initially, and then employs (y12, y15, y18) along with the history (y1, y4, y7, y10) to generate (y13, y16, y19)
We understand you to be extrapolating the process in Fig. 1(d), for **Regular-non**. Indeed, it is a *non*-alternating model: the first sub-series prediction is generated end-to-end while only using historical values for that subseries.
> If (y12, y15, y18) turn out to be incorrect predictions, would these errors accumulate and affect the subsequent predictions of (y13, y16, y19)?
Errors do still accumulate, but to a lesser extent. Consider generating y19. In a standard RNN (Fig. 1a), we must generate y11, y12, y13, y14, y15, y16, y17, y18 - 8 steps - in order to generate y18, i.e., the value that precedes y19. However, in Regular-non, we can generate y13, 15, y18 - 3 steps - in order to generate y18. So the prediction of y19 can be based on proximal values (y18) themselves generated with $K$X less error accumulation.
> Why wasn't (y10, y11) used to generate y12? It seems predicting y12 using (y10, y11) would be much easier
Regular-non by definition generates y12 using y9 and earlier, while standard RNNs (Fig. 1a) and Regular-alt (Fig. 1c) generate sequentially as you are suggesting. Yet interestingly, Regular-non sometimes works better than Regular-alt in experiments. Why? Well, During *training*, sequential RNNs sometimes find it effective to just repeat the previous gold/true value, i.e., to predict y12 based on gold y11. But in *inference*, if y11 is generated after a long chain of error accumulation, repeating y11 to predict y12 is problematic. By forcing the network to attend to values $K$ steps in the past, we reduce train/test "discrepancy" and improve forecasting accuracy.
> since y12 is decoded using y0, y3, y6, and y9, does it imply that the model assumes the time series to be periodic?
SutraNets do not require data to be periodic to be effective, e.g., dataset `mnist`$^{\pi}$ comprises images after a fixed random permutation, and `azure` and `wiki` comprise many trending but non-seasonal series. However, for time series that *are* periodic, we asked a similar question to yours: does it matter whether y0, y3, y6, etc. captures seasonality, i.e., does it matter whether $K$ divides into the seasonal period? The results are in Table 3 and Finding 5: non-alternating models are comparatively weaker than alternating versions.
> it is important to note that this strategy is still limited
While *non-alternating* RNNs generate future values without "seeing" all the history, keep in mind that **all the alternating RNNs do see all the history in their generative process**. That is, the orange RNN in Fig. 1(f) will have seen y0, y1, y2, ..., y7, y8 when it goes to generate y11. True, it does not see y9 or y10, but that's the point: all values from y9 onward must be generated, and Backfill-alt generates y11, then y10, then y9. Since this seems to work quite well, we would like to share these findings with the community.
> There is a related work TimesNet [1], which shares a similar idea to SutraNets. Both approaches transform long time series into multiple subseries
We can certainly include discussion of TimesNet. It is interesting that TimesNet transforms univariate time series into 2D tensors of *multiple* periods, which are then processed via Inception-style 2D kernels (i.e., now using a 2D computer vision backbone, and outputting point predictions). SutraNets on the other hand transform time series into $K$ sub-series of the *same* period, and proposes an autoregressive factorization (with different generative orderings) for generating these sub-series using their own and previous values (with a 1D sequential generative model that produces probabilistic outputs).