Effectively modeling long spatiotemporal sequences is challenging due to the need to model complex spatial correlations and long-range temporal dependencies simultaneously. ConvLSTMs attempt to address this by updating tensor-valued states with recurrent neural networks, but their sequential computation makes them slow to train. In contrast, Transformers can process an entire spatiotemporal sequence, compressed into tokens, in parallel. However, the cost of attention scales quadratically in length, limiting their scalability to longer sequences. Here, we address the challenges of prior methods and introduce convolutional state space models (ConvSSM) that combine the tensor modeling ideas of ConvLSTM with the long sequence modeling approaches of state space methods such as S4 and S5. First, we demonstrate how parallel scans can be applied to convolutional recurrences to achieve subquadratic parallelization and fast autoregressive generation. We then establish an equivalence between the dynamics of ConvSSMs and SSMs, which motivates parameterization and initialization strategies for modeling long-range dependencies. The result is ConvS5, an efficient ConvSSM variant for long-range spatiotemporal modeling. ConvS5 significantly outperforms Transformers and ConvLSTM on a long horizon Moving-MNIST experiment while training 3X faster than ConvLSTM and generating samples 400X faster than Transformers. In addition, ConvS5 matches or exceeds the performance of state-of-the-art methods on challenging DMLab, Minecraft and Habitat prediction benchmarks and enables new directions for modeling long spatiotemporal sequences.
Paper
Similar papers
Peer review
Summary
This paper presents ConvS5, a convolutional state-space model that aims to model long-range spatiotemporal dependencies in video data. They extend the prior S5 model to operate in a convolutional state-space, and retain the long-range benefits of state-space models by using a point-wise convolutional kernel for diagonalizable and S5 HiPPO initialization. The resulting model can then be efficiently trained using parallel scans due to linearity in the recurrent mechanism. They show strong results on several complex long-range video benchmarks in Moving MNIST, DMLab, Minecraft, and Habitat.
Strengths
- The paper is generally clear and well written - Comprehensive experiments on a range of datasets focused on long-range understanding in videos, showing that their proposed method is better than or competitive with SOTA methods - Experiments show that better scaling in sequence length allows for training on longer sequences, which results in better / more stable generation - Faster inference speed compared to existing methods while retaining high generation fidelity
Weaknesses
- Overall technical novelty is a little low, as the model itself is a straightforward extension of the S5 model to a convolutional setting. - A primary concern I have about recurrent-based models for long-range understanding in video generation is limited capacity in the recurrent state, as these models are required to store all fine-grained details about the scene in the recurrent state (since potentially every part of the scenes will be visited / need to be reconstructed at a future timestep), whereas transformer-based models can dynamically retrieve from all prior states for each timestep. A rigorous study on this aspect would benefit the paper greatly, as this issue is more present in visually complex videos (minecraft -> habitat -> real videos), and could be a potential bottleneck to the scalability of these models.
Questions
- For Moving MNIST, how would ConvS5 potentially compare against temporally hierarchical methods in modeling long-range dependencies (e.g. Clockwork-VAE)? Or would an analogous hierarchical version of ConvS5 be better than a hierarchical GRU model? - How would ConvS5 compare for Minecraft / Habitat? I believe the primary difference is without the MaskGit / iterative decoding on the prior, so would ConvS5 struggle with more stochastic environments? - What is the potential cause of the difference in performance between FVD and PSNR / SSIM / LPIPS in Minecraft? Is FVD lower due to better frame fidelity of TECO-ConvS5, and TECO-transformer has better long-term consistency? Or perhaps other factors? - Are generated video samples available anywhere to view? It is hard to evaluate temporal coherence in the rows of frames provided in the appendix.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Yes, the authors discuss limitations.
Summary
This paper proposes a new state space model for spatiotemporal modeling by introducing inductive bias of spatial locality. The core idea is to extend SSMs to ConvSSMs (just like extending FC-LSTM to ConvLSTM), which has a inherent convolutional structure. The new model also establishes an equivalence between the dynamics of particularly structured ConvSSMs and SSMs. Based on recent state space method S5, a model instantiation ConvS5 combines the stateful autoregressive generation with the ability to be parallelized across the sequence. A experimental evaluation shows that the proposed method captures spatiotemporal information better.
Strengths
* The idea is simple but effective as it combines fast autoregressive generation and parallelized process. * The proposed method achieves better performance than Transformer and ConvRNN on Moving-MNIST prediction task. It also strikes a balance on computational complexity between the two methods.
Weaknesses
* The lack of different ConvSSM variants. The paper proposes ConvSSMs to address long-range spatiotemporal modeling. However, only a variant, ConvS5, is present and evaluated. It is not convincing to support the basic idea of ConvSSMs. More variants are needed. * The experiments about comparison with Transformer and ConvLSTM could be more extensive. In Table 2, we see that the ConvS5 outperforms the other methods across the metrics. We notice that ConvLSTM achieves a slightly lower performance result. However, ConvLSTM shows a slightly better sampling speed than ConvS5. It is not sufficiently significant that ConvS5 has a better quality-speed tradeoff than ConvLSTM. * There lacks explanation and experiments to the design of convolutional operator upon SSMs. The comparison results of whether convolution or not are missing.
Questions
* The idea of parallel scan for SSMs is not new. This design is considerable overlap with S5, which also leverages parallel scans to efficiently compute the states. The authors are suggested to discuss more about this. * In Table 3, ConvS5 shows a significant improvement than S5 without TECO. When trained using the TECO, ConvS5 only achieves a comparable result with S5. It makes me wonder if the proposed method is effective consistently on various settings (e.g. framework, SSMs backbone, etc). * As in Weakness 3, I wonder if the proposed method is applicable to other SSMs. The authors are suggested to discuss more about this.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
yes
Response to authors
Thank the authors for providing the response and addressing my concerns. The motivation of this work looks much clearer to me now. My current main concern preventing further increase is the lack of different ConvSSM variants (not only the particular formulation of ConvS5). That being said, I would be happy to raise my score to 5.
We thank the reviewer for taking the time to read our rebuttal and for increasing their score! Re. ConvSSM variants: Since the reviewer requested to introduce more ConvSSM variants, we can add another possible variant. Here, motivated by the SSM to ConvSSM connection made in Section 3.3, we can consider a "ConvS4" variant related to S4 [3]. S4 uses single-input, single-output SSMs (one for each feature), so this would require applying a single-input, single-output SSM to every pixel and feature channel independently. From the ConvSSM point of view, this would require restricting the input/output kernels (B and C) and dynamics kernel A to $1\times1$ kernels and applying a different single-input, single-output ConvS4 to every channel independently. This stack of ConvS4s could be shared/convolved across each pixel of the frame. For efficient sequence parallelization, FFTs would need to be used with this structure similar to S4 rather than using a parallel scan as used in ConvS5. In addition, an additional mixing operation would be required to mix the information of all the independent channels and pixels. It can be viewed as a depthwise-separable ConvSSM that could potentially reduce the number of parameters and operations. However, this approach has restricted kernel sizes, independent dynamical evolution of features, and a requirement of using FFTs and time-invariant dynamical systems. We can add a separate discussion section with "ConvSSM Variants" to the final paper which includes this ConvS4 variant, in addition to H3 [1] and Liquid SSM [2] variants that we mentioned in our rebuttal. We can also include an experiment for one of these variants. In addition to the proposed variants, we think the connection we make in this work between ConvRNNs and recent SSM methods can open up a new line of research to consider these alternative ConvSSM architectures. We hope this addresses the Reviewer's concerns. Please let us know if we can provide further clarification or information regarding ConvSSM variants and help to further increase the score. References: [1] Hungry Hungry Hippos: Towards Language Modeling with State Space Models. [2] Liquid Structural State-Space Models. [3] Efficiently Modeling Long Sequences with Structured State Spaces.
Summary
The paper builds upon ConvRNNs and proposes the use of SSM (State Space Models) as a replacement for RNNs. This allows for efficient computation using parallel scan. The proposed method is evaluated on the Long Horizon Moving-MNIST Generation and Long-range 3D Environment Benchmarks datasets, where it achieves promising results.
Strengths
The organization of this paper is clear and easy to understand. It starts from ConvRNN and replaces RNN with SSM, leading to the instantiation of ConvSSM known as ConvS5.
Weaknesses
I think the main issue with this paper is the lack of motivation for the proposed approach, as replacing the matrix multiplication of SSM with convolution seems trivial.
Questions
1. The motivation for the proposed approach should be made more explicit, whether it is based on intuitive explanations or considerations of efficiency, among other factors. 2. The experiments conducted in the paper are not comprehensive enough. It would be beneficial to compare the proposed method with other efficient attention-based approaches, such as kernel-based linear attention, 1+elu, performer, and cosformer, on long sequence tasks. 3. More ablation studies are needed to validate the rationale behind the design, specifically regarding the convolutional aspect. The existing ablation studies primarily focus on validating the initialization, but it is equally important to investigate and verify the effectiveness of the convolutional components. References: [1] Fast Autoregressive Transformers with Linear Attention [2] Rethinking Attention with Performers [3] cosFormer: Rethinking Softmax in Attention
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes.
Summary
This paper proposes convolutional state space models (ConvSSMs), that combine ConvLSTM with the long sequence modeling approaches like S4/S5. In particular, authors propose ConvS5, that allow parallelizing the stateful autoregression of convRNNs across the sequential direction. Naively applying parallel scan speedup tricks like in S4/S5, leads to convolutional kernels of exploding kernel size. Hence, the authors propose an interesting way of structuring modeling complexity, choosing to have simple linear dynamics in the sequence direction that are learnt with 1x1 conv kernels, and complex non-linear operations in depth that can be learnt with entire resnet blocks. The authors benchmark the proposed ConvS5 model on the Moving-MNIST dataset by training on 300 and 600 frames and predicting 400/800/1200 frames conditioned on the initial 100 frames. Also benchmarking results are reported on the DMLab, Minecraft, and Habitat long-range benchmarks. Across the axes of image quality such as FVD, PSNR, SSIM and LPIPS as well as efficiency axes such as sampling speed.
Strengths
+ **Well motivated and principled design** : The design considerations in ConvS5 is very well motivated with both theoretical (most of which are directly inspired from Deep SSMs) and practical arguments that make an interesting and worthwhile contribution to the community. + **Broad benchmarking and strong results** : The authors perform benchmarking across a number of datasets and benchmarks across all of which ConvS5 outperforms prior works such as ConvLSTM and a vanilla transformer on image quality as well as sample speed. + Paper is also well written with good exposition of the prior work ideas that are necessary to grasp convSSM which is difficult to balance given the large amount of history in S5 model family development.
Weaknesses
- **Model Size** such as FLOPs / parameters and **Training Efficiency** such as Training Speed / Number of Epochs need to reported for both convS5 and also all the baselines that are being compared to. Without these details, quality results on their own are meaningless since it is unclear if the best footing was also provided to the baselines as the proposed method. It is mentioned as an offhand remark that convS5 uses ResNet blocks as intermediate non linearities but that can lead to major slowdown and burdens (as also alluded to by the authors in limitations). If so, this should be made clear in the tables through the metrics mentioned before. - **Boarder Ablations** : The paper is quite weak on ablation studies presented. Since the authors propose a new architecture, boarder ablations on more than just initialization are requires to be convincing of choices proposed. - Experiments on non synthetic real data/video. While the authors do experiment on a bunch of datasets, they're all either toy like Moving-MNIST or, synthetic like DMLab/Minecraft/Habitat. For ConvS5 to be successful we need to make sure that it does well on real world benchmarks as well.
Questions
- What hyperparameter sweeps and effort went into optimizing the Transformer performance on the proposed benchmarks? Was it commensurate with the proposed convS5? Also see weaknesses.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
Yes
Summary
This paper introduces a method for long-term sequential modeling. The authors extend prior RNN-based work for sequential modeling, i.e. S5[20], by substituting the linear operations with convolutions. The authors show the superiority of their method on the future prediction task with transformers. A favorable property of the RNN methods, in general, compared to transformers is that they scale linearly with respect to the sequence length, while the transformer's complexity is quadratic with respect to time.
Strengths
1) The proposed method makes sense and is a natural extension of prior work. 2) The paper is well-written and easy to follow.
Weaknesses
1) Compared to the transformers, the linear complexity with respect to the sequence length is clearly favorable. However, it would be informative to compare the actual computational cost, e.g. in terms of flops. 2) Despite transformers, RNNs are notorious for being difficult to train. It would be great to compare the training of two architecture as well. This can be a potential blocker for scaling the method for more complex datasets. 3) I am not an expert in this domain to have a proper evaluation on the impact of the experiments, but the contributions of this paper sound marginal with the NeurIPS standards. Also, it looks like the paper is missing some prior work to compare[1]. [1] Gao et al, Simvp: Simpler yet better video prediction, CVPR 2022.
Questions
Please see the weaknesses.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
No, they have not. However, I do not see a particular negative social impact associated for this work.
Response
Thank you for your rebuttal, and the extra experiments. **re: ConvS5 in Minecraft / Habitat** Could the authors clarify on the ConvS5 architecture? Is it autoregressive per token, or autoregressive over time and predicts all tokens of the next image simulatenously, or something else? **re: novelty** Please correct me if I am misunderstanding something, but the proposed method / architecture seems very similar to a simple baseline of a standard S5 layer that is vmaped across space with shared parameters for each spatial token - i.e. (`jax.vmap(s5_layer, axis=1)(x) # x is of shape [T, H * W, C]`) - equivalent to ConvS5 with $1\times 1$ kernels for both $A$ and $B$. The only difference I see is that $B$ is replaced $k\times k$ kernel, motivated by a technical contribution of using the connection between convolutions and matrix multiplications applied to S5 / ConvS5. If my understanding above is correct, the contribution over just using S5 this way does not seem too significant, or is there large gain in the extra $B$ being some $k\times k$ instead of $1\times 1$ convolution? That being said, I would be happy to raise my score to a 6. My current main concern preventing further increase is the degree of impact / novelty of the contribution.
We thank the reviewer for reading our rebuttal and increasing their score. --In these experiments, ConvS5 is autoregressive over time and predicts all tokens of the next image simultaneously. But it can also be plugged into other settings as you mentioned (e.g. autoregressive per token at each timestep, or using MaskGit as in the TECO version). Please let us know if we can provide further clarification regarding this. --The reviewer's understanding regarding the $B$ kernel is correct. To clarify, if all kernels (both $B$ and $C$) are replaced with $1 \times 1$ kernels, then it is equivalent to vmapping S5 across the (reshaped) pixels/tokens of the image. You can consider ConvS5 to be a generalization of simply vmapping S5. It allows for modulating the amount of convolution/weight sharing and shared spatial information that is fed to the dynamical system. However, our novel contribution is not only this generalization but also the development of parallel scans for convolutional recurrences and the parameterization scheme that retains SSM's properties. This provides a practical extension of S5 for spatiotemporal data. We are working to include additional ablations that address this. An additional contribution is developing the connection of ConvRNNs with recent SSM methods, and providing a parallelizable and efficient ConvRNN-like model which overcomes difficulties of both Transformers and traditional ConvRNNs. We think this can open up a new line of research to consider these alternative architectures.
Response to Reviewer xT6A's questions regarding kernel size
We would like to further address the reviewer's question: is there a large gain in $B$ and $C$ kernels being $k \times k$ instead of $1\times1$? The table below includes the ablation with different kernel sizes and nonlinearities on DMLAB. We ensure similar parameter counts for all models. These models have not fully converged yet (they are about halfway through and we evaluate all models at the same training step), but the results indicate that a larger kernel size significantly helps to improve the performance, especially for the FVD. We note that the rows with $1\times1$ kernels for both B and C are equivalent to vmapping an S5 layer which the reviewer mentioned. We will include the final results in the final version of the paper. These results reflect our observations during previous pilot experiments, though we did not fully quantify the gain. We thank the reviewer for highlighting this point as these results clearly improve the paper. We hope these results highlighting the value of the convolutional structure of ConvS5 fully address the reviewer's concerns and we hope the reviewer may further increase their score based on this clarification. Please let us know if we can provide further clarification. **Kernel size ablations on DMLAB** | B Kernel | C Kernel | Nonlinearity | FVD ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |:-----------------------:|:-----------------------:|:------------:|:----------:|:-----------:|:-----------:|:------------:| | $\boldsymbol{3\times3}$ | $\boldsymbol{3\times3}$ | **ResNet** | **77\*** | **22.3\*** | **0.740\*** | **0.090\*** | | $1\times1$ | $3\times3$ | ResNet | 104 | 21.9 | 0.729 | 0.092 | | $1\times1$ | $1\times1$ | ResNet | 207 | 21.9 | 0.726 | 0.094 | | $3\times3$ | $3\times3$ | GLU | 122 | 20.7 | 0.677 | 0.118 | | $1\times1$ | $5\times5$ | GLU | 144 | 20.4 | 0.670 | 0.126 | | $1\times1$ | $1\times1$ | GLU | 381 | 19.95 | 0.644 | 0.134 | **bold**: final model, **\***: best number
Response
Thank you for running further experiments. I have a few quick clarifying questions: 1) How did you ensure similar parameter counts? i.e. between row 1 (B, C 3x3) and row 3 (B, C 1x1)? 2) What architecture was this? TECO-ConvS5 or ConvS5? 3) Nonlinearity here refers to the block in-between SSM layers?
Thank you for your questions: 1. Between row 1 and row 3 we increased the size of one of the convolutions in the ResNet block by using a $5\times5$ kernel. 2. This was the ConvS5 architecture (No TECO framework). 3. Yes, nonlinearity refers to the nonlinear connection/block between linear layers. Please let us know if we can provide any further clarifications.
Response
Thank you for the clarifications. After further consideration, I believe my concern about technical impact has been addressed (re: further experiments / ablations performed by the authors + other theoretical analysis regarding the ConvS5 formulation), and recommend acceptance (7). In the revised paper, I would encourage the authors to also include some measure of computational requirements for each row (e.g. wall clock time, GPU-hours, FLOPs, etc.) as additional columns in the table to better demonstrate the benefits of a convolutional state-space compared to just adding convolutions to a nonlinearity.
We thank the reviewer for carefully considering our rebuttal, asking thoughtful questions, and increasing our rating. We will include additional computational details in the table as suggested.
Official Comment by Reviewer R5qx
Thanks for the author's response. Most of my questions have been addressed, and I am upgrading my rating from 4 to 5.
We thank the reviewer for taking the time to read our rebuttal and for increasing our rating. Please let us know if we can address any additional questions/concerns to further increase the score.
We thank the reviewers for the productive rebuttal discussion. To address questions from Reviewer xT6A regarding the convolutional structure, we performed additional ablations on the kernel sizes of ConvS5. Since we think the additional experiments are also relevant for Reviewer Kz49 and Reviewer R5qx's requests regarding additional ablations, we move the answer we provided to Reviewer xT6A to the general response also: We would like to further address the reviewer's question: is there a large gain in $B$ and $C$ kernels being $k \times k$ instead of $1\times1$? The table below includes the ablation with different kernel sizes and nonlinearities on DMLAB. We ensure similar parameter counts for all models. These models have not fully converged yet (they are about halfway through and we evaluate all models at the same training step), but the results indicate that a larger kernel size significantly helps to improve the performance, especially for the FVD. We note that the rows with $1\times1$ kernels for both B and C are equivalent to vmapping an S5 layer which the reviewer mentioned. We will include the final results in the final version of the paper. These results reflect our observations during previous pilot experiments, though we did not fully quantify the gain. We thank the reviewer for highlighting this point as these results clearly improve the paper. **Kernel size ablations on DMLAB** | B Kernel | C Kernel | Nonlinearity | FVD ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |:-----------------------:|:-----------------------:|:------------:|:----------:|:-----------:|:-----------:|:------------:| | $\boldsymbol{3\times3}$ | $\boldsymbol{3\times3}$ | **ResNet** | **77\*** | **22.3\*** | **0.740\*** | **0.090\*** | | $1\times1$ | $3\times3$ | ResNet | 104 | 21.9 | 0.729 | 0.092 | | $1\times1$ | $1\times1$ | ResNet | 207 | 21.9 | 0.726 | 0.094 | | $3\times3$ | $3\times3$ | GLU | 122 | 20.7 | 0.677 | 0.118 | | $1\times1$ | $5\times5$ | GLU | 144 | 20.4 | 0.670 | 0.126 | | $1\times1$ | $1\times1$ | GLU | 381 | 19.95 | 0.644 | 0.134 | **bold**: final model, **\***: best number
response to the authors
I thank the reviewers for their rebuttal, specifically for the comparisons in terms of FLOPs. However, I still believe the technical contributions of this paper do not match NeurIPS standards. I'll keep my rating as it is.
Final summary on the originality and significance of your work
Dear authors, Thank you for submitting your work to NeurIPS, responding to the initial reviews provided by the reviewers, engaging in subsequent discussions with them and even conducting further experiments during this author-reviewer discussion phase. As you can see, although most reviewers acknowledge the merits of your work, there is still reservation on recommending for paper acceptance due to various reasons even after further deliberation. As such, may I invite you to give a final summary in light of the further discussions with the reviewers? In particular, please address specifically the originality and significance of this work to justify its acceptance for presentation to the NeurIPS community. Originality and significance, among others, are major review criteria for acceptance, as you can see from the NeurIPS reviewer guidelines: https://nips.cc/Conferences/2023/ReviewerGuidelines. We look forward to your summary (or “pitch”) soon. Best, AC
As requested by the AC, below we provide a summary motivated by the discussion period focused on the originality and significance of our work. ### Summary We presented the ConvS5 spatiotemporal sequence model. We introduced the idea of ConvSSMs, models with convolutional inductive biases of ConvRNNs, and linear, continuous-time dynamics as in SSMs such as S5. However, to address ConvRNN weaknesses (slow training and vanishing/exploding gradients), we addressed two technical challenges: **1. How to parallelize training across the sequence:** We are the first to introduce the parallelization of convolutional recurrences. In Section 3.2, we show both theoretical (Proposition 1) and practical results required to make this efficient and feasible. **2. How to effectively model long-range spatiotemporal dependencies:** To ensure ConvSSMs can capture long-range spatiotemporal dependencies, we developed a theoretical connection between the dynamics of SSMs and ConvSSMs (Proposition 3, Section 3.3). Based on this result, in Section 3.4, we are able to introduce a parameterization and initialization design that allows a ConvSSM's convolutional recurrence to capture long-range spatiotemporal dependencies. **Resulting Method**: The methodological result is ConvS5 which is parallelizable and overcomes difficulties during training on long sequences that traditional ConvRNNs experience. It also provides fast autoregressive generation compared to Transformers. Our experimental results, including new ablations on the convolutional structure suggested by reviewers, support the design decisions and illustrate the effectiveness of the method compared to a number of strong baselines. We now address the Originality and Significance dimensions described in the Reviewer Guidelines linked above: ### Originality: *Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited* ConvS5 is a new method for spatiotemporal modeling that retains the favorable **convolutional inductive biases** and **fast inference** abilities of ConvRNNs, but unlike ConvRNNs, also has **parallelizable training** and can effectively capture **long-range spatiotemporal dependencies**. While ConvS5 can be viewed as a novel combination of the ideas of well-known ConvRNN methods and the relatively new (but increasingly popular) SSM methods, it required novel techniques to combine them in a practical and effective way. We propose a **novel parallel scan** for convolutional recurrences and introduce a **parameterization scheme** that retains the favorable dynamical properties of recent deep SSMs. The theory and discussions in Section 3.2 and 3.3 (and outlined above) provide a rigorous framework for how to think about these choices. We emphasize that ConvS5 is **not** a simple extension of SSMs such as S5. ConvS5 is different than simply replacing a Transformer with an S5 model that acts on a sequence of spatial tokens. We show experimentally that ConvS5 outperforms this S5 baseline in spatiotemporal modeling. More specifically, we show a particular **convolutional structure choice and kernel sizes** (including ablations suggested by xT6A, R5qx, and rhqo) are necessary to obtain the best performance. ### Significance: *Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?* Spatiotemporal modeling is an important and impactful domain. Unlike ConvRNNs, ConvS5 has **parallelizable training** and effectively captures **long-range dependencies**. Unlike Transformers, ConvS5 supports **fast autoregressive generation** and **unbounded context**. ConvS5 shows either state-of-the-art or comparable performance to state-of-the-art baselines on the most challenging long-range video prediction benchmarks available. We compare to a host of baseline methods including additional advanced RNN and efficient transformer baselines requested by reviewers xT6A and R5qx. Besides the method itself, an important contribution is developing the connection of ConvRNNs with recent SSM methods. In addition, during the rebuttal we proposed **additional ConvSSM variants** (ConvS4, H3 and Liquid ConvSSM variants), as suggested by Reviewer rhqo, to further connect ConvRNNs/ConvSSMs with recent advances in sequence modeling. More broadly, the connections and framework developed in this paper can open up a new line of research into these types of alternative, efficient spatiotemporal architectures. **We continue to welcome** concrete feedback from reviewers on these points to further improve the paper.
Decision
Accept (poster)