Slot State Space Models

Summary

This paper introduces Slot State Space Models where instead of having a single monolithic state, the SSM state is divided into K different - each ideally representing a separate object in the scene. Each SSM state is evolved independently while interacting with each other through self attention. The complete architecture consists of Slot Encoder - to encoder images into slots, SlotSSM - to evolve the state of each SSM state, and Slot Mixer - to capture interactions between slots. Through experimentation they show the proposed architecture is useful for various long range reasoning and object-centric tasks.

Strengths

The idea of incorporating modularity in State Space Models is interesting and combines the strenghts of SSMs (fast parallel training, constant memory inference) with that of modular architectures (factorization, compositionality, ood generalization). The paper is well written and experimental results demonstrate the efficacy of the approach.

Weaknesses

The paper has introduced a multi-layer architecture where the number of slots can vary per layer but this architecture has not been used in any of the experiments. Also if this architecture requires the user to specify the number of slots separately for each user, its utility seems limiting since it is hard to estimate this beforehand and would require abundant hyperparameter tuning. The authors have stated that the interaction between slots is sparse. If the interaction is done using QKV self-attention, then each slot can interact with every other slot and hence the interaction should be dense. I don't understand how such an interaction can be sparse. It would be nice to see an ablation which investigates the effect of number of slots. It seems that most experiments use 6 slots. Specifically, it would be interesting to see how the performance scales with increasing number of slots.

Questions

- One of the issues with applying object-centric models to videos is the problem of temporal consistency - a given object may be represented by different slots into neighbouring timesteps. Did the authors face this issue in their video experiments and is there a way to address this in SSM based models which consider parallel training across timesteps? In RNN based method, this is addressed by initializing the slots of the future timestep by slots from the previous timestep. - Would it be possible to apply top-k attention like used in RIMs in the proposed architecture? Do the authors see any benefit to doing that in this architecture.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed limitations.

Summary

This paper presents SlotSSMs, an extension of SSMs such that their states encourage information separation. This is in contrast to conventional SSMs whose state is monolithic. The authors evaluate slotSSMs in object-centric video understanding and video prediction tasks involving multiple objects and long-range temporal dependencies.

Strengths

- As far as I know, this is the first work to propose the use of slots and object-centric modeling in SSMs. - The paper is well structured and easy to read. The exhibition and definition of the different SlotSSM components is very clear (up to some small observations). - The paper contains multiple experiments that illustrate the power of their proposed method. It also incorporates multiple ablation studies that illustrate the contribution of each of the proposed modules.

Weaknesses

- My biggest concern lies in the evaluation of the method. For example, in Fig 4, the loss of SlotSSMs is relatively low and comparable to that of SlotTransformers. However, if we look at the predictions, these do not look good for any of the methods. I wonder if these methods and dataset are the right place on which SlotSSMs should be evaluated. Note that this is also the case in Fig 7, where all of the predictions are also quite bad. I would encourage the authors to think about settings on which SlotSSMs could really solve an open issue as this would increase the impact of the paper. - Next, it seems that the authors evaluate other features than what is often evaluated in the literature. For example, in the MOVi paper, the authors evaluate object segmentation, for which SAVi gets 82% accuracy. I do not see this back in the paper, unless this is the first column of Fig 7 right, in which case the matrics are very low in comparison to the original paper. This, in combination with the reconstructions shown in the paper, I doubt that the Slot SSMs would be any better than SAVi. Also, the authors should consider including more recent methods in the comparison as well. - In addition, it would be nice to see –as is often the case in slot papers– to check what the slots are actually learning. If this is not as easy for SSMs as for Transformers, then this should be stated as well in the limitations. - I also have concerns regarding the reproducibility of the method. There are several experimental details missing which, in its current form, I would argue would make this paper irreproducible. Given that the metrics in this paper are also lower than what is shown in other papers, I consider this to be of vital importance. I would encourage the authors to add these details in the appendix.

Questions

Aside from the previous weaknesses, there are some other aspects I would like to mention / clarify: - In Eq. 8, are these Linear layers different per slot or are they the same everywhere? - As far as I understand, in Eq.7 the “CLS tokens” are basically learnable embeddings. Is there a reason to call these CLS tokens instead? This might be confusing, as this implies that the input is always the CLS, which I think is not the case here. - The authors go about how their method can use a different number of slots at each layer, but then go and use the same number of them at all layers. It would perhaps be better to introduce the slots refinement module as it is. In the current setting, when reading this it feels as if having a different number of slots will be required, which can be counterproductive for the impact of the paper (readers can think that the method is too complicated to use in practice). - There are several typos / misspellings in the paper. - As far as I know, the A matrix of Mamba is not input dependent. Could you confirm if you made it input dependent in Slot SSMs? Also, if you are using only Mamba, why not call the paper Slot Mamba? - In many aspects, your proposed method is similar to the idea of using heads but for SSMs. Could you please comment on this? Also, I am aware that multiple works have introduced heads to SSM / long-conv models as well, e.g., in multi-headed Hyena [1]. [1] https://arxiv.org/abs/2310.18780

Rating

7

Confidence

4

Soundness

4

Presentation

2

Contribution

3

Limitations

The authors clearly state a limitations section for the method. However, the limitations stated there obey more to a future work section and does not really discuss the limitations of the existing method. I would encourage the authors to revise this section and state the limitations of the submitted work. ### Conclusion Whilst I acknowledge the novelty of this paper, I as of now have many concerns that I believe should be addressed before the paper is ready for publication. I therefore am unable to support acceptance. With that being said, I am happy to increase my score should these concerns be addressed.

Summary

This paper presents Slot State Space Models (SlotSSMs), a novel framework that integrates modular structures and inductive biases into State Space Models (SSMs) to improve sequence modeling. SlotSSMs maintain a collection of independent slot vectors and perform state transitions independently per slot, with sparse interactions managed through self-attention. This approach effectively captures the inherent modularity present in many real-world processes. Authors demonstrate substantial performance gains in object-centric video understanding and video prediction tasks, highlighting the importance of modularity in sequence modeling. Additionally, the authors introduce Object-Centric SlotSSMs (OC-SlotSSMs), which leverage inverted attention to further enhance the discovery of modular structures. Extensive experiments across multiple tasks, including multi-object video prediction, long-context reasoning, unsupervised object-centric learning, and 3D visual reasoning, validate the effectiveness and versatility of the proposed models. The paper also includes a detailed analysis of the emerging modularity in SlotSSMs, showcasing their ability to naturally discover and exploit the underlying structure of the data.

Strengths

- This approach is original in its design, utilizing independent slot vectors with sparse interactions managed through self-attention. - A thorough evaluation of the proposed models across multiple challenging tasks, including multi-object video prediction, long-context reasoning, unsupervised object-centric learning, and 3D visual reasoning. The experiments are well-designed and provide robust evidence of the models' effectiveness. - The authors provide a detailed analysis of the emerging modularity in SlotSSMs, offering valuable insights into how the models learn to capture the underlying structure of the data, with visual aids. - The success of SlotSSMs in capturing modular structures suggests promising directions for further research in modular and object-centric sequence modeling. This could lead to the development of even more advanced architectures capable of handling complex, real-world data.

Weaknesses

- While the potential for applying SlotSSMs to other modalities (e.g., text, audio) is mentioned, the paper does not provide experiments or theoretical analyses in these areas. Including preliminary results or theoretical discussions on applying SlotSSMs to different modalities would strengthen the paper and demonstrate broader applicability. - In the related work section, mention also another interesting work utilizing SSMs for Neuromorphic Cameras: "State Space Models for Event Cameras". Nikola Zubić, Mathias Gehrig, Davide Scaramuzza.

Questions

1. Given the current computational constraints mentioned in the paper, what are the potential strategies for scaling SlotSSMs to handle larger datasets and model sizes? Are there specific optimizations or approximations that you are considering to improve scalability? 2. Have you conducted any preliminary experiments or theoretical analyses on applying SlotSSMs to other data modalities, such as text or audio? What challenges do you anticipate in these domains, and what benefits might SlotSSMs bring? 3. How do you anticipate SlotSSMs would perform on datasets with higher visual complexity? Have you identified any specific challenges or limitations in such scenarios, and what strategies might you employ to address them?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Already discussed by the authors.

Thank you for rebuttal

I thank the author for their rebuttal. My questions have been addressed. I hope that the authors will add the clarifications they mentioned in the rebuttal to the paper. I am happy to raise my score.