Moving Off-the-Grid: Scene-Grounded Video Representations

Summary

This work presents a video representation learning approach where tokens are decoupled from explicit grid locations in the video sequence. Rather than simply extract patches to construct tokens and apply self-attention across blocks of frames, this proposed model predicts sub-pixel motion given a history of frames to advect latent features throughout the video. Once those grid-aligned features are advected to the next frame to their non-grid aligned positions, cross attention can be applied between the grid aligned observed features and the advected features in order to correct the result. The authors claim three contributions: * Introduction of MooG * Qualitative validation of the approach * Quantitative evaluation on a number of different downstream tasks

Strengths

Idea seems like a fundamental one that the community will use. It applies to a wide variety of tasks. The concept is novel and general. There's a lot of great qualitative experiments that help me understand that this method is implicitly learning a tracking representation.

Weaknesses

I had a hard time understanding the point of the corrector for awhile. If the predictor functions as intended, shouldn't a reconstruction loss applied to D(P(z)) be enough? But after some more thought, I guess the intent isn't to build a an open loop video predictor (although I suppose the authors could evaluate it as such?), but instead the predictor is just a way to establish alignment between features in neighboring frames so that the cross attention of the corrector is more effective? I think the prior work could go farther back in history in order to frame this proposed prediction-correction framework as being related to old school recursive filters. For example, I think if I were told how this is designed similar to a Kalman filter, then I'd have a much clearer picture of what is trying to be done. I don't think Fig 1 does enough to clarify the system. I felt it was still necessary to read eqn 1 and 2 to understand what exactly was going on. Some ideas include: (1) include arrows going up in the direction of image -> encode -> state and state -> decode -> image. (I originally read this as left-to-right top-to-bottom and realized I went the wrong direction once I saw encode and decode -- arrows would prevent this). (2) Make the relationship between prediction and correction clear. It's currently presented as two colorful feature map sets, which doesn't communicate much. Once again, if it's established early on what the predict-correct framework looks like, then this is obvious, but I failed to grok it on the first read through.

Questions

I didn't get a good impression of what kind of motion dynamics the predictor properly supported. It looked like there was some overshooting happening in the gifs, but it's not clear to me if this is a real issue. Can this be clarified? One ablation that would be useful is to change the backprop through time length. It would be great if there was evidence that only 2 or 3 frames were enough to get the predictor to estimate dynamics. Another question could be, what kind of dynamics are modeled with 2, 3, 4, 8, etc. frames and are there diminishing returns? Can this be clarified? Given that this model is deterministic and subject to over-blurring in the reconstruction task, have the authors explored other losses such as an LPIPS loss which might be more invariant to small misalignments?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

looks good to me

Summary

This work discusses Scene-Grounded Video Representations. Compared with current vision models that make each layer consist of tokens organized in a grid-like fashion, the authors introduce Moving Off-the-Grid (MooG), a self-supervised video representation model that proposes an alternative approach. The novelties are: Introducing Moving Off-the-Grid (MooG), a novel transformer-based recurrent video representation model designed to learn off-the-grid (OTG) representations using a straightforward next-frame prediction loss; illustrating how this representation enhances a range of downstream vision tasks, including point tracking, monocular depth estimation, and object tracking.

Strengths

+ This work allows individual tokens to consistently track elements of the scene through videos of arbitrary length and “anticipate” where a scene element will be observed next. + The authors designed MooG to process an unlimited number of video frames while consistently maintaining a scene-grounded representation of the video. + Tokens within the latent state are inherently associated with particular pixel positions. The model can obtain corresponding token representations based on different features of the image.

Weaknesses

- In line 135, you said “It is, however, crucial that the image is decoded only from the predicted state—decoding the current frame from the corrected state reduces the problem to simple auto-encoding and hurts representation quality considerably.” So how to prove the effectiveness of the Corrector module? - What’s the difference between MooG and Deformable DETR? Does the Corrector play a same role as the offset modal in Deformable DETR? Besides, do you treat the states as the tokens? - It seems that you used transformer (attention module) in Corrector and Predictor, and how does the algorithm perform in tracking tasks? What is its speed? Can it meet real-time requirements? How’s the performance in the Multi-object tracking field? - In table 3, the performance of TAPIR on Davis-full is better than MooG, so could you please provide more detailed explanations?

Questions

Please refer to the Weaknesses

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do some analysis and the work will not have any potential negative societal impact.

Summary

The authors present a self-supervised video representation learning strategy. A grid-structure free feature extractor is trained using a next frame prediction objective. A corrector module extracts per-frame features. A predictor module predicts the next frame features. A decoder (with suitable grid free architecture) reconstructs frames. A sparse (for efficiency) reconstruction loss is applied on decoder output as the learning signal. The method, MooG, appears to bind features to specific structures in the visual inputs and tracks these across frames. The authors present evaluations for tracking and depth estimation.

Strengths

1. Interesting and novel idea for self-supervised representation learning from videos 2. Clear explanation of methodology and through details

Weaknesses

1. **Evaluations:** Learned representations are evaluated on a) niche tasks, b) with only image SSL baselines. Proposed method uses videos to learn unlike the baselines. Please compare a) on generic tasks (evaluate learned representations for classification, object detection, or segmentation) and b) compare against video SSL methods (on the identified tasks and generic ones). In fact, in introduction, the authors argue that tasks like object detection and segmentation need the kind of off-the-grid architectures being proposed. Evaluation on such tasks will strengthen the authors case. 2. **Prior Video SSL works:** It is unclear how these representations compare against video SSL methods [2,3,4,5]. Maybe try to apply some of these methods on the selected tasks to compare how MooG representations compare to them? Or evaluate MooG on tasks these are commonly used for. 3. **Related Works:** Consider discussing [1] which explores a similar idea of next frame prediction in a latent space to learn good representations. Also [6] which learns local features using language which shows DINO like grouping / tracking behaviour. [1] Sequential Modeling Enables Scalable Learning for Large Vision Models, CVPR 2024 [2] Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations, ICCV 2023 [3] Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video, ICLR 2024 [4] Self-supervised Video Transformer, CVPR 2022 [5] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, NeurIPS 2022 [6] Perceptual Grouping in Contrastive Vision-Language Models, ICCV 2023

Questions

See weaknesses

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Ok

Summary

This paper proposes a field-based method for video representation learning called MooG. Instead of propagating a discretized grid of features for every pixel location in the video, the method updates an arbitrary set of state tokens. These state tokens are used to parametrize a context dependent PerceiverIO-style “field”: a network that decodes the RGB values at a particular location (x, y) in the target frame conditioned on the “predicted” state tokens from previous frames and a position encoded (x, y) input. Since the state tokens parametrize a field, they are dubbed “off the grid”. The paper proposes to train such a model in a recurrent fashion using next frame prediction. Further, it describes methods to read-out the (potentially pre-trained) representations for downstream tasks. Results show qualitatively that the off-the-grid representations are “well behaved” through compelling visualizations and quantitatively work better than on-the-grid counterparts for a range of dense visual tasks, such as object tracking and depth prediction.

Strengths

- The idea of combining recurrent state token estimation together with an off-the-grid field-like representation that is repeatedly queried across time and trained with next frame prediction is interesting and novel. - The methods section is simple and intuitive and the writing is clear and concise. At important steps (such as 130 – 132) where there could be room for confusion (eg. decoding from predicted tokens vs correct tokens), the details are highlighted to elucidate the key elements that make the method work. - A wide range of experiments are shown for three important dense prediction tasks: monocular depth estimation, points tracking and boxes tracking. Baselines are chosen appropriately, representing widely used and ubiquitous recent methods in the respective fields. Quantitative experiments show that MooG is a powerful representation learning method, and that it outperforms baselines in the frozen setting, which should be the dominant paradigm to evaluate a video pre-training model. - The qualitative visualizations are impressive and creative. That tracking (in the token attention sense) and segmentation features (in the PCA sense) emerge from using the limited token and off-the-grid inductive bias are exciting results. The paper shows many examples of these results.

Weaknesses

- While appropriate baselines are chosen for all downstream evaluations, and comparison is made with on-the-grid representations that simply propagate features, baselines with architectures similar to MOOG have not been explored in a lot of detail. In the related work, the “off-the-grid” similarity between Perceiver IO and MooG is acknowledged, with the key difference being that MooG additionally implements recurrence of state tokens. One baseline that would be a closest comparison would be to train a Perceiver IO like model, except with spatio-temporal queries: in particular, this would entail conditioning on time and position. This could be trained simply like an auto-encoder (essentially reducing to Perceiver IO for video), or with next frame prediction. - Papers that use similar architecture but specialized for domain-specific video prediction tasks are missing from the related works. While MooG is a general purpose representation learning method, a thorough discussion of some domain specific models would be important to highlight key differences between those papers and the potential general purpose nature of MooG representations. Some of these papers are already cited (eg. CONDITIONAL OBJECT-CENTRIC LEARNING FROM VIDEO, Kipf et al. 2022), but are missing deeper discussion on fundamental differences to MooG.

Questions

- Is the paper the first to show token binding to task related activities emerging unsupervised in a field-based transformer? - Are the baselines chosen for point tracking (TAP-Net, TAPIR) and for depth prediction (DPT, etc) the state of the art results in the domains? (Given MooG is a general purpose video representation learning method, I agree it would not be fair to compete against the best task-specific engineered methods, but a discussion on limitation and deficit to state of art (if any) would be useful).

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations section is adequate and thorough (discussing disocclusions, occlusions, and alternate coarse / fine tasks possible on video), and future directions for improvement are also duly noted in Appendix A.

Most concerns addressed; update to weak accept

I thank the authors for the detailed rebuttal. The authors address most concerns here. I believe this paper will be valuable to the community, hence support accepting. Suggestions: Please update the paper with the explanation on why action recognition benchmarks is unfair to MooG. Highly recommend adding at least one line to the intro to explain this (while highlighting its unique strengths). It is useful to highlight the kind of tasks these learned representations are ideal for alongside limitations. Also, please mention this in detail somewhere in the paper, Note on comparison to video SSL methods: MAE style pre-training is known to not emerge strong grouping / tracking behavior (i.e. what DINO has). Consider evaluating against Video-SSL works building off DINO (e.g. [4] or preferably a similar more recent work). Maybe in later work or as an addition to the appendix.

Concerns addressed

My concerns were addressed in the rebuttal and I will keep the rating of weak accept.