Discussion on the difference with Voltron
Thanks for your prompt reply to our response and acknowledgement to our evaluation efforts. On the difference of our STP with Voltron, we would like to make the following clarifications and hope this can well address your concern.
1. Our STP shares a simpler design than Voltron just as you mentioned from multi-modal setting to uni-modal setting, which makes our STP a more scalable approach than Voltron as uni-modal data is more easily obtained.
2. The key difference is that our STP decouples the current frame and future frame for separate modeling, while Voltron employs the MAE-ST (VideoMAE) pre-training to jointly model the whole clip (2 frames). Our decoupled design leads to several important differences in technique design:
a. The encoder design is different. Our STP encoder processes each frame independently and there is no attention operation between frames. However, the Voltron encoder operates on a clip to learn a video-level representation and there is cross-frame attention operations. Our design aims to encourage our encoder to learn an image-level representation that is temporally-sensitive for prediction. We find this image-level encoder is more friendly for the downstream adaption compared with the video-level representation (video encoder has higher computational cost).
b. The decoder design is different. Our STP has two decoders: one for spatial prediction and the other for temporal prediction, to treat current frame prediction and future frame prediction separately. For the spatial decoder, we use `joint self-attention` to process the current frame; for the temporal decoder, we add `cross-attention` to capture the interaction between the current frame and future frame (See Fig. 2 in the paper). `This design ensures the predictive property of STP, in sense that the representation of current frame will not see the future and acts as condition for future prediction`. The Voltron only employs a single decoder to reconstruct the whole video directly.
c. The masking strategy is different. The decoupled design allows STP to assign different masking ratios to current frame and future frame. Specifically, we use ratio of `75%` and `95%` for them, while Voltron uses the same ratios for all frames.
3. The ablation studies in in Tab 2(a), Tab 2(b) and Tab 2(c) demonstrate that these different technique designs as mentioned above are `crucial` for achieving excellent performance. Meanwhile, during rebuttal, we add a direct comparison with MAE-ST (The architecture is the same with Voltron without language input). Our STP is better (63.7 vs. 52.6, see response to Reviewer QdYc).
4. Finally, our evaluation is more comprehensive than Voltron on the backbone scaling, pre-training data, and robotic motor control downstream tasks. In terms of pre-training data, Voltron only uses `small-scale` Something-Something-v2 dataset for pre-training. At the same time, the model size of Voltron (V-gen) is only `small`. In contrast, STP uses larger scale training data and trains the `ViT-Large`. Our self-supervised and uni-modal settings ensure the scalability and generality of STP.
If you have further concerns, please feel free to comment. We would like to answer your question.