Summary
This paper considers the problem of using self-supervised learning from video to obtain a representation that is well-suited to the task of estimating correspondence between a pair of images. They propose a significant modification of the MAE training procedure which is adapted for estimating correspondence: one image is not masked at all while the other has most (90%+) of its tokens masked, and the same (i.e. a siamese) encoder is applied to both. This is designed to require the model to internally establish correspondence. The model is trained for pixel prediction on Kinetics-400 and evaluated on several propagation tasks (object mask, part masks, human pose) using kNN inference to establish a dense correspondence field. SiamMAE is shown to greatly outperform existing self-supervised learning procedures with comparable backbone architectures, including methods trained on video. Ablative experiments confirm the importance of combining a siamese encoder with asymmetric masking. Visualization of the attention maps show that the model pays strong attention to object boundaries, seemingly a novel attribute.
Strengths
1. Good motivation and contextualization with respect to past work.
1. The modification of the MAE procedure is simple but clever, and manages to extract much more information for correspondence from video than past methods.
1. Visualization of predicted images is quite impressive (Figure 2), despite the main goal being to learn a feature extractor for correspondence.
1. Comprehensive evaluation with 3 different tasks and wide selection of relevant baselines.
1. Ablative experiments verify the importance of each component of the design.
1. Hyper-parameters provided for reproducibility.
Weaknesses
1. I'm not sure about the emphasis on predicting the _future_. It seems that the temporal order could be reversed (i.e. predict the past given the future) or randomized and I would expect similar results. Has this already been tested?
1. It wasn't abundantly clear how the patch-patch similarity was obtained. It seems to be taken from the cross-attention values within the decoder (line 239). However, this could be more clear since the decoder may contain multiple cross-attention layers with each having multiple heads?
1. It wasn't clear how k-NN and the queue were used to perform propagation. This should be explained in more detail or a reference provided.
1. No code provided at this stage.
1. Lack of confidence intervals (not a major issue - delta is quite large in most cases).
1. No evaluation of ViT-B and ViT-L (not a major issue - impressive results obtained with smaller model).
Questions
(Please also address or correct weaknesses above.)
1. What is the purpose of the [CLS] tokens? Is the model much less effective without it? (It is surprising that the attention masks for these tokens were so salient given that they are not involved directly in the loss.)
Suggestions:
1. I wonder whether it's possible to identify a general principle/strategy for preventing shortcut learning, of which this is just one instance?
Minor edits: (no need to respond)
1. Bold values in Tables 2d and 3b seem incorrect (e.g. 58.4 > 58.1, 56.7 > 56.5).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Additional limitation:
1. Even though Kinetics-400 was used without labels, it is possible that its image distribution is quite similar to that of the downstream tasks. It would be good to discuss this potentialityu, and say that the impact of dataset similarity has not been investigated?
I do not see any negative societal impacts stemming from this individual paper.