Self-supervised video pretraining yields robust and more human-aligned visual representations

Summary

The research question addressed by the paper is whether self-supervised training on videos leads to image features that are better aligned with human perception. To that end, the authors propose a contrastive learning method that leverages natural changes over time as different views, beyond adapted image augmentations. Moreover, an attention mechanism is introduced that allows the model to compare image pairs based on features from selected regions only. The model is trained on videos that have been curated to better match the ImageNet category distribution than existing video datasets. Trained this way, the model performs better than previous methods when transferred to several standard benchmarks and shows a better alignment with human vision.

Strengths

- Training computer vision models to better align with human perception is an active area of research with the potential to improve, e.g., the robustness of current approaches. Self-supervised training on natural videos is closer to how humans learn than standard training methods. Exploring this direction therefore is well motivated. - The authors compare to a broad range of previous models on several computer viion benchmarks. The newly proposed method improves over previous approaches in most cases. - Several ablation studies are performed and reported in the supplement to disect the influences of the contributions on the improved performance.

Weaknesses

The paper aims at better aligning neural network features with human perception. However, the range of models tested for alignment with human perception is much smaller than the range of models tested for performance on computer vision bechmarks. In particular, none of the other video based self-supervised methods is tested and none of the ablation studies considers alignment with human vision. The title of the paper, "Self-supervised video pretraining yields human-aligned visual representations", and the introduction therefore raised wrong expectations for me. Due to the missing ablations, one cannot judge whether the improved human alignment is due to the video pretraining or other contributions, such as the new training set or the multi-scale contrastive attention pooling. Moreover, I find the title too bold given that other methods perform much better in terms of human error consistency according to the official benchmark repository (e.g., CLIP). **Update:** Both points raised above have been addressed during the rebuttal. Therefore I improved my rating of this paper.

Questions

As described above, the paper has a much stronger focus on performance in terms of computer vision (where the proposed method is very competitive) than alignment with human vision. I would recommend to change the title and introduction to better align with this focus and I am happy to increase my score if this issue is addressed. L186ff needs clarification for me: When evaluating video understanding, the paper claims that "VITO learns features that capture finegrained temporal deformations of objects". However, VITO is based on image networks so that spatio-temporal features cannot be learned.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss limitations of the paper due to the focus on a single architecture (ResNet-50). Potential societal impacts are not discussed.

Summary

This paper introduces a new video dataset (VideoNet), and architectual adjustments (VITO) that enables allows pretraining on video datasets to improve performance on downstream transfer tasks, such as segmentation, object detection, and generalization. The VITO network uses a ResNet backbone, and architectural changes include extracting spatial feature maps at the two penultimate blocks, applying a learned softmax function to obtain a weighting over features, and then projecting this into a final feature map. This feature extractor is trained using the InfoNCE loss on two sampled video frames, with additional image-space augmentations applied. Additional experiments in the supplementary material investigate the use of transformer backbones. The VideoNet dataset retrieves video clips matching the ImageNet categories, and applies an image classifier on these videos to filter for the correct category. Additional experiments compare the attention maps from the VITO model and other baselines to human ground truth from the ClickMe dataset.

Strengths

- The proposed method and training dataset presents strong empirical results. It outperforms, or is competitive with, image pretraining on ImageNet, video pretraining, or robust image pretraining on a variety of transfer tasks. Notably, it demonstrates improved robustness on datasets with image corruptions even when compared to models trained for robust classification. - Ablations evaluate the independent contributions of both the dataset on the model design. With the same VITO model, but different datasets, VideoNet can outperform AudioSet as a prior video dataset on segmentation and object detection tasks. On the same dataset (VideoNet) but different models (MoCLR vs VITO), VITO also outperforms the prior method. (Table B.1) - On additional experiments on Swin transformers show promising improvements. I think additional details on the transformer setup, and which spatial feature maps are extracted in the transformer would be helpful. - This paper is well written and clear to follow.

Weaknesses

- The benefit of VITO seems to be primarily on transfer tasks. On standard classification, it falls short of ImageNet pretraining by ~10%, but it is still outperforming other video tasks. It is also slightly lower on object detection compared to the DINO representation. However, I think there are still valuable insights to be gained from the methodology and dataset presented here. - There are some additional parts that could benefit from clarification. Please see the below questions section.

Questions

- L124: what is the softmax normalized over? - L130: is $a_\xi$ also a target network? It may be helpful to also illustrate the network $g$ in Fig. 1 - Fig 3: Why was the CLIP Resnet chosen for the attention map, as opposed to the attention maps extracted from CLIP or DINO transformer models?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitation are adequately addressed. The current setup explores primarily the ResNet backbone with the InfoNCE loss, but there are promising initial results with transformer models.

Summary

The paper proposes a self-supervised method for learning image representations from videos. Towards this end, a procedure to curate video datasets most suitable for such pre-training is proposed by selecting videos that best match the distribution of visual classes found in ImageNet. Secondly, this dataset is leveraged through a contrastive training approach where the common global average pooling of video frame features is replaced with an attentional module that learns to select spatial regions of several network layers to construct the image embedding. The learned representations are evaluated on a large set of downstream tasks, including spatial scene understanding (segmentation + detection), video understanding (segmentation + action recognition), and robust image classification.

Strengths

- The method is very well presented, and the paper is well written - The downstream evaluation of the learned image representation is extensive, and the model achieves good performance across a variety of tasks - The finding that a video dataset with similar category distribution to ImageNet appears to perform considerably better than many existing video datasets is interesting (see Fig 4)

Weaknesses

- I would have preferred to see more extensive ablation experiments to support the technical contributions of the paper (e.g., the attentional pooling module for contrastive learning). These ablations are currently only reported on a single task and dataset. It would be more convincing to see the influence on all the downstream tasks in Table 1 and to summarize these ablation results in a Table. As it is, it is unclear if the benefits of the proposed training are consistent across tasks or if most of the benefit is due to the data curation process. - The data curation process relies fundamentally on reproducing the object category distributions of ImageNet in a video dataset. While it is interesting that this appears to lead to a benefit across many downstream tasks, it is unclear what explains these benefits. Is it greater visual diversity, lesser class imbalance, or maybe some other property?

Questions

I would appreciate it if the authors could address the weaknesses outlined above. I'm particularly interested in how the technical contributions affect downstream results across multiple benchmarks (i.e., how do the ablations look when evaluated across the multiple benchmarks in Table 1)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are adequately addressed.

Summary

This paper studies how to take advantage of natural temporal distortions in video to learn image spatial representations. The paper first proposes a VideoNet dataset that filters the video data from common video datasets by an ImageNet classifier. The paper further proposes to multi-scale attention pooling to improve the baseline algorithm. The algorithm trained on the proposed the dataset is evaluated on both image and video datasets. The paper also studies the human alignment of the learned attention.

Strengths

1. Pre-training image representations with video datasets is a natural and valuable idea to explore. 2. The VideoNet dataset could be a valuable contribution to the community provided the authors have a plan to release it publicly. 3. The paper provided abundant experiments to demonstrate the effectiveness of the proposed algorithm.

Weaknesses

1. Although using an ImageNet classifier to filter the video data is a practical idea to reduce the domain gap between video and image datasets, this also introduces unfairness to the comparison to other self-supervised methods. This is because the filtering process with an ImageNet classifier implicitly takes advantage of ImageNet labels, and ImageNet labels are very effective on downstream image tasks as validated by the authors in Figure 4. 2. It is unsure how much the temporal deformations provided by a video dataset help with the pre-training VITO, which I think is one of the central questions for video contrastive learning for image downstream tasks. Because if temporal deformations are not useful, then why bother filtering a video dataset which costs much more than a image dataset. I am not sure if T=0 in Figure B.2 is a baseline that generate two views from the same frame in VideoNet. If so, this would be a helpful ablation that can be highlighted in the main text.

Questions

I have a few questions on the experiment settings: 1. In Table 1, why are some results different from what are reported in their original papers? For example, VFS [8] on DAVIS is reported to be 68.9 in its paper, but 67.8 in this manuscript. Is this 67.8 number produced by the authors with a different experimental setting? Are most of numbers in Table 1 produced by the authors by fine-tuning on the publicly available checkpoints? 2. Why are the UCF-101 results in Table 1 significantly lower than other self-supervised learning algorithms on video? For example, in Feichtenhofer et al. [34] a ResNet-50 pre-trained on K400 can easily obtain an over 90% accuracy on UCF101? 3. The experiments on ImageNet-3DCC in Figure 2 are interesting. I am wondering why only evaluate MoCLR as the representative self-supervised algorithm, while not others like DINO? 4. I am wondering the training cost on VideoNet. For example, how many ImageNet-equivalent epochs have been used and how many crops are there in each step?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations potential negative societal impact of their work.

Summary

The paper introduces a SSL method using video pre-training to produce general visual representations for both image and video tasks. The proposed VITO pipeline includes a data curation process and a video SSL technique based on MoCLR. The authors conduct a comprehensive evaluation of VITO's performance on diverse benchmarks, spanning detection, segmentation, video segmentation, classification, and out-of-distribution object recognition. Additionally, the paper includes a comparison of human-alignment on two benchmarks.

Strengths

• The paper is well-organized and easy to follow. The proposed components for the method, including data curation and various modifications on MoCLR, all make sense to me. • Ablation studies are conducted by the authors to validate the effectiveness of their proposed method.

Weaknesses

1. The main method is based on the image SSL method MoCLR. While the authors demonstrate the effectiveness of multi-scale contrastive attention pooling, this component appears applicable to image SSL as well. Therefore, the technical novelty of applying an image SSL method to video datasets seems limited, especially considering the existence of other established video pre-training works. 2. The motivation behind choosing MoCLR as the starting baseline should be clarified. Given the availability of better contrastive learning methods and the superior performance of Masked-based pre-training (e.g., MAE), it would be beneficial to provide comparisons or discussions to justify this selection. 3. The data curation process plays a critical role in the overall pipeline. To gain more insights, additional ablation or analysis would be helpful. 3a. For example, what are the the effect of the number of curated videos and different filtering strategies. 3b. For the ablation (Figure 4), considering that performance can be significantly influenced by pre-training data and downstream dataset distribution, it would be better to verify on more downstream tasks instead of only one dataset. 3c. It would be interesting to Investigate the scaling behavior on the training schedule, as 300 ImageNet epochs might be insufficient for large-scale video datasets. 4. In Table 1, incorporating more standard benchmarks would enhance the evaluation's comprehensiveness. For instance, Kinetics is a more widely adopted benchmark for video understanding compared to the small-scale UCF-101. 5. In B.5, VITO doesn’t show superior performance on image-based tasks, especially it has significant drop on ImageNet compared to ImageNet pre-trained methods. It might be worth to have some discussions around that as this is also important observations. 6. The paper lacks comparisons with some video-specific SSL methods, including contrastive learning-based and Masked-based methods (e.g., VideoMoCo, VideoMAE, MAE_ST, etc).

Questions

Overall I think the video pre-training direction and results are interesting. However, my main concerns revolve around the limited evaluation and results presentation, which lack sufficient ablation studies and comparisons, as mentioned in the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

See weakness

Summary

The paper proposes VITO, a new method for self-supervised video pretraining that learns general and human-aligned visual representations. It made several modifications over existing contrastive learning frameworks, including larger crop sizes, improved temporal sampling scheme, and multi-scale attention feature pooling for the projector. The paper also creates a video dataset (VideoNet) that aligns the class distribution with ImageNet, and partially redresses the imbalance between image and video learning. This improves spatial understanding compared to other video datasets. Experiments are conducted on various tasks, including semantic segmentation / object detection on image, video object segmentation and classification, as well as OOD object recognition.

Strengths

This paper is well-organized and easy to follow. The technical contribution and the way to do data curation make sense to me. The results shows that VITO matches or exceeds image pretraining on spatial tasks like object detection and segmentation while outperforming other video pretraining methods. This shows it learns a general representation. The ablation is comprehensive.

Weaknesses

The human alignment evaluations are somewhat weak. Human alignment is a big area and whether saliency and shape bias tasks can serve as representative tasks for the evaluation concerns me. More comprehensive alignment benchmarks could be explored. It would be great if the authors could also show baselines with the masked image modeling objective. Overall I think this paper shows good results and presents things clearly. Thus my rating is weak accept. I encourage the authors to perform the above things I suggest to improve the quality of this paper further.

Questions

Please see the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

quantitative comparisons with masked image modeling approach

The reviewer requested comparisons with a masked image modeling baseline. While we had initially not found any such baselines which use a comparable architecture, in fact the recent work of Li et al. 2022 (A2-MIM [1]) develops a masked-image modeling methodology that can be applied to ResNet-50 architectures and is highly performant. To facilitate the discussion, we evaluated the A2-MIM models on multiple benchmarks. On the Geirhos human alignment benchmarks, we find it does not improve over standard supervised training in human alignment and still significantly underperforms VITO. On the ImageNet-3DCC benchmark, while A2MiM provides additional robustness (at the level of the stylized and robust resnets), it still underperforms VITO significantly. For detailed numbers see the tables below. We hope this provides an additional data point that demonstrates that VITO outperforms a variety of state-of-the-art SSL objectives in this respect. **Geirhos Human Alignment** | **Method** | **Dataset** | **Accuracy difference (↓)** | **Observed. Consistency (↑)** | **Ceiled Error Consistency ( ↑)** | | :--- | :----: | :----: | :----: | :----: | | VITO | VideoNet | **0.157** | **0.564** | **0.422** | | A2-MiM | ImageNet | 0.197 | 0.520 | 0.325 | | DINO | ImageNet | 0.236 | 0.504 | 0.291 | **ImageNet-3dcc** | **Method** | **Dataset** | **$\Delta$ Accuracy Severity 1** | **$\Delta$ Accuracy Severity 2** | **$\Delta$ Accuracy Severity 3** | **$\Delta$ Accuracy Severity 4** | **$\Delta$ Accuracy Severity 5** | | :--- | :----: | :----: | :----: | :----: | :----: | :----: | | VITO | VideoNet | **-14.3** | **-19.6** | **-24.8** | **-29.6** | **-34.1** | | L2-Robust | ImageNet | -15.2 | -23.5 | -30.1 | -35.8 | -40.5 | | A2-MiM | ImageNet | -16.3 | -23.4 | -30.5 | -36.6 | -42.2 | |DINO | ImageNet | -19.4 | -27.7 | -34.7 | -40.6 | -45.5 | [1] Li, Siyuan, et al. "Architecture-Agnostic Masked Image Modeling--From ViT back to CNN." arXiv preprint arXiv:2205.13943 (2022).

quantitative comparisons with a masked image modeling approach

The reviewer requested comparisons with a masked image modeling baseline. While we had initially not found any such baselines which use a comparable architecture, in fact the recent work of Li et al. 2022 (A2-MIM [1]) develops a masked-image modeling methodology that can be applied to ResNet-50 architectures and is highly performant. To facilitate the discussion, we evaluated the A2-MIM models on multiple benchmarks. On the Geirhos human alignment benchmarks, we find it does not improve over standard supervised training in human alignment and still significantly underperforms VITO. On the ImageNet-3DCC benchmark, while A2MiM provides additional robustness (at the level of the stylized and robust resnets), it still underperforms VITO significantly. For detailed numbers see the tables below. We hope this provides an additional data point that demonstrates that VITO outperforms a variety of state-of-the-art SSL objectives in this respect. **Geirhos Human Alignment** | **Method** | **Dataset** | **Accuracy difference (↓)** | **Observed. Consistency (↑)** | **Ceiled Error Consistency ( ↑)** | | :--- | :----: | :----: | :----: | :----: | | VITO | VideoNet | **0.157** | **0.564** | **0.422** | | A2-MiM | ImageNet | 0.197 | 0.520 | 0.325 | | DINO | ImageNet | 0.236 | 0.504 | 0.291 | **ImageNet-3dcc** | **Method** | **Dataset** | **$\Delta$ Accuracy Severity 1** | **$\Delta$ Accuracy Severity 2** | **$\Delta$ Accuracy Severity 3** | **$\Delta$ Accuracy Severity 4** | **$\Delta$ Accuracy Severity 5** | | :--- | :----: | :----: | :----: | :----: | :----: | :----: | | VITO | VideoNet | **-14.3** | **-19.6** | **-24.8** | **-29.6** | **-34.1** | | L2-Robust | ImageNet | -15.2 | -23.5 | -30.1 | -35.8 | -40.5 | | A2-MiM | ImageNet | -16.3 | -23.4 | -30.5 | -36.6 | -42.2 | |DINO | ImageNet | -19.4 | -27.7 | -34.7 | -40.6 | -45.5 | [1] Li, Siyuan, et al. "Architecture-Agnostic Masked Image Modeling--From ViT back to CNN." arXiv preprint arXiv:2205.13943 (2022).

Response to author rebuttal

Thanks to the authors for the response and the additional clarifications and ablation experiments. I agree with reviewer jN1F that perhaps human alignment was not the principle focus of this paper, but overall I find the methodology presented here to demonstrate compelling results across a variety of tasks. I will retain my original rating.