Video self-supervised learning (VSSL) has made significant progress in recent years. However, the exact behavior and dynamics of these models under different forms of distribution shift are not yet known. In this paper, we comprehensively study the behavior of six popular self-supervised methods (v-SimCLR, v-MoCo, v-BYOL, v-SimSiam, v-DINO, v-MAE) in response to various forms of natural distribution shift, i.e., (i) context shift, (ii) viewpoint shift, (iii) actor shift, (iv) source shift, (v) generalizability to unknown classes (zero-shot), and (vi) open-set recognition. To perform this extensive study, we carefully craft a test bed consisting of 17 in-distribution and out-of-distribution benchmark pairs using available public datasets and a series of evaluation protocols to stress-test the different methods under the intended shifts. Our study uncovers a series of intriguing findings and interesting behaviors of VSSL methods. For instance, we observe that while video models generally struggle with context shifts, v-MAE and supervised learning exhibit more robustness. Moreover, our study shows that v-MAE is a strong temporal learner, whereas contrastive methods, v-SimCLR and v-MoCo, exhibit strong performances against viewpoint shifts. When studying the notion of open-set recognition, we notice a trade-off between closed-set and open-set recognition performance if the pretrained VSSL encoders are used without finetuning. We hope that our work will contribute to the development of robust video representation learning frameworks for various real-world scenarios. The project page and code are available at: https://pritamqu.github.io/OOD-VSSL.
Paper
Similar papers
Peer review
Summary
The authors studied the behavior of six popular self-supervised methods in response to various forms of natural distribution shift. And the study uncovers a series of interesting findings and behaviors of video self-supervised learning (VSSL) methods. The experiments and results are beneficial for the video representation learning community.
Strengths
1. Extensive experiments are conducted. And sufficient experimental data and analysis are provided. It seems to the reviewer that all the conclusions are grounded.
Weaknesses
1. It seems that the effects of data size and model size are not discussed. 2. Many experiments are conducted and some conclusions are obtained. But no improved VSSL are provided, which might be more interesting.
Questions
1. Any suggestions for the future research to design better DSSL method? 2. Providing a table of summarisation of the 6 methods might be helpful for the readers. There are many experiments and data. 3. The methods are pre-trained on K400 and K700 datasets, which might not be large enough. If the models are pre-trained on a much larger dataset, will the conclusions in this paper be changed? 4. Does the model sizes effect the conclusions in this paper ?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Limitations have been discussed in the text. And no obvious negative societal impacts.
Summary
The paper studies the generalizability of many self-supervised training approaches in the video domain. The paper uses 17 tasks to completely test different aspects of the models, including view-point change, temporal modeling, and open set generalizability.
Strengths
1. Most of the cutting-edge video pre-training approaches are covered. 2. Generalizability is an important problem for current large foundation models.
Weaknesses
1. It's good to have a hyper-parameter table in the appendix. However, I am not sure if the authors have swept hyper-parameters for each method, and make sure each pre-training is complete. I understand the training for each approach is expensive, but the value will be reduced if the pre-training recipe hasn't been fully explored. 2. Given the paper studies the pros and cons of each approach, I would expect there will be a proposed approach that performs better in most cases or a guideline that can help readers to develop such a method. 3. Now the conclusion of each question scatters in different sections. This is fine but I think having a summary table to compare all approaches and show the findings at the beginning or at the end could help readers to understand the pros and cons of each approach better.
Questions
1. How is the v-supervised trained? Does it use kinetics' labeled data for pre-training? 2. In lines 194 - 198 and Table 1, I didn't see the OoD performance of v-supervised and v-MAE as better than the others if they learn better time-variant information. Most of the time Table 1 shows mixed results in linear probing and fine-tuning (e.g. v-MAE performs better in OoD with FT but worse in Lin.). 3. Maybe this paper can consider citing [1, 2] in lines 219 - 222. 4. In Table 5, why the trend of fine-tuning and linear probing is very different? [1] Lei, Jie, Tamara L. Berg and Mohit Bansal. “Revealing Single Frame Bias for Video-and-Language Learning.” [2] Buch, S., Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei and Juan Carlos Niebles. “Revisiting the “Video” in Video-Language Understanding.” --- **Post-rebuttal** Thank you for the authors' response. I have read it and it addressed my questions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The limitation is adequate.
Summary
The paper proposes a set of benchmarks to assess different robustness properties of video representation learning models, including contrastive, non-contrastive, and generative models. The paper trains multiple of these models using a common training protocol and reports multiple empirical findings on different forms of distribution shift.
Strengths
The paper outlines a very clear training and evaluation protocol and nicely outlines the results. I like the "highlights" section at the end of each addressed question section (up to comments below). The considered datasets and models are extensive. Benchmark and models could become a very valuable resource for further robustness benchmarking on video datasets.
Weaknesses
- One major weakness of the paper seems to be the missing control for the baseline performance of the proposed methods. For instance, in Table 2, several claims are made with respect to the performance of contrastive vs. other models under viewpoint shifts (l. 246 etc). However, in Table 2 it is unclear whether these improvements are originating from the improved OoD, or in fact the difference in InD performance. Given that the authors computed error bars for all results, I propose to equip every statement in the paper with a suitable hypothesis test capable to test the influence of the different factors. - Another major weakness is the missing link to published results. All models in the paper are trained from scratch by the authors, and no numbers are reported by applying existing model checkpoints from the literature and confirming that the trained models reach a comparable performance level. What is the rationale for not verifying model performance e.g. against the released best models trained on image datasets? Other weaknesses: - In the training protocol, parameters like the batch size were kept constant across methods. However, it is known that methods like SimCLR depend on availability of large batch sizes, while models like MoCo have mechanisms build in to more effectively leverage data in small batchsize training setups. Hence, I disagree with the author's statement that the considered setting is "fair". Happy to discuss the rationale for these choices (vs. for example using the best available model configurations). Minimally, it would be good to discuss this point more in the paper, e.g. in the limitation section. - For a purely empirical paper, I would recommend to back up the fairly general statements at the end of each section with statistical tests. - The plots are unreadable without heavily zooming in, e.g. Figure 4 and Figure 6. Figure 5 is borderline.
Questions
- If I read the methods correctly, you trained all models from scratch on the respective datasets (Kinetics400 and -700). Did the video models you trained readily outperform the best "static image" models available for the respective methods? - Will you open source the model checkpoints and source code for all models you trained in this study? The paper only makes a statement regarding the code. - The results in Table 2, Viewpoint (ego) seem to be very bad (around 11%). Did you investigate possible causes for this performance drop? How do you justify to infer conclusions from this part of the table (e.g. in l. 246 etc) given the bad InD performance to begin with?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Yes, limitations have been addressed (up to the additional comments I made above). However, given the amount of datasets used in the paper, I am missing license and copyright statements for these datasets, e.g. in the appendix, that go beyond the citations provided in section 5 in the main paper. This could e.g. be included into Appendix C. If the benchmark is intended for later release as outlined, I think it would be very useful to have such an overview directly in the paper for future reference by users of the benchmark.
Summary
This works studies various video SSL methods under distribution shifts. 6 different video SSL methods (SiMCLR, MoCO, BYOL, SimSiam, DINO, MAE) are trained on Kinetics-400 under the same experiment settings (e.g. ViT-B, fixed number of epochs, etc) and then evaluated on various distribution shifts (e.g. viewpoint shift, context shift) and report the findings (e.g. contrastive methods more robust to viewpoint shifts, while MAE is a stronger temporal learner).
Strengths
1. Writing and presentation is clear. 2. Same experimental and test bed setup for all the baselines. The test bed and evaluation code could be useful to the community. 3. Useful empirical observations characterizing the strengths and weakness of prominent approaches.
Weaknesses
No experiments or analyses on model scaling or different architectures. Some findings are pretty much expected (e.g. contrastive methods not being good temporal learners happens by design), but it's nice have an easy test bed to quantify them. Some of the settings controlled for can be strong caveats, e.g. fixing the number of epochs can bias evaluations towards methods that converge faster. It is unclear how much of the results hold when models used their respective optimal settings - which is an important setting for evaluations. Some of the differences could stem not from core differences in pretext task, but in use of a teacher. e.g. MAE can also use an EMA teacher and the representations would converge much faster.
Questions
Could the authors please discuss some of the limitations and caveats to their findings (e.g. some are listed above)?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes.
Thanks for your response. I've updated my score to reflect the additional experiments and discussion. I should add that some of these findings are not really surprising, e.g. v-MAE's poor performance in linear evaluations is reflective of the mis-match between the pretext task and linear eval, while this is less so for contrastive learning. So my view is that an easy test bed to quantify these is the more valuable contribution here and hope the authors release the code and make it easy to use - which will also be strongly impactful to the paper.
Thanks again for all the valuable feedback, going over our experiments and discussions, and updating your score accordingly! We will release all the code and all model checkpoints and will strive to make them an easy test bed for understanding the robustness of VSSL methods.
Reviewer E4vF
The author has posted their rebuttal, but you have not yet posted your response. Please post your thoughts after reading the rebuttal and other reviews as soon as possible. All reviewers are requested to post this after-rebuttal-response.
Reviewer jxy6
The author has posted their rebuttal, but you have not yet posted your response. Please post your thoughts after reading the rebuttal and other reviews as soon as possible. All reviewers are requested to post this after-rebuttal-response.
Reviewer D3xd
The author has posted their rebuttal, but you have not yet posted your response. Please post your thoughts after reading the rebuttal and other reviews as soon as possible. All reviewers are requested to post this after-rebuttal-response.
Re: Rebuttal
Thank you for the comprehensive rebuttal. I especially appreciate the efforts to perform experiments comparing image and video models, as well as the statistical analysis performed. I wanted to follow up on a few points: - **Statistical analysis**, I have follow up questions on Table R4: - Could you provide the details on how you compute the correlation coefficients, i.e. for which table in the paper? It would help to read a few more details on the analysis (e.g., like you would also put it into the methods/supplement of the paper), like which model was employed, which test was performed, etc. In general, thanks for going in this direction, I think this will greatly strenghten the claims in the paper. The way how I read it right now is that you find a correlation between IID and OOD performance across all methods, which was one of my concerns --- I would like to better understand the effect of model (e.g. contrastive vs. non-contrastive) vs. the confounder that the IID performances between the models vary. (maybe I am also not fully understanding your analysis, hence, please expand) - To make it more concrete: Could you again make a very clear example how e.g. the statement "contrastive methods (v-SimCLR and v-MOCO) are robust to viewpoint shifts as they consistently achieve better performance in all three setups in both linear and finetuning schemes." (l. 223 in the paper) maps to Table 2 and is supported by your statistical analysis? There are a few more of these kind of strong statements in the paper that are not fully clear to me yet, I might follow up with a few additional examples. - **Source code release**: The additional info to open source model checkpoints and evaluation code are very useful. Regarding the code, is this already in a state ready to release? If so, I think there is an opportunity to send the AC a link to the codebase which they can pass along to me. I would be interested to have a look, if this is well setup it would strengthen the contribution. - **Figure R1**: The color somehow needs to map to a metric/number. I think this overview is great and could e.g. go into the supplement, but you should work on making this quantitative (vs. "low" to "high") I apologize for the late response just before the weekend --- please feel free to post multiple replies as they get ready in case this speeds up the further discussion.
Re: Statistical Analysis
To make my previous comment more clear, here is a quick re-analysis of Table 2. The simplest control for IID performance is to consider the statistic "OOD - IID" (lower = more robust), for which we get with linear evaluation (bold = lowest degradation): | | method | method_type | ego | surv | view+act | |---:|:----------|:----------------|------:|-------:|-----------:| | 0 | v-MOCO | contrastive | **-1.7** | -15.2 | -26.8 | | 1 | v-SimCLR | contrastive | -2 | -13 | -25.4 | | 2 | v-BYOL | non-contrastive | -2.4 | -15.1 | -28.3 | | 3 | v-DINO | non-contrastive | -2.4 | -13 | -27.6 | | 4 | v-MAE | non-contrastive | -2.8 | **-8.5** | **-20.2** | | 5 | v-SimSiam | non-contrastive | -2.5 | -11 | -25.5 | and for fine-tuning, we get: | | method | method_type | ego | surv | view+act | |---:|:----------|:----------------|------:|-------:|-----------:| | 0 | v-MOCO | contrastive | **-3.3** | -21 | -17.2 | | 1 | v-SimCLR | contrastive | -4 | **-19.5** | -15.7 | | 2 | v-BYOL | non-contrastive | -3.6 | -22.2 | -22 | | 3 | v-DINO | non-contrastive | -3.7 | -20.9 | -27.2 | | 4 | v-MAE | non-contrastive | -7.2 | **-19.5** | **-15.1** | | 5 | v-SimSiam | non-contrastive | -3.9 | -19.6 | -22.4 | I did not run a statistical test here as I only had access to the mean values and standard deviations for either ood or iid. It would be insightful if the authors could run a full statistical analysis or share all results that were used to compute the standard deviation. But even without an analysis, I question that the statement "contrastive methods (v-SimCLR and v-MOCO) are robust to viewpoint shifts as they consistently achieve better performance in all three setups in both linear and finetuning schemes." is conclusivly backed by the experimental data. Another example is the statement "Overall, the results of zero-shot recognition presented in Table 4 show that no single method dominates in all three benchmarks of zero-shot recognition.", where again absolute performance values are compared in Table 4. However, the performance difference in InD performance on K400 varies as much as 58.4 - 35.9 = 22.5% points on linear eval, and 69.8 - 64.3 = 5.5% points for finetuning, which makes the statement almost trivial (the difference in in domain performance seems to be so drastic that it is impossible to make an informed statement about OOD performance). I hope these examples make my concerns even clearer. Finally, to reference the author rebuttal, > To ensure our observations are statistically significant, we run these experiments 3 times with different seeds and report the average and standard deviation. **This is not sufficient to claim statistical significance.** What is missing in multiple sections in the paper is the link from numbers in tables and plots to (minimally) a summary statistic that is ideally free of confounding information (e.g., difference between ood and iid performance in the example above, and other statistics in other cases). And/or ideally, there needs to be a suitable test to make these statements really convincing. The alternative is: (1) Remove the word "significant" from the abstract and text when no test is performed (2) Make it clearer on how you back up each statement in the "Hightlights" with numbers, that goes beyond comparing the OOD performances on the different datasets (as these are confounded by choices like fine-tuning vs. linear tuning, the IID performance on the source data, and so forth). I am insisting on this point from my original review as I fear that in case the paper gets published as-is, a lot of statements will be referenced in the literature that are not backed by experimental data due to this missing link between "raw" numbers and the respective statements in the text. Happy to discuss further.
We would like to sincerely thank you for all your time and efforts in engaging with our paper and rebuttal, providing constructive suggestions, and allowing us to respond again. - We have created an anonymized repository that contains the codes for self-supervised pretraining and finetuning. The repo also contains a link to an anonymized google drive that contains all the necessary checkpoints for the reproducibility of the results. The link to the repo is now provided to the AC, so that they’d share it with you. Please kindly refer to the README.md files in the main folder along with the two main subfolders (`vssl-train` and `vssl-eval`) for details on how to navigate the codebase. We appreciate your patience while we were putting the cleanup touches on the code. - The repo also contains the code (`table_r4.ipynb`) used in creating Table R4, per your request. - The repo also contains an updated Figure R1 (`figure_r1.pdf`) where we have added all the numbers in the different columns, per your suggestion. - We are currently working on providing all the statistical tests in a unified and comprehensible manner, along with a detailed response to your comments about the claims, so that we can share them with you. Thank you.
Re: Rebuttal & Statistical analysis
We would like to thank you again for engaging with our work and your feedback on our paper. We have now provided an anonymous google sheet document to the AC to be passed along to you. We have also added a link to this gsheet in the README file of the anonymous repository. The provided gsheet contains all the statistical analyses for every major claim in the paper. Following we provide a point-by-point response to your questions. > Comparing OOD performance after compensating InD performance. As you suggested, we use OoD - m * InD as the basis for compensating for InD performance, where $m \in [0, 1]$. Further, to consider statistical significance, we modified the following equation, and `accept` **method1 is more robust than method2** if **Adjusted $\Delta_{ood}$>0**: Adjusted $\Delta_{ood}$ = (method1_ood - method2_ood) - m$\times$(method1_ind - method2_ind) - 1.96 $\times$ method1_ood_stderr - The first term **(method1_ood - method2_ood)** measures the difference between the OOD performance of two methods. - The second term **m $\times$ (method1_ind - method2_ind)** compensates for InD performance, where *m* refers to the slope of a linear fit between InD acc and OoD acc. - The third term **1.96 $\times$ method1_ood_stderr** considers the standard error to ensure statistical significance at 95% confidence. The first two terms of the formula correspond to the plots in **Figure S12** in the **Appendix**. In **Figure S12**, the method that lies above the linear fit with the highest margin is considered to be the most robust to OoD shift. While the statistical confidence is eliminated from **Figure S12** to prevent creating a clutter, we now present those statistical analyses in the shared gsheet. The shared google sheet contains the results using this equation for the claims in the paper. Overall, the sheet confirms that all our claims hold. While in the paper, we had originally given a higher-level overview of the claims, as per your suggestion, we will add further details to each claim to make them more clear as per our discussions below. > Could you again make a very clear example how e.g. the statement "contrastive methods (v-SimCLR and v-MOCO) are robust to viewpoint shifts as they consistently achieve better performance in all three setups in both linear and finetuning schemes." (l. 223 in the paper) maps to Table 2 and is supported by your statistical analysis? Based on our statistical analyses, v-SimCLR is more robust to all three viewpoint shifts and v-MOCO is robust to egocentric and top-down viewpoint shifts. Additionally, our claim that v-SimCLR and v-MOCO learn better viewpoint invariance is also supported by statistical analysis. The statistical results are presented in `tab: viewpoint`. This is also corroborated in **Figure S12 (c, d, and e)** in the **Appendix**. As can be seen, v-SimCLR consistently clears a positive margin from the linear fit suggesting its relative robustness to viewpoint shifts. To add more clarity and detail, we will add further details to our claim in L. 223 to read as ``` v-SimCLR exhibits robustness to all three viewpoint shifts. v-MOCO shows robustness to two out of three viewpoint shift setups, i.e., egocentric (viewpoint (ego.)) and top-down (View+Act (t-down+syn.)). ``` > "Overall, the results of zero-shot recognition presented in Table 4 show that no single method dominates in all three benchmarks of zero-shot recognition.", where again absolute performance values are compared in Table 4. However, high-performance difference in InD makes the statement almost trivial. We test robustness between all possible pairs after compensating for InD performance and the results confirm our claim that there is no single method that exhibits superiority in all three setups of zero-shot recognition either in linear or finetuned setup. The statistical results are presented in the google sheet `tab: zeroshot`. > It would be insightful if the authors could run a full statistical analysis or share all results that were used to compute the standard deviation. We provide our statistical tests to support all our major claims in the paper in a separate comment. > (1) Remove the word ''significant'' from the abstract and text when no test is performed. We will remove the word "significant'' from the paper where appropriate, and replace with the actual numbers for better clarity. Kindly note that in some cases, e.g., the abstract, the term "significant" is used in the sentence: ''Video self-supervised learning (VSSL) has made significant progress in recent years''. > (2) Make it clearer on how you back up each statement in the "Hightlights" with numbers Thanks for this suggestion! We will summarize the gist of the google sheet (also discussed in our separate comment) with all the statistical tests in a new table and add it to the supplementary material of the paper. We will also provide the detailed results of the tests with the code.
Statistical analysis of all major claims in the paper.
Below, we provide our statistical tests to support all our major claims in the paper. - L.223: *The results on viewpoint shifts, presented in Table 2, reveal that contrastive methods (v-SimCLR and v-MOCO) are robust to viewpoint shifts as they consistently achieve better performance in all three setups in both linear and finetuning schemes.* Our statistical test confirms that v-SimCLR is robust to all three viewpoint shifts. Additionally, v-MOCO shows robustness to two out of three viewpoint shift setups, i.e., egocentric and top-down viewpoint shifts. Please see the results in `tab: viewpoint` and accordingly we will add further details to our claim: ``` The results on viewpoint shifts, presented in Table 2, reveal that contrastive method v-SimCLR is robust to viewpoint shifts as they consistently achieve better performance in all three setups in both linear and finetuning schemes. Additionally, v-MOCO exhibits robustness to egocentric and top-down viewpoint shifts. ``` - L.259: *(b) Contrastive methods (v-SimCLR, v-MOCO) exhibit robustness to viewpoint shifts.* Our statistical test confirms that v-SimCLR is robust to all three viewpoint shifts. Additionally, v-MOCO shows robustness to two out of three viewpoint shift setups, i.e., egocentric and top-down viewpoint shifts. Please see the results in `tab: viewpoint` and accordingly we will add further details to our claim: ``` v-SimCLR exhibits robustness to all three viewpoint shifts. v-MOCO shows robustness to two out of three viewpoint shift setups, i.e., egocentric (viewpoint (ego.)) and top-down (View+Act (t-down+syn.)). ``` - L.259: *(a): Video models generally struggle in out-of-context generalization, while v-Supervised and v-MAE exhibit more robustness as they are strong temporal learners.* Our statistical test confirms the robustness of v-MAE and v-Supervised in Context (10 class) in both linear and finetune setups. Additionally, In Context (50 class) setup, v-Supervised is robust in linear evaluation and v-MAE is robust when finetuned. We will clarify this in the paper, please see the detailed results in `tab: context`, and accordingly we will add further details to our claim: ``` (a): Video models generally struggle in out-of-context generalization, whereas v-Supervised and v-MAE exhibit higher robustness to **Context (10 class)** in both linear and finetuned setups. This could be explained by the fact that they are strong temporal learners. ``` - L.244: ``` While single-stream networks learn better temporal dynamics, Siamese frameworks learn better viewpoint invariance. ``` Our experiments support this claim. Please see the results in `tab: single_and_siamese`. - L.229-233: ``` It is worth noting that while v-Supervised exhibits comparable performance to VSSL methods in egocentric and surveillance camera viewpoint shifts, its performance decreases significantly when multiple shifts are applied concurrently, as evident in the case of synthetic top-down viewpoint shift. ``` Our experiments support this claim. Please see the results in `tab: multiple_shift_supervised`. - L.327: ``` (a) Contrastive methods demonstrate superior performance in open-set recognition when finetuned. ``` Our experiments support this claim. Please see the results in `tab: openset`. - L.319: ``` However, the pretrained encoders, v-SimSiam, v-DINO, and v-Supervised, which are considered weaker in closed-set, perform better in open-set recognition. ``` Our experiments support this claim. Please see the results in `tab: openset`. - L.273 ``` For example, as shown in Figure 4 (a), while finetuning is significantly beneficial under source shifts of UCF to HMDB, it interestingly hurts the performance when this shift is reversed (HMDB to UCF). ``` Our experiments support this claim. Please see the results in `tab: source_shift`.
Decision
Accept (spotlight)