Language-based Action Concept Spaces Improve Video Self-Supervised Learning

Summary

This paper proposes to transfer CLIP to the video domain for self-supervised learning. Textual features of video categories are used to obtain text classifiers and fixed during pre-training to obtain transferable information. Multiple complementary loss functions are designed for pre-training. Experimental results on three datasets demonstrate the effectiveness of the method.

Strengths

Pros: 1. A new paradigm is proposed to transfer the knowledge of CLIP to the video domain for self-supervised learning. 2. Impressive performance has been achieved on multiple datasets. 3. The results of the ablation experiments demonstrate the effectiveness of the method.

Weaknesses

Cons: 1. The authors claim that they did not use labeled or captioned videos in the paper. But they used the labels of the Kinetics-400, UCF101, and HMBD51 data sets during training. Does this violate the self-supervised setting? 2. Unfair comparison. The proposed LSS leverages the pretrained CLIP for weight initialization while existing methods such as SVT，VideoMAE do not. 3. LSS is pre-trained on the Kinetics dataset. Why did the label of HMBD51 and UCF101 be added to the pre-training? Does this break the downstream transfer setup? Since the target label has been leaked in the pre-training. 4. The fully-finetuned experiments and SSv2 results are missing.

Questions

Please refer to Weaknesses for more details.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations have been listed.

Summary

The paper proposes a new self-supervised approach to adapt image-level CLIP features to video. The key idea is to use a teacher-student self-supervised learning framework, and distill the knowledge in the action concept space, derived from text action concepts using the CLIP's text encoder. The resulting framework produces SoTA self-supervised video features.

Strengths

- The work tackles an important problem - adapting the CLIP visual encoder to video. Doing it without supervision is nice a bonus here. - The approach is simple and sound. - The evaluation is extensive and the results are strong.

Weaknesses

- Since the set of concept vectors is taken from Kinetics-400 UCF-101 and HMDB-51, where the evaluation is performed, it is a little unfair to call the method completely unsupervised. While it is true that no video labels are used during training, the training process does use the knowledge of the actions vocabulary, which would aid the evaluation. To avoid this shortcoming, the authors may consider constructing the action concepts in a dataset-agnostic way. - I could not find the ablation of the uniform distribution prior regularization. It is important to understand the influence of that prior on the training. Could the authors please include this? - The writing could be significantly improved.

Questions

in line 79, the authors probably meant “self-supervised learning” instead of “semi-supervised”? Semi-supervised assumes you have access to some labeled data.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

It is not very clear whether knowing the text action concepts is important for the method to work well. This may be a potential limitation.

Summary

The paper introduces a language-tied self-supervised learning approach to adapt an image CLIP model to the video domain. The method employs two video-specific self-supervised learning objectives: concept distillation and concept alignment, for training the model. The authors showcase that the proposed method achieves good zero-shot and linear probing performance on three action recognition benchmarks.

Strengths

1. The motivation behind using language for video self-supervised learning is good as it addresses the existing challenges associated with video datasets and holds the potential to offer effective solutions. 2. The paper presents good zero-shot performance 3. The paper is well written and the overall message is well understood.

Weaknesses

1. While this paper refers to itself as a self-supervised method, it relies significantly on labeled information. For instance, the text classifier utilizes language embeddings derived from the dataset categories. Additionally, the construction of action concept spaces and category concept spaces takes into account the awareness of dataset categories. This information leak introduces a potential unfairness in comparing the proposed self-supervised learning method with other SSL methods. 2. Consider adding experiments that involve removing the awareness of dataset-level labels to align with other SSL methods. 3. It is not fair to directly compare the proposed method, which utilizes pre-trained CLIP weights, with other SSL methods that are trained from scratch. This distinction should be discussed, and the attribute of pre-training should be added to Table 1.

Questions

One of the main issues with this paper is the misleading setting employed by LSS. As mentioned in the weaknesses section, LSS is not a purely self-supervised method as it requires some dataset-level labels and also uses CLIP pre-trained weights. It is important to differentiate this setting from traditional SSL settings and explain the practical utility of the proposed LSS setting in applications.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have addressed the limitations.

Summary

The paper presents a method to adapt a vision-language model (CLIP) to represent videos. The method extends the image encoder of CLIP to a video encoder via factored space-time attention. The paper introduces a self-distillation-based objective to adapt the video encoder to train on unlabelled videos (no video captions are required). This self-distillation is performed in a so-called "action concept space" which results from projecting visual embeddings into the space spanned by 0-shot action classifiers or text embeddings of action descriptions. The approach is evaluated on Kinetics 400, UCF101, and HMBD in 0-shot and linear action classification experiments.

Strengths

- The adaptation approach does not require additional video-level captions (only knowledge of action categories in the training data is required) - The method performs well in "zero-shot" and linear probing experiments in the considered action datasets - The use of fixed projections obtained through 0-shot classifiers for distillation is interesting and seems effective when the classes observed in transfer are known - The paper is overall well written and the method well presented

Weaknesses

- The experiments are limited to 0-shot and linear classification of the same action classes also observed and used in the adaption training. It is unclear if the method can generalize to new actions (ones not used to define the action concept spaces) - The experiments also lack any text-to-video retrieval benchmarks. These would be essential to demonstrate the claim that the method "preserves and improves the strengths of CLIP ... for video operation" L49 - The comparisons to most methods in Table 1 are unfair since they are all fully self-supervised (no captions used in pre-training), and most are trained exclusively on videos (much less effective training data). Also, it is unclear why many methods are included without any performance numbers. Instead, it would be better to compare to other CLIP-based methods as in Tab 2 in this benchmark as well - The difference in Tab 2 and 3 suggests the method is very sensitive to the set of actions used to define the projection space

Questions

I would appreciate it if the authors could address my concerns listed in the weaknesses above. Most importantly, I would like to know: - How does the model perform for text-to-video retrieval? - How does the model generalize to actions not used during pre-training? For example, what if Tab 5 only consists of K400 classes? Additionally, I'm wondering: - What is the importance of w_s (Eq 5)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Some of the limitations I see (see weaknesses) have not been addressed in the paper.

Summary

The paper proposes a novel language-based self-supervised learning framework (LSS) for video representation learning. It extends the self-distillation based SSL approaches like BYOL and SimSiam by replacing the randomly initialized project network by the text classifier defined by language embeddings extracted from the image CLIP text encoder. With two novel self-supervised learning objectives, the pretrained video model retains and improves transferability and generality of image CLIP representations better in comparison to existing video SSL methods. LSS achieves state-of-the-art results under linear probing settings and competitive zero-shot transfer performances on HMDB-51 and UCF-101.

Strengths

+ The paper is overall well written and easy to follow. + The idea of replacing the project network with the text classifier defined by image CLIP embeddings is really interesting and makes sense. The video representation learning suffers from relatively expensive and noisy annotations; we can easily distill knowledges obtained from abundant and diverse image-based datasets using LSS. + The experimental results show the high transferability and generality of the proposed method, LSS.

Weaknesses

- LSS uses action categories of Kinetics-400, UCF-101 and HMDB-51 for defining the action category and description concept spaces. However, if those action concept spaces can be used only for pretraining on one of the three datasets used for concept space construction, this means that we can only use manually annotated action recognition datasets for pretraining. This limits the data scalability of LSS. The authors should provide experimental results with pretraining on more general, non-labeled video data using the same action concept spaces. Otherwise, for using abundant web videos without labeling, the authors should come up with the action space construction method without manually annotated action categories. - Please fix the typos; e.g., the l2 norm, not the squared l2 norm, should be used in Eq. (1) and (3) for normalization.

Questions

The key idea of the paper, replacing the project network with the text classifier really makes sense and is very interesting. I will lean towards acceptance if the authors address my concerns in the rebuttal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and potential societal impact of the work.

Response

I thank the authors for their response. I think point 1 makes the paper look stronger and point 2 gives important intuition about the problem. I advise to include them in the final paper. Otherwise, the rebuttal answers all my questions. I keep my score as week accept.