Opening the Vocabulary of Egocentric Actions

Summary

In this paper, a new method is proposed for open vocabulary object understanding within egocentric videos for the task of action recognition. The method uses CLIP features to generalise to new objects which haven't been seen during training by focusing on objects using a HoI model, combining nouns via object mixing and creating learned prompts. The method is tested on three datasets, EPIC-Kitchens-100, Assembly 101, and Something-Else. The proposed method outperforms other methods across all datasets on both seen and unseen nouns.

Strengths

Expanding action recognition beyond a closed vocabulary of objects is good to see and the proposed method makes sense to use the CLIP aligned image text space. The results perform well on all three datasets and most aspects of the method are ablated.

Weaknesses

Figure 2 could be improved such that it shows the examples of the anchor, positive, and mixed objects (as well as the negatives). Even with labels, currently, the frames in the figure are quite small and it's difficult to understand what is going on. Section 4.2: What are the values of k1 and k2 and how were these chosen? How sensitive is the model to these values? In a similar vein, has an investigation between the learnt prompts and what these represent from the learnt embedding space been found? Line 210: It isn't clear to me what it means by fine-tuning the object embeddings but the CLIP text encoder is kept frozen? Section 5 the value of alpha and lambda haven't been explored within the paper, I assume these values set to 0.5 have been empirically chosen? If so, how sensitive is the method to these different hyperparameters?

Questions

Here are the list of questions that I would like to see answered, for more context of each question please see the weaknesses section. 1. More information regarding k1 and k2, what are the values and how were they chosen? 2. What do the learnt prompts look like? Do they have a relation to any words that are within the context of the datasets? How much do the learned prompts change per dataset? 3. Could line 210 be clarified as to how the object embeddings are tuned? 4. How were the values of alpha and lambda chosen and how sensitive is the model to these?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the authors have addressed some limitations in the method such as the assumption that every verb/object pair makes a viable action and that CLIP has a strong object bias.

Summary

The paper tackles open vocabularly action recognition where novel objects may be seen at test time and will be composed with a known verb. Note that the paper does not investigate the open vocabularly of verbs. To address this problem the paper proposes 'object agnostic pretraining' to become less reliant on the objects seen in training as well as 'active object prompting' a method to prompt CLIP for the specific problem of recognizing unseen objects in combination with seen verbs. This approach is tested on compositional action recognition with SomethingElse and open vocabularly action recognition on EPIC-Kitchens-100 and Assembly101.

Strengths

- The set up of open vocabularly benchmarks is thoughtful. For instance, keeping objects from imagenet and kinetics in the base split to avoid leak from pretraining to open vocabularly in test and using tail clases as the open vocabularly ones to make the setting more realistic ensures it will be a useful baseline for future work. *** - Reporting results on SomethingElse allows comparison to prior work on the related task of compositional action recognition. This is useful as there is no prior work on open vocabularly action recognition to compare to. *** - The ablations demonstrate the effectiveness of the majority of proposed components

Weaknesses

- Only focusing on the open vocabulary of objects, not verbs, limits the usefulness of the baseline to future works. I appreciate the paper is focused on unseen objects however setting up the baseline in a way that allowed future works to study unseen verbs would have been valuable *** - The approach is tailored to the specific scenario of egocentric vision and recognizing unseen objects in combination with seen verbs. Testing on SomethingElse, which is not egocentric, did partially alleviate this concern however, the method is still quite specific to the target task. *** - The baselines in Table 2 are mostly standard action recognition backbones rather than works for compositional action recognition. - [A] and [B] provide results on SomethingElse and are more recent than [35] so it is unclear why these results weren't compared to. *** - Limited open vocabulary object recognition baselines in Table 3 - It is unclear why [63] and [18] aren't compared to as the introduction sets these up as some of the most relevant works. - CLIP performs quite well on novel objects already so makes sense that an existing open vocabulary object detection method built on CLIP would perform even better. *** - Table 6 ablation, some components ineffective - From Table 6 the benefit of temporal gradients and L_in seem negligible: +0.2% top-1 accuracy - Furthermore, it would be useful to assess the impact of these two components separately. For instance, can the object mixing be added without L_in to assess whether L_in is effective or whether the object mixing is the only effective component in this section - I'm also curious as to whether the success of the object mixing is mainly due to the imbalance of objects in training. How would class-balancing the negative object selected compare to the object mixing. *** [A] Is an Object-Centric Video Representation Beneficial for Transfer? ACCV 2022. [B] Revisiting spatio-temporal layouts for compositional action recognition. BMVC 2021.

Questions

Why weren't prior works [A] and [B] on compositional action recognition compared to in Table 1? Why aren't additional open vocabularly object recognition methods compared to in Table 2? E.g. [63] and [18] Does L_in have any impact to the final results? Is object mixing better than class-balancing the object in the (verb) in-class positives? After reading the rebuttal some of my concerns have been alleviated, namely the comparison with compositional action recognition, the additional ablation, the promise of an open-vocabulary verb baseline in the final version and the clarification on the open vocabulary object detector comparisons. As reading the other reviews has not raised any additional concerns I have upgraded my rating to borderline accept. I have not raised it higher as I am not convinced by the arguments for not considering an open vocabulary of verbs. I do think it is an interesting problem for the paper to focus on an open vocabulary of objects in actions however, the title, abstract and introduction should be updated to clearly indicate this is the focus of the paper if it is accepted. I was also not convinced by the rebuttal response on the proposed object mixing vs. class balancing. An experiment would demonstrate the point more clearly.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are addressed well in a separate section of the main paper.

Summary

The paper proposes a novel open vocabulary action recognition task, and this task aims to generalize the ability of the model to an open vocabulary of actions with seen and novel objects. To do this, the authors propose a new method, which mainly contains two components: Object Agnostic Pretraining (OAP) and Active Object Prompting (AOP). The authors also create open vocabulary benchmarks on the EPIC-KITCHENS-100 and Assembly101 datasets, and the results on these benchmarks demonstrate the effectiveness of this method.

Strengths

1) The paper proposes a novel open vocabulary action recognition task, which is important in practice. 2) The method presented in the paper is simple and straightforward. 3) The paper presents sufficient experiments. The results of comprehensive experiments enhance the validity and reliability of the proposed approach.

Weaknesses

1) Motivation is unclear, and many modules are designed without proper motivation, e.g., why conduct object agnostic pretraining, why interested in recognizing active objects. 2) The authors claim that we are the first to explore end-to-end contrastive learning on the video backbone of egocentric videos, but contrastive learning is already widely employed in the fields of video and image. 3) Since this paper focuses on open vocabulary action recognition, why can't the verb category be open vocabulary as well? 4) The proposed methods lack novelty. Firstly, mixup operations are widely used in image augmentation [1]. Secondly, the idea of Parameter-efficient training for Active Object Prompting appears to resemble a modified version of COCOOP [2]. Thirdly, Active Object Prompting (AOP) is not very novel, just a simple combination of the HOI model and CLIP. CLIP has been widely used in open-vocabulary tasks. 5) The description of the paper's method is insufficiently detailed, making it difficult to reproduce. For example, in Line 205, the author mentions ΦL as a projection network without further interpretation or elaboration. [1] Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation, CVPR2022 [2] Conditional Prompt Learning for Vision-Language Models, CVPR 2022

Questions

It seems that only the objects in this task can be open-vocabulary. Can the verbs in the task be also open-vocabulary?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have discussed limitations in chapter 5.6, such as, bias in CLIP.

Summary

Existing approaches for egocentric action recognition considers the problem as recognizing actions from a closed set. The manuscript instead proposes to consider this as an open set recognition problem where some objects are unseen during training. Towards, a novel benchmark for evaluating open set egocentric action recognition is proposed using two egocentric action recognition benchmarks. The manuscript also proposes an approach for open set egocentric action recognition. The key idea behind the approach is to decouple the verb representation from interacting objects followed by using a prompt based object encoder. The proposed approach shows strong improvements in the proposed benchmark when compared against existing closed set action recognition baselines.

Strengths

The open set egocentric recognition problem addressed in the manuscript is novel, non trivial and highly relevant to the research community. The development of the two benchmark datasets will be highly beneficial for future research. The paper is written in a clear and concise manner making it easy to understand the motivations and the proposed approach. An extensive set of ablations are also presented in the manuscript making it clear the impact of the various contributions. Evaluation of the proposed approach on the benchmark and comparison against the existing close set recognition approaches shows its effectiveness in addressing open set egocentric action recognition.

Weaknesses

The main weakness of the manuscript is that the proposed approach is compared only against simple baselines based on standard action recognition approaches. The proposed approach makes use of the CLIP model for object recognition. CLIP is widely known for its performance on zero shot and few shot object recognition. One important baseline would be to use the S3D model for verb recognition (without the proposed OAP training) and the CLIP model for object recognition. The CLIP model can also be used for both verb and object recognition. Without this, it is not clear how effective the proposed approach is for open set egocentric action recognition. Similarly since the novel objects are composed of the tail classes from the respective datasets, using CLIP model would be highly effective since it is good at recognizing few shot and zero shot classes. So the comparisons reported in the paper do not reveal the true impact of the proposed approach. Some of the design choices behind the proposed approach are not motivated well or empirically proven. For example, why use pre and post context prompts? Why not add them as prefix or suffix to the fixed prompt? For AOP, the verb features are added to the context tokens. Why is this preferable over concatenating the verb features to the input embeddings of the text encoder? After the OAP pretraining, the verb encoder is further finetuned for supervised verb recognition. Wouldn’t this result in the model losing its object agnostic verb prediction capabilities? Even though the manuscript presents various ablation studies to validate the effectiveness of various contributions, the experiments are conducted on different datasets. This makes it difficult to analyze the impact of the different contributions of the proposed approach. The reviewer recommends the authors to select a single dataset to present in the ablation analysis.

Questions

Please see the weaknesses section for the main questions regarding the manuscript. The main problem that the proposed approach is trying to address is related to the fact that existing approaches consider that there is only a fixed set of actions (verb-noun pairs that are observed in the training set). Instead the proposed approach is trying to develop an approach for open set recognition (some object classes are not observed during training). However, the AOP based object encoder still functions in a way that the set of possible objects are already known since the object names are required for the fixed prompt. This contradicts the main objective of the paper to develop an approach for open vocabulary egocentric action recognition. A true open set recognition approach requires a generative model instead of the discriminative one used in the proposed approach.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

One of the limitations of the proposed approach is the fact all verb-noun pairs cannot constitute a plausible action. This is discussed in the limitations section.

Summary

Learning representations that generalise across novel objects and actions is an important topic of research that is of great practical utility. This is especially true due to most datasets containing only a few combinations of object-action pairs or triplets, which often leads to model overfitting. A key question is, how to overcome this dataset limitation. The approach tackles this by decoupling the object representation and verb representation by using object word embeddings and verb-agnostic visual representations. - Task: Given objects and actions seen during training, generalize to new actions that are performed with either seen or novel objects. - Approach: Decompose predictions of objects and verbs. This overcomes the verb-object overfitting that might cause lack of generalization to novel verb-object combinations. - Object-agnostic pretraining of a verb encoder - Prompt-based object encoder (CLIP features for a given prompt). This enables an open vocabulary (of objects that CLIP is trained with). - Contributions: - Open vocab. benchmarks on EPIC-KITCHENS-100, Assembly101 - Approach that decomposes object and verb encodings is better than prior approaches at recognizing novel verb-object combinations.

Strengths

1. Decoupling the object representation into a function of (a) object word embedding, and (b) object-agnostic verb embedding is a promising formulation. While the specific details of the verb-agnostic encoder training algorithm and prompting of the text encoder might change and improve, the basic formulation that decouples the two different semantic concepts, makes sense. This leverages CLIP’s pretrained object-text representation that is trained over large datasets containing many variations of objects. 2. CLIP text encoder’s input “context prompt” into a function of (noun CLIP word embedding, object-agnostic verb encoding ) is interesting. The paper mentions that the “pre” and “post” are necessary for the encoding to work. Is this a novel observation or is there prior literature on the topic? 3. In addition to the verb encoder, the paper aims to more accurately represent the differences between fine-grained objects via CLIP’s text encoder. To achieve this, the approach also involves fine-tuning the object embeddings (on seen object classes). The paper demonstrates that this achieves a significant improvement in performance.

Weaknesses

1. There are many small decision choices involved in the successful implementation of the approach. There are different types of augmentation and contrastive sampling. Further, there are different representations for nouns, verbs. The noun object representation is also fine-tuned. It makes the overall approach appear to be the sum of a series of smaller tricks. While it makes the paper a bit harder to read and understand, from a practitioner’s standpoint, this might indeed contribute to new knowledge / understanding of what’s possible. Further, it’s nice that Table 5 evaluates the effects of these different decision choices. 2. The visual feature extraction heavily relies on a HOI detector to obtain the “active object”. This only works on hand-based interactions. This is a fairly narrow set of actions compared to overall possible object-verb interactions. However, one can imagine that this can also be extended to object detectors and relationship-detectors in non-ego-centric RGB videos. Yet again, the approach would have to rely on the high performance of these detectors. A concrete question along these lines is: how good is the HOI detector? What’s the UB performance of the proposed approach assuming perfect HOI detection? 3. There’s an assumption that non-static ⇒ active objects ⇒ objects involved in action. And vice-versa, i.e., static objects are *not* of interest. The masking (or not) of objects is based on this assumption? Is this a valid assumption? I raise this point because the validity of this assumption requires some discussion, and the implementation of this assumption (masking), is also understandably imperfect, and results in noisy augmentations. 4. To produce a training set containing diverse objects for the same verb, the approach does data augmentation by masking out certain objects, and alpha-blending other objects in. This results in unrealistic, noisy images. The MIL-NCE-like loss appears to counter this effectively. However, is it possible to perform better augmentation like [54, 55]? How much will performance improve with better augmentation? 5. Limitations: The approach is only as “open” as the vocabulary of objects that CLIP’s feature space encodes. It is also heavily dependent on the performance of the HOI detector.

Questions

**Clarity / Details** 1. Fig. 4 and text don’t go well together, or I’m confused. f_\theta = f_v and g_theta (f_theta(.)) = f_v^L ? 2. In Fig. 4, the verb encoder is shown to be frozen. This confuses me. My understanding was that the training procedure in Sec. 4.1 indeed updates the parameters of the verb encoder. 3. The input to the verb encoder is the whole video. Is it the same for the HOI detector? Then how are image crops sampled from this video (to pass through CLIP Image Encoder)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, they discuss limitations.

Response to authors

I thank the authors for the rebuttal which addressed most of my concerns. As a result, I change my rating to borderline accept.

Response to Rebuttal

Thank you for responding to my initial review. I have no further questions and will enter the review discussion more positive than my original rating.