AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

We appreciate your detailed and constructive feedback! >***Q1: The novelty is limited*** We respectfully disagree. We view the simplicity of our design and the effectiveness of our approach as desirable properties and contributions. In order to arrive at our simple and effective design, three fundamental research questions need to be addressed: + Does goal conditioning benefit LTA? If so, can we utilize LLM to infer the goals without supervision? + Do LLMs capture prior knowledge to model temporal dynamics? + Can we condense LLM’s prior knowledge and temporal modeling ability to a compact neural network for efficient inference? Our paper strives to answer these questions and design corresponding experiments accordingly. Despite its simplicity, to the best of our knowledge we are the first to demonstrate the effectiveness of the action-based video representation for long-term action anticipation, and how to properly leverage and condense the prior knowledge encoded by LLMs for the task. >***Q2: The improvement of table 1 is marginal*** We would like to highlight that Table 1 only studies one of our contributions, namely goal-conditioned LTA, where the goals are inferred by LLM via few-shot in-context learning. We show that the goals inferred by LLMs (Llama-2 and GPT-3.5) lead to consistent improvements across all three benchmarks, especially for the rare actions in EK-55 and EGTEA. We also show that the models based on inferred goals even approach the performance when oracle goals are used. When the whole system is evaluated and compared against the previous state-of-the-art methods, we show in Table 6 and 7 that AntGPT outperforms all previous methods by large margin. We also report the variances of AntGPT’s performance to further reinforce that the gains are statistically significant. >***Q3: Rely on classification results*** The reviewer is correct that as expected, action recognition accuracy would have an impact on the LTA performance. We recognized this and performed an ablation study when oracle (ground truth) actions instead of recognized actions are used by the LLM. This oracle model achieves 0.679 verb ED and 0.617 noun ED on the validation set of Ego4D v1. This result shows that: (1) The LTA performance is indeed reliant on the action recognition accuracy; (2) AntGPT with recognized actions achieves competitive performance when compared to the oracle actions; (3) The LTA task remains challenging even when oracle actions are used, we attribute this to the challenge of modeling temporal dynamics. In addition, Table A2 (Appendix) studies on the impact of different visual backbones (hence action recognition performance). We observe that as expected, better recognition models do benefit future action prediction but this influence is moderate. Besides, the final results also show that our proposed method achieves strong performance on all 3 benchmarks even with imperfect recognition results.

Thank you for your constructive feedback! We address your concerns in detail below and revise our paper accordingly: >***Q1: Few-shot learning*** We would like to clarify the purpose of our few-shot learning experiments. Since we use LLMs as the temporal model, the experiment aims to understand if in-context learning is also more desirable than direct gradient updates (as have been shown for language-based tasks with LLMs), when one only has a few examples to “supervise” the LTA task. The few-shot learning scenario arises when one has access to a reasonable action recognition model (e.g. those trained on Kinetics, an action recognition dataset of temporally trimmed 10-second video clips) but only has a limited amount of examples to demonstrate the temporal dynamics. We acknowledge that the Ego4D dataset is not an ideal benchmark for this scenario, since the actions are already densely annotated over time, and can be used to provide “full” supervision for both action recognition and anticipation. Nonetheless, we believe that we can realistically simulate the few-shot LTA scenario by **randomly** select the LTA examples (i.e. simulate the scenario where one only has access to those few examples), in order to understand the desirable approach to perform few-shot LTA with LLMs. And that is indeed the setup we adopt in the experiments. >***Q2: Minor comments*** We would like to express our appreciation for these comments. + For the “manually designed goals”, we describe how we design the goals manually in our response to Reviewer Utid Q3 and update the description in our revision. + For ego-specific video-language models , we compare with HierVL, which is a recent sota ego-specific video-language model (based on EgoVLP). Besides, we also conducted an ablation study to compare CLIP and EgoVLP as backbone vision models in Table A2 in the supplementary, the results show that incorporating egocentric visual features indeeds brings some improvement. + For experiments on EK100, thank you for the suggestion! We’ll consider EK-100 as another major benchmark for future work, we picked EK-55 for the ease of comparisons with previously published results. + In table 5 the “from-scratch” means the 91M model is initialized randomly. In fact in this paper the 91M model is always trained from scratch with random initialization. We’ll clarify it in the revised version. + We have made the corresponding changes in our revision to address the other points raised by the reviewer.

Thanks for your constructive feedback and inspiring questions! > ***Q1: The performance gain is evident but not surprising since additional labels of goals and additional knowledge from LLMs are introduced.*** We would like to clarify the significance of the performance gains achieved by AntGPT as follows: + We consider our proposed goal conditioned framework (top-down) as a novel perspective for the LTA task and a contribution on itself. Prior work either adopts the bottom-up approach or uses latent goals. Instead, our Table 1 demonstrates the effectiveness of explicit goal-conditioning, where the goals are inferred by LLMs through in-context learning. Although additional goal annotations are used, it is worth highlighting that we only use few-shot examples for the goal generation: Only 12 examples are used for Ego4D (over 3,000 hours), EK-55 (55 hours), and EGTEA (26 hours), respectively. + Although it might appear to be intuitive for some that LLMs might be helpful for the LTA tasks, we believe our work strives to understand if and how they benefit for the LTA tasks, via extensive ablation studies (Table 1 for goal generation, Table 2 and 3 for temporal modeling, and Table 4 for input representations). These studies allow us to propose a compact model (91M) that is able to outperform its LLM counterpart (91B). Finally, we did compare with two LLM-based approaches in Table 6, where AntGPT outperforms both approaches. >***Q2: The role and importance of visual modality. Fundamental positioning of the LTA task.*** We thank the reviewer for the excellent and thought-provoking questions! We view demonstrating the effectiveness of action-based representations for modeling temporal dynamics of human activities as one of the major contributions of AntGPT. This provides a natural interface for video understanding tasks to benefit from prior knowledge (e.g. through goals, or few-shot in-context learning) encoded by LLMs. Nonetheless, we view AntGPT as a **multimodal** framework that first leverages the visual inputs to generate interpretable and discrete action tokens, which are then processed by LLMs to model temporal dynamics. In this multimodal framework, text serves as a representational bottleneck between perception and temporal “reasoning”. AntGPT can be naturally generalized to relax the representational bottleneck, for example by adding visual embeddings alongside the action labels as inputs to the temporal model. We conduct such an analysis by concatenating CLIP embeddings to the discrete action labels, and add a linear projection layer to map the visual embeddings to the language input space, a standard practice adopted by BLIP-2 (Li et al.), AnyMAL (Moon et al.), and other work. The results are shown in the table below. | Input | Verb | Noun | Action | |------------------------|-----------|-----------|-----------| | Action (AntGPT) | 0.661 | 0.651 | 0.878 | | Action + CLIP features | **0.643** | **0.650** | **0.868** | After combining visual features and discrete action labels, we observe a moderate level of performance improvement compared with the original AntGPT results (from Table A4). This shows that: (1) our proposed framework can indeed be naturally extended with a hybrid “bottleneck” that includes both text and visual inputs; (2) for the LTA benchmark, we observe the gain of directly incorporating vision features is moderate, which confirms the strength of our proposed original AntGPT with discrete action representation. In summary, we believe action anticipation should still be studied as a standalone task. Despite AntGPT’s strong performance across multiple benchmarks, we believe better and more detailed visual representation (e.g. gaze and pose of the actors) are essential for solving the task, and AntGPT can be naturally extended to achieve this goal as demonstrated above. Other possible improvements include incorporating fine-grained, object-centric visual representations, and even to directly backpropagate the gradient to the visual encoders.

We thank reviewer Utid for your helpful comments and constructive feedback! Due to character limits, we'll answer your question with two comment pages. > **Q1: The paper writing/organization can be improved.** Thank you for the feedback! We have updated Figure 1 as suggested in revision, to emphasize the connections among different panels and made the fonts larger. Notably, Figure 1(a) provides an overview of the AntGPT framework, which illustrates how the action recognition model (blue), temporal model (orange), and goal generator (red) are used in the bottom-up, and top-down LTA frameworks, respectively. The color coding is reused for Figure 1(b-d) to indicate their connections. Figure 1(b) provides a zoomed-in view of the goal generator to illustrate how “goals” are defined in the context of our work. Figure 1(c) discusses a special case of Figure 1(a), when in-context examples (as opposed to explicitly computing gradient updates) are provided to the temporal model for few-shot LTA. Figure 1(d) discusses another use case for knowledge distillation. We acknowledge your suggestion on the state-of-the-art results, our intention was to first provide answers to our core research questions, and context information on how AntGPT achieves state-of-the-art performance. We hope our edits address both of your feedback on the paper’s presentation, and we look forward to your additional feedback! > **Q2: Lack of error analysis** We provided error analysis in the appendix due to space limitations. Specifically, Figure A1 illustrates how the goal-conditioning would improve long-term action anticipation performance. Figure A4 presents two positive and negative examples when the LLM for temporal dynamics modeling is fine-tuned. Figure A5 presents examples on how chain-of-thought prompting may help (a) or hurt (b) LTA performance. Finally, Figure A6 (Section B1) presents the counterfactual examples. Through these qualitative analysis, we can see that long-term action anticipation is a very challenging task, and nonetheless AntGPT is capable of predicting correct future actions, or making reasonable mistakes, even when the recognized actions are not perfect. However, we do observe that when most of the recognized actions are incorrect (Figure A4c), the predictions would also fail as expected. Other failure modes include the hallucination effect of LLMs (Figure A4d). Figure A5b shows a failure example of chain-of-thoughts prompting. In this video of “playing tablet”, the top-down model predicts the goal as “using electronic devices”. As expected, an incorrect goal is likely to mislead the predicted actions conditioned on the goal. We believe adding additional supervision (when available) to the goal generator, and improving the LLM backbone to reduce hallucination, would both help mitigate the errors.

(continues) > **Q3: How are the goals defined?** A goal is defined as the intent / purpose of a human actor’s behaviors. In the context of daily household activities, a goal often corresponds to the task the actor is trying to accomplish (e.g. take the action “mix eggs” to accomplish the goal of “making fried rice”). The desired goals are hence not supposed to be always a simple repetition of the last observed action, but to provide a description of the task. To meet our definition of desired goals, we collect the few-shot examples for goal generation with in-context learning as follows: For Ego4D, we manually annotate goals as the task descriptions by watching the videos. For EGTEA, we use the video title (e.g. “cook cheeseburger”). For EK-55, we use the video descriptions from the dataset (e.g. “cook instant noodles for lunch”). The newly added **Figure A3** in appendix shows goals we used to construct the in-context examples for goal generation. Figure 2 shows the predicted goals based on the in-context examples. In Figure 2a, the last observed action “paint (with) paintbrush” is to accomplish the task “paint the door”; in Figure 2b, the last observed action “clean mat” is to accomplish the task “clean car and mat”. Although the inferred goals appear to correlate to the future actions, we still view them as positive examples since they accurately describe the high-level tasks nonetheless. > **Q4: Distillation results are better than LLMs.** The student model is optimized by both distillation loss and language modeling loss as shown in Figure 1d. We conducted comparing experiments showing that distilling results with only distillation loss without loss from ground-truth future actions are strictly worse than the teacher model’s performance (shown in the table listed below). We thus attribute the better performance of the student model to: (1) the integration of knowledge from the teacher model through distillation; (2) the optimization of a student model with task-specific ground-truth labels (in comparison, the LLMs are pre-trained as “generalists”, then adapted to the LTA tasks). | Loss | verb | noun | action | |-----------|------------|------------|------------| | dist | 0.7044 | 0.6788 | 0.9041 | | dist + lm | **0.6649** | **0.6752** | **0.8826** |

Thanks for the reply. The authors have addressed all my questions and concerns.