Response to Reviewer FnrC
Thank you for your detailed review, below we address the key concerns.
**Q1: *The computational cost of performing multiple inferences far
exceeds that of the basic video understanding model.***
Kindly note that the VideoPrompter offers a flexible framework, allowing
for the substitution of various modules, including the video-2-text
model (VGPT), text-2-text model (GPT-3.5), and vision-language model
(CLIP). During our initial submission, Video-ChatGPT (VGPT) \[1\] was
chosen as the video-2-text model due to its competitive performance when
compared to Video-Chat and VideoLLaMA. However, the recent introduction
of Video-LLaVA \[2\], *a model released this week*, has set a new
state-of-the-art (SOTA) in zero-shot question-answer and video-based
generative benchmarks. We performed an experiment where we replaced VGPT
with Video-LLaVA and generated video-textual descriptions for HMDB-51
and UCF-101 benchmarks. Conducting a single inference (as opposed to the
10 inferences with VGPT) without any filtering applied (as now we have
only one description per video), we observed a 0.21% and 0.75%
improvement on previous scores for HMDB-51 and UCF-101 benchmarks,
respectively, as shown in the table below. The adoption of Video-LLaVA
over VGPT reduced the number of required inferences from 10 to 1,
leading to a significant decrease in computational cost. Additionally,
the per-video inference time for Video-LLaVA is approximately 6 times
less than that of VGPT, with 3.79s per video, while, for reference, the
inference time of the CLIP vision encoder is 0.74s per video. We
anticipate that ongoing advancements in multi-model methods will further
reduce the inference cost of VideoPrompter. We will include the updated
results with Video-LLaVA on all benchmarks and models in our final
manuscript.
| **Dataset** | **Baseline (CLIP)** | **Video-Prompter (VGPT)** | **Video-Prompter (LLaVA)** |
|:---------------------------:|:-------------------:|:--------------------------:|:---------------------------:|
| HMDB-51 | 37.5 | 52.51 | **52.72** |
| UCF-101 | 61.72 | 73.88 | **74.63** |
**Q2: *Except for CLIP, the proposed approach shows relatively limited
performance gain in other video-based models.***
Please note that the large gain on the vanilla-CLIP can be linked to the
fact that its not adopted for videos and when its used with our proposed
framework, without any further training, the VideoPrompter brings
significant boost in the performance. The video-based models such as
ViFi-CLIP are already trained on Kinectics-400 and are adopted to videos
to some extent. These models when combined with VideoPrompter increase
the performance by 6.12%, 4.27%, and 5.3% for ViFi-CLIP, AIM, and
ActionCLIP respectively on HMDB-51 benchmark and similar trend is seen
on other benchmarks as shown in Table-2 and Table-6. Kindly, considering
that no further training is performed, and the computational cost has
been substantially reduced after Video-LLaVA, these score increments can
be considered significant.
**Q3: *The configurations of adopted pre-trained models (AIM,
ActionCLIP, ...) remain unclear. For AIM, do the authors directly remove
the classification layers?***
Kindly note that all of the pre-trained models used in this study
(ViFi-CLIP, AIM, and ActionCLIP) are pre-trained on Kinectics-400.
Secondly, the original AIM model removed the text classifier from the
vanilla CLIP and only trained the visual encoder. We removed the
classification layers from AIM and used a text classifier from the
vanilla CLIP with the AIM-visual encoder to get zero-shot results. These
points have been made more clear in the original paper, as highlighted
in red.
\[1\] Maaz, et al. \"Video-ChatGPT: Towards Detailed Video Understanding
via Large Vision and Language Models.\" arXiv.
\[2\] Lin, et al. \"Video-LLaVA: Learning United Visual Representation
by Alignment Before Projection.\" arxiv.