Thanks for highlighting the strengths of the proposed paper and for providing valuable feedback! As all weaknesses are complemented with respective questions, we will focus on those specifically.
#### Questions:
1. *Visualization to demonstrate the OT for key frame selection?:* Thanks for the suggestion! We will work on the visualization and report it soon.
2. *Modeling the temporal relation between key frames:* Thanks for the question. We would like to answer in multiple ways here: First, we actually evaluate the framework with both, 3D and 2D backbones in Table 2, namely a S3D backbone which captures low-level temporal dynamics as the requested ones. Table 2 also shows that even while CLIP does not capture such temporal dependency it still performs slightly better than the S3D backbone. This can be considered a phenomenon known from various current video approaches using CLIP backbone, such as [1,2]. To address this issue, we offer to include an additional experiment with the TimeSformer backbone, which also utilize a visual transformer that encodes temporal dynamic, for the camera-ready version.
To reason why CLIP might perform better than the S3D backbone, we want to point out that our dataset is targeted to be used for grounding instead of classification. Hence, our text queries seldom contain explicit complementary semantics such as pick up and put down as it would be labeled in action classification tasks.
[1] Luo, Huaishao, et al. "Clip4clip: An empirical study of clip for end-to-end video clip retrieval and captioning." Neurocomputing 508 (2022): 293-304.
[2] Bain, Max, et al. "A CLIP-Hitchhiker's Guide to Long Video Retrieval." arXiv preprint arXiv:2205.08508 (2022).
3. *Justification or empirical evidence for the preference of key points:* We have conducted an annotation study in supplement Figure 6,7,9. Our finding is that our annotation reaches high consistency and captures meaningful variety for evaluation. In general we found in the initial phase of the annotation that the center point of action is more intuitive than the bounding box since it is hard to give any guidance of the scale of the bounding box, thus the physical outline for an action, which varies e.g. with respect to annotations and viewpoints of the hands, human, or instrument. As an alternative view, if we would predefined certain label criteria (e.g. always include the hands), the model would not be aware of those criteria without being trained on it, which would defy the idea of a weakly-supervised training without the need for annotation. Instead, we used the current spread of center points to define the outlines of the region that we want the model to localize on.
Please let us know if you have any other questions or concerns regarding our paper or response.