Single-Stage Visual Query Localization in Egocentric Videos

Summary

The paper proposes VQLoC, an end-to-end trainable framework for Visual Query Localization (VQL) on long-form egocentric videos. Compared with Ego4D's multi-stage approaches, VQLoC proposes a single-stage process that efficiently localizes visually specified objects. It establishes both query-to-frame and frame-to-frame relationships for spatial-temporal localization. The proposed method achieves new SOTA performance on the public leaderboard with a notable improvement.

Strengths

VQLoC can streamline the VQ2D localization process. The method is end-to-end trainable and achieves faster inference speed, as shown in Figure 4. The results are verified on the public server. Furthermore, from the quantitative ablation study, it seems like the boost comes from the model design rather than parameter tuning or training tricks, which is good. The related work covers the field well. And the authors made comparisons to previous baselines in terms of size, speed, and performance.

Weaknesses

My major concern is the completeness of the ablation study. This paper can be stronger if more ablations can be done. For the spatio-temporal transformer, Tab. 4 only shows the window size = 5 gives the best performance among the numbers greater than 5. It's not convincing 5 is the best choice. How about a window size smaller than 5? It's possible that if you don't do any frame-to-frame correspondence, i.e., a window size of 1/0 gives even better performance. Also, there are multiple thresholds in the methodology. Are the results sensitive to the selection of those hyperparameters? For example, the peak response plot in Figure 5 looks noisy, so it's not sure whether an empirical choice of φ makes sense. The same also applies to the anchor box threshold theta.

Questions

(P6 L207) Why not trim complete negative video clips to augment the data for the training? (P6 L228) Are the top K anchors selected from the entire video clip n' and mixed with anchors from n? I need clarification here since the loss is frame level, but the text is about video level. Ego4D Episodic Memory also has similar challenges like VQ3D, NLQ, and MQ. Is it possible to adopt similar strategies for these tasks by changing the query encoder and predictor? It will be quite interesting to develop a unified framework for video query tasks.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I don't think the authors can provide more ablations for a large benchmark like VQ2D within the rebuttal date. Overall, this work has pushed VQ2D task forward with a remarkable step. However, more complete ablations will make this paper stronger in the future.

Summary

The author proposes a new single-stage end-to-end method for visual query localization. The proposed method builds a holistic understanding of query-video relationship, then spatial-temporal localization is performed. The proposed achieves major performance and speed gain over previous methods on major benchmark datasets.

Strengths

In general, the proposed method is simple and performs well on the major egocentric Ego4D dataset. The proposed method utilizes simple pipelines with cross attention/Transformer, however, achieves state-of-the-art results. In general, I am satisfied with the paper.

Weaknesses

However, I have some concerns about the paper. 1. The author only uses the Ego4D dataset as the benchmark for egocentric dataset. Since it is a new task, it is OK to only have few baselines/datasets cause it is a brand new task. However, It would be better if the author's method can be applied to other tasks/sub-tasks like visual tracking datasets (or modify the tracking datasets 2. More experiment details should be reported. Because it is an end-to-end framework. Though the author reports the FPS of the model, I think the author could report the more detailed metric like FPS/GPU memory. I think the proposed method could have larger GPU memory consumption/larger FLOPs while with fast execution time.

Questions

Please mainly see the weaknesses section for details.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I think the author has adequately addressed the limitations.

Summary

The paper introduces a new approach to address the visual query localization problem in egocentric videos. The major contribution is that the proposed method, a single-stage model, simplifies the previous multi-stage frameworks and eliminates the need for off-the-shelf object detectors, tracking, and similar components by utilizing an end-to-end trainable model. Notably, the proposed method has achieved the top position in the Ego4D VQ2D challenge as of the submission date, demonstrating its effectiveness. Besides, the method demonstrates a considerable enhancement in inference speed, further highlighting its advantages.

Strengths

1. The paper introduces a novel and technically sound method that formulates the visual query localization task as a unified framework. By taking a video and a template query image as input, the proposed method can directly produce the track of bounding boxes, eliminating the need for off-the-shelf object detectors and tracking methods. This is achieved by decoupling the task into spatial reasoning (finding per-frame responses) and temporal reasoning (consultation within the clip), followed by a prediction head (box&score) upon the learned features. In summary, the idea behind the method is both original and rational. 2. The proposed method demonstrates notable computational efficiency, a crucial aspect for real-world applications. By simplifying previous multi-stage frameworks into a one-stage process, the method achieves a remarkable 10x improvement in speed while maintaining a satisfactory level of performance. This advancement represents a significant stride towards practical applications. 3. The paper is well-written. Readers with a relevant background will have no difficulty following and comprehending the content. 4. The paper presents solid experimental results, achieving the top position on the Ego4D VQ2D challenge leaderboard at the date of submission.

Weaknesses

1. L12-L13: "Our experiments demonstrate that our approach outperforms prior VQL methods 12 by 20% accuracy while obtaining a 10× improvement in inference speed". Without delving into the entire paper, a glance at Table 1 and Figure 4 might cause confusion regarding the reported results, as the speed does not appear to be 10x faster than STARK [51] (if my understanding is correct, this is an adapted tracking approach upon the proposed framework). It would be beneficial to include some explanation and details in the caption to prevent such confusion. 2. It is unclear how the number of frames (T=30) per clip is determined? Is this choice based on the limit of computational resources or driven by experimental performance considerations. If the latter, it would be helpful to know if any ablation studies have been conducted to investigate its aspect. 3. Discussion on the long videos. The input is fixed to a maximum of 30 frames (~6 seconds with a frame rate of 5fps). To handle longer videos, the proposed solution is to chunk the untrimmed video into fixed-length clips. In this case, is there an additional step to ensure smooth predictions between these clips, such as incorporating tracking between them? Alternatively, another solution to address long videos is to use a streaming mode. Suppose we have processed the past 30 frames $\{ t_{i-29}, …, t_{i-1}, t_i \}$, we would only need to process the new incoming frame $t_{i+1}$. Spatial transformer naturally lends itself to this scenario since it is computed at frame level. The spatio-temporal transformer might be more complex. But it operates within a bounded window, which suggests that it might not require significant re-computation or adjustments for the streaming mode. It would be valuable to have the authors' insights and comments on this matter. 4. L177-L193: The symbol $w$ is used ambiguously, representing both the spatial width and the temporal window.

Questions

Please refer to the previous section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors addressed the limitations.

Summary

This paper tackles the problem of query localization of visually specified objects in egocentric human-activity videos. The main technical contribution for the solution is to design a single-stage transformer-based architecture to model the query-to-frame correspondence matching and frame-to-frame correspondence propagation. Ablation studies verify the effectiveness of the proposed components and comparisons show the superiority of the method in performance and speed with respect to prior works.

Strengths

1) The network is single-stage and end-to-end trainable, which alleviates the issues in existing multi-stage approaches that different stages are separate from each other for conducting non-differentiable stage-wise predictions. 2) Model designs, including the frame-wise spatial transformer and the subsequent spatio-temporal transformer, are technically sound and easy to follow. 3) Good performance is achieved by the proposed method and speed-accuracy trade-off is also taken into consideration.

Weaknesses

1) The technical contributions in terms of the model designs are short of novelty, and most of the building blocks can be viewed as simple adaptations from existing techniques used in methods for similar tasks (e.g., visual tracking, temporal sentence grounding, spatio-temporal video grounding, etc., can refer to STARK[1], LGI[2], STCAT[3], QD-DETR[4]). Apart from that, the unique challenges in the task of VQL, such as the egocentric characteristics presented in the videos, are less considered and are scarcely reflected by the model designs. 2) A powerful pre-trained model DINOv2 is adopted as the visual backbone, which plays a crucial role for ensuring the success of the query-to-frame correspondence matching. However, no related discussions are included in the manuscript. What if the method gets rid of (by replacing it with a weaker but common one) the pre-trained DINOv2 model? Would it simply fail or drop significantly in performance? And it’s unclear how much gains are brought by the proposed architecture except for the utilization of a stronger feature backbone. 3) According to the ablation studies, the proposed model heavily relies on a high input resolution to achieve decent localization results. This could be a negative factor hindering the VQL model from scaling up to a larger amount of training data. 4) Some literal expressions, such as the writing of Section 3.3 could be reorganized to make it clearer. 5) As shown by the temporal plot of predicted object-occurrence probabilities in Figure 5, there are still a lot of spurious responses with large magnitude (some are even larger than responses within GT) outside the target temporal region, which implies the current model’s deficiency in accurate temporal localization to some extent. [1] Bin Yan, et al. "Learning spatio-temporal transformer for visual tracking." In ICCV, 2021. [2] Mun, Jonghwan, et al. "Local-global video-text interactions for temporal grounding." In CVPR, 2020. [3] Jin, Yang, et al. "Embracing consistency: A one-stage approach for spatio-temporal video grounding." In NeurIPS, 2022. [4] Moon, WonJun, et al. "Query-dependent video representation for moment retrieval and highlight detection." In CVPR, 2023.

Questions

None

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

YES

Summary

The paper proposes VQLoC, an end-to-end method for egocentric visual query localization based on a holistic understanding of the query-video relationship. The key component is a spatio-temporal transformer, which can effectively model the relationships between the query and frames. The presented approach can perform single-stage inference and achieve the winning entry on the leaderboard. Moreover, the method produces a set of models to balance the speed and balance, while the top-performing one is more advanced from both perspectives than the state-of-the-art. Moreover, the visualization shows that VQLoC is excellent at the visual query 2D detection task and can even deal with challenging cases such as clutter, occlusions, and motion blur.

Strengths

1. This paper positively impacts the egocentric community, especially in episodic memory tasks. Unlike Moment Query and Language Query, the visual query task is more complicated in spatial-temporal localization, and the existing solutions are redundant. The paper provides a single-stage solution, and the chosen model is more accurate and efficient than the state-of-the-art. This will encourage more researchers to work on the valued research problem. 2. The proposed architecture for the spatio-temporal transformer is able to effectively model the relationships between the query to each frame as well as between the frames. Therefore, the model can leverage rich semantics during feature embedding. Locally-windowed temporal self-attention is applied to improve the model efficiency without losing too much information. 3. Code and the architecture details are attached to the paper, which helps people to re-implement the code. Also, the high-quality video in the supplementary material makes the paper clear and easy to follow.

Weaknesses

1. My main concern is that although the paper aims to solve the Visual Query Localization task, all the experiments are conducted in the VQ2D setting. According to the definition of the Ego4D [13] paper, visual query localization can be done in the video domain (VQ2D) or the real-world coordinate (VQ3D). Therefore, it is not precise to only validate this method on VQ2D, and evaluating the proposed method on the 3D setup is highly recommended. Otherwise, the paper should revise the task as visual queries 2D localization. 2. Further experiments could be conducted to improve the model further. For instance, the authors find the local windows of size 5 works the best in the set {5, 7, 9}, then they should experiment with smaller windows for the local optima of their hyper-parameters.

Questions

1. In the ablation study L299, using high-resolution frames (448 × 448) leads to way better results than the low-resolution ones (224 × 224). Is it possible to further increase the resolution for better performance, even if we can slightly sacrifice the clip length? 2. It seems that the prediction head only gives framewise bounding box locations and confidence scores, but the VQ2D task requires a response track of the query object. Is there any mechanics in the head to make the predicted boxes consistent with the predictions from the temporal neighboring frames? 3. According to Table 1, the recovery ratio and success rate improved significantly, but the AP only raised a little. Is it because the single-stage detection pipeline is weak in predicting precise bounding boxes?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitation is already discussed in the paper.

Re: Rebuttal by Authors

I appreciate your response. I'm pleased to hear that you agree with my points (especially, intend to incorporate the steaming function into your future work) and their inclusion will undoubtedly enhance the clarity and quality of the work.

Response to the rebuttal

Thanks to the authors for the detailed explanation and additional info on the rebuttal. The table on local window size ablation is helpful to me. A unified framework for other tasks like NLQ, VQ3D, MQ, etc, could be interesting following this work. The authors have addressed my concerns and questions I raised. I have no further questions at this point.