Hawk: Learning to Understand Open-World Video Anomalies

Summary

This work proposes a novel video-language framework, HAWK, aiming at understanding video anomalies, which incorporates motion modality to enhance its capability. The work generates rich language descriptions for seven different video anomaly datasets, and also generates question-answer pairs to tackle potential user inquiries. The proposed framework demonstrates SOTA performance for video anomaly understanding and question-answering across multiple scenarios, which will advance the open-world anomaly understanding field.

Strengths

The work proposes a novel vision-language framework to address open-world video anomaly understanding, which is a very different pipeline from previous classification-based anomaly detection pipelines. To build a vision-language model for open-world video anomaly understanding, the work adopts seven different video anomaly datasets for training, where rich language descriptions and question-answer pairs are generated. Experiments for video anomaly understanding and question-answering across multiple scenarios demonstrate the effectiveness of the proposed framework. I believe the work will advance the field of open-world video anomaly understanding.

Weaknesses

1. The work adopts Gunnar Farneback’s algorithm to obtain the motion modality. Is this algorithm efficient? 2. The Ubnormal dataset consists of virtual anomaly videos. Do the authors consider the gap between real and virtual anomaly videos, which is mentioned in previous works [1, 2]? [1] Ubnormal: New benchmark for supervised open-set video anomaly detection, CVPR 2022 [2] Generating Anomalies for Video Anomaly Detection with Prompt-based Feature Mapping, CVPR 2023

Questions

See the Weakness part.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The paper has discussed the limitations and potential impacts of the work.

Summary

The manuscript proposes a new framework that uses an interactive large-scale visual language model (VLM) to accurately explain video anomalies. The framework can explicitly integrate motion modalities to enhance anomaly identification. The manuscript also constructs an auxiliary consistency loss to guide the video branch to focus on motion modalities. The manuscript annotates more than 8,000 anomaly videos with language descriptions, enabling effective training in different open-world scenarios, and creates 8,000 question-answer pairs for users' open-world questions. The final results show that HAWK achieves SOTA performance, surpassing existing baselines in both video description generation and question answering.

Strengths

1.This manuscript proposes a novel video language framework HAWK, which aims to understand video anomalies, and combines motion modalities to enhance its capabilities. 2.This manuscript collects seven video anomaly datasets from different scenarios and generates rich language descriptions for each video. At the same time, considering the diversity of open-world questions, question-answer pairs are generated to solve potential user queries. 3.The proposed method achieves state-of-the-art performance on three public anomaly detection datasets.

Weaknesses

1. There are already many studies focusing on the background information in abnormal events. Have the authors considered describing the background information when describing the action information? [1] Scene-aware context reasoning for unsupervised abnormal event detection in videos[C]//Proceedings of the 28th ACM international conference on multimedia. 2020: 184-192. [2] Few-shot scene-adaptive anomaly detection[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020: 125 -141. [3] Hierarchical scene normality-binding modeling for anomaly detection in surveillance videos[C]//Proceedings of the 30th ACM international conference on multimedia. 2022: 6103-6112. 2.The dataset proposed in the manuscript also supports question answering in open scenes, but the method does not reflect how to use these questions and answers to improve the model's anomaly detection and understanding capabilities. 3.When constructing a dataset, if a description is generated for each second in the dataset, it may result in a lot of repeated descriptions. In order to avoid this redundancy problem, have you considered using keyframes instead of second-by-second descriptions to reduce the complexity and computational cost of data processing? 4.Most previous studies pay equal attention to various parts of the video, such as background, motion information, character appearance, etc. This paper focuses on motion information. Section 4.3 mainly extracts language descriptions related to motion. Whether background information is not considered. Moreover, the 7 datasets proposed in the manuscript contain more abnormal scenes. I am very curious why the scene information is not paid attention to.

Questions

As mentioned above.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors address possible limitations of their study.

Summary

The paper proposes a new variant of the video anomaly detection. Prior methods were vanilla classification methods, and this paper proposes more descriptive anomaly description and also QA along with that. The paper first gives the dataset creation strategy and then introduces the motion model and the video architecture to provide a text description of the anomaly. The authors also have baselines to compare against recent methods.

Strengths

1. The problem setup is novel and fills the gap in current VAD methods. 2. The dataset creation strategy is very intuitive and the quality looks good from the examples provided. 3. The evaluation protocol is correctly explained.

Weaknesses

My main concern is regarding the experimentation and the lack of some crucial baselines. 1. The proposed method benefits from the training data, whereas all the baselines are zero-shot. It is difficult to now evaluate the contribution of your motion model and the training data. It is essential to show how the baseline methods like VideoChat will behave when they are fine-tuned with the training data the authors propose. Without these baselines, it is difficult to evaluate the effectiveness. 2. The GPT-4 output shown in Fig. 2 (a car with red lights has lost traction....) is already pretty good. Due to this, the authors must show that the trained model is able to perform better than the basic dataset creation strategy. It is understood that the dataset is created using off the shelf models and expected to perform worse than the trained video and motion model, but a baseline that uses the pseudo-dataset creation strategy is important. (Or an appropriate justification of why it cannot be added)

Questions

Please answer the question regarding the experimental details. Overall, I am positive about the paper and hence, will increase the rating once the clarification on the experiments is provided. Minor comment Guieded -> Guided in Tab 4

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes, limitations are discussed and they correctly reflect the shortcomings that I see.

Summary

This paper presents HAWK, a framework that uses large Visual Language Models (VLMs) to accurately interpret video anomalies. HAWK integrates motion and video modalities through a dual-branch framework, enhanced by an auxiliary consistency loss to focus on motion-related features. The authors annotated over 8,000 anomaly videos with language descriptions and created 8,000 question-answer pairs to train the model across diverse scenarios. HAWK demonstrates state-of-the-art performance.

Strengths

1. The framework benefits from extensive annotations and question-answer pairs, improving training quality. 2. HAWK achieves superior results in video description generation and question-answering tasks, outperforming existing baselines. 3. The paper is well-written and easy to follow.

Weaknesses

1. In Table 1, it would be better to indicate the LLM backbone of the methods for a fair comparison. 2. From the ablation study in Table 3, it appears that even without the motion modality, the baseline model achieves comparable performance to other methods without motion modality. This may be because the high-quality dataset and strong LLM backbone contribute more to the performance, which weakens the perceived technical contribution of the motion modality.

Questions

See the weaknesses.

Rating

3

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

See the weaknesses.

Response to author response

I would like to thank the authors for their response. The experiments for both my concerns are now making the comparisons stronger. Regarding Q2, it is good to see that the training improves the naive weakly supervised training set. Similarly, in Q1, the fine-tuning significantly improves the numbers. These are the numbers that should have been ideally reported. I am updating my rating to weak accept since these new experiments finally show the effectiveness of the method. Please update the paper with these new numbers.