CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos

Summary

This work focuses on the VidSGG task. Specifically, it built a new UAV-based VidSGG dataset named AeroEye, and also further propose a CYCLO approach for using cyclic attention over the VidSGG task.

Strengths

1. The paper is well-written and easy to follow. 2. From my perspective, a high quality dataset and large scale dataset is of critical imporantance over the VidSGG area. I thus appreciate the effort by the authors.

Weaknesses

(See the questions section below)

Questions

Overall, while I tend to accept this work, I still have the following concerns and suggestions w.r.t. the current version of the submission: (1) It seems to me that, while the dataset focuses on the UAV scenario, the direct and long-range temporal relationships focused by the proposed method is not closely related to the UAV scenario. I appreciate if the connection (if any) can be more clearly explained. (2) Besides comparing with those existing UAV datasets, I suggest the authors to also include a statistical discussion (in a table format maybe) between AeroEye and those existing VidSGG datasets. (3) In line 43, the authors claim that "They [5, 19] usually struggle with long-term dependencies due to the diminishing influence of inputs over time." This claim should be made with better support. (4) Moreover, I believe that the connectivity between the proposed CYCLO method and scene graph needs to be better discussed. This is important since the usage of "cyclic attention" solely in the video context seems to be not a new stuff already (e.g., Transformer Tracking with Cyclic Shifting Window Attention CVPR 2022). Thus, for authors to clearly indicate the novelty of their CYCLO method, from my perspective, it is necessary to (a) discuss how CYCLO is designed for scene graph more clearly and (b) discuss the difference between CYCLO and existing cyclic attention methods in the video context.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed their limitations.

Summary

This paper tackles an interesting task for understanding video scenes that focuses on modeling object relationships in aerial videos. Specifically, it introduces a new dataset, the AeroEye dataset, and proposes a novel approach, CYCLO, to better model the video object relationships. Experimental results demonstrate the effectiveness of CYCLO. CYCLO also achieves state-of-the-art performance on two scene graph generation benchmarks.

Strengths

- The paper is well-written, easy to follow, and presents many key points clearly. - Introducing the AeroEye dataset is a valuable contribution, as it fills an important gap in video scene graph generation datasets by providing a drone perspective relation dataset. - The design of CYCLO is both interesting and inspiring. The paper also demonstrates its superior performance compared to prior solutions.

Weaknesses

- It would be great to report the inference cost of the proposed approach. - It is unclear how the proposed solution works in live (online) mode, e.g., video streaming.

Questions

- In the ablation study (line 265), when a frame is discarded, is it something like that the frame is set to some random noise?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

4

Limitations

I don't have concerns about the potential negative societal impact of this work.

Summary

This paper presents a new problem: modeling multi-object relationships from a drone's perspective. To address this, the authors propose the AeroEye dataset and introduce the Cyclic Graph Transformer (CYCLO) method. This method captures both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The authors not only validate the CYCLO approach on the AeroEye dataset but also test it on the PVSG and ASPIRe datasets, demonstrating the effectiveness of their method.

Strengths

1. The authors have introduced the problem of multi-object relationship modeling from a drone's perspective for the first time and constructed the AeroEye dataset, which effectively fills a gap in the field of multi-object relationship modeling and has significant application value. 2. The CYCLO method proposed by the authors is not only useful for the dataset introduced in this paper, AeroEye, but is also a versatile method that can be applied to general Video SGG tasks. It has been tested on datasets like PVSG and achieved good performance. 3. The structure of the paper is clear, the writing is standard and fluent, making it easy to understand.

Weaknesses

1. The method proposed in this paper could benefit from a clearer network architecture figure, which would allow everyone to better understand the method presented. 2. Is there an issue with non-differentiability in the Cyclic Attention described in Eq. 3? It would be helpful if the authors could provide further explanation of the Cyclic Attention mechanism. 3. It is suggested to include some analysis of bad cases, which would help future researchers understand from which directions further optimizations can be made.

Questions

1. The current model proposed by the authors shows low mR@K scores on the AeroEye dataset. However, from the figures in the supplementary material, the long-tail distribution of relations in this dataset does not appear to be very severe. What could be causing the low mR@K scores? If possible, I would like to see some typical bad cases.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

It is recommended that the authors consider the limitations of this paper not only from a technical perspective but also from a societal standpoint. Given that relationship modeling from a drone's perspective may lead to widespread applications in surveillance and potentially significant impacts, this aspect requires special consideration.

Summary

This paper proposes a video scene graph generation dataset called AeroEye on aerial videos and a framework called cyclic graph transformer to tackle the problem of video scene graph generation. The authors annotated the ERA and MAVREC dataset with keyframes at 5FPS. They manually annotated the frames for bounding box localization for each frames along with the tracking. The relationship annotations were done using a GPT4RoI model. The proposed cyclic graph transformer uses a cyclic attention mechanism which the author claims, are able to capture the direct and long-term temporal dependencies.

Strengths

1. The proposed video scene graph dataset for aerial videos with a diverse set of predicates can offer more granular and nuanced understanding of dynamic interactions and relationships within aerial footage 2. The dataset will be publicly available 3. The proposed approach for circular attention for dynamic online scene graph generation seems promising

Weaknesses

1. the paper is very difficult to follow. There are separate discussion sections which somewhat disrupts the flow. 2. line 82 seems incomplete 3.Line 118: 'no temporal edge is treated a boundary' can it not be a disadvantage as well since it does not takes into account for an event boundary? 4. A very brief discussion of annotation procedure should be in the main paper such as manual annotation of bounding boxes and relationship annotations done by GPT4RoI model. 5. For loss function section, it would be better to include the overall loss equation. Does the object distribution refers to the object detection loss using the DETR? 6. For table 2, you can refer to the shift value term from equation 3. It will be easy to follow. Table 2 refers to the ablation studies 'Semantic Dynamics in Cyclic Attention'. I think this section and the section followed by it should have a detailed explanation.

Questions

1. Did you do evaluation on the 5 keyframes for each videos? 2. point 5 in weakness section

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

yes, the limitations are addressed.