Drones Help Drones: A Collaborative Framework for Multi-Drone Object Trajectory Prediction and Beyond

Summary

This paper presents a collaborative framework for multi-drone object trajectory prediction, named DHD, which consists of two specifically designed components: the GBG module, aimed at generating more accurate BEV representations in aerial scenes, and the SISW module, which adaptively selects regions for collaborative interactions. Additionally, a simulated multi-drone collaborative observation dataset is created to demonstrate the effectiveness of the proposed DHD framework.

Strengths

- This paper successfully implements an end-to-end multi-drone object trajectory prediction based purely on visual inputs and extends it to the collaborative object detection task. This work fills in the blank of end-to-end multi-drone collaborative prediction, demonstrating notable originality. - The proposed GBG module innovatively leverages the unique inclined observations from drone perspectives as geometric priors and replaces traditional depth estimation with height estimation. These enhancements significantly improve the accuracy of BEV representations for long-range aerial observations compared to the listed baselines, such as LSS and DVDET. - This paper presents an efficient communication strategy called SISW, which considers both the limited inter-drone communication and the prediction task's dependency on foreground and contextual information. Experimental results indicate that this design outperforms previous sparse interaction strategies like where2comm. - A simulated dataset, "Air-Co-Pred," is created to support multi-drone object trajectory prediction. The dataset enriches the available resources within the multi-drone field and can serve as a valuable benchmark for future research.

Weaknesses

- The authors need to carefully review the paper for grammar mistakes, typos, and formatting errors. Specific issues include: - Line 82: BEV's full-term appears redundantly. - Line 119: The subscript in$ Y_{k}^{o} $ is incorrect and should be$ Y_{k}^{t_o} $. - Line 152: Figure reference is incorrect. - Line 221: "2HZ" is improperly formatted and should be "2Hz." - Equation 6: The subscript next to $ \Sigma $ should be $ xy $ instead of $ hw $. Addressing these issues will improve the paper's readability. - The transmission volumes for various collaboration strategies in Table 2 are unclear. It would be beneficial to supplement this information to better reflect the balance between the performance improvements in prediction and the cost of collaborative interactions. Additionally, it appears that the results of DHD in Table 2 are not the best in every column, so why are they bolded? - Please consider exploring scenarios where the number of drones varies from 1 to 4. This would help investigate the impact of drone quantity on perception enhancement and also reflect the influence of potential drone failures on performance metrics. - In reality, extrinsic parameters are derived through computation and approximation, leading to certain biases. Please explore how the extent of extrinsic bias affects the performance of collaborative prediction.

Questions

- Could you provide more details about Air-Co-Pred, including the drone attitude settings, flight patterns, the speed of objects, etc.? Why were four drones chosen to collect data? - In Section 3.3, the equation regarding the derivation of the depth upper-bound is presented directly without any intermediate steps in the main text or supplementary materials. Please provide the detailed derivation process to help readers understand it better. - The reviewer found that the proposed GBG module is highly relevant to the applications of camera intrinsics and extrinsic in UAVs. The reviewer encourages the authors to acknowledge the existence of related works and clarify the differences. For reference, see: Shen H, Lin D, Yang X, et al. Vision-Based Multi-Object Tracking through UAV Swarm[J]. IEEE Geoscience and Remote Sensing Letters, 2023. Pan T, Dong H, Deng B, et al. Robust Cross-Drone Multi-Target Association Using 3D Spatial Consistency[J]. IEEE Signal Processing Letters, 2023. - Considering the complexity of the proposed framework, could you provide additional details or code for better understanding and reproducibility?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors acknowledge that the current focus of this paper is on simulated environments to investigate multi-drone object trajectory prediction without considering latency and camera noise. These are reasonable limitations, and addressing them in future work would be beneficial

Summary

This paper studied the problem of collaborative multi-drone object trajectory prediction. The authors proposed a framework named "Drones Help Drones" (DHD) that can improve accuracy compared to existing methods while reducing the required communication bandwidth. To be more specific, the authors leveraged a Ground-prior-based BEV Generation module that can generate more accurate BEV representations, and used a Sparse Interaction via Sliding Windows module that enables efficient information sharing between drones. The authors also constructed a new dataset named "Air-Co-Pred" for multi-drone collaborative prediction tasks.

Strengths

The paper is well-structured and clearly written, with helpful visualizations that aid in understanding the proposed methods and results. The DHD framework introduces a solution to key challenges in multi-drone collaborative perception, particularly the GBG module for more accurate BEV generation and the SISW module for efficient information sharing. The introduction of the "Air-Co-Pred" dataset addresses a gap in existing resources for multi-drone collaborative prediction tasks, potentially benefiting the broader research community.

Weaknesses

* The primary evaluation is conducted on simulated data. While the authors acknowledge this limitation, it raises questions about the framework's performance in more complex, real-world scenarios with sensor noise, communication delays, environmental variability, and other common difficulties in real-world experiments. * It would be interesting to see how the performance scales with the number of collaborating drones, which could provide valuable insights into the method's scalability. Is there an optimal or maximum number of drones for effective collaboration?

Questions

see above

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed

Summary

The paper presents the "Drones Help Drones" (DHD) framework for improving collaborative trajectory prediction in multi-drone systems. It addresses challenges with aerial observations and communication bandwidth by: (i) Using ground priors from inclined drone observations to enhance Bird's Eye View (BEV) accuracy; (ii) Implementing a selective mechanism to prioritize essential information, reducing communication needs; (iii) Introducing the "Air-Co-Pred" dataset for multi-drone collaborative prediction. Experiments show that DHD improves BEV accuracy, reduces required transmission bandwidth, and maintains high prediction performance.

Strengths

The paper presents an innovative approach to multi-drone collaborative trajectory prediction. Here are some observations: S1. The module for estimating depth from multi-view drone perspectives is convincing and well-formulated. The use of object height contributes significantly to research in this field. S2. The strategy requires only 25% of the transmission ratio without affecting performance. This might be due to the SISW module, which retains only essential information. S3. The three strategies (depth estimation, interaction, and prediction) are trained end-to-end. This is somewhat surprising given the architecture design.

Weaknesses

W1. The strategy follows the classic collaborative framework composed of three modules: BEV estimation, construction of interaction, and prediction. The methodology utilizes well-known strategies from the literature for this task. Specifically, for BEV estimation, it uses part of the framework from [13], although it is well adapted with the inclusion of height in the depth estimation. The Selective Sparse Interaction used in the SISW module appears similar to the approach in [14], as well as the aggregation performed. Finally, the prediction mechanism is taken from [34], as mentioned by the authors in L206. Although the construction of a unified framework is not trivial, the overall strategy deviates little from existing ones. W2. The authors have contributed to the creation of a new dataset that could benefit future research, but this will require appropriate adjustments. There is no comparison with other datasets used in the same or related fields, making it impossible to evaluate the quality of the dataset. W3. The lack of additional datasets also makes the effectiveness of the framework less convincing, as there is no evidence of generalization on other established benchmarks. I understand the uniqueness of the task and wonder if it is possible to test the framework on established benchmarks (e.g., DAIR-V2X, V2XSet, and OPV2V datasets). W4. In Table 2, the reasoning for separating V2X-ViT and V2VNet from other methodologies is not clear. The distinction between fully/partially connected is not a convincing rationale. Could the authors clarify this point? W5. [Minor] Some findings in the appendix, such as Figure 9 and Section 6.4, could have been included in the main paper to strengthen certain claims.

Questions

See Weaknesses section for questions, especially I would appreciate if the authors highlighted the differences with the other mentioned methodologies.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors adequately addressed the limitations.

Summary

The paper proposes a ground-prior-based BEV generation for drone trajectory prediction. The methods also propose an efficient information interaction via a Sliding Windows module. It accesses information volume across different areas through sliding windows. The authors provide experiments on the simulator-based Air-Co-Pred dataset and 3D object detection on CoPerception-UAVs.

Strengths

1. The flow of the paper is thorough and clear. 2. The proposed ground-prior-based BEV generation and Sliding Windows module are innovative.

Weaknesses

The Ground-prior-based BEV Generation (GBG) module relies on certain geometric assumptions for depth estimation. In real-world scenarios, these assumptions might not hold due to variations in terrain, drone stability, and other environmental factors, potentially affecting the accuracy of BEV representations. Although the paper mentions using EfficientNet-B4 for feature extraction due to its low latency and optimized memory usage, it does not provide a detailed analysis of the computational efficiency and scalability of the overall framework. The Air-Co-Pred dataset, while a valuable contribution, is still limited to simulated data. The paper could benefit from additional datasets or validation for the main task. The proposed Sparse Interaction via Sliding Windows (SISW) module, although innovative, might oversimplify the information selection process. The method assumes that feature discrepancies are the primary indicators of valuable information, which may not always be the case, especially in highly dynamic or cluttered environments. There is a lack of baseline in the main experiment; the authors have compared their method with only two other methods for BEV generation. There is a lack of trajectory prediction evaluation; the authors should provide the average displacement error as well. Typo in Table 3 caption. SISW has shown little improvement, and the performance declines when the transition rate is 0.25. Additionally, the authors do not provide the individual result of SISW on 1 with GBG still present.

Questions

Can the authors provide a component ablation of only the SISW component, like three rows on a 1:1 ratio? Furthermore, the results of missing GBG and SISW on a 0.25 transition rate are not provided.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The paper relies on simulations using the CARLA platform. Although the simulated environment provides controlled and varied scenarios, it may not fully capture the complexities and unpredictabilities of real-world environments. It seems like the sliding window module is not that significant, and the authors have not provided a complete ablation study to show its efficacy. The component ablation is not thorough.

Summary

This paper proposes a framework, Drones Help Drones (DHD), which tackles trajectory prediction of objects in the scene. DHD consists of a Ground Prior Based Bird’s Eye View (BEV) Generation (GBG) module, which provides depth estimation from the drone to an object using ground priors to create an accurate BEV representation of the features. It also utilises Sparse Interaction via Sliding Window (SISW) to minimize the data transmission cost between drones. In addition, the authors develop a new dataset for multi-drone collaboration prediction, "Air-Co-Pred". The paper is interesting in the field of collaborative AI.

Strengths

The paper is well-structured, referencing a well-established research stream in computer vision and collaborative object trajectory prediction. The following are some strengths of the paper: - The research problem is clearly described with good visuals and diagrams, which aid in the explanation. - There are several contributions, including BEV generation, sliding windows for sparse interaction, and Air-Co-Pred simulated dataset - The paper provides both quantitative and qualitative assessments of their framework, compared to baseline and state-of-the-art, such as Who2com and Where2com. It demonstrates good improvements. - The provided appendices are useful for further details and ablation studies.

Weaknesses

The paper has several weaknesses: - It investigates methods to overcome single-drone issues, such as occlusions and blurs; nevertheless, it is also important to discuss a bigger picture of their use cases, including accident prevention and path planning, in greater details with certain limitations. For example, in accident prevention, if multiple drones collaborate and predict an accident is about to happen, what can it do? Does it then communicate/inject commands over the air to the vehicle causing certain actions? - The DHD framework consists of feature extraction, BEV, Sparse Interaction via Sliding Windows, and Trajectory Prediction. The idea of having sliding windows for sparse interaction is quite interesting; nevertheless, relying on a local coordinate system and pixel-level weight fusion can be a weak spot when it comes to real-world settings. It is also important to examine how each of the modules contributes to the overall performance in ablation studies. - The development of Air-Co-Pred remains questionable, based on CARLA. There are indeed many CARLA-based datasets and it is important to compare your dataset with others to prevent overlappings/duplications. - It is vital to discuss the following papers, in relation to the work: 1. Wei, S., Wei, Y., Hu, Y., Lu, Y., Zhong, Y., Chen, S., & Zhang, Y. (2024). Asynchrony-robust collaborative perception via bird's eye view flow. Advances in Neural Information Processing Systems, 36. 2. Lu, Y., Hu, Y., Zhong, Y., Wang, D., Chen, S., & Wang, Y. (2024). An extensible framework for open heterogeneous collaborative perception. arXiv preprint arXiv:2401.13964. 3. Liang, J., Jiang, L., & Hauptmann, A. (2020). Simaug: Learning robust representations from simulation for trajectory prediction. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16 (pp. 275-292). Springer International Publishing.

Questions

There are many questions to address as the following: - With the DHD framework, given objects to track/predict, what is the optimal number of drones that need to be “watching” the object in order to produce the best prediction? Is it always the case that the more drones that have the object in view, the better the trajectory prediction? Why four collaborative drones in Air-Co-Pred? Can we get away with just 2 drones? - Can you explain more about why CARLA is used to simulate and produce the dataset? Has anyone attempted trajectory prediction with multiple drones in real-world settings? Is it feasible? What needs to happen before DHD can be deployed in production in the real-world setting? Maybe it can be included in a discussion section. Is there an optimal drone height/altitude? Is 50 meters the best height value, so it’s used in the dataset? - How often should the drones be communicating/transmitting data to each other? Is it the same as the aerial observation samples being collected (frequency of 2 Hz)? Can it be reduced to further lower transmission data? - How does the DHD framework respond to noises, such as flight turbulence? - What will be the performance if we turn on/off multiple components, such as BEV or Sparse Interaction? This should be included in ablation studies. - How far are we from real-world experiments? - What will be the impacts of your study to the field?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have raised concerns about the use of simulated settings, which limit the research's practicality. It is important to develop real-world scenarios for future validations of the research.

Comment after reading the response

Thanks for the authors’ time and effort in responding to the weaknesses and questions raised. Overall, this revised version is more comprehensive, incorporating some previously missing details and effectively resolving all my concerns. Moreover, the innovations presented in the paper have been fully recognized by reviewers CzWU, nEkT, and rEnT, and partially acknowledged by reviewer nhVA. Consequently, I am convinced to maintain my initial judgment.