Uni3DETR: Unified 3D Detection Transformer

Summary

The authors explore and analyze the existing LiDAR-based 3D object detection framework, and propose to adopt transformer-based method to perform detection from differnt LiDAR inputs. The experimental results on a number of datasets are better compared to the existing methods.

Strengths

1. The task of 3D object detection is very important in the 3D community. Interestingly, the authors propose to adopt transformers to generalize 3D detection across diverse scene inputs. 2. The paper is easy to follow. 3. The authors conduct the experiments on widely-used indoor and outdoor datasets, including SUN RGB-D, ScanNet, S3DIS,m nuScenes, and KITTI.

Weaknesses

1. Missing important SOTA works on KITTI. The manuscript does report validation results on KITTI, however, the authors forgot to compare with LiDAR-only SOTAs 'PV-RCNN++' (PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection). However, it would be much convincing if the authors can compare with this SOTA. 2. Results on KITTI and nuScenes test set. In table 2 & 3, the proposed Uni3DETR is comparable/better with the cited methods. However, it would be much convincing if the authors can report the test set results on KITTI and nuscenes. 3. Computation/memory footprint comparison. The authors didn't make a comparison of their work in terms of memory/speed with the existing 3D detection methods. The time consumption might be large since the memory/time are espeically heavy for the transformer-based methods from my point.

Questions

Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA.

Summary

The authors propose a unified detr-style detector for both the indoor and outdoor 3D object detection. Besides common learnable query, they also adopt unlearnanle query sampled from raw point cloud.

Strengths

The proposed method is simple and easy to follow.

Weaknesses

The novelty is limted, and the experimental results are not fair/convincing.

Questions

1. Uni3DETR follows the design of DETR, which is a ''sparse'' and end-to-end detector. However, Uni3DETR uses 4 groups of query points for indepedent prediction and needs extra pos-procedure to filer redundant boxes. The authors claim that the learnable/unlearnable queries capture the local and global information of the scene. If so many different queries are neccesary to cover the whole scene, why not use a dense detector like pointpillar or centerpoints instead? 2. I am surprised that the results in Tab. 6. I hope the authors could provide more explanation about why multiple groups of queries brings no performance gain? Why this ablation is not conducted on other datasets (e.g. KITTI) like Tab.5 and Tab. 7? 3. Using 4 groups of query points (learnable & unlearnable) as proposed in the paper, or simply use more learnable queries, which one is better? 4. The authors are suggestd to provide results on the KITTI and nuscenes test split. Besides, KITTI provides a more fair and relibale evaluation setting with 40 recall positions. 5. What is the diffence between Uni3DETR and DAB-Deform in Tab. 4. 6. With differenent backbone, model size, inference speed, ect., it is not fair for the comparison with other sota methods in Tab.2 & Tab.3.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The novelty is limited, and more comprehensive analysis is required.

Summary

Uni3DETR is a unified 3D object detector that is capable of handling both indoor and outdoor scenes within the same framework. This is significant as many existing detectors are specialized for either indoor or outdoor environments, but not both. The method employs a detection transformer with a point-voxel interaction mechanism and a mixture of query points. This allows it to exploit both global and local information effectively. Uni3DETR introduces a decoupled Intersection over Union (IoU), which serves as an easy-to-optimize training target for localization. This is aimed at improving the accuracy of the detector. The paper demonstrates that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection tasks. Moreover, it shows strong generalization ability under heterogeneous conditions, which is beneficial for real-world applications.

Strengths

Versatility Across Environments: One of the major strengths of Uni3DETR is its ability to handle both indoor and outdoor environments within the same framework. This is a significant advancement as it eliminates the need for different models for different environments, which is common in existing approaches. Optimization of Localization: The introduction of a decoupled Intersection over Union (IoU) as a training target for localization is a notable strength. This provides an easy-to-optimize objective that can potentially lead to more accurate localization of objects, which is often a challenging aspect of object detection. Empirical Validation: The paper provides empirical evidence showing that Uni3DETR consistently performs well on both indoor and outdoor 3D detection tasks. This is important for establishing the credibility and effectiveness of the proposed approach.

Weaknesses

Innovative Element. The core pipeline appears to be a Deformable DETR complemented by non-learnable queries (voxel features derived from FPS). It essentially resembles a simple combination of two model architectures. Efficiency. The model's efficiency is not entirely at the cutting-edge level. For instance, in nuScenes, several competitive 3D detection techniques are not listed in your tables. (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Lidar). Increase in Precision. According to Table 5, the non-learnable query appears to have a marginal impact on outdoor detection. This aligns with our prior understanding that FPS is not well-suited for outdoor detection. The mixed query's design might be superfluous for outdoor scenarios. Computational Complexity. It is widely recognized that greater computation often results in superior performance. The authors should include a breakdown of the mixed query's complexity, such as inference latency and FLOPS, in comparison to other techniques.

Questions

As evidenced in Table 5, the non-learnable query makes a small contribution to outdoor detection. Can you provide a comparable analysis of the impact of the mixed query points on nuScenes? I'm curious to determine if the performance on nuScenes is primarily driven by your learnable query. Could you supply a comparison of computational complexity and inference latency between your approach and alternative methods?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Integrating two distinct 3D detection algorithms in this method might potentially increase computational complexity. Nonetheless, computation overhead is a key factor for outdoor detection tasks, such as autonomous driving. Given this, the inclusion of a non-learnable query design seems to be superfluous in such outdoor environments.

Summary

The paper proposes Uni3DETR, a unified architecture suitable for indoor and outdoor scenes. Uni3DETR employs the DETR with point-voxel interaction for object prediction. It uses a mixture of query points to exploit global and local information. Finally, Uni3DETR uses a decoupled IoU loss by disentangling the depth and other dimensions to ease out training. Training a per-dataset model and testing on the same dataset improves over the baselines.

Strengths

+ The problem of using a unified architecture for indoor and outdoor point clouds is super relevant. + The idea of employing DETR, a mixture of query points, and decoupling 3D IoU is nice. + The per-dataset results are good.

Weaknesses

- The first claim that the architecture is uniform is only partially correct. L228 says that the authors chose a 0.02m grid size for indoor datasets, while they chose a different voxel size for the KITTI and the nuscenes datasets. The authors should pick a single voxel size and then run it on all the datasets. (It is OK to change the range for nuScenes since it is a multi-camera dataset, but the authors should choose a single voxel size) - The claim of "strong universal ability of our model (L336)" is an overstatement, in my opinion, since the paper only carries out Per-dataset model training and testing. A related paper that trains a single RGB 3D detector is Omni3D + Cube R-CNN [A]. I would urge the authors to follow a similar protocol where the train and test sets are - KITTI + nuScenes front and then test on indoor indoor/outdoor - Train on indoor scenes and then test on outdoor indoor/outdoor - Finally, train on all indoor/outdoor scenes, then test on indoor/outdoor scenes. By training Uni3DETR on KITTI + nuScenes front and testing outdoors, one can compare against the Cube R-CNN model of [A]. - How well does the method generalize in zero-shot settings? E.g., How does the Uni3DETR model trained on outdoor scenes work on an unknown lidar scan from an unknown say, Waymo dataset. - A sad part of 3D detection papers is to use a bigger backbone and show improvements on nuScenes. Please quantitatively compare the flops, model size, training time, and inference time for Uni3DETR against the baselines in Table 3. - The authors say the learnable query points mostly contain local information and fit the outdoor detection well. In contrast, non-learnable query points emphasize global information and are more effective for dense indoor scenes. Another way to enforce local and global information is to use a dual-path architecture where one branch is attention-based for global information while the other is convolution-based for local information. One can later fuse their outputs by concatenation or predicting a coefficient to add them. Why did the authors not consider this design choice? - An alternative to using the decoupled IoU loss is the disentangled loss [B], where one constructs a 3D box from GT parameters except the one in consideration. That way, one can avoid explicitly weighing the losses for respective parameters (e.g., the authors use weighing xy and z terms by 0.5). The authors should, therefore, quantitatively compare the decoupled IoU loss against the disentangled loss. - The details of the 3D feature extractor (Section 2.1) need to be included. - The authors use the mean average precision (mAP) under IoU thresholds of 0.25 and 0.5 for evaluating datasets (L216). Using one or two thresholds is quite sensitive. A better alternative is to use exact IoU3D over a series of thresholds of 0.05:0.5:0.05 and then average them as in [A]. Minor: - Do the authors plan to release the code? - Typo: L203: final References: A. Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild, Brazil, Kumar, Ravi, et al., CVPR 2023. B. MonoDIS: Disentangling monocular 3D object detection: From single to multi-class recognition, Simonelli, Bulo, Porzi, et al., PAMI 2020.

Questions

Please see the weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors do not list out the limitations.

Summary

This paper proposes Uni3DETR, a unified 3D detection transformer that addresses indoor and outdoor 3D detection within the same framework. The paper provides a specific analysis of the inconsistency in the structure of current indoor and outdoor scene detection models. Due to the differences in data distribution between indoor and outdoor environments, indoor datasets have a smaller range of point clouds, with objects being closer together and denser, occupying a larger portion of the point cloud. On the other hand, outdoor scenes have smaller and sparser objects, with background point clouds dominating the overall point cloud. Utilizing the mixture of query points which consist of the learnable and non-learnable query, the detector is able to exploit the global information for dense small-range indoor scenes and local information for large-range sparse outdoor scenes. Besides, decoupled IoU is proposed for easier and faster training target localization optimization by disentangling the xy and z space.

Strengths

1. The paper provides a detailed analysis of the inconsistent models of indoor and outdoor 3D detectors, with the difference in data distribution being the most important reason. The author explains how to design a unified 3D detector and solves the problem of inconsistent architectures between indoor and outdoor scenes. 2. The overall logic of the paper is clear, and it is well written. 3. Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions while maintaining caomparable performance.

Weaknesses

1. The experiments of model architectures are a bit insufficient. In line 99, the paper mentions, "Then, we convert the extracted sparse features into dense ones and apply 3D dense convolutions for further feature processing." However, no experiments show the performance gain brought by 3D dense convolution to support the claim. Furthermore, will the model architectures vary from datasets? For example, are the numbers of layers different for models trained on KITTI and S3DIS? 2. The author claims that learnable queries primarily capture the local information of the scene, and non-learnable queries usually concentrate more on global information. Although experiments show that combining different queries could bring better performance, there is no evidence in the paper (e.g., visualization or other analysis) to support the claim.

Questions

1. Decoupled IoU is only used in Uni3DETR, and it can be seen from the ablation study that this loss has a significant performance improvement for the detector. However, Decoupled IoU loss was not used in other mothods when compared to other methods. Can Decoupled IoU improve other detectors, such as GroupFree3D, and also bring significant performance improvements? 2. The learnable query mentioned in the paper lacks its initialization process, such as how to obtain $P_q$ and $C_q$ mentioned in line 107. 3. How about the generalization ability of feature extraction based on this structure for indoor scenes? For example, how it performs zero-shot testing by training on SUN RGBD and directly testing the trained model on ScanNet? 5. The author uses a mixture of query points to obtain detection results through different categories and a sufficient number of queries. Will this result in noticeable time consumption during inference? 6. It would be better to illustrate more visualization results to support the claim that the learnable and non-learnable queries have different capabilities. 7. According to the paper: ConQueR: Query Contrast Voxel-DETR for 3D Object Detection: DETRs usually adopt a larger number of queries than GTs (e.g., 300 queries v.s. ∼40 objects in Waymo) in a scene, which inevitably incur many false positives during inference. Could this problem occur in Uni3DETR since many more queries are used in inference?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

It seems that the paper now provides a unified paradigm for solving 3D object detection tasks of indoor and out door scenes and do not use one model for testing on all the datasets. Furthermore, the model is separately trained with different hyper-parameters on different datasets.

Response to authors

Thank you, authors, for putting an excellent rebuttal. The benefits of your architecture are not fully visible if the experiments are not cross datasets/cross domains. - I understand that scaling to datasets with different voxel sizes and domains is an issue with lidar-based 3D detection. Why do not we keep a single voxel size and do a quantitative cross-dataset evaluation with the Omni3D metric over all combinations of dataset **within** the indoor and outdoor domains for Uni3DETR and your baselines? (such as KITTI --> NuScenes, nuScenes --> KITTI, ScanNet --> SUN, SUN --> ScanNet) In other words, please compare with Cube R-CNN table 5. The comparison will let us know how good the within-domain generalization is keeping the voxel size unchanged. - If possible, please also quantitatively report the following results with a single voxel size. Train on indoor and test outdoor for Uni3DETR, your lidar baselines, and Cube R-CNN with the Omni3D metric. Train on outdoor and test indoor for Uni3DETR, your lidar baselines, and Cube R-CNN with the Omni3D metric. - Your Omni3D numbers in Table 2-2 of the rebuttal differ from Table 5 of the Omni3D paper. They report AP on KITTI as 42.5, while you report as 15.0. Why is it so? - It would also be good to quantitatively compare the flops, model size, training time, and inference time for Uni3DETR against the baselines in Table 3. Please note that it is completely OK if the Uni3DETR numbers on some of the experiments are bad. We are not here to compete on benchmarks and penalize for the bad results. All we want is to clearly understand the limitations of the methods and advance science :)

Response to authors

Thanks, the response addresses my concerns.

Authors have addressed most of my concerns.

Thanks for the answers and clarification in the rebuttal, which covered most of my concerns.