SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

Summary

The paper introduces SyntheOcc, a framework utilizing diffusion models to synthesize photorealistic images for autonomous driving simulations. The proposed method addresses limitations in the existing 2D diffusion model to generate multi-view driving videos by integrating detailed 3D geometric data. The authors effectively employ 3D semantic multi-plane images (MPIs) for precise geometric control, enhancing the realism and utility of generated images for training perception models. The paper also proposes re-weighting strategies to address the imbalance problem between foreground, background, and object categories. The experiments prove the effectiveness of the proposed MPI encoder and the reweighting strategies.

Strengths

- The paper introduces an innovative approach by incorporating 3D semantic Multi-Plane Images (MPIs) to capture both geometric and semantic details of a scene. This approach allows for the precise modeling of 3D environments in a 2D image synthesis context, enhancing the photorealism and depth accuracy of the generated images. - The design of the MPI encoder is very effective in handling the input conditions with a large number of channels while maintaining spatial consistency to the latent features of diffusion UNet. - Additionally, SyntheOcc incorporates sophisticated reweighing strategies to address class imbalance and ensure focus on critical features. These include foreground enhancement, depth-aware reweighing, and class-balanced reweighing. - The paper outlines a comprehensive set of evaluations to demonstrate the effectiveness of the proposed method. Qualitative evaluations visually demonstrate the photorealism and environmental accuracy of the generated images compared to real scenes from the nuScenes dataset. Quantitative analyses leverage metrics such as Frechet Inception Distance (FID) to measure image quality and evaluate perception model performance, offering solid empirical evidence of the framework's effectiveness. Ablation studies further dissect the impact of various components and design choices in the proposed method. Additional robustness tests are conducted to evaluate how changes in the MPI settings (like variations in depth or semantic labeling) affect the output quality and the training effectiveness of perception models.

Weaknesses

The contributions for reweighing strategies seem to be minor improvements over existing methods (Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong, Zhenguo Li, and Dit-Yan Yeung. Integrating geometric control into text-to-image diffusion models for high-quality detection data generation via text prompt. arXiv preprint arXiv:2306.04607, 2023, Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection (arXiv preprint arXiv:1908.09492), which limits the perceived novelty of the paper's contributions. The paper focuses on scene editing capabilities, but there is a noticeable underrepresentation of object-level editing in the experiments. Magic drive’s data augmentation is evaluated on two perception tasks BEV segmentation and 3D object detection, with CVT (Zhou & Krahenbuhl , 2022) and BEVFusion (Liu et al., 2023a) as perception models, respectively. Hence, evaluations on same downstream tasks are encouraged for better comparisons to the state-of-the-art baseline. The paper doesn't provide any evaluations comparing the re-weighing solution proposed by GeoDiffusion. There are also several noticeable view inconsistencies, eg: Fig 14 - row 2 column 2-3 (clouds seem different), row 7 column 4-5 there is a mismatch in building structures, which are not discussed in the paper.

Questions

I do not have any further questions. I would like the reviewers to comment on the weaknesses that I listed in the previous paragraph.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors acknowledge some key limitations in the proposed method. First, it relies heavily on existing data for generating scenes, which means it doesn’t create as much variety as it could. This limits how well it can train models to handle different driving conditions. The paper also struggles with complex scenes, like crowds, where it fails to accurately identify individual people. This is a big deal for autonomous driving, where accurate representations of the scene are crucial for predictions. The authors suggest that future improvements could include better methods for creating diverse scenes and making the model more capable of handling dynamic environments, which would help make the system more practical and effective for real-world applications.

Summary

This paper introduces SytheOcc, a method that employs a diffusion model with 3D occupancy as conditions to generate street view images.

Strengths

1. Unlike previous methods that use box conditions, this paper proposes the use of 3D occupancy, resulting in finer geometric control ability. 2. The paper further suggests the use of 3D semantic multi-plane images to represent the 3D occupancy. 3. The text and figures are well-presented, and the provided examples are very promising.

Weaknesses

1. The main concern is the inconsistency between views and frames. Despite using Cross-View and Cross-Frame Attention and 3D occupancy as conditions, the spatial and temporal consistency results are unsatisfactory (e.g., Fig 5 (b) and the video demos). This is not the expected outcome, as incorporating 3D occupancy as a consistent world representation should result in better spatial and temporal consistency. Additionally, it would be preferable to have metric results such as FVD for the temporal experiments. 2. In Table 1, why does SytheOcc-Aug show worse results for certain categories (e.g., bicycle, moto)? 3. Table 1 lacks experiments for ControlNet-Aug or ControlNet+depth-aug. 4. Some discussions regarding 3D occupancy as a 3D geometry condition: 4.1. The field of view (FOV) for 3D occupancy is limited as it is generally generated using lidar, which leads to inconsistency issues for high-rise buildings when considering cameras of larger FOVs. 4.2. The current annotation of 3D occupancy has limited category coverage. It would be beneficial to explore open-vocabulary approaches. 4.3. When using only 2D semantic masks as conditions, the paper mentions the presence of ambiguity (i.e., Fig 6 a0). Can the use of instance-level semantic masks alleviate this problem?

Questions

See Weaknesses. Overall, I have a positive view of the paper and am willing to increase the score if the authors can address my concerns.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have provided a comprehensive list of limitations and future work of their paper.

Summary

The paper propose a new 3D semantic multi-plane images (MPIs) based image generation pipeline, which enables finer geometric control for 3D editing, dataset generation, and long-tailed scene generation. Through extensive experiments, the work demonstrates substantial advancement in generation quality and better alignment between condition and synthesized images.

Strengths

1. The work explores a new 3D semantic Multi-Plane Images (MPIs) as a condition, which provides better spatial alignment compared with baselines and enables 3D editing. 2. The comparison results are comprehensive and demonstrate the effectiveness the proposed method, the ablation is relatively complete to validate the MPI Encoder and the reweighting strategy. 3. The paper is well-written, and the experimental results are presented clearly.

Weaknesses

1. The MPI encoder, which is the major contribution, is not novel for me. Although the proposed 3D MPI enables finer control than BEVGen, but the diffusion model also operates on the 2D domain and generates each view and frame separately without strict geometry constraints. 2. The importance of reweighing is tricky and hard to tune, considering many hyperparameters. Are the m and n in Eq. 6 the same for different datasets? 3. It’s hard to decide how the method works without a supplementary video, I doubt the view-consistency of the generated video across frames and views.

Questions

1. How are the hyperparameters of the “progressive foreground enhancement” determined? Is it calculated using the distribution of different categories? 2. Will the appearance of the same objects change across different frames and views? Because the MPI encoder only provides semantic layout, not texture features, and there is no strict geometry constraint.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

None

Summary

In this paper, the authors propose a new controllable diffusion-based image generation method named SyntheOcc, which takes an occupancy map as input and generates camera images. SyntheOcc enables the application of scene editing and long-tail corner case generation and shows a strong capability of data augmentation for autonomous driving systems.

Strengths

1. Compared with previous controllable image generation methods for traffic scenarios like Panacea or MagicDrive, the occupancy map contains more 3D spatial information than the BEV layout. 2. The paper is well-organized and easy to follow. 3. The extensive experimental results demonstrate the effectiveness of the proposed data generation pipeline.

Weaknesses

1. The control signal in Panacea or MagicDrive is BEV layout, which only contains lanes and foreground objects and is more easily acquired than occupancy. However, the SyntheOcc relies on sophisticated collected occupancy.

Questions

It's interesting to note that diffusion models typically don't generate identical images in repeated runes, even when given the same control inputs. However, in Figure 1, we see that the generated images corresponding to both the original and edited occupancy maps show remarkably similar street structures. This raises an important question: Could this consistency be a result of overfitting to the nuScenes dataset? Diffusion models are known for their variability, but this particular case shows an unusual level of structural similarity. It's worth considering whether the model has perhaps learned the specific patterns of the nuScenes data too closely, leading to this unexpected consistency in output. This observation could have implications for the model's generalizability and its performance on diverse, real-world scenarios outside the training dataset.

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The proposed SyntheOcc faces challenges in real-world application scenarios. For instance, to generate planning-level long-tail corner cases, other methods like Panacea or MagicDrive simply require editing the object's trajectory. However, SyntheOcc demands not only inputting the background occupancy but also constructing a pseudo occupancy for the foreground object. This raises the question of whether using occupancy as a control signal is an advantage or a disadvantage. The crux of the issue lies in the complexity of this method compared to alternatives. While other approaches need some adjustments to object trajectories, this technique necessitates providing comprehensive background and foreground occupancy data. This additional overhead prompts us to ponder whether occupancy-based control offers tangible benefits or introduces unnecessary complications.