ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models

Summary

This paper tackles the task of controllable image generation from bounding boxes and text prompts for training object detectors. To fine-tune diffusion models to specific domains effectively and deal with the challenge of concept bleeding, this paper proposes a method called ODGEN. In the proposed method, the diffusion model is first fine-tuned on both entire images and cropped foreground regions. Then, text lists and image lists are utilized as input for ControlNet to avoid the concept bleeding and generate high-quality images. The experimental results show that the proposed method outperforms the state-of-the-art methods for controllable image generation on seven types of datasets in terms of FID score. In addition, the object detectors trained on the generated images from the proposed method achieve better performance than other methods.

Strengths

i) The proposed method is quite simple yet effective. Its relative ease of implementation is helpful for computer vision practitioners and future researchers. Also, the task tackled in this paper is of practical benefit. ii) This paper is well-written and easy to follow. The motivation for tackling the task is clearly described in Sec. 1, and related works are well-summarized in Sec. 2. The proposed method is clearly explained at both the idea level and procedure level in Sec. 3. iii) The experiments demonstrate the high performance of the proposed method. The proposed method achieved better FID scores than other controllable image generation methods on seven types of object detection datasets. Additionally, training with synthetic images from the proposed method improved the performance of YOLO-based object detectors significantly.

Weaknesses

My concerns are mainly about the lack of detailed analysis: I) This paper claims that fine-tuning with both cropped foreground regions and entire images is one of its contributions. However, an ablation study to confirm the effectiveness of this approach is missing. ii) From Fig. 4, it looks like the corrupted label filtering is performed only for the proposed method (ODGEN). Because the corrupted label filtering is a simple technique and can be applied to other methods as well, the comparisons between ODGEN and other methods not only without filtering but also with filtering should be provided for fair comparisons. iii) This paper provides only the results with 200 real images. I'm curious to know how the performance of the proposed method and other methods change when more number of real images are available for training. iv) In Sec. 1, the concept bleeding is raised as one of the challenges in this task. Although Fig. 6 provides an example, I would like to see a quantitative evaluation to check whether it was effectively addressed by the proposed method.

Questions

See iii) in Weakness. How is the performance of the proposed method and other methods when more real images are available?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are adequately discussed in Sec. A.

Summary

This work is about generating synthetic (annotated) data for object detection with diffusion models that are tuned for a specific image domain. Generating synthetic annotated data can be useful for model training in situations where data is scarce. The authors propose a specific pipeline and new modules to generate such data. First, an off-the-shelf diffusion model is fine-tuned on a specific domain, for instance a specifc object detection dataset. This is done both with the full images and crops of foreground objects (via the given bounding boxes). The statistics of object categories, number of instances, bounding boxes are estimated. Then, a new layout is sampled from the distribution, the class-specific objects are individually synethesized with the tuned diffusion model. These are individually placed at their sampled bounding box location to build a so-called image list. Then, a ControlNet model is used that encodes this image list along with a text list of the object categories, which outputs the final synthesized image. The newly designed text and image encoders for the controlnet are trained. Finally, a verification step is done at the object level (classifying if the generated image contains the desired object inside a certain bounding box). The synthesized data is evaluated with the FID metric, and also used to train object detectors, which are then evaluated with mAP.

Strengths

- The problem the paper tries to address is important and has several use cases, maybe even beyond what the paper lists. - I especially like the adaptation to specific domains, which is certainly relevant for many real-world use cases. - I enjoyed reading the paper, it's well written, has a good flow, it was easy to understand. - The results seem to be clearly better than prior works like GLIGEN.

Weaknesses

- Positioning with respect to some prior work. - In the related work section, the paragraphs on layout-to-image generation & dataset synthesis contain some highly related works (like InstanceDiffusion or others listed as the second group of data synthesis approaches). However, these works are only briefly described but not contrasted against the proposed method. So, what's the difference to InstanceDiffusion, for example? - Why are there no comparisons to InstanceDiffusion in the experiments, or even simpler data augmentation techniques like "copy-paste"? - The object distributions are simplified compared to real distributions. - Using a Gaussian distribution for a discrete variable seems odd. There will be some probability to sample a negative number and if you clip, you ultimately end up with a different distribution than what was estimated, I guess. - It seems that all estimated distributions regarding the bounding boxes are independent. I assume that location and areas are dependent. Are the distributions modeled independently on purpose? As in, is it better for simulation to have independent distributions? Or is this just done for simplicity? And was the impact of this evaluated empirically? - Missing details on the usefulness of synthetic data - For the domain specific domain, it seems that the diffusion models were also tuned only with the 200 selected domain-specific images (according to line 212). I found this information to be crucial in judging whether or not the experiments are valid. I suggest to highlight this aspect already in Section 4.1 - and maybe also discuss why this information is curcial. My thinking here is that if you used more data for tuning the diffusion model than you use to train the detectors, it would be unclear where performance gains come from. It could just come from the additional data used to train the diffusion model, rather than the images being synthetic. - Also, does the same conclusion on object detector performance gains from Table 2 also hold when scaling up real and synthetic images? From a practical point of view, annotating 800 more images with bounding boxes would likely be affordable in most cases. So, I'm missing an experiment like in Table 2, but with more images for both real and synthetic images, e.g., 1000 and 25000 real and synthetic images. - For the general domain, it's great to see in Table 3 that the detector is better when trained on ODGEN data compared to any other synthetic data. However, the real reference points should be (a) the 10k real images and (b) the combination of 10k real and K synthetic images.

Questions

- When you fine-tune the diffusion model with driving data, don't you have issues with limited diversity? I understand that you might only be interested in the same label spaces as defined in the dataset. But for real applications, one is often interested in simulating something that is missing in a dataset. For instance, a driving dataset may contain lots of regular cars, vans and trucks. But you also want to generate emergency vehicles like police cars or ambulances. - Does the filtering rate of the "corrupted label filtering" step correlate with object occlusions? As in, are there more issues (during either the generation or even the classification/filtering) for occluded/overlapping objects? - An interesting application of this paper could be to improve language-based object detectors like in [A]. This work relies on GLIGEN and could benefit from ODGEN. The difference would be that the general semantic knowledge of the diffusion model would be leveraged. References: - [A] Generating Enhanced Negatives for Training Language-Based Object Detectors. Zhao et al. CVPR'24

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations are adequately discussed in the appendix.

Summary

ODGEN uses a diffusion-based generation model to create novel images to train object detectors. Object bounding boxes along with the object's textual description are given as a conditioning for the generation step. With these generated images they can improve the detector's performance.

Strengths

The proposed generation pipeline can be used to enrich the training dataset with more high-quality images. Since multiple objects can exist in the same image, using bounding boxes along with the textual description seems to be the better approach. The paper is well written and the evaluation is done with standard datasets. The improvement over the baselines shows the efficacy of the approach.

Weaknesses

1. There is a high generation cost when there are multiple objects in the same image, which reduces the practicality of the proposed approach. 2. It would be interesting to see if the generated images improve the detector's performance where it originally failed due to lack of data, such as when objects are partially occluded. 3. How are the statistics of the object layout taken? 4. Why only 5k images were generated? Can we see some study of gradually adding synthetic data, like 1k,5k,10k,20k, and some plateauing of the detector performance? 4. It will be only fair to compare with other generation approaches with the same post-generation filtering being applied to them. How does ODGEN compare then? 4. In line 111, it is misleading to say "a new method to fine-tune the diffusion model", as it is nothing more than just fine-tuning.

Questions

Please refer to the weakness section.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the weakness section.

Summary

This paper proposes the ODGEN method to generate high-quality images conditioned on bounding boxes. ODGEN fine-tunes Stable Diffusion (SD) on domain-specific datasets to enhance image quality in specialist domains, designing a novel strategy to control SD with object-wise text prompts and synthetic visual conditions to alleviate 'concept bleeding'. Experimental results on the COCO-2014 dataset demonstrate that ODGEN surpasses other methods in control capability. Additionally, the authors design a dataset synthesis pipeline using ODGEN, showing that using additional generated training data can improve the performance of state-of-the-art (SOTA) object detectors.

Strengths

+ The ODGEN method proposed in this paper outperforms previous methods and shows significant improvements. + This paper is well organized and written. The overall paper is easy to follow.

Weaknesses

There are some unclear implementation details, and a few experiments are missing. Please refer the Questions.

Questions

- When fine-tuning the pre-trained diffusion model, the authors use not only the entire images from the dataset but also the crops of foreground objects (i.e., resized to 512 x 512). I'm curious if this approach would help in generating particularly small objects. - During actual inference, what is the visual relationship between the image list $c_{il}$ (i.e., in Eqn. 2) and the final generated image? The authors should provide some visual examples to illustrate this. - The authors mention that the object-wise conditioning used in the paper can alleviate "concept bleeding" in multi-instance generation. I believe it is necessary to conduct experiments on the COCO-MIG benchmark [65], which assesses the model's ability to control both positioning and attributes, to elaborate on the attribute binding capability of the ODGEN. [65] Migc: Multi-instance generation controller for text-to-image synthesis, CVPR'24 - Compared to GLIGEN and MIGC, ODGEN achieves better results in overlapping cases. Which module's design primarily contributes to this improvement? As this is a major issue in the industry, I hope the authors can emphasize this in the methods section. - Have the authors considered open-sourcing the code and model? I believe this would greatly benefit the community.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- The proposed method requires fine-tuning the entire Stable Diffusion model and training an additional ControlNet. This often requires significant computational resources, making reproduction challenging. - I hope the authors can open-source the corresponding model and code, as this would greatly benefit the community.

Summary

It proposes a novel data synthesis pipeline, ODGEN, which uses diffusion models to generate high-quality and controllable datasets for object detection. They first fine-tune the pre-trained diffusion models on both cropped foreground objects and entire images. Next, they control the diffusion model using synthesized visual prompts with spatial constraints and object-wise textual descriptions. For each image, the object number per category, bounding box location and size are sampled from estimated normal distributions. The sampled values are used to generate text lists, image lists, and global text prompts which are fed to ODGEN to generate the image.

Strengths

The method addresses a practical problem in machine learning, and new effective techniques are employed in the application of diffusion models to achieve significant better synthesis quality than existing approaches. For example, two-step encoding (CLIP text tokenizer -> stacking -> encoder) of the textual condition enables the ControlNet to capture the information of each object with separate encoding and alleviates the concept bleeding problem of multiple categories. Also, encoding individual image lists rather than pasting all objects on a single image can effectively avoid the influence of object occlusions. The experimental validation is thorough, covering multiple benchmarks and providing detailed comparisons with baseline methods. Ablation analysis is insightful.

Weaknesses

As mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed. For the data scarcity experiment, authors only sample 200 images as the training set for all datasets, which is interesting but not sufficient for the analysis. More extensive evaluation can be done with higher ratios of real:synthetic data for datasets like ApexGame, Robomaster, and Underwater (and even larger datasets like COCO) . Restricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this. ODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper. There are no comparisons provided with the fully supervised approach (models trained on complete real training set) on representative datasets. This would help evaluate the trainability of the proposed approach over data augmentation techniques such as copy-paste. The computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.

Questions

As mentioned in section D2, the encoder architecture (used for image and text embedding) differs for different datasets, which potentially limits the generalizability of the approach. This should be discussed. Restricting real images to 200 might lead to sub-optimal “Domain-specific Diffusion Model Fine-tuning”, which maybe the reason why “synthetic only” version in Table 7 doesn’t perform very well. Kindly comment on this. ODGEN’s and RICO have similar FID score on COCO (while the mAP is quite different), which could be discussed/explained in the paper. The computational complexity (and training time) of the proposed approach should be discussed and compared with existing methods.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Supplemental: synthesizing overlapping objects

Compared with other cases, cases with overlapping objects are more complex and challenging to synthesize. As shown by the visualized samples provided in Fig.5, 10, 11, and 12 in our paper, our approach has made great progress in generating overlapping objects and outperforms prior works on layout-image consistency.