Dear Reviewer 3GaN, thank you very much for your further feedback. It is glad to know part of the concerns have been addressed. As for the remaining concern about whether the strength of our method comes largely from FreestyleNet instead of our proposed two processing techniques, we would like to provide some further clarifications here.
- First, directly using the synthetic pairs from FreestyleNet *without any processing* **will not bring any improvement** to the challenging fully-supervised real-image baseline. This has been supported by Table 5 in our main paper. We borrow the results below (the real-only result is borrowed from main Table 3 with SegFormer-B4). The inferior result of blindly integrating synthetic images (FreestyleNet, -0.2 mIoU) also explains why existing works [1, 2] mostly only address the few-shot scenario, which contains a few real images and is easy to boost. In comparison, with our filtering and hardness-aware synthesis strategies, we can significantly boost the challenging but practical fully-supervised scenario (**+1.8 mIoU**).
| | Real data | Synthetic data *w/o processing* | Synthetic data *w/ processing* | validation mIoU |
:-: | :-: | :-:| :-:| :-:
| Real only | ✔ | | | 48.5 |
| FreestyleNet | ✔ | ✔ | | 48.2 (-0.3) |
| **Ours** | ✔ | | ✔ | **50.3 (+1.8)** |
- Second, we have further validated the necessity and superiority of our two strategies in Table 6 of our main paper. Results are borrowed below. We observe that, *without our proposed techniques*, scaling up the number of synthetic images *does not* yield consistent improvement, even gradually downgrading the performance. In addition, our proposed method tremendously outperforms the plain FreestyleNet which does not include any processing techniques. The performance gap can be as large as **+5.0 mIoU**.
| $N_{\max}$ (scaling ratio) | 6 | 10 | 20 |
:-: | :-: | :-:| :-:
| w/o processing (FreestyleNet) | 43.7 | 43.6 | 43.3 |
| **w/ processing (Ours)** | 47.2 | 47.7 | **48.3** |
| Superiority over FreestyleNet | +3.5 | +4.1 | **+5.0** |
- Lastly, as we provided in the [rebuttal](https://openreview.net/forum?id=XOotfgPiUF¬eId=k6Zw1ZF8r9), our hardness-aware synthesis strategy is not trivial. It is much better than online hard example mining (OHEM). We also borrow the comparisons below (44.2 *vs.* 45.4). As for your opinion that similar filtering strategies also exist in label-efficient learning, we agree with this. Indeed, this is a very common motivation. Similar practices also exist in noisy label learning. However, as we clarified in the global response, *we use similar motivations for totally different purposes*. We aim to recognize the synthesis failure cases, which is proved fundamental to the success of later perception task. The necessity of "cleaning" synthetic images is almost completely ignored in previous works [3] that utilize synthetic images to benefit perception tasks. *We believe that, drawing attention to "more effectively" learning from synthetic images, rather than always focusing on better synthesis quality, is also our contribution to the community.*
|Baseline|+ OHEM (thresh=0.7, min kept=100K)|+ Our hardness-aware synthesis
|:-:|:-:|:-:|
|44.0|44.2|**45.4**|
Please tell us if you have any further concerns about this response. We are more than willing to provide any further explanations. Thank you very much for your precious time and great contributions.
[1] He, Ruifei, et al. "Is synthetic data from generative models ready for image recognition?." *ICLR* 2023.
[2] Wu, Weijia, et al. "Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models." *ICCV* 2023.
[3] Azizi, Shekoofeh, et al. "Synthetic data from diffusion models improves imagenet classification." *ICCV* 2023.