Thank you for reading our rebuttal! We are more than happy to share our thoughts on your questions.
> Other than Figure 3, do you have some intuitive explanations for this?
>
Our approach is largely inspired by recent advancements in fine-grained control for style transfer using SDXL UNet [1,2,3,4]. We hypothesize that the blocks processing low-resolution image feature maps—particularly those near the bottleneck in the UNet—exert more control over the overall coarse structure or core formulation of the image. In contrast, the blocks processing high-resolution image feature maps play a crucial role in controlling high-frequency details, a task that is difficult for low-resolution blocks to manage.
Regarding why the first upsampling block influences the spatial layout the most, rather than a middle or final downsampling block, we conjecture it is because the other blocks are not yet sufficiently close to the final output image to have a significant impact. We conducted several ablation studies and found empirically that excluding the first upsampling block is the most effective in generating more diverse poses.
> And has such idea of excluding the first upsampling block explored in previous works?
>
We are indeed inspired by a series of works that have demonstrated how different blocks of the SDXL UNet affect the final image generation. These works include, but are not limited to, InstantStyle [1], B-Lora [2], and discussions within the open-source community [4]. Most of these studies [1, 3, 4] focus on injecting style from a reference image and investigating which blocks most influence style generation. Meanwhile, B-Lora also analyzes which blocks have the greatest impact on content generation. In particular, B-Lora identifies the first upsampling block as the most influential for content generation, significantly controlling the spatial layout—a finding that aligns with our observation.
However, these studies are primarily concerned with style transfer for text-to-image models. The key difference is that they aim to preserve the spatial layout or style from the reference image as much as possible, while we seek to exclude the spatial layout information from the reference image as much as possible. **Regarding consistent generation, we have not yet observed any work that uses this method to diversify object poses.** Previous consistent generation method Consistory [5] introduced several techniques to avoid the issue of similar poses, such as self-attention dropout and using vanilla query features. We believe that our approach of excluding the first upsampling block offers a simpler and more principled solution in practice.
> Do you have some quantitative analysis to prove that removing the first upsampling block is more effective than removing other blocks?
>
We also include the quantitative results here. Specifically, we conducted 11 groups of experiments, where in each group, we excluded one of the 11 blocks of the UNet from applying Reference Feature Guidance. And in each group, we generated 20 sets of consistent objects, with each set containing 5 images. This resulted in a total of $11\times 20 \times 5 = 1100$ images for metric calculation. We then used `DreamSim` and `LPIPS` to measure the distance between the generated images and the reference image, and report the mean and standard deviation. Higher values of these metrics should indicate more diverse poses, as these scores tend to favor similar layouts [5]. From the table, we can see that excluding the first upsampling block greatly boosts the spatial layout diversity.
| Excluded block | DreamSim distance | LPIPS distance |
| --- | --- | --- |
| Down1 | 0.2283 ± 0.0102 | 0.5484 ± 0.0052 |
| Down2 | 0.2261 ± 0.0118 | 0.5484 ± 0.0043 |
| Down3 | 0.2298 ± 0.0082 | 0.5475 ± 0.0058 |
| Down4 | 0.2332 ± 0.0095 | 0.5551 ± 0.0072 |
| Mid | 0.2484 ± 0.0048 | 0.5673 ± 0.0070 |
| Up1 | **0.3077 ± 0.0092** | **0.5880 ± 0.0047** |
| Up2 | 0.2567 ± 0.0087 | 0.5603 ± 0.0077 |
| Up3 | 0.2440 ± 0.0088 | 0.5580 ± 0.0032 |
| Up4 | 0.2441 ± 0.0079 | 0.5611 ± 0.0074 |
| Up5 | 0.2368 ± 0.0106 | 0.5540 ± 0.0055 |
| Up6 | 0.2455 ± 0.0103 | 0.5639 ± 0.0075 |
**Reference**
[1] Wang, Haofan, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. "Instantstyle: Free lunch towards style-preserving in text-to-image generation." *arXiv preprint arXiv:2404.02733* (2024).
[2] Frenkel, Yarden, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. "Implicit Style-Content Separation using B-LoRA." *arXiv preprint arXiv:2403.14572* (2024).
[3] Jeong, Jaeseok, Junho Kim, Yunjey Choi, Gayoung Lee, and Youngjung Uh. "Visual Style Prompting with Swapping Self-Attention." *arXiv preprint arXiv:2402.12974* (2024).
[4] https://github.com/huggingface/diffusers/discussions/7534
[5] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consistent text-to-image generation. arXiv preprint arXiv:2402.03286, 2024.