Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation

Summary

The paper introduces Stable-Pose, a novel adapter model designed to enhance controllable T2I diffusion models by improving pose guidance, particularly in complex human pose conditions. Utilizing a coarse-to-fine attention masking strategy within a Vision Transformer , Stable-Pose effectively refines pose representation and improves performance over existing methods on the LAION-Human dataset.

Strengths

The paper proposes an adapter model using a coarse-to-fine attention masking strategy within a Vision Transformer (ViT) to significantly enhance pose guidance for text-to-image (T2I) diffusion models. This approach not only refines pose representation but also demonstrates performance improvement over existing methods on the LAION-Human dataset. Additionally, the comprehensive evaluation across multiple datasets underscore the robustness, generalizability, and collaborative potential of the proposed method.

Weaknesses

1. although this method shows good results in condition-following ability, but it has performance drop on image quality and text alignment. 2. compare to controlnet and other pose-guided t2i models, there is no significant contribution both in training recipe and architecture.

Questions

1. It is interesting to see when pose-mask guidance strength is larger than 7, the AP for pose actually got dropped. 2. for all the examples shown in the paper, are they using the same seed for each of the method? (seed is important)

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Stable-Pose has slightly longer inference times due to its use of self-attention mechanisms within the Vision Transformer (ViT). It has also not been thoroughly evaluated under various conditions, such as edge maps, leaving some robustness aspects unexplored. Further testing is needed to confirm its reliability and effectiveness across diverse scenarios.

Summary

The authors introduce a novel approach for pose-guided human image generation by developing a new pose encoder block. This block incorporates a coarse-to-fine attention masking strategy within a Vision Transformer architecture, leveraging the self-attention mechanism of Vision Transformers to analyze the interconnections among various anatomical parts of human pose skeletons. Furthermore, their loss function is specifically designed to place greater emphasis on the pose region. The model's performance is assessed both qualitatively and quantitatively, demonstrating promising results.

Strengths

The paper is well-written and has a clear motivation. It offers an interesting method for pose-guided human image generation. The innovative aspect of the paper is how it encodes pose information into the latent vector during the denoising process, using the PMSA ViT block. The design of this block is novel and seems effective, as demonstrated by the ablation study in Table 4 and the attention maps shown in Figure A.2. The qualitative and quantitative results look promising, and the ablation study appears thorough. Overall, this paper is solid and could be considered for acceptance.

Weaknesses

The coarse-to-fine approach does not appear promising based on the ablation study. The improvement from constant variation is negligible. Limitations and failure cases are not adequately discussed.

Questions

The claim in lines 34-35, stating that pose-guided image generation requires paired human data during training, is not entirely accurate. Previous works, such as [1], [2], [3] etc., do not require paired information and can be trained in unpaired and unsupervised settings, though they do need appearance information. This requirement is not necessarily a disadvantage, as it suits applications like animation from a single image. It would be beneficial to revise this section to reflect these nuances and include the relevant citations. How do the results appear for extreme poses, such as bending the upper body backward? [1] A variational u-net for conditional appearance and shape generation. CVPR 2018 [2] Learning realistic human reposing using cyclic self-supervision with 3d shape, pose, and appearance consistency. ICCV 2021 [3] Towards purely unsupervised disentanglement of appearance and shape for person images generation. 2020

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are not adequately discussed.

Summary

The authors proposed a pose guided image generation pipeline. By introducing the PMSA ViT as pose encoder, the pipeline can generate more controllable images with corresponding 2D pose conditions.

Strengths

- According to Fig 4, the proposed method is the only one which distinguish the front and back side of the 2D pose condition, which is much better than other SOTAs. - The generated images follows the pose condition better than other methods.

Weaknesses

- The proposed method is more like an incremental work of previous methods instead of a new innovative pipeline. - Although the generated images follow the pose guidance better, the image quality(FID) and T2I Alignment(CLIP Score) are worse than those SOTA methods.

Questions

- Could the authors discuss more about why image quality(FID) and T2I Alignment(CLIP Score) are worse than those SOTA methods? - Why the proposed methods are better at distinguish the left and right of the pose condition? During PSMA ViT, it seems like the authors treated all the keypoints the same, without any special design to improve the left-right consistency.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitation has been discussed in the paper.

Summary

To obtain more refined pose condition control, especially for challenging condition (pose & text) generation, this paper designs a coarse-to-fine Pose-Masked Self-Attention (PMSA) module with the use of pose masks, fine-tunes Stable Diffusion with pose-mask guided loss, and ultimately achieves more precise control with little sacrifice of visual generation quality.

Strengths

- The paper shows a good accurate control effect consistent with the pose and text guidance. - The method makes sense because pose is a more sophisticated control condition than text, so it requires a refined and localized design. It is simple and relatively easy to understand. - There are some experimental verifications in terms of controllability and robustness.

Weaknesses

I carefully evaluated this module paper and did not find particular flaws except for no consideration of the multi-person condition. I will refer to the review suggestions of other reviewers and the author's feedback later. [Method] - (No consideration of different persons & parts) Surprisingly, the conditions may include multiple person, but PMSA does not explicitly consider modeling different people and different parts (in this sense the DensePose is a better choice), but only models the relationship between patches and semantics are considered by high-level pose decoder $\beta$. (Although the qualitative results shown look fine, there is no wrong association, such as A's hand is generated at B's hand position, or A's hand is generated at A's foot) I hope the author could provide more detailed explanations and discussion, even on the potential improvement to the proposed PMSA. [Experiments] - (Inferior PCE) Can the authors try to analyze the disadvantages of PCE (People Count Error) of the image generated by this method (Tab. 1)? - (Add module architecture ablation studies) Since this is a work that proposes a module, the ablation studies of the module structure may still be helpful, e.g. the number of the PMSA-ViT block. - (Add FPS results) Because the proposed PMSA is a plugin, to give readers a better comprehension, it is recommended that the author report the inference time (e.g. FPS) together with other metric results (although the author's claim method has disadvantages). - (Add failures) It is also important to understand the boundaries of a method, e.g. the failure cases. It is recommended to visualize and analyze some of them. - (Opinions on masked generation) Pose-mask guided is used as training loss in the paper. To further enhance control, it seems that the generation of humans can be similarly replaced within the mask area during the inference time. I am curious about the author’s opinion on this.

Questions

Please see the weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors state the limitations in Sec. 5 as 1. longer inference time caused by the Self-Attention, 2. not evaluated with other condition guidances. It is recommended to add the visualization and discussion of failure cases.

Reviewer-author discussions

Dear Reviewers ypoD, qFuv, pWbK and RKmi Would you please look at the rebuttal, discuss with the authors and finalize your score after discussion? Thanks, Your AC

Reply to the authors

Thank you for taking the time to reply to my queries. I have carefully read the concerns about the other authors and the rebuttal. I still think this is a good paper to get accepted and would support it.

Reply

Thanks for the author rebuttal, it address most of my concern. I would like to adjust my rating to borderline but slightly tend to reject side. The method and motivation in this paper are reasonable, but indeed it is kind of incremental compared to the previous method. I believe this paper is good enough to be accepted by a conference in computer vision, but whether it meets the standards for NeurIPS depends on the judgment of the area chair.