Recently, prompt learning has emerged as the state-of-the-art (SOTA) for fair text-to-image (T2I) generation. Specifically, this approach leverages readily available reference images to learn inclusive prompts for each target Sensitive Attribute (tSA), allowing for fair image generation. In this work, we first reveal that this prompt learning-based approach results in degraded sample quality. Our analysis shows that the approach's training objective -- which aims to align the embedding differences of learned prompts and reference images -- could be sub-optimal, resulting in distortion of the learned prompts and degraded generated images. To further substantiate this claim, as our major contribution, we deep dive into the denoising subnetwork of the T2I model to track down the effect of these learned prompts by analyzing the cross-attention maps. In our analysis, we propose a novel prompt switching analysis: I2H and H2I. Furthermore, we propose new quantitative characterization of cross-attention maps. Our analysis reveals abnormalities in the early denoising steps, perpetuating improper global structure that results in degradation in the generated samples. Building on insights from our analysis, we propose two ideas: (i) Prompt Queuing and (ii) Attention Amplification to address the quality issue. Extensive experimental results on a wide range of tSAs show that our proposed method outperforms SOTA approach's image generation quality, while achieving competitive fairness. More resources at FairQueue Project site: https://sutd-visual-computing-group.github.io/FairQueue
Paper
Similar papers
Peer review
Summary
This paper examines the prompt learning method used in fair text-to-image (T2I) generation, highlighting its impact on image quality. The authors identify that aligning prompt embeddings with reference image embeddings introduces noise due to unrelated concepts, leading to degraded image quality. They conduct an in-depth analysis of the T2I model's denoising subnetwork, introducing novel prompt switching analyses (I2H and H2I) and new quantitative metrics for cross-attention map characterization. Their findings reveal abnormalities in the early denoising steps, which affect global structure synthesis. To address these issues, the paper proposes two solutions: Prompt Queuing, which applies base prompts initially and ITI-GEN prompts later in the denoising process, and Attention Amplification, which balances quality and fairness. Extensive experiments demonstrate that these methods improve image quality while maintaining competitive fairness and diversity. The paper's contributions include identifying issues in the current prompt learning approach, providing a detailed analysis of the denoising process, and proposing effective solutions to enhance T2I generation quality.
Strengths
1. The paper embarks on an in-depth examination of the denoising subnetwork within the T2I model. The goal is to understand how learned prompts influence image generation throughout the denoising steps. This involves dissecting the entire process from noisy input to the final image, which allows for pinpointing where and how the learned prompts might be causing degradation in image quality. This deep dive includes analyzing how noise is progressively reduced and how the model’s intermediate representations evolve at each step. By doing so, the authors can track the influence of different prompt tokens on the generation process, providing a granular view of the internal dynamics of the model. 2. The cross-attention maps are visual tools that show how attention is distributed across different parts of the image in response to various tokens in the prompt. By mapping out these attention distributions, the paper identifies specific patterns and irregularities. The analysis reveals that certain tokens, which are not directly related to the target sensitive attributes (tSAs), exhibit abnormally high levels of activity. For instance, non-tSA tokens like "of" and "a" show higher attention scores, indicating that the model's focus is scattered and not properly aligned with the relevant attributes. This misalignment disrupts the formation of coherent global structures in the early stages of image generation. 3. The paper highlights that abnormalities in cross-attention maps are particularly pronounced during the initial steps of the denoising process. At these stages, the model is supposed to establish the foundational global structure of the image. However, the distorted prompts from ITI-GEN interfere with this process, leading to poorly structured outputs. For example, the analysis shows that when using ITI-GEN prompts, the model often fails to correctly focus on key image regions, resulting in artifacts and unintended features. This is contrasted with the behavior of hard prompts, which show more targeted and consistent attention patterns that help maintain image quality. 4. This paper includes extensive experimental results demonstrating the superiority of FairQueue over ITI-GEN in terms of fairness, image quality, and semantic preservation. These experiments are conducted on multiple datasets and with various tSAs.
Weaknesses
1. The proposed innovations, Prompt Queuing and Attention Amplification, are relatively incremental and might be seen as a combination of existing techniques rather than a breakthrough innovation. The combination of these two mechanisms does improve performance but lacks a strong theoretical underpinning to justify why this specific combination is superior. 2. The proposed method, FairQueue, while innovative, could benefit from a more detailed explanation of how it addresses the issues identified in the paper. In the current form, the explanation of Prompt Queuing lacks depth regarding why the base prompt T is more effective in the early denoising steps. The authors should delve deeper into the reasons behind this, possibly by providing theoretical analysis and specific experimental results that demonstrate how the base prompt gradually establishes the fundamental elements of the image during the denoising process. 3. Additionally, the transition to ITI-GEN prompts in the later stages should be better explained, particularly in terms of how it enhances the detailed expression of tSAs. Comparative illustrations of image generation results at different stages using different prompts would effectively show the benefits of this transition. 4. The mechanism of Attention Amplification needs a clearer explanation. The authors should detail why scaling the cross-attention maps enhances tSA expression, possibly through mathematical derivations or experimental data showing the impact of different scaling factors on image quality and tSA expression. 5. Although the experimental results are comprehensive, the paper lacks detailed descriptions of the experimental settings and specific parameters used, which could hinder reproducibility.
Questions
1. The authors provide a more detailed explanation or evidence on why the base prompt T is more effective in forming global structures during the early denoising steps compared to ITI-GEN prompts. Including theoretical justifications or experimental comparisons that illustrate how the base prompt helps establish the foundational elements of the image more effectively would be beneficial. Specific examples or visualizations from your experiments could clarify this point. 2. How does switching to ITI-GEN prompts in the later stages of denoising improve the detailed expression of tSAs without compromising the overall image quality? 3. The author should elaborate on the mechanism by which attention amplification enhances tSA expression. How is the scaling factor chosen and validated? 4. The paper would be strengthened by more detailed experiments and results analysis, especially comparisons between using only the base prompt, only ITI-GEN prompts, and the FairQueue method. A thorough analysis of these results would help demonstrate how FairQueue effectively addresses the issues identified with the original method. By incorporating these detailed explanations and analyses, the paper would more convincingly show how FairQueue resolves the problems it aims to address, enhancing its credibility and scientific rigor.
Rating
5
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
1. Although the authors have identified and analyzed the issues with the current methods, the explanation of how their proposed method, FairQueue, effectively addresses these issues lacks detail. Regarding the effectiveness of Prompt Queuing, the authors should provide a deeper explanation of why the base prompt is more effective in the early denoising steps. This could be supported by theoretical analysis and specific experimental results, such as demonstrating how the base prompt gradually establishes the fundamental elements of the image during the denoising process. Further explanation of how using the base prompt in the early steps helps form a robust global structure, allowing the application of ITI-GEN prompts in later steps to better add details without compromising the overall structure, would be beneficial. 2. For the mechanism of Attention Amplification, the authors need to elaborate on why scaling the cross-attention maps effectively enhances tSA expression. This section could include mathematical derivations or experimental data to show how different scaling factors impact image quality and tSA expression. Providing specific examples of how attention amplification compensates for the reduced tSA expression due to prompt queuing in various scenarios would strengthen the argument. 3. The authors should include more experimental results, particularly comparisons between using only the base prompt, only ITI-GEN prompts, and the FairQueue method. Detailed analysis of these results would help demonstrate how FairQueue addresses the issues identified with the original method, such as image quality degradation and insufficient tSA expression. Providing specific quantitative metrics and visual comparisons can more intuitively show the advantages of the proposed method. 4. The paper lacks a thorough analysis of the computational complexity and the necessary resources for implementing FairQueue. Understanding the computational demands is essential for assessing how feasible and scalable the method is in real-world applications. Without this information, it is difficult to determine whether FairQueue can be efficiently deployed in various environments, particularly those with limited computational resources.
Clarifying FairQueue’s as the Best Performing Combination
>R4Q8: Explanation of why the specific combination of Prompt Queuing and Attention Amplification is superior to other possible combinations could be more detailed. Deeper analysis of why this particular combination works best would strengthen your argument We thank Reviewer 9buQ for insightful comments. Here we provide analysis of all possible combinations of Prompt Queuing (PQ) and Attention Amplification (AA). We note that all these analyses are already in the paper to derive the best combination, but we fully agree with Reviewer that it is better to have them in one place, to clearly support why our PQ+AA is superior. Recall: 1. PQ enables proper global structure generation, leading to **improved sample quality**. 2. AA supplements PQ by enhancing exposure to tSA tokens, leading to **state-of-the-art fairness**. With these, we provide analysis of why PQ+AA works better than all other combinations. $ $ Tab i: Analysis of **all possible different combinations** for Prompt Queuing (PQ) and tSA Attention Amplification (AA). We summarize our findings from main paper for the tSA “Smiling”. Note $\alpha (S)$ notates AA for tSA tokens and results in **bold** and *italics* are the best and second best. Notice that C6:FairQueue (PQ+AA) provides the best combination: it achieves **both** outstanding sample quality (C6: TA=0.674 & FID=80.02 similar to C1: TA=0.681 & FID=76.9 with the best quality but poor fairness) and fairness (C6: FD=0.069 similar to C4: FD=0.05 with the best fairness but poor quality). | | Prompt Queueing (PQ) | Attention Amplification (AA) | Stage 1 Prompt | Stage 2 Prompt | FD$\downarrow$ | TA$\uparrow$ | FID$\downarrow$ | DS$\downarrow$ | Remarks | |---|---|---|---|---|---|---|---|---|---| | C1: **no PQ, no AA** for Base Prompt | No | No | $T$ | $T$ | 0.211 | **0.681** | **76.9** | - | | | C2: **no PQ, no AA** for ITI-Gen | No | No | $[T;S]$ | $[T;S]$ | 0.124 | 0.605 | 88.63 | 0.557 | | | C3: **AA only** for Base Prompt | No | Yes | $T$ | $T$ | N.A | N.A | N.A | N.A | An **unimplementable combination** due to the absence of tSA tokens for AA. | | C4: **AA only** for ITI-Gen | No | Yes | $[T;S]$ | $[T;\alpha (S)]$ | **0.05** | 0.61 | 89.41 | 0.55 | | | C5: **PQ only** | Yes | No | $T$ | $[T;S]$ | 0.145 | *0.674* | 80.15 | **0.24** | | | C6: **PQ + AA** (Our specific combination) | Yes | Yes | $T$ | $[T;\alpha (S)]$ | *0.069* | *0.674* | *80.02* | *0.284* | Both PQ and AA are present i.e., our proposed FairQueue | $ $ With the results summarized from main paper (above), we provide deeper analyses below with additional explanations for improved clarity: - **C1: Base Prompt $T$ Only** (no AA no PQ): It lacks tSA-specific knowledge and results in poor fairness. Additionally, without tSA tokens $S$, AA is not applicable for **C3**. - **C2: ITI-Gen prompt $P$ Only**, in Tab.1 (no AA no PQ). Our analysis in Sec 3.2. shows it has poor quality due to distortion in global structure during sample generation. Without PQ, the issue of distorted global structure persists for some tSAs. - **C4: Attention Amplifcation (AA) Only**, in Supp. A.2. Fig.22 when PQ transition point=0. It results in poor quality since only ITI-Gen is used. We remark that utilizing only AA for ITI-Gen may deceptively improve fairness, but the generated samples have poor quality e.g., Smiling cartoons. The reason is (similar to C2): without PQ, the issue of distorted global structure persists for some tSAs. - **C5: Prompt Queuing (PQ) Only**, in supp. A.2 Fig 22 when $c=0$. By replacing the distorted ITI-Gen prompt with the Base prompt in Stage 1, PQ leads to improved quality, but without AA, the fairness remains poor given reduced exposure to tSA tokens in the denoising process. - **C6: FairQueue (PQ+AA)**, in Tab.1. Our proposed solution with optimal quality and fairness. Specifically, it combines the effects of Prompt Queuing– enabling the global structure to be properly formed resulting in good quality samples, and Attention Amplification–enhancing the tSA-specific expression for better fairness. $ $ The results in Tab.i present a summary of related quantitative results: - Comparing C4, C5, and C6 reveals that both PQ and AA are necessary to obtain high-quality samples with good fairness performance. Specifically, C4 has poor quality, while C5 has poor fairness. - Comparing C1, C2, and C5 reveals the necessity of Prompt Queuing because utilizing either only ITI-Gen or Base Prompt results in quality and fairness degradation, respectively. Overall, our quantitative analysis demonstrates **C6:FairQueue** is the superior combination, balancing both fairness and quality. It achieves a TA=0.674 and FID=80.02, the closest to the best quality by C1 with TA=0.681 and FID=76.9 (but poor fairness), while achieving fairness of FD=0.069, the closest to the best fairness by C4 with FD=0.05 (but poor quality). We hope this further discussion addresses Reviewer’s concerns.
Appreciation for the Discussion
As we will conclude the Rebuttal’s discussion session in 5 hours, we sincerely thank the Reviewer 9buQ for their valuable insights and thoughtful discussion. We hope that our efforts to address the Reviewer’s comments have been satisfactory. *If the Reviewer finds our responses appropriate, we kindly ask for a re-consideration of our scores based on the new additional understanding of our work.*
Summary
The authors propose FairQueue, a simple and effective solution to solve the quality degradation problem in ITI_GEN through prompt queuing and attention amplification.
Strengths
1. The paper is well-written and logically structured. 2. The proposed FairQueue effectively solves the quality degradation problem in ITI_GEN. 3. The paper is very rich in experiments
Weaknesses
Some of the contribution points of the article can be optimized.
Questions
1. In Chapter 4, the author proposed a method for attention amplification. Is the amplified area Si related to tSA? If so, please explain the specific impact relationship. 2. Are there specific metrics to measure the relationship between generation quality and fairness?
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The author can further improve the description of the method part of the paper and describe the innovation more clearly.
Summary
This paper introduces an approach to improving fairness and quality in text-to-image generation models by rethinking prompt learning. The current Inclusive Text-to-Image Generation (ITI-GEN) methods often degrade image quality due to suboptimal training objectives and embedding misalignment between learned prompts and reference images. Key innovations include Prompt Queuing, ensuring regular and fair prompt usage, and Attention Amplification, enhancing the impact of prompts on image generation. Experiments show that FairQueue improves both image quality and fairness compared to state-of-the-art methods.
Strengths
- Through detailed analysis, they revealed the limitations of the previous approaches, specifically how suboptimal training objectives and embedding misalignments degrade image quality.
Weaknesses
- The readability of the paper is poor. - The evaluation of the model is insufficient, as there are only evaluations on the datasets of human faces, which lacks in evaluating the generalizability of the model. More experiments on other datasets are needed. - The paper emphasizes quantitative metrics but does not give enough attention to qualitative assessments. Including user studies or expert reviews could provide valuable insights into the perceived quality and fairness of the generated images, complementing the quantitative data.
Questions
- I'm curious why the experiments were conducted on only one dataset. - Some of the metrics used in the paper are unfamiliar. Are these metrics validated, and are they commonly used in other research papers as well? - Additionally, I would like to know how you plan to address the limitations mentioned in the limitations section.
Rating
5
Confidence
1
Soundness
3
Presentation
1
Contribution
3
Limitations
They mentioned limitations and broader impact in the paper.
We appreciate the Reviewer Raising our rating and for the kind words
We are very thankful for the Reviewer's increased rating and that we were able to “addressed all the concerns that I (the reviewer) raised” We respectfully seek the reviewer to consider adjusting their presentation and/or confidence score to reflect their new understanding, should the reviewer find it appropriate.
Summary
In the field of fair text-to-image generation, the ITI-GEN paper demonstrated good performance by learning and embedding tSA. However, this paper argues that the embeddings learned in ITI-GEN can include unrelated attributes, resulting in noisy embeddings and significantly degrading the quality of the generated samples. To address this, the paper proposes using prompt queuing to apply tSA after a certain timestep and employs attention amplification to prevent the weakening of tSA.
Strengths
The paper provides a highly detailed analysis of how the learned tSA embeddings are actually reflected in the images using attention maps. It also demonstrates that the learned embeddings do not properly reflect differences in the images.
Weaknesses
First, the degradation seems severe compared to the ITI-GEN paper. If I understand correctly, ITI-GEN appears to aggregate all learned tSA embeddings by summing them, while this paper seems to concatenate a large number of tokens, as indicated by (p+q)*d. This difference in implementation may be causing a decline in baseline performance and requires an explanation. Additionally, the proposed method to address this issue seems naive, and there is a lack of ablation experiments. Also, even with the proposed method, there appears to be a degradation in sample performance compared to HP. For example, in Figure 2, an originally female image appears to have been changed to a male image using the proposed method.
Questions
- Is there a difference between the original ITI-GEN and your re-implementation as I mentioned or did I just misunderstand? (I am not an expert of ITI-GEN) - Wouldn't this problem be naturally resolved if a large number of reference images were used to learn the embeddings?
Rating
5
Confidence
3
Soundness
2
Presentation
4
Contribution
2
Limitations
The authors describe the limitations and potential societal impact in their supplemental material.
Addressing Question on Research Gap and Reproducibility
Firstly, we really appreciate Reviewer 89En’s prompt and insightful feedback. >Q7: Thank you for clarifying my questions and comments. I agree with the rebuttal that a simple method could be proposed due to thorough analysis, and I believe this paper is indeed a suitable follow-up to ITI-GEN. while I still think it may be incremental because it heavily depends on ITI-GEN method. R7: We would like to emphasize that, as one of our main contributions, we had also proposed new qualitative and quantitative analysis of the effects of learned tokens in individual denoising steps of diffusion models. The proposed method thus grounded on our novel analysis. $ $ >Q8: which experiment in this paper achieved similar scores under the exact same settings R8: We thank the Reviewer for the question and clarify that our work considers a more fine-grained performance analysis than ITI-Gen which aligns better with existing SOTA literature [27,28,30,31]. **Quality evaluation**: Reproducibility of ITI-Gen’s quality is indeed a question raised by other researchers in ITI-Gen’s GitHub. To address this, ITI-Gen authors clarified (in GitHub issues #1 “Missing Performance Metric” and #7 “The gap between FID indicators is too large”, currently closed) that the FID results in ITI-Gen’s paper Tab.2 are achievable when evaluation is performed on data of all 40 tSAs, for both real and generated images (with learned ITI-GEN prompt) gathered as one large dataset (200*40), and FID is calculated on this large dataset. We have also verified this. (We will include the links to the GitHub page to AC per NeurIPS guidelines.) However, as remarked in global responses, such high-level analysis is susceptible to missing the quality degradation existing in specific tSAs (e.g., High-Cheekbone). To address this, in more fine-grain analysis, we instead focus on individual tSA and evaluate their respective quality, aligning with existing fairness literature [27,28,30,31]. With this more fine-grain analysis, we were able to see the distortion in individual learned prompts. In addition, we considered an additional Text-Alignment (TA) quality metric which confirms our findings. Note that, Tab. 1 in the original ITI-GEN paper has only fairness comparison for specific tSAs, while FID or other quality evaluation for these tSAs is missing. **Fairness evaluation**: As mentioned in our paper, following existing literature, we utilizes Fairness Discrepancy (FD) metric [27,28,30,31]. Specifically, FD is the same as that implemented with ITI-Gen but instead uses L2 distance in place of KL divergence. $ $ >Q9: compare the scores with the original model and HP in Table 1 in the proposed paper? R9: We provide the following results as per reviewer’s request for HP evaluation: | | FD | TA | FID | DS | |---|---|---|---|---| | Gender | $1.4e^{−3}$ | 0.699 | 77.1 | 0.318 | | Young | 0 | 0.674 | 76.4 | 0.392 | | Smiling | $8.4e^{−3}$ | 0.672 | 79.8 | 0.323 | | High C. | $3.68e^{−3}$ | 0.672 | 80.3 | 0.332 | | Pale Skin | $591e^{−3}$ | 0.660 | 96.2 | 0.397 | | Eyeglasses | $670e^{−3}$ | 0.670 | 79.1 | 0.468 | | Mustache | $554e^{−3}$ | 0.674 | 81.4 | 0.372 | $ $ As mentioned in our paper e.g., Tab 2, many of the HPs perform poorly w.r.t. Fariness due to their tSA having linguistic ambiguity. However, their sample quality is competitive. Overall, FairQueue still achieves best performance–with competitive fairness as ITI-Gen and high-quality samples as HP. Due to space limitations, we can provide side-by-side comparison with ITI-Gen and FairQueue in another reply if Reviewer allows us (per NeurIPS rules). (ITI-Gen and FairQueue comparison is already in main paper) $ $ >Q10: Wouldn’t HP perform better with more recent models that use improved text encoders, such as stable diffusion XL or version 3? Is there a possibility to apply this method to these latest models? R10: We clarify that we have verified that the problems indicated by the ITI-Gen paper (e.g., in Fig.1) would still persist in more recent versions of Stable Diffusion e.g., SD 3.0. This issue can be observed by simply inputting the prompt “A headshot of a person without glasses” to SD 3.0 demo and generating several samples (provided through the AC comments). The generated samples still frequently have wrong tSA class (“with glasses”) indicating this ambiguity still exists. We provide some results showing the poor fairness of HP with SD3.0. Here, we utilize the same setup as Tab.1 (main paper) with 500 samples per tSA class and HP per Tab.3 (Supp.B.3) to report FD: $ $ | | FD | |---|---| | Eyeglasses | $670e^{−3}$ | | Pale Skin | $580e^{−3}$ | $ $ Regarding applying ITI-Gen or FairQueue to more recent models: while in principle our ideas are model agnostics, it requires some effort to implement in the new SDXL or SD3 code base, beyond the duration of this rebuttal. **Once again, we thank Reviewer 89En’s prompt and insightful feedback.**
I see. Now I understand that while the ITI-GEN performance is strong across the entire dataset, there are some attributes where its results suffer from significant degradation, which this paper addresses. Thank you for clarifying my comments. I am raising my score to 5. However, I did not raise it further because I still feel this work is too dependent on the ITI-GEN method, even though it provides valuable improvements for the community.
We appreciate the Reviewer Raising our rating and for the kind words
We are very thankful for the Reviewer's increased rating and kind words in considering our work “... valuable improvements for the community”. We respectfully seek the reviewer to consider adjusting their soundness/contributions score to reflect their new understanding, should the reviewer find it appropriate.
Thank you for clarifying my questions and comments. I agree with the rebuttal that a simple method could be proposed due to thorough analysis, and I believe this paper is indeed a suitable follow-up to ITI-GEN. while I still think it may be incremental because it heavily depends on ITI-GEN method. There is one more thing I’d like to ask. While I also trust that the authors reproduced the exact results since the code is released, I ask for your understanding in confirming the results because the quality of the results in the paper seems to vary significantly. Could you clarify which experiment in this paper achieved similar scores under the exact same settings as ITI-GEN? In the ITI-GEN paper, most results (Table 1 and 2 in their paper) show better or comparable performance in FID compared to the original stable diffusion, but the reproduced results seem to fall short compared to the original model. To be clarifying this, could you also compare the scores with the original model and HP in Table 1 in the proposed paper? Additionally, I have a concern that could be raised not only with this paper but also with the ITI-GEN paper. The poor performance of the HP method might be due to stable diffusion v1.4 not properly reflecting the prompts. Wouldn’t HP perform better with more recent models that use improved text encoders, such as stable diffusion XL or version 3? Is there a possibility to apply this method to these latest models?
Thank you for the rebuttal. I think the author has addressed all the concerns that I raised. I updated my score to 5.
Response for authors
Thank you for the rebuttal. I still have one concern. The authors conducted ablation studies to validate the contribution of each component, which is well-handled. However, the explanation of why the specific combination of Prompt Queuing and Attention Amplification is superior to other possible combinations could be more detailed. A deeper analysis of why this particular combination works best would strengthen your argument.
Updating Reviewers on All Addition Experiments in the Rebuttal
Once again, we thank all the Reviewers for their insightful comments and for those who considered raising their scores after further clarifications. To help keep all Reviewers updated, in the following, we provide a summary list of the experiments conducted by us, as suggested by the Reviewers (which we will include in the final manuscript): $ $ 1. **Assessing the persistence of linguistic ambiguity in a newer version of T2I generative model**: Given that newer versions of the text-to-image models (like SD 3) are using improved text encoders compared to version 1.4, this experiment aims to assess whether these newer versions have a better understanding of prompts with potential linguistic ambiguity e.g., “without eyeglasses”, and therefore whether HP can be used for all/several tSAs in these T2I generative models. Our results show that linguistic ambiguity *still exists in the latest versions* of SD. This aligns with our findings in the main paper (experimented for SD 1.4) showing that HPs are still not viable solutions to address fairness for all tSAs (FD scores for these tSAs still remain large). Therefore, enforcing fairness across all tSAs requires alternative approaches like prompt learning in our proposed FairQueue. $ $ 2. **User study with human evaluation (Amazon-Mechanical Turk) demonstrating FairQueue’s preferred performance**: In addition to extensive qualitative and quantitative experimental results (using four well-known metrics) already provided in our paper, we have included a user study to compare the quality and fairness of ITI-Gen vs FairQueue based on human assessment. The results show that for all tSAs, the majority of human evaluators also prefer FairQueue sample quality, and the fairness of FairQueue remains competitive with ITI-Gen. These results align with our qualitative and quantitative analysis in the main paper. $ $ 3. **Additional qualitative and quantitative analysis** on the attention maps between BasePrompt vs FairQueue vs ITI-Gen. Our analysis further demonstrates the effectiveness of FairQueue’s mechanism to first utilize the natural language, Base prompt $T$ to first form a good global structure followed by (tSA aware) learned prompt to synthesize tSA expressions (via localized attention). $ $ 4. **Qualitative demonstration of the effectiveness of Attention Amplification (AA)**: This is a qualitative ablation study aimed to show how without AA ($c=0$), tSA expression (tSA=’Smiling’) can still be lacking in many samples. Then, with the gradual introduction of AA ($c>0$), the tSA expression is emphasized (samples slowly begin to smile) while maintaining sample quality. $ $ 5. **Evaluation on a larger reference dataset further supports our proposed solution (2K samples per class of tSA)**: This experiment aims to assess whether the problem of encoding unrelated knowledge can be resolved by increasing the reference dataset size. Our experimental results show that the sub-optimality of the learning objective can not simply be addressed by increasing the reference dataset size, and therefore the quality degradation issue still exists using even larger datasets necessitating using our proposed FairQueue as a solution. $ $ Overall, all the experiments support FairQueue as an effective solution, achieving both high-quality and fair sample generation, when compared against the existing state-of-the-art.
Decision
Accept (poster)