**Weaknesses:**
1. We acknowledge the difficulty in fully capturing the diversity and complexity of stereotypes in real-life scenarios. In our study, we attempted to increase the diversity of evaluated samples by drawing from toxic text content datasets on various social platforms. Regarding the imbalanced distribution of subgroups, we would like to clarify that Figure 3 illustrates the proportional distribution of instruction pairs obtained during the initial text extraction process, which is biased towards specific subgroups. It is important to note that this distribution does not represent the training data. Instead, it highlights the strong stereotypes associated with certain subgroups in everyday expressions.
2. In the Appendix section, we provide a comprehensive evaluation of the proposed agent framework, including an analysis of false positives, false negatives, and the impact of different parameters. We apologize for not making this clear in the main text and will revise accordingly.
3. We compare our proposed agent framework with existing stereotype detection methods. It's important to highlight that these conventional methods rely on manually crafted prompts for comprehensive testing.
| Bias dimensions | Ours | Cho et al. [1] | Bianchi et al. [2] | Naik et al. [3] |
| --------------- | -------- | -------------- | ------------------ | --------------- |
| Gender | ✔ | ✔ | ✔ | ✔ |
| Race | ✔ | ✔ | ✔ | ✔ |
| Religion | ✔ | ✖ | ✖ | ✖ |
| Bias subjects | Ours | Cho et al. [1] | Bianchi et al. [2] | Naik et al. [3] |
| ----------------- | -------- | -------------- | ------------------ | --------------- |
| Person | ✔ | ✔ | ✖ | ✔ |
| Occupacations | ✔ | ✔ | ✔ | ✔ |
| Traits | ✔ | ✖ | ✔ | ✔ |
| Situations | ✔ | ✖ | ✖ | ✔ |
| Verb-noun prompts | ✔ | ✖ | ✖ | ✖ |
4. Our agent framework emphasizes the use of existing models to complete complex tasks. We add a more comprehensive justification for the selection of specific tools.
5. We understand the importance of discussing the potential impacts and implications of stereotype detection in practice. In the revised paper, we provide a more thorough discussion of the unintended consequences of bias mitigation strategies, the role of human judgment in determining stereotypes, and the balance between freedom of expression and risk mitigation.
6. In response to the concern about providing explicit recommendations or strategies for mitigating biases, we include a section discussing potential bias mitigation approaches and their implications in the revised paper.
**Questions:**
1. Language agents for stereotype detection in text-to-image models may face limitations in accurately detecting subtle stereotypes or those embedded in complex contexts. Additionally, they may struggle with cases where stereotypes are expressed using sarcasm or irony. We discuss these challenges and potential solutions in the revised paper.
2. For manual annotation, annotators were provided with guidelines that included descriptions and examples of stereotypes. They were instructed to determine the presence of stereotypes based on these guidelines. We provide more information about the criteria used for manual annotation in the revised paper.
3. We apologize for the lack of detail regarding the annotation process. In our study, multiple annotators were involved, and we conducted an inter-rater reliability assessment to ensure consistency. We include this information in the revised paper.
4. The toxic text datasets used to construct the benchmark dataset were selected based on their relevance to the task, diversity of content, and representation of various social platforms. We provide more details about the selection criteria and guidelines in the revised paper.
[1] Jaemin Cho, Abhay Zala, and Mohit Bansal. 2022. DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Models. https: //doi.org/10.48550/ARXIV.2202.04053.
[2] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2022. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. https://doi.org/10.48550/ARXIV.2211.03759
[3] Naik R, Nushi B. Social Biases through the Text-to-Image Generation Lens[J]. arXiv preprint arXiv:2304.06034, 2023.