Thank you for your feedback. We sincerely apologize for the concise response and any misunderstanding it may have caused.
**We initiated testing of our method using the referenced work [1] at the start of the rebuttal period. However, due to time constraints and limited GPU resources, we were unable to complete the experiments before the rebuttal phase concluded.** Consequently, we plan to present the full extent of our findings in the final draft.
Below, we provide details of the experiments conducted and **our current observations** to offer insights into the remaining misalignments we've identified with the assistance of [1].
First of all, we briefly summarize [1]: the paper introduces an attack method to automatically find out what prompt caused Stable Diffusion model to generate misalignment images. Based on the successful attack prompts, [1] summarizes four prompt patterns where Stable Diffusion model often fails to generate aligned images. The four patterns are: a) Variability in Generation Speed; b) Similarity of Coarse-grained Characteristics; c) Polysemy of Words; d) Positioning of Words.
Our experiment and analysis contains the following 3 parts:
**Experiment 1. Do the discovered four patterns of the original impairment of Stable Diffusion model still exist in CoMat?**
Based on the example four patterns in [1], we manually create 20 prompts for the patten (a), (b) and (c), and 5 meta patterns for pattern (d), which generates 30 prompts. Then we use them as prompts for the original Stable Diffusion model and CoMat. The generated images are evaluated by humans. We show the ratio of successful generation below:
| Model | Pattern (a) | Pattern (b) | Pattern (c) | Pattern (d) |
| --------------------- | ----------- | ----------- | ----------- | ----------- |
| Stable Diffusion v1.5 | 45.0% | 35.0% | - | 13.3% |
| CoMat-SD1.5 | 65.0% | 50.0% | - | 30.0% |
This result align with the enhancements targeted by CoMat. Pattern (a), (b) and (d) all involves multiple objects in the prompt. The Concept Activation Module contributes to the object's existence. Besides, the Attribute Concentration Module restricts the diffusion model to only attend to one object's token for each object area in the image. This further prevents the object combining problem found in pattern (b).
The pattern (c) is about the polysemy of words. We argue that without sufficient contextual information, it is infeasible to evaluate the generation result.
This experiment demonstrates that CoMat greatly mitigates the original vulnerabilities discovered in Stable Diffusion. However, we acknowledge that there remains a gap between CoMat's performance and perfect alignment, particularly in addressing pattern (c) where prompts contain more than two objects. We are actively working to address these remaining challenges. We apologize that the visualization result cannot be provided in the discussion stage, we will include the visualization result in our final draft.
**Experiment 2. Can the learned Gumbel Softmax distribution of Stable Diffusion v1.5 still effectively attack CoMat-SD1.5?**
We implement the method of [1] using its open-sourced code in GitHub. Following the setting in [1], we learn a Gumbel Softmax distribution for each class from ImageNet-1K [2]. Due to the time limit, instead of generating 50 images for each class, we generate 5 images for each class and calculate the attack success rate. Then, we directly use this learned Gumble Softmax distribution to attack CoMat-SD1.5. The result is below:
| Model | Short Prompt Success | Long Prompt Success |
| --------------------- | -------------------- | ------------------- |
| Stable Diffusion v1.5 | 47.6% | 51.1% |
| CoMat-SD1.5 | 39.8% | 45.3% |
As the result shown, CoMat-SD1.5 suffers less from the attack. This result is reasonable since CoMat has gone over the fine-tuning process so the distribution for SD1.5 may be not valid for CoMat-SD1.5.
We will include the result following the original setting in [1], where 50 samples are generated for each class, in our final draft.
**Experiment 3. How does CoMat act under the auto-attack method proposed in [1]?**
Finally, we directly apply the method introduced in [1] to CoMat-SD1.5. Again, due to the time and resource limit, we only test on 5 samples for each class. The result is below:
| Model | Short Prompt Success | Long Prompt Success |
| ----------- | -------------------- | ------------------- |
| CoMat-SD1.5 | 45.2% | 52.3% |
Our experiments reveal that CoMat demonstrates superior robustness compared to SD1.5 in scenarios involving short prompt attacks. However, we observed a decline in performance when dealing with long prompts. This discrepancy may be due to the lack of long prompts in CoMat's training dataset.