Further responses
**Q: Which CLIP model is used?**
We used the OpenAI CLIP model `openai/clip-vit-large-patch14' to compute the alignment score. We will add this information to the revised version.
**Q: How does your proposed method differ to the "Forget-Me-Not" paper? Any specific reason this is not covered in comparisons?**
Our method significantly differs from the Forget-Me-Not (FMN) method. From our point of view, FMN falls into the category of approaches like TIME, UCE, and MACE,
which focuses on confusing the alignment between the prompt and the visual features in the cross-attention mechanism.
Specifically, FMN introduces an attention resteering method that attempts to alter the attention maps related to the target concept (i.e., by minimizing the L2 norm of the attention maps).
Our method, on the other hand, stands out by focusing on identifying which concepts are most affected by the erasure of a target concept and then preserving these concepts to maintain the model's capability on other concepts.
We will cite and discuss the FMN in the revised version.
Regarding additional experiments with the FMN method, we attempted to erase the same set of concepts from the Imagenette dataset and evaluated the erasing and preservation performance of the FMN method within our setting.
However, despite our best efforts given the time constraints, FMN did not effectively erase the target concepts.
Notably, their open-source code provides hyper-parameter settings for single-concept erasure but not for multiple concepts.
We also noticed an open issue on their GitHub repository questioning this same problem.
This issue can easily be verified by running their code with a modified configuration file, `attn.yaml`,
to erase multiple concepts, given some representative images of the target concepts.
**Q: Coding question: Implementations of Equation 4 and Equation 5**
Equation 4 describes our naive approach, which utilizes Projected Gradient Descent (PGD) to search for the adversarial concepts in the continuous space. We have uploaded the code for this approach to the anonymous repository. Please refer to the code for more details.
Regarding the implementation of Equation 5, our method involves bilevel optimization. In this process, the inner maximization step maximizes the L2 loss (i.e., minimizes -L2 loss) to select the most affected concepts to be preserved. The outer minimization step minimizes the combined L1 + L2 loss to erase the target concepts while preserving the most affected concepts.
For the L1 loss, we inherited this implementation from the ESD paper and did not attempt to modify it, including the aspect of negative guidance.
**Q: Coding question: Why using K-means?**
In the implementation, we investigate the use of K-means to control the tradeoff between the computational cost and the size of the vocabulary when searching for adversarial concepts. More specifically, we first compute the similarity between the target concept and the entire vocabulary, then select the top-K most similar concepts. We then use K-means to choose the K most representative concepts from these top-K most similar concepts. This approach allows us to cover a wide range of concepts while keeping the computational cost low.
**Q: Coding question: How to calculate the text embedding for adversarial concepts in Stable Diffusion version 3**
To the best of our understanding, as described in the `encode_prompt` function in the Stable Diffusion v3 pipeline (lines 310-311),
as similar to the same function in the SDXL pipeline (line 397), we still can obtain the standard prompt embedding as the output of a text encoder which is independent of the time step. Therefore, we can still use our current approach to calculate the `emb_r` for the adversarial concepts.