Summary
The authors propose an explainability method to interpret the weights of a diffusion model. In particular the method is meant to assess which weights contribute to a certain concept. They demonstrate applications of concept erasing and localizations. They provide experiments to demonstrate the quality of the interpretability method and visual results for editing.
Strengths
1. The proposed method was able to remove weights while keeping the model performance for allowed outputs.
2. The proposed method demonstrate superior qualitative results for removing nudity.
3. The authors chose a challenging task, as the autoregressive nature of diffusion models prevents comparing final generation results to some ground truths or input them to classifiers and train. The general approach they take is elegant: Comparing one step results to one step results with a different condition.
Weaknesses
## 1) Method:
As far as I understand, the method boils down to grant importance to each weight by looking at the derivative of the objective w.r.t. that weight (scaled by the weight itself, which is equivalent to normalizing i.e. instead of quantifying the change in w, quantifying the change in percents of w). This exposes several weaknesses:
1. __Limited novelty__: quantifying importance by gradients has been used in many cases. Sometimes w.r.t. inputs, sometimes with weights. applying it in the particular case of diffusion models weights is not novel enough for this venue, to my taste.
2. __Another novelty point__: This is almost the first part of simple fine-tuning. Fine-tuning would be to take this score (but unnormalized, this is why I wrote "almost"), multiply by some learning-rate and subtract from the weight. This almost feels like gradient descent without the descent.
3. __Quality of the method__: I think this measure is a too loose approximation. While expressing the objective as a linear function of the weights is reasonable, it is important to make the distinction: determining the importance of each weight in this calculation according to the derivative w.r.t. it is different, an I believe- too local. It is equivalent to assuming that the model itself is linear. It is far less accurate than the mentioned Shah et al.- By randomly zeroing weights and learning the effect you get much better assessment of the importance as it is not local. To make this claim clearer, here is an example. say the objective as a function of one of the weights saturates, e.g., the function is Sigmoid. Around high values and values close to zero for this weight this measure will be almost zero. However for mid-values it can be high. An extreme case is a shifted heavy-side (step function), so that Alpha=1 for x>3 and Alpha = 0 for x<=3. If the relation is close to it, it means that around 3 the derivative is extremely high, but zero in all places. It is not uncommon for weights to act somewhat binary in some range, especially with ReLUs around. In such a case, Shah et al. would perform reasonably, as there will be reasonable difference between turning this weight "on" and "off". Shah et al. also makes more sense as it corresponds with the applications. The applications zeroize weights, just like in Shah's measure. They don't change them by epsilon. It could be that the authors could have gotten better results if they took several gradient steps instead of zeroizing these weights.
## 2) Experiments:
1. __Figure 1__: I wouldn't say Pearson correlation of 3.5 counts as high, but actually weak to moderate. Also examining the figure visually, I think it looks only slightly correlated. I think this highlights the weakness of the the proposed approach rather than its success.
2. __Qualitative results__: In general, I think the results are not in par with recent editing methods. The results by the ablated model don't have any content. I would have been impressed, if the entire image mostly would remain the same except for the concept. At the very least, I would expect the final image would look like something realistic. The visual results for removing concepts generate images with no content. Actually in some cases, there is even still a trace of an inappropriate result (e.g., fig3 rightmost)- This implies that it is not the concept that was deleted. One could claim that it is a reasonable goal to detect inappropriate prompts and just not output anything reasonable. However, in such a case all you have to do is detect inappropriate prompts and block them. It would have made sense if the authors would have demonstrated cases where a regular prompt yields inappropriate image that can be detected and blocked efficiently.
3. __No baselines__: For assessing the quality of the interpretability mechanism, I would expect some reasonable baselines that show that indeed this measure has a distinctive meaning: Most basic is removing random weights. But also doing some gradient steps instead of zeroizing weights (as mentioned in 1.3). Also in Figs 2 and 3, a comparison of the allowed prompt of the ablated model with the non ablated model is missing. If the image changes drastically between them it shows that the process was far from cleanly removing the concept.
## 3) Readability and strength of claims:
1. __"Editing" is over-claiming__: I would not describe the experiments as applications. I would describe them as ways to showcase the quality of the interpretability method. There are very strong editing methods and I don't think this should compete with them.
2. __Not defining the objective__: J is only defined for the case of concept removing. There is no definition of it I could find before that. If it's always that J then have it defined early on- it is the most important value dealt with. If there are many possible Js, then have an explanation early on that J could be many things and provide some examples.