Summary
This work focuses on uncovering a principled, objective approach to evaluating existing interpretability tools in identifying humanly interpretable attributes in image classification. They approach this via the following synthetic task:
1. A known, humanly interpretable trojan (a special patch, style transfer, a key object) is injected into a model's training pipeline, leading to a false classification by the model when the trojan is present in an image.
2. Different interpretability tools are applied to this model on an uncorrupted image to identify "interpretable features" that would lead to a misclassification. An interpretability tool that performs well should ideally output something about the underlying trojan scheme, since this is a clearly interpretable feature that changes the classification.
3. The quality of outputs from these interpretability tools are compared by seeing which tool's outputs best help humans identify the original trojan scheme from a multiple-choice set of possible options.
A key challenge of this task is to find these trojan backdoors with access to the model alone and not to the adversarially corrupted examples.
The authors also propose two novel variants on the existing "robust feature-level adversaries" approach of [Casper and Nadeau et.al. Robust feature-level adversaries are interpretability tools] to solve this task.
Their main findings/contributions are:
1. The authors propose this trojan identification task as a synthetic benchmark for assessing the usefulness of human-level interpretability tools
2. They claim that naive approaches like saliency maps are not useful, even with access to the adversarial examples,
3. In a human study, robust feature-level adversaries and its variants are shown to be best at helping people identify the backdoor scheme, suggesting that these are the most effective interpretability tools.
4. No technique worked well for identifying style-transfer based, non-local backdoors. This suggests that current interpretability tools might not do a good job of explaining "textural" features.
Strengths
• The paper investigates a useful and important research question in the field of interpretability research: "how do we objectively quantify and measure the quality of interpretable explanations?".
• The trojan task proposed by the authors is a clever proxy for measuring human interpretability of explanations.
• The authors run an actual human study to measure the usefulness of interpretability tools to people, and their proposed evaluation is a solid measure of whether these tools provide utility.
• The authors are upfront about the limitations of their work and the need for diverse techniques other than theirs to create an effectve benchmark for interpretability tools
Weaknesses
• The trojan tasks are fairly simplistic and might not measure the actually important capabilities needed by interpretability tools in the real world. For example, if a tool performs well on this benchmark, it's not clear what that would tell me about how to use it in interpreting my model outputs for a misclassified, nonadversarial image. The trojan task is a proxy, and I would have loved to see how the "best" models under this proxy can actually be used for a useful interpretability task like identifying why a model is misclassifying certain images. I would have particularly liked to see how their proposed SNAFUE would work in a real interpretability task.
• I'm not sure the authors' evidence for the ineffectiveness of saliency maps is well-founded. They use $\ell_1$ error between the saliency map and the location of true patch, which seems like a strange objective. If the patch is small, the $\ell_1$ distance of a perfect saliency map would decrease with patch size, meaning an all-zeros baseline would appear great. If you used a different metric like bounding box intersection you wouldn't get such a strong negative result. *This isn't a huge deal* because this entire section seems not incredibly relevant to the paper, since your main goal is to do this task without access to the adversarial examples.
• It took me several reads to make sense of this paper. The writing is not completely clear and the figures took a while to interpret correctly. I would suggest really working on finding a better way to explain the human study task to your readers.
Questions
• On the first few reads of the paper, I thought the goal was to show which tools are best at identifying backdoors in DNNs (the title of the paper further corroborates this interpretation). If that is the case, I would have many questions about why we should care about these artificial tasks rather than, say, a cryptographic planted sequence that can't be inspected or interpreted visually. After several reads, I think what you actually care about is just evaluating how good these "humanly interpretable" tools are, and the trojan task is just a proxy for this. This was not obvious to me and I would suggest a better title and a clearer intro to better emphasize the goal of your paper.
• I wasn't sure about the details of your human study. Was it 100 participants per method or 10 per method? It would also be great to see some error bars here.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have a healthy discussion about the limitations of their work. Namely that:
1. They do not cover all possible types of features that may cause model failures.
2. Evaluations are based only on multiple choice questions, which may be sensitive to aspects of survey and experimental design