Response to Reviewer WKXw
**Thank you for your detailed review. We think there are some misunderstandings and we will clarify them.**
> All revisions are highlighted in purple.
## **For Weakness 1**
**First, we would like to clarify that GPT-4-judge is not commonly employed in safety interpretability research.**
The safety interpretability of LLMs, is still an emerging field with relatively few studies.
In the existing research, for example, two closely related works[1, 2], they both used keyword detection instead of GPT-4-judge.
Other related works, such as [3, 4], [3] used the model in [5] to evaluate toxicity, while [4] utilized a fine-tuned Llama from [6] without employing GPT-4-judge.
The primary reason for this is that evaluating ASR with GPT-4 is prohibitively expensive for interpretability research, particularly when compared to jailbreak studies.
Jailbreak evaluations typically require only **a few hundred data points**, while safety interpretability requires comparisons across **hundreds of thousands of data points** under different settings to analyze the parameters.
To illustrate, we estimated the cost based on the prompt template from [7] and the paper you referenced.
Their prompt templates have more than 400 tokens, so, every request needs approximately 600 input tokens for evaluating 128 tokens generation (the outputs may not have a fixed number, we omit them now).
If we were to produce with GPT-4-judge, we would require roughly 120M tokens: 600(input tokens) * 1024 (head) * 100(dataset size) * 2(dataset) tokens, resulting in an estimated cost of 3,600 dollars for a single figure (30.00 dollars per 1M tokens for GPT-4)
Since our work is more fine-grained than other works, it would cost over 10,000 dollars solely for input tokens across the entire paper, excluding any additional debugging or analysis costs.
In contrast, a jailbreak evaluation typically costs around $3.60 for a similar sample size (0.12M tokens).
**Thus, GPT-4 is rarely, if ever, used in safety interpretability research, which differs significantly from jailbreak studies**
We sincerely hope that you will understand our choice of keyword detection, which is commonly employed in interpretability research due to both cost and efficiency. We believe it aligns well with the objectives of our work.
## **For Weakness 2**
**Our improvement is not quiet limited, the increase from 0.27 to 0.55 is a significant improvement-it has effectively doubled.**
In fact, the performance on Vicuna might seem limited because it is inherently not as safe as Llama, resulting in a smaller multiple of improvement compared to Llama.
Nonetheless, this improvement is substantial enough to support our conclusions.
## **For Weakness 3**
Thank you for pointing out the numbering issue.
There is indeed an oversight here.
**Figure 5a and Figure 5b have been correctly numbered as 6a and 6b**.
## **For Weakness 4**
We apologize for the oversight regarding Figure 6b.
The experimental results presented in Figure 6b are from Llama-2-7b-chat, and we have added this clarification in Section 5.3.
We believe your concern may stem from a misunderstanding.
A harmful answer must also be helpful for malicious user[8], and our helpfulness evaluation on benign questions indicates that the model’s understanding of questions drops by about 15% overall.
This level of degradation is generally considered to be insignificant in pruning methods [9, 10], which are chasing utility.
We argue that our helpfulness analysis actually strengthens our findings rather than undermining them.
## **For Q1**
**We appreciate your suggestion, but adding a metric based on GPT-4 is challenging for us due to the significant cost involved, which is one of the reasons GPT-4 is rarely used in safety interpretability research, particularly in our case.**
Regarding human judgment, we understand your concern and appreciate your suggestion.
While we did conduct small-scale manual reviews, they were not extensive enough to cover all generations due to similar cost constraints (as mentioned in Weakness 1).
We also considered sampling before human evaluation, but this would still require us to process 1024 groups. Even if we were to sample just 10 items per group (which we acknowledge would be an inaccurate approximation), we would still need to evaluate 10,240 data points.
Given these challenges, we decided not to include this approach in the paper.
We hope you understand the difficulties we faced in this regard, and we sincerely appreciate your understanding.
## **For Q2**
We believe it would be helpful to first address Weakness 2, as it is directly related to your concern. Once we have clarified that point, we will then provide a detailed response to this question.
Limited to character, references are given in an additional comment.
---
We sincerely hope that our response addresses your concerns and encourages you to reconsider your score. We would be more than happy to engage in additional discussions.