Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models

Summary

This paper explores an interesting issue in current applications of LLMs: false refusals. Unreasonable refusals can harm the user experience. This paper proposes **Auto-Regressive Controllable Prompt Generation** for generating false refusal text, based on AutoDan and three objectives. This method can generate natural prompts and control the harmfulness of the prompts and the probability of being refused. Based on this method, this paper proposes a new dataset, PHTest, containing 1.5k prompts usable for detecting false refusals. The authors made comprehensive evaluations of state-of-the-art large models, including commercial-level close-sourcde LLMs and open-source LLMs.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

* This paper brings our attention to the issue of false refusals. * The author proposes an automatic method for generating text for false refusals. This method can be easily applied to generate more prompts to evaluate LLMs * This paper already provides a new dataset covering several categories of false refusals, with a preliminary evaluation conducted on a range of LLMs. Overall, I think this work makes a positive contribution to the community.

Reasons to reject

Although this has been discussed in detail in the paper, I still think that the definition of harmfulness in the text is not promising. Since the method used in the text is fully automated, the harmfulness of the generated text is also annotated by a model, which inevitably leads to unreliable harmfulness annotations in the dataset.

Questions to authors

Equation 1 includes two hyperparameters $\alpha_0$ and $\beta_0$. How did you pick these two parameters?

Summary

This paper focuses on an interesting problem LLMs may refuse pseudo-harmful user prompts and affect users’ experience. To accelerate the development of this domain, the authors propose auto-generating pseudo-harmful prompts, leveraging a white-box LLM to generate natural, varied, and controllable prompts. They construct a larger dataset PHtest and conduct extensive experiments on 14 models. They also discuss a trade-off between false refusals and safety against jailbreak attacks.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The problem that LLMs refuse pseudo-harmful user prompts is interesting and important for user experience. 2. The authors generate a larger evaluation dataset to evaluate the performance of many LLMs on False Refusal Evaluation. 3. The writing is clear and the logic of this paper is easy to follow.

Reasons to reject

1. The authors have one hyperparameter $\beta$ during the generation process of their dataset. At the end of section 3.2, the authors discuss the potential function of $\beta$. It’s better to give specific details about beta you choose or some simple experiments about the effect of $\beta$ to generate datasets. 2. In the Appendix, the authors set $y_{refusal}$ as “Sorry, I cannot assist with that.” It’s also better for authors to discuss the effect of different $y_{refusal}$ for generating datasets or give some case studies about this. 3. In observation 1 and Figure 9 about calculating AUC values, it’s better for authors to give a bit of detail about experiments.

Summary

This paper presents an investigation about the False Refusals in defending harmful prompt. The authors propose a method to generate a pseudo-harmful dataset, and use it to evaluate several popular LLMs. Analysis are presented to show the tradeoff between false refusals and successful defendings.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. While many work improve the successful defending rate, much less work put false refusal in an important position. This work brings out an interesting direction. 2. A dataset are created for evaluating the above problem, with detailed labeling of the type and reason of false refusal, which may be potentially helpful for further study. 3. The authors present analysis to help understand the different categories of false refusal (including newly identified categories), as well as the balance between false refusals and successful defendings.

Reasons to reject

1. The generation process of the dataset seems pretty standard. 2. It is interesting to find new categories of fasle refusal. However, even with the listed 10 categories, there might still be other cases that have not been studied yet. I am wondering how it is useful to reveal these detailed categories. For example, is it possbile to improve the alignment/defence on a given category

Summary

In this paper, the authors study the problem of falsely refusing pseudo-harmful prompts by LLMs. Specifically, the work proposes an approach to automatically generate pseudo-harmful prompts. Using this method, the authors construct a new dataset called PHTest which is larger in scale and more finely annotated than existing datasets for this purpose. They evaluate current LLMs on PHTest, revealing insights such as the correlation between model size and false refusal types, and the trade-off between safety against jailbreak attacks and low false refusal rates.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. The studied problem is highly practical and important for developing safer and more user-friendly LLMs. 2. The PHTest dataset is a significant resource, being 10x larger than prior datasets. A natural and fine-grained dataset can help identify the detailed drawbacks of existing LLMs and provide developers with insights on their capabilities, which is valuable for advancing the community.

Reasons to reject

1. The content-steering process could be explained in more detail. The writer LLM may not be able to generate desired pseudo-harmful prompts with x_{natural, harmless}. I think the generation quality is severely limited by the writer LLM and the reference prompt. How can the authors ensure that the writer LLM generates prompts within some specific categories, like "Compound term" or "Shared Property vs Privacy"? Additional experiments or analysis on the effectiveness of content-steering would strengthen this part of the methodology. 2. The generated prompts may not transfer to other LLMs as being pseudo-harmful. As the authors mention in Section 2, the definition of harmfulness is inherently complex and controversial. Using a single target LLM to guide the generation process could introduce biases specific to that model. To improve the generalizability of the generated prompts, the authors could consider using an ensemble of different target LLMs to mitigate model-specific biases.

Questions to authors

1. How can the authors ensure that the writer LLM generates prompts within specific desired categories, such as "Compound term" or "Shared Property vs Privacy"? 2. Did the authors experiment with using different target LLMs in the generation process to improve the transferability of the generated prompts?

reply

Thanks for the clarification. The datasets are indeed different in size. However the methodology in generating is similar to Zhu et al. I think the paper may have better influence in the dataset and evaluation rather than the method of generation.

Official Comment by Reviewer VMEF

Thanks for the response, my concerns have been addressed and I have raised my rating.