Toxicity Detection for Free

Summary

This work proposes to leverage logits of the first token in LLM responses to identify toxic prompts. The experiments against LLaMAGuard and multiple open-sourced LLMs show satisfactory performance in ToxicChat and LMSYS-Chat-1M datasets.

Strengths

- The proposed method is simple and easy to implement. - The presentation is clear. - Toxicity detection is a significant problem in LLM safety.

Weaknesses

- I don't think toxicity detection based on the first token makes great sense. For example, in the appendix, "sorry'' is one of the refusal tokens. However, it is also possible that the LLM expresses toxic contents in a "sorry ... but ..." format. - The success of the proposed method heavily depends on the refusal token list. How to ensure the generalizability of the trained SLR model? - The baselines to compare are relatively weak. The authors fail to include state-of-the-art LLM models such as LLaMA3, GPT-4, GPT-4 Turbo, etc. - There also exist a few works training a neural network to detect toxic contents which the paper did not mention such as [1]. How does the proposed method compare with them? - The datasets for evaluation are skewed and not very popular. Most of the samples are non-toxic. Is it possible to have a try on hh-rlhf datasets [2]? References [1] He, Xinlei, et al. "You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content." 2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2023. [2] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022).

Questions

- How to determine the list of refusal tokens? - How to extend your work to multi-class toxicity classification tasks (e.g., settings like LLaMAGuard)?

Rating

4

Confidence

4

Soundness

1

Presentation

3

Contribution

1

Limitations

The authors discussed one limitation in Section 7. But it can be more boardly.

Summary

This work proposes a toxicity detection method for LLMs that incurs negligible additional inference cost, and shows superior performance compared with two existing methods. The main observation of this work is that the logits of the first generated token (after prompt) are informative about toxicity of the prompt. The authors propose 2 baseline methods, PoR (crude probability of refusal) and PoRT (probability of refusal according to the logits of a pre-selected token, eg. "sorry"). They show that PoRT is effective, paving the path to their flagship method MULI. In MULI, a sparse logistic regression classifier is trained on top of the first token logits. L1 regularization on the weights is applied, as well as a non-linear function on the logits (ablations for both the regularization strength and the non-linear function are provided). MULI shows excellent performance compared with LlamaGuard and OMod, with lower computational complexity. The improvement over these methods supposes a jump in performance in toxicity detection. MULI also shows more robustness than the other methods to modality shifts (tested with 2 toxicity datasets).

Strengths

*Originality:* * This work proposes simple yet original solution to toxicity detection. Analyzing the logits of the first token is surprisingly effective, sound and simple. *Quality:* * This work contains experimental results on 2 toxicity datasets, a comparison with 2 existing methods and a comparison among several LLMs. The experimental setup is of great quality. * I really appreciated the experiment comparing LLMs (fig 6) and the ablation of dataset size to estimate MULI (fig 7). *Clarity:* * The paper is well written, all the proposed evaluations and methods are sound and well explained. *Significance:* * This work tackles the important topic of toxicity detection in intruction-tuned LLMs. This is an important topic nowadays, since practically the everyone is using LLMs in daily life, with millions of queries per day. Being able to better distinguish toxic queries from non-toxic with low computational methods is key to improve quality and also to reduce costs associated with LLM inference.

Weaknesses

*Quality:* * One possible weakness is the lack of justification to analyze the first token only. I missed some discussion on how is this token "attending" to the prompt or how can this method fail? Intuitively, this method can suffer from adversarially designed prompts, some analysis in this sense would be interesting. In general, a limitations section is missing.

Questions

* One question that immediately came to my mind while reading is what happens beyond the first token logits. My intuition is that including the 2nd token would disambiguate many more answers. I believe the same MULI formulation could be applied to the 2nd token by appending the 2nd token logits to the 1st token ones, and using that 2x larger vector to train the logistic regression. Some discussion on this would be great. * I missed some discussion on the possible jailbraking of MULI. How could MULI suffer from adversarially designed prompts, such that they are toxic but can deceive the logistic regression? * How does MULI perform with sub-work tokenizers, or tokenizers that strongly split words? In general, how is MULI impacted by the tokenizer, since only the 1st token is used. --- **Overall comment:** Despite the method's simplicity, it provides excellent results and major improvement in terms of compute. I find this combination very interesting, being able to combine simplicity and a leap forward in terms of results is hard to achieve. Moreover, the experimental results are solid and reproducable. Overall, I find this paper of good quality for NeurIPs and I recommend acceptance.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors mention that the method limitations are discussed in Section 7 (conclusion). However, section 7 only contains: _"Nevertheless, there are limitations, as the information hidden in the output of LLMs is much more than what we extract by MULI. It encourages researchers to look deeper into the information hidden in LLMs in the future."_ I encourage the authors to include a dedicated limitations section, with discussion on the topics mentioned in previous sections of this review. For example, how MULI could be jailbroken, limitation of using only the 1st token vs more, etc. Additionally, the authors should expand on _"the information hidden in the output of LLMs is much more than what we extract by MULI"_. What, in the authors opinion, could be extracted that MULI does not? Please, take these only as constructive suggestions.

Summary

The paper introduces Moderation Using LLM Introspection (MULI), which leverages the LLM's first token logits for toxicity detection. This is a novel approach compared to traditional methods that require an additional LLM for toxicity detection, thereby reducing computational costs and latency.

Strengths

- Efficiency: MULI achieves near-zero additional computational cost, which is a significant improvement over existing methods that often double the computational overhead by using a separate LLM for detection. - Performance Metrics: The paper demonstrates that MULI significantly outperforms state-of-the-art (SOTA) detectors in multiple metrics. - Practical Implications: By focusing on detecting toxicity based solely on the prompt, MULI allows for real-time blocking of toxic prompts before the LLM generates any response. This is particularly useful for streaming APIs and web interfaces. - Reproducibility: The paper stated that the author(s) will release code (L197).

Weaknesses

- While the results are promising, the paper does not provide enough information on how well the method generalizes to different types of toxic prompts. - The toy models make some simplifying assumptions, such as using specific refusal tokens like "Sorry," "Cannot," and "I." These assumptions might not hold in all scenarios (e.g. training data, models). - While the paper provides a high-level overview of the method and results, it lacks a detailed analysis of failure cases or scenarios where MULI might not perform well. Understanding these limitations is crucial for practical deployment.

Questions

- Line 212: is it should be “a tolerance of 0.1% FPR” instead of TPR? - What are the specific scenarios or types of prompts where MULI might fail to detect toxicity accurately? Qualitative analysis would be beneficial for future work. - How does the choice of refusal tokens affect the performance of MULI? Would the method be robust to changes in these tokens, and how can it be adapted to different languages or dialects as foundation models like Llama-3 now are trained on multilingual data? - How does MULI handle edge cases, such as prompts that are borderline toxic or ambiguous in nature? What strategies can be employed to improve its robustness in such scenarios? [1] introduces a type of implicit hate speech that is very ambiguous in nature and [2] recently shows that LLMs are struggling to detect this implicit one. - Overall, having qualitative results on failure cases would be beneficial for the paper besides the existing great quantitative performance. [1] https://arxiv.org/pdf/2109.05322 [2] https://arxiv.org/pdf/2403.16685

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

NA

Summary

This paper introduces a novel approach to detecting toxic prompts in LLMs using a method called Moderation Using LLM Introspection (MULI). The authors highlight the limitations of SOTA toxicity detectors, which often have low true positive rates (TPRs) at low false positive rates (FPRs) and incur high computational costs. The main motivation to develop this approach is that information is hidden in the LLMs' outputs that can be extracted to distinguish between toxic and benign prompts. Therefore, MULI leverages the logits of the first response token from the LLM to detect toxicity, eliminating the need for additional classifiers and reducing computational overhead.

Strengths

- The paper introduces a novel method that uses the introspective capabilities of LLMs to detect toxic prompts, which is both cost-effective and efficient. - Results show that MULI significantly outperforms existing SOTA detectors in various metrics, particularly in achieving high TPR at low FPR, which is crucial for real-world applications. - The biggest strength to me is that MULI eliminates the need for additional classifiers, reducing computational costs and latency. - The approach can be applied in real-time settings, including streaming APIs, making it highly practical for deployment.

Weaknesses

- One weakness of this work is its limited scope of evaluation. While the paper evaluates MULI on specific datasets (ToxicChat and LMSYS-Chat-1M), it would benefit from testing on a broader range of datasets to ensure generalizability. - Another issue of this approach is its dependency on LLM Quality. The effectiveness of MULI is highly correlated to the alignment quality of the underlying LLM. It is not clear whether poorly aligned models can provide reliable logits for toxicity detection.

Questions

The method relies on logits, which may not be easily interpretable. Do authors have an explanation for why certain logits indicate toxicity? Can you investigate the impact of different alignment techniques on the performance of MULI to understand its dependency on the quality of the underlying LLM?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- As I mentioned above, the main limitation of this approach is its dependency on LLM quality and alignment. - Besides, generalizability across different types of toxic content is still an open question and maybe a limitation. - The method based on output logits, but more information can be hidden in LLMs' outputs. Further investigation on toxicity detection with LLM outputs is required.

Summary

The authors of this paper propose an approach for detecting toxicity of prompts from strongly aligned models (that are trained to refuse toxic prompts) using their Moderation Using LLM Introspection (MULI). Key to this approach is the observation that even though LLMs may not refuse toxic prompts always at very high probabilities (thus leading to actual refusal response), the probability of certain tokens associated with refusal rises above the average when they see toxic prompts. They build a sparse linear regression model using the probabilities for the first token output to detect toxic prompts and show they can sometimes achieve high TPRs at low FPRs using standard datasets.

Strengths

The idea itself is very interesting and is a refreshing take on toxicity detection. The method does seem original and the paper is easy to read - I appreciated how they started building up their problem with small toy examples and then proceeded to develop their SLR method. Some of the other key strengths that I noted: - The insight that even though the refusal response may not rise to the top, it may have substantially high probability for toxic prompts. - Easy way further enhance the performance of well-aligned LLMs like llama-2-7b or build detectors using them.

Weaknesses

Please construe these broadlyPlease see weaknesses section. as comments and try your best to respond to them: - Need to mention somewhere that this will work only for current LLMs that are safety aligned in a certain way and using specific refusal responses. - Let’s say we want to guardrail toxic prompts for a specific fine-tuned LLM using MULI. There will be a certain additional inference cost when doing this (even if it producing just one token output). This can be discussed somewhere. - The interpretation of SLR weights was a bit murky for me. Are the authors trying to show that the refusal tokens typically have coefficient values that lead to positive predictions (toxicity labels)? Also how was the $\lambda$ in SLR (eqn. 5) chosen? Standard cross validation?

Questions

Please see weaknesses section.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations are discussed in just one sentence - perhaps this can be expanded. One limitation I could think of was that you need a well-aligned model already and an infrastructure to run it (even if it is for just producing a single token output).

Acknowledgement

Thanks to the authors for their responses. I keep my score as-is and recommend acceptance.

Thanks.

Thanks for the clarification. I read the other reviews as well and am inclined to recommend acceptance. However, a few more clarifications: For Q2 - What I meant was this. Lets say a user wants to creates guardrails for a specific LLM using MULI. However, if that LLM itself is not safety aligned, they will have to use another safety-aligned LLM to get the first token logits. This is only one token inference but does add to the cost since the safety-aligned LLM has its own inference cost. For Q4 - This is interesting. If it is not sensitive, does an un-regularized logistic regression work? May be there is a broad range of $\lambda$ values.

Reply to your rebuttal

Thank you for your clarification. I still have further questions about your rebuttal. Regarding your response to **Q3**, although GPT-4/GPT-4 Turbo cannot allow users to access the logits of the first token, they are indeed strong baselines to detect toxic content. Calling GPT-4 API is expected to be much faster than your proposed method (although it may introduce some cost) since your method still relies on the inference of LLMs. I am wondering how your (free) method compares with commercial ones so that the users can balance the financial cost and the detection performance. Regarding your response to **Q7**, as you mentioned "MULI learns a simple classifier that is specific to a single LLM", MULI has to be re-trained from scratch for each new LLM which implies a non-trivial computation cost. For example, if we want to obtain a MULI-based toxicity detector for LLaMA-70B, re-training from scratch might be very time-intensive and computationally inefficient. A single A40 GPU may not be able to finish this task. I am wondering if it is possible to reuse between different versions of LLMs if they share the same vocabulary list? I will raise my score if my concerns are addressed.

Thanks for the rebuttal

I thank the authors for their clear rebuttal answers, and the commitment to add a limitations section. I will maintain my score, I believe this paper deserves an accept.

Thanks

Thank you for the thoughtful suggestions. We will include them in the limitation and conclusion in the final version of the paper.

Additional adjustments to our paper

We thank the reviewers for the additional comments. Here are some additional adjustments to our paper. 1. We will include two more commercial models, GPT-4o and GPT-4o-mini, as baselines. They are both suboptimal compared to MULI in detecting toxicities on ToxicChat and LMSYS-1m. 2. We will include ethical concerns in the limitation section.