Summary
The paper introduces a benchmark designed to evaluate the safety alignment of large language models (LLMs) when used in scientific domains like chemistry, biology, medicine, and physics. The authors highlight the critical risks LLMs pose in these fields, such as the generation of potentially harmful compounds, pathogenic sequences, or instructions for synthesizing such substances. The benchmark includes a robust and comprehensive dataset of over 31k examples and evaluates models under various prompting settings, including zero-shot, few-shot, and chain-of-thought.
Strengths
1. The paper makes a substantial contribution by providing a large-scale, multidisciplinary benchmark tailored to the needs and risks of scientific language modeling. It addresses a timely and critical issue in the application of LLMs to scientific domains.
2. The introduction of multi-disciplinary safety evaluations and the jailbreak prompt feature seems valuable.
Weaknesses
1. The motivation for creating instruction-tuning datasets using *only harmful compounds* from hazard databases is not well justified in terms of real-world applicability. It seems the authors collected substances associated with hazard-related tags and created an instruction dataset using templates from prior work, followed by prompting experiments. However, the practical impact and real-life utility of these experiments remain unclear, especially since real-world hazard assessments are often multi-layered especially for chemistry.
2. The authors aim for the LLM to not respond at all when there is a risk of mentioning hazardous or toxic substances. If the LLM refuses to respond even to legitimate and safe queries simply because the query involves a topic or substance that can be hazardous in different contexts, it could hinder useful and informative interactions. An improved strategy could involve the LLM generating safe, context-aware responses that include warnings or safety instructions. Then it raises the question: how to evaluate such responses.
3. The notion of hazard or toxicity in fields like chemistry or medicine is nuanced. For example, certain levels of toxicity might be acceptable under specific conditions: dose, exposure duration, specific patient contexts, etc. Thus, suggesting that a simple binary approach to harmful versus non-harmful substances might be insufficient.
4. The evaluation framework appears to be incomplete -- it focuses solely on accuracy in cases where the model fails to refrain from responding to instructions involving hazardous substances. What about the cases where the model fails to respond to instructions related to non-hazardous substances? Such cases would equally impact the model's practical utility and should be accounted for in the evaluation.
5. While the paper successfully identifies vulnerabilities in current LLMs, there seems to be little contribution to actually address this issue. The identified issues are somewhat already familiar to researchers, hence limiting the major contribution of this work to the introduction of the dataset which is crowdsourced from existing work.
6. Since the evaluation heavily relies on adversarial prompting (jailbreak prompts), there is a risk that the models may have been inadvertently tuned or overfitted to specific types of prompts used in the benchmark. What if the prompts do not adequately represent the full range of possible adversarial strategies?
Questions
1. Could the authors provide insights into the underlying mechanisms that make certain models more vulnerable than others, such as differences in training data or tokenization strategies?
2. Could the authors provide more details on the diversity of jailbreak prompts?
3. For few-shot experiments, how exactly are the "3 representative examples demonstrating effective strategies for handling malicious prompts" selected? Any quantifier? The example in G.1 shows a case for DNA classification. What about the other tasks
Ethics concerns
How do the authors ensure safe and responsible usage of the dataset?