JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks

Summary

The authors investigate whether the jailbreaks that are effective in eliciting problematic responses from LLMs, also transfer to multimodal LLMs. To this end, the authors build upon an existing benchmark and expand it to include similar unsafe prompts (textual expansion), and various related/unrelated images (visual expansion), providing a multimodal benchmark comprising 28k examples. The authors find that the prompts that elicit unsafe responses from LLMs, also transfer to multimodal LLMs. Furthermore, these unsafe prompts are effective regardless of the image input associated with them – whether related or not. This work adds to the existing volume of work on safety evaluations on LLMs, and to this end presents interesting insights about the transferability of unsafe prompts between LLMs and multimodal LLMs. However, I have some concerns about the motivation and phrasing of the work, as well as the methodological choices of the work.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- Evaluating the tendency of multimodal LLMs to respond to unsafe multimodal prompts is an important and timely problem - The paper is easy to follow, and well-structured - I particularly liked how the authors expanded the unsafe textual prompts to include related/noisy images to increase the coverage of their experiments

Reasons to reject

- Major concern: The phrasing of the work is confusing and somewhat inconsistent . The main contributions of the work is to assess the transferability of unsafe textual prompts for LLMs to multimodal LLMs. To this end, the authors construct a synthetic benchmark that adds images to known unsafe prompts. I think the claims (and the title of the paper) should reflect that the benchmark is for evaluating the transferability of unsafe textual prompts, and is not a general-purpose benchmark. This is important because, in principle, multimodal LLMs, have a greater attack/jailbreak “surface”, as the input can span multiple modalities. However, from that broad attack/jailbreak surface, this “benchmark” only evaluates for a very specific, and perhaps, narrow subset of jailbreaks. - Methodological weakness: Now, moving to the methodology that is adapted to construct the synthetic benchmark. First of all, the textual expansion of unsafe prompts is not IID, as the prompts to generate the unsafe prompts asks the LLM to generate 100 examples at once. Even though the authors have subsequent steps to filter out highly similar unsafe prompts from the generated set, it is unclear why the authors did not generate one unsafe prompt at once, which would have been a close approximation to the IID condition. Furthermore, there is not evaluation of the quality of these expanded unsafe prompts. - Lastly, as a minor point, while discussing the results pertaining the role of image input, I would suggest the authors to clarify that in situations where the image encoder is not frozen, it is possible to leverage the image input much more prominently to design a jailbreak. Very minor comment: - The citations are rendered without parentheses in several places throughout the text

Questions to authors

**Q1:** Could the authors clarify why the prompt to expand the unsafe prompts instructed the LLM to generate 100 examples at once, instead of generating them one-by-one? **Q2:** Are there any evaluations that establish the quality of these expanded prompts – perhaps on a subset of the data?

Summary

The paper proposes JailBreakV-28K, a comprehensive dataset consisting of 28K test cases to evaluate the susceptibility of multimodal large language models (MLLMs) to jailbreak attacks from textual and visual inputs. By generating 20,000 text-based prompts that successfully jailbreak LLMs and combining them with different image types, as well as constructing 8,000 image-based jailbreak instances leveraging state-of-the-art techniques, the paper demonstrate the high transferability of LLM jailbreak methods to MLLMs. Extensive experiments on different open-source MLLMs show the significant vulnerability, with text-based attacks proving really effective, regardless of the image content.

Rating

7

Confidence

3

Ethics flag

2

Reasons to accept

- This paper is well written, and its motivation is very clear. - The JailBreakV-28K benchmark can serve as a valuable resource for researchers and developers working on improving the safety and alignment of MLLMs - The experimental results obtained using JailBreakV-28K reveal critical vulnerabilities for today's MLLMs. The high success rates of transferred LLM jailbreak attacks, particularly text-based attacks, underscore the pressing need for improved defense mechanisms. - The paper's insights have practical implications for the development and deployment of safe and robust MLLMs.

Reasons to reject

- Given the existence of numerous jailbreaking MLLM benchmarks in prior research (Table 1), it appears that consolidating them could potentially create a robust jailbreak benchmark. Therefore, it can be argued that the originality of the benchmark proposed in this paper might not be particularly significant. - Although the benchmark was proposed to uncover jailbreaking issues, it is crucial to consider the potential problems of releasing the dataset, as it could lead to numerous security challenges. However, I did not find any statement on how to release the dataset in a responsible manner.

Questions to authors

- The majority of current MLLMs are derived from LLM (i.e., build MLLM after the pre-training of LLM such as LLaVa), which suggests that they should be sensitive to LLM's jailbreak techniques (the conclusion of this paper). Have you conducted any comparative analysis on different MLLM solutions to see which technical approach is the most robust against LLM attacks?

Ethics concerns details

The benchmark may contain a lot of harmful queries, but this paper lacks discussion on the potential problems of releasing the benchmark.

Summary

This work collects various high-quality jailbreak attack examples from existing works and generates a larger machine-crafted example set to enlarge the JailBreak datasets. Also, they incorporate images into several examples and create a new multimodal dataset for safety research. During the process of collecting/generating data examples, the authors provide several heuristic rules and deduplication methods for improving the quality of the dataset. Based on the generated dataset, authors find that the MLLMs do inherit the vulnerability to specific jailbreak examples from the LLM counterparts.

Rating

7

Confidence

4

Ethics flag

2

Reasons to accept

1. Research about inherited vulnerability between MLLMs and their LLM counterparts is interesting. 2. Authors made complex data generation and filter pipelines for getting high-quality datasets on jailbreak. 3. A set of fully model-generated jailbreak examples is an interesting dataset.

Reasons to reject

The llama-Guard-based jailbreak evaluation method may introduce many biases. A more complex evaluation method may give further possibilities.

Questions to authors

How long does it take for each generation steps take? Please list the GPU type, clear time schedule for evaluating the cost of each step for this data generation process.

Ethics concerns details

Generate Jailbreak data examples may help misuse open-source LLMs

Summary

This work investigates the vulnerabilities of MLLMs towards jailbreak attacks, specially on the transferability of llm jailbreak attacks. To this end, the authors construct a benchmark including (1) RedTeam-2K malicious query set (2) JailbreakV-28K jailbreak prompts, and conduct experiments across 10 MLLMs to assess their robustness against jailbreak attacks. The results indicate that the MLLMs are still quite vulnerable towards LLM based jailbreak methods.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The gathered malicious queries and chosen multimodal jailbreak prompts cover a comprehensive range of different safety policies. If the benchmark is made public, they could be a valuable resource to the community.

Reasons to reject

1. Though this benchmark includes numerous MLLMs, the results on state-of-the-art (SOTA) closed-source models like GPT-4V and Gemini remain intriguing. Could the authors provide some analysis of these models? 2. It's expected that MLLMs with the same LLM backbone would exhibit high vulnerability, but a more thorough analysis could be beneficial. For instance, what about the transferability from different LLM models to different backbone LLM models? Figures 4 and 5 only provide partial insights. 3. The jailbreak methods based on visual input are not fully comprehensive. Some common methods, such as VisualAdv[1], ImageHijack[2], and Compositional[3], are absent without explanations. 4. The availability of the code and evaluation datasets is unclear. Will they be made public?

Questions to authors

1. The paper does not report the variances of ASR and can the authors give some observations about the variances of different jailbreak prompts? 2. Sometimes, the generated output is harmful but can be unrelated to the input. In this case, do the authors treat it as successful or failed attack? 3. Is the high ASR across these MLLMs due to the backbone LLMs not being strongly aligned? 4. The authors use Llama-guard to calculate the ASR. Some policies in this benchmark are not included during the Llama-guard training, such as economic harm and malware. Can the system prompt be simply extended to add new policies? Further explanations on this matter would be beneficial. 5. There are some minor typos: - 3.2.1 first line: it should be "the first stage." - The citation format is incorrect and inconsistent. - The authors should include a citation for Llama-guard. - In Table 2, some text is bold but lacks explanations.

Thanks for you clarification

Thanks for the rebuttal. I would like to raise my score based on the clarified novelty. Best, Reviewer

We will be glad to receive your further feedback

Dear Reviewers, As you might know, we are approaching the discussion period deadline. We deeply appreciate your comments and made detailed responses to all your concerns. If your concerns have been addressed, we sincerely appreciate that you can update the score. Please feel free to reach out in case you need any clarification. Best Regards, Authors.

We will be glad to receive your further feedback

Dear Reviewers, As you might know, we are approaching the discussion period deadline. We deeply appreciate your comments and made detailed responses to all your concerns. If your concerns have been addressed, we sincerely appreciate that you can update the score. Please feel free to reach out in case you need any clarification. Best Regards, Authors.

Response to rebuttal

I would like to thank the authors for their response, and for presenting additional results. Regarding the point on generating 100 samples at once, as opposed to one-by-one, I now see the authors' rationale, and would encourage them to include this in the updated version of the paper. Regarding the comprehensiveness of the benchmark, I am still on the fence regarding whether or not to the attacks covered in this work are comprehensive for multimodal models. Multimodal models do have an increased attack surface area due to multiple input streams (covered in this work to some extent), but ALSO because of how data in these input streams interact with each other (unexplored in this work). The benchmark takes existing unimodal attacks and seeming mix-and-matches them to build multimodal attack, without leveraging the interplay that could exist in these modalities. I believe this is a limitation that should be made more prominent among the major claims of this work, and also plays against the novelty of this work. Nonetheless, since the rationale for generating 100 samples at once makes sense, I am increasing my score to reflect that.