Effective Prompt Extraction from Language Models

Summary

The paper introduce a new threat to language model that the system prompt could be extracted through user prompt query. Specifically, the paper designs several prompts aiming on letting the model to output what have been given in the input. He then uses a gathered dataset to train a detector to check whether the extracted prompt is the system prompt or not. For the closed source model such as GPT, the paper further proposes a translation based attack that utilized the translated designed prompt to evade the filter. The experimental results show the proposed method could successfully extract system prompt with a high accuracy in a series of LLM including both open-sourced and real-world systems.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. The proposed setting is very interesting and practical. The prompt extraction attack could be regarded as a new threat to language model system. 2. The experimental results show the proposed method could extract the system prompts with high accuracy in a wide range of LLMs including both open-sourced and real-world systems.

Reasons to reject

1. The paper needs some serious polish. Several important details are missing which makes the paper not that convincing. Please refer to the questions section. 2. The system prompt definition is not clear. The open-sourced model's system prompt is just the first paragraph of the user prompt however the real system is something hidden in the back-end. If the same standard is applied, the real system could still add the first paragraph of the user prompt act as the system prompt. 3. The proposed finding is interesting however it is purely based on the heuristic. All prompts tested are letting model to repeat itself. I am not sure whether this threat could be fixed very easily by letting the LLM paraphrase his own words. Minor: 1. Section 4.2 we analyze conduct -> we conduct

Questions to authors

1. The indictor function in metric definition is not defined. If it is the indictor function, it should get 0 or 1 results. Why does the result later show it is a real number? 2. How to create the Dev dataset of 16000 extractions? Details are needed. 3. How to set the permutation $\pi$? Details are needed. 4. Since there is no opened system prompt in GPT-3.5 and GPT-4, how to calculate the accuracy in Section 5?

Summary

The paper introduces a method for prompt extraction that employs attack queries to extract several candidate prompts, and then chooses the candidate with the highest confidence estimate as the most likely prompt.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

The prompt extraction task is interesting, and the author has conducted comprehensive experiments across multiple models.

Reasons to reject

1. This method lacks novelty. [1] has already proposed a similar approach for extracting prompts via multiple "Jailbreak" queries interacting with LLMs, and they also introduced the "Inversion Model" (IM), which not only surpasses "Jailbreak" but also achieves SOTA results. However, the advantages of this paper’s method compared to "IM" and "Jailbreak" remain ambiguous, and it lacks a comparative analysis of performance. 2. The importance of each step in the method is unclear and lacks theoretical support. For example, the necessity of training a DeBERTa model in Section 3.2 is unclear. What theory it is based on? If the goal is to identify common parts across multiple extractions, simpler methods or the LLM itself might suffice. The authors should provide adequate theoretical explanations, ablation studies of different parts, and comparisons with more straightforward strategies. 3. It extracts prompts based on 105 attack queries. Why choose 105 attacks? Why not more or fewer? The impact of this specific quantity on the overall performance of the extraction process remains unclear and needs further investigation. 4. The test sets featured in the main Table 1—including Unnatural Instructions, ShareGPT, and Awesome-ChatGPT-Prompts—comprise only a few hundred prompts each. This limited quantity is insufficient to demonstrate the effectiveness of the method. 5. The author emphasizes extracting the system prompt of LLM and presents the extraction performance of GPT-3.5-turbo and GPT-4 in Table 1. However, this performance is evaluated by comparing the extracted prompts to those in the benchmarks, rather than the real system prompts of LLMs. This evaluation appears unreasonable and does not effectively demonstrate the validity of system prompt extraction across various LLMs.

Questions to authors

See above.

Summary

This paper presents a framework for systematically measuring the effectiveness of system prompt attacks. They use the simple templates to extract the system prompt first, then use a pre-trained small models to decide which system prompt has the highest probability. In experiments with 3 different sources of prompts and 11 underlying large language models, they find that simple text-based attacks can in fact reveal prompts with high probability. Their framework can determine with high precision whether an extracted prompt is the actual secret prompt, rather than a model hallucination. Except the smaller LLMs, they also find ChatGPT and Claude's system prompts can be revealed by an adversary despite existing defenses in place.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

I think this paper is quite interesting, especially the proposed evaluation framework. I think people know that we can extract the system prompt to some extent, but the author propose a novel framework for this, and the framework is quite effective.

Reasons to reject

N/A

Questions to authors

Thanks for this great submission,I have the following question for you: (1) Have you tried other models except DeBERTa in your experiment? I think if you can do model ensemble, it potentially may further increase the precision.

Summary

This paper present a study of prompt extraction, which is to extract the system prompt based on designed interaction. A framework is designed to systematically measure the effectiveness of these attacks, where one major difficulty is to discriminate a successful extract with a hallucination. To solve the problem, the authors propose to treat the majority extraction results as the oracle secret prompt. And trained a model to predict the probability of one extracted prompt to be successful. Experiments on several popular LLMs such as Alpaca, Vicuna, Llama-2, GPT3.5, GPT4 are presented, showing that the prompt attacks are successful.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

It is interesting to see the research on prompt extraction, which is a safety direction that has been less considered.

Reasons to reject

I have concerns about the correctness of using the majority of extracted prompt as the oracle. It is indeed possible that the model generate the similar results for the similar queries for the prompt. It is strange that different type of prompts have very different rate of being extracted (In Table 1). For example, Unnatural is much higher than the other two prompt datasets. It might be useful to include more discussions.