Logits of API-Protected LLMs Leak Proprietary Information

Summary

This paper presents an algorithm to extract proprietary LLM information from an API-protected black-box LLM. The algorithm uses the observation that LLM outputs are "softmax-bottlenecked", or they occupy only a low-dimensional subspace of the `R^vocab_size` output space. Using this observation, along with logit bias access to APIs, the algorithm can efficiently extract this low-dimensional subspace. This subspace can be leveraged to 1) infer the hidden dimension of the softmax layer; 2) extract a fingerprint of the API, which can be used to monitor API updates (like LLM changes). The algorithm is tested in a real-world setting on the OpenAI GPT3.5 Turbo model, and is shown to work with high accuracy.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. This paper presents technically solid work, with a very creative algorithm to extract proprietary information with high accuracy from a black-box LLM. 2. The presented attack is on a real-world LLM used by millions of people (the OpenAI GPT3.5 Turbo model), and not just in toy settings. This is quite interesting and impressive, given how secretive OpenAI has been about their model's internals. Moreover, the proposed algorithm is economically feasible (< 1000$ API credits). 3. This paper is very well-written, with a good mix of low-level linear algebra details of the algorithm, and high-level discussions on the implications of the attack. 4. The biggest implication of the method (in my opinion) is forcing more transparency from API providers on silent backend changes. Section 6 shows that even consecutive checkpoints can have very different images, which means that extracted fingerprints are fairly sensitive, and can be used to monitor API updates.

Reasons to reject

This is a great paper overall and a good fit for COLM. I have a few concerns, but overall I think the strengths outweigh the weaknesses. 1. The method (as well as prior works), rely on API providers allowing users to adjust logit biases. Based on Section 8, this method will not work without logit biases. However, I think logit bias is a fairly limited hack from OpenAI's perspective, rather than a long-term solution for controlled decoding. With rapidly improving instruction following capabilities, I think models maybe able to achieve effects similar to logit bias with just natural language instructions in the prompt, and get more natural-looking output. For reference, I don't see logit bias access to the Claude API (https://docs.anthropic.com/claude/reference/messages_post) or the Gemini API (https://ai.google.dev/api/python/google/generativeai/GenerationConfig). 2. Overall, I think the amount of information leakage from the attack is fairly limited, which limits the implications of the work. The only major information extracted is the dimensionality of the softmax matmul layer, which leaves an auditor speculating about the rest of the giant LLM (the more important proprietary information). The signature / fingerprint aspect of the attack is definitely interesting, but Section 6 and Table 2 makes me think there are too many variables that could change the API signature. This could include innocuous changes like timestamp in the hidden prompt, or search results in the hidden prompt of a RAG-enabled API. Watermarking, routed models, prompt preprocessing, output post-processors (safety filters?) could also change signatures (see questions below). Given that the APIs are rapidly changing due to industrial competition, frequent changes in API signature could leave auditors confused. 3. (minor, likely an unreasonable request) I was a bit surprised that experiments were only done on one proprietary model, and one family of open-source models. In particular, were there some cost bottlenecks to run the experiment on GPT-4 Turbo models?

Questions to authors

Questions: - What happens if the logits are postprocessed before being sent to the softmax? In particular, I think the addition of a watermark [1] could change the image of the LLM without changing the underlying LLM model. I also have a similar question for temperature changes in the API. - How well does the method work with APIs which are not a single LLM (they route queries between two or more LLMs, like [2, 3, 4]). Does that muddy the image procured by this method, possibly adding to the estimated "effective dimension"? What if routing weights are changed? Similar question for the mixture-of-softmax method from [5]. Typos / presentation: - The paper is missing a related work section, and I think more references would help in better grounding the work. Notably, the work is closely related to hyperparameter extraction and model extraction efforts (some examples: [6, 7, 8, 9]). - page 3: "we turn out attention" --> "we turn our attention" - page 3/ 4: Figure 3 is confusing and I don't think the translation to the clr space is helping my understanding (possibly better for the Appendix). Please correct me if I'm wrong, but my understanding reading the text is that everything after the softmax matmul matrix (logits, softmax, log-softmax) are in a d-dimensional subspace of R^v. Figure 3 feel like there is further reduction in image dimensionality from logits --> softmax-out (which I don't think is true). Here's an alternative suggestion: Present a table with three columns being (transformer operation, vector space dimension, effective image dimension). Something like the table below: (transformer operation, vector space dimension, image dimension) (end of transformer sub-blocks, R^d, R^d), (softmax MatMul `l = Wh`, R^V, R^d), (softmax, `p = e^l / sum(...)`, R^V, R^d), (log-softmax, `q = log(p)`, R^V, R^d) etc. Accompany this table with a simulation / points in a space where V = 3, d = 2, and one figure accompanying each of the rows above. The first row could be a 2-d plot, and all others 3-D plots with the points in a 2-d surface. [1] - https://arxiv.org/abs/2301.10226 [2] - https://arxiv.org/pdf/2310.12963 [3] - https://arxiv.org/abs/2207.07061 [4] - https://arxiv.org/abs/2211.17192 [5] - https://arxiv.org/pdf/1711.03953 [6] - https://arxiv.org/abs/1711.01768 [7] - https://arxiv.org/abs/1802.05351 [8] - https://arxiv.org/abs/1910.12366 [9] - https://arxiv.org/abs/2305.15717

Summary

This paper discussed an interesting method to partially recover some important meta data of black box LLM, primarily the hidden dimension before embedding matrix. The core idea is that the output logit vectors reside in dimension $d$ which is much smaller than vocabulary size $V$, thus a singular value decomposition over a collection of inferred logits matrix successfully recovers the actual hidden dimension. The authors further discussed extensions such as inferring the model that generates certain outputs or auditing for any model updates, etc.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

Though the core idea of this paper appears to be similar to [Carlini 2024], I think following are reasons to accept this paper: 1. This paper made a clearer algorithms and their complexities under different assumptions (such as w/ logprobs, numerically stable, scholastically robust, etc.) 2. This paper opened up more discussions on impact of the discovery. Including applications on detecting for any backend model change, detecting the source of the LLM mode, and ways to mitigate the stealing of meta information, etc. 3. I appreciate the authors revealed the actual number of estimated hidden dimensions of GPT3.5 (as oppose to [Carlini 2024], which is hidden from public).

Reasons to reject

The primary concern on this paper is the contribution on top of [Carlini 2024]. After comparing with two papers I feel like authors need to make it clearer on what additional value this paper brings to research community. Right now the authors did not have such comparison thoroughly. Thus, I would like the authors to include a "Contribution" section to clarify this. Is the main breakthrough of this paper lies in the algorithmic complexity of doing LLM invocations? If so, I expect to make it clear by comparing big Os of this paper vs. [Carlini 24]. Though I appreciate the detailed discussions on potential future applications, I doubt the feasibility of quite a few. For instance "detecting the LoRA updates", the reality is people could simply skip lora updates on embedding layer; and the wording "It may be possible to detect these types of updates" sounds sneaky. It should be avoided in academia papers.

Questions to authors

My questions are mostly tied to "reasons to reject" section. Please respond to that section.

Summary

This paper explores the vulnerabilities in the API deployment of large language models, specifically focusing on how a limited number of API queries can expose non-public information about such models. The authors demonstrate that softmax bottleneck deduces extensive details about a model's structure, such as its embedding size. The study further suggests methods for LLM providers to shield against these vulnerabilities, while also proposing that these weaknesses could enhance transparency and accountability in LLM operations.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

+ The findings are interesting, albeit it is a concurrent work to [1]. + The techniques for + There are suggestions for: (1) employing this method for model auditing, and (2) mitigating this type of attack. [1] Stealing part of a production language model, published on arXin in March 11th.

Reasons to reject

+ The attack can be easily defended by blocking logits or adding noise to the outputs. + Although several types of attack has be discussed in this work, but the main outcomes and purpose of this attack is unclear. How these findings can be orchestrated as a comprehensive attack that harms LLM applications. I would consider discovering size of h in LLM is minor as it is just a number and we still need to train the models. Recovering the exact logits of the top K tokens is still minor given the 'biased' top K logits already provides essential information. + Improvements are needed in the clarity and organization of the writing: + A comprehensive overview of the algorithm and underlying theory would enhance understanding. + Clarification is needed on how this work differs from [1]. Specifically, if Sections 7 and 8 are intended to address these differences, a more thorough analysis in these sections could help establish a clear distinction from prior work + The suggested applications (in Section 7) lack sufficient detail and are not adequately supported by theoretical frameworks or experimental validation. + The countermeasures suggested are not substantiated by experimental evidence. + The paper was submitted several weeks after [1], (and arXiv version is also several days after [1]) yet it does not adequately distinguish its theoretical contributions or empirical analysis from those found in [1]. For example, an algorithm to recover the last layer was proposed in [1] but not discussed in this work. I would consider recovering the 'shape' is far less interesting than recovering 'parameters' of the linear module. This means the missing piece to [1] is the most essential part of this work. I am aware that this is a concurrent work to [1], and my evaluation is based on the quality of this paper.

Thank you for your comments, keeping my score

Thank you for your detailed replies! I will keep my original score, and continue to support the acceptance of the paper.

Response

Thanks for your response. I checked the timestamps of both papers, seems indeed a concurrent case. I'll maintain my rating because 1/ COLM review guideline didn't mention how to deal with such senario. 2/ As today's APIs either do not provide logits (as Anthropic) or patched (as OpenAI), it's not very clear to me about the longer term impact. But other than that I think this paper is solid and interesting enough.