Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models

We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts for black-box models. We derive the distortion-rate function for this setup as a linear program, and provide an efficient algorithm to compute this fundamental limit via the dual of the linear program. Using the distortion-rate function as the baseline, we study the performance of existing compression schemes on a synthetic dataset consisting of prompts generated from a Markov chain, natural language queries, and their respective answers. Our empirical analysis demonstrates the criticality of query-aware prompt compression, where the compressor has knowledge of the downstream task/query for the black-box LLM. We show that there is a large gap between the performance of current prompt compression methods and the optimal strategy, and propose Adaptive QuerySelect, a query-aware, variable-rate adaptation of a prior work to close the gap. We extend our experiments to a small natural language dataset to further confirm our findings on our synthetic dataset.

Paper

References (52)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer sxbS7/10 · confidence 3/52024-07-17

Summary

The paper formulates hard prompt compression as a rate-distortion problem. An algorithm is provided for hard prompt compression, as well as another to estimate the RD curve (Algorithm 1). Experiments on synthetic as well as benchmark datasets are provided with comparisons to previous methods. Note: - I'm more than willing to change my score significantly if the questions are addressed appropriately. - I don't know much about LLMs. - I'm not familiar with previous literature on prompt-compression. - I'm familiar with convex optimization, but it's hard to evaluate the correctness or novelty of Algorithm 1. - I'm familiar with info theory in general.

Strengths

- Overall the paper is very well written. I was able to follow everything to a reasonable degree even without knowing much about LLMs. - The paper takes a principled approach, through the lens of info theory, to prompt-compression to investigate fundamental limits. - The overall conclusion points towards a space for improvement in current methods (summarized in Figure 1.), with the caveats mentioned in the questions/weakness.

Weaknesses

- The synthetic data is very limited as it has a binary support (although this is highlighted in as a limitation in the appendix). - The gold standard for the distortion functions, as far as I understand, would be to have a distortion function that is low when two sentences have "the same meaning", as measured by humans, and high otherwise. Of course, this is function doesn't exist (as far as I know). It isn't obvious why the chosen distortion metrics (i.e., log-loss and 0/1) capture "semantics" as originally intended by the authors. Smaller issues - Line 123: notation for a pair of objects inconsistent with the previous parts of the text (i.e., used $[M, Q]$ instead of $(M, Q)$) - Line 147: typo, "taken with respect" -> "taken with respect to" - Line 158: typo, "The dimensions of problem" -> "The dimensions of the problem" Suggestions (this is conditioned on there being reasonable explanations for the questions in the Questions section): - I would suggest focusing the paper more about the empirical results and why the RD formulation makes sense for this problem (the ML community is not too familiar with info theory, so the formulation itself is a contribution in my opinion). - The entirety of section 3.3 is not too important for the story and could be entirely moved to the appendix. Instead, replace 3.3 with a discussion as to why this RD formulation captures the "rate-semantics" trade-off originally intended by the authors, by providing evidence that the distortion functions proposed can actually measure semantic distortion.

Questions

> Line 261: "taking $P_{XQY}$ to be the empirical distribution on the dataset" If the empirical distribution is used in the RD formulation, isn't this just overfitting to the sample? > Line 108 and Equation (1) I don't understand why the log-loss, or 0/1 loss, under the LLM is a reasonable distortion function to measure semantic distortion. Please see "Weakness" section.

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

There's a large limitations section in the appendix which I appreciate.

Reviewer Jydp7/10 · confidence 4/52024-07-17

Summary

The paper studies the distortion-rate function for prompt compression and proposes an linear programming based algorithm that produces compressed "hard" prompts and is suitable for black box models. The authors provide empirical results on a synthetic dataset illustrating a gap between the existing prompt compression approaches and the theoretically optimal distortion-rate tradeoff and show that the proposed algorithm outperforms the existing methods.

Strengths

- The paper provides sufficient detail on related work and motivates the problem well. - The proposed distortion-rate formulation is simple yet easy to understand/interpret. It also allows for query-adaptive solutions with simple modifications to the formulation. - The empirical results on the synthetic dataset demonstrate the improvements over existing prompt compression methods, which are provably (since the dataset is synthetic) worse than the optimal compressor.

Weaknesses

- It would be great to see some experimental results with real datasets. - What is the reason that the rate is normalized but the distortion is not? Have the authors tried different combinations? - How does the efficiency of the proposed method compare with existing prompt compression methods? Did the authors compare how long each method takes to compress the same prompt?

Questions

In addition to the questions above, out of curiosity, do the authors expect any challenges when the compressor is conditioned on multiple queries?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors mention limitations in the last section.

Reviewer cHot7/10 · confidence 5/52024-07-23

Summary

This paper proposes an information-theoretic framewok for token-level prompt compression for large language models, where the rate is characterized by the expected ratio between the compressed prompt and the original prompt, and the distortion is a cross-entropy based distortion or an accuracy based distortion. The RD problem is then converted to a LP problem, where optimal solutions are given for both query-aware and query-agnostic settings. An algorithm "LLMLingua-2 Dynamic" is proposed that outperforms other LLMLingua-based algorithms.

Strengths

Token-level semantic compression has been an active area of research. The current paper builts an information-theoretic framework inspired by the rate-distortion trade-off in source coding. Algorithms for solving the RD problems are given. Therefore, the proposed framework will be useful for comparing prompt compression methods in the future.

Weaknesses

The motivation for query-aware compression is not clear. If query is provided, then why not compress the answer directly, so the optimal rate would be a constant? The main content focuses on the query-aware setting (we assume that a query is provided in addition to the compressed prompt during the target LLM inference call), but the theorecital discussions in section 3.3 are on the query-agnostic setting. While the query-aware parallel is provided in the appendix, is there any reason for not focusing on the query-aware setting in the first place? The experiments are not convincing, in that the prompt lengths in the dataset is too short, and that the dataset only contains 7 queries. In practice, the token-level semantic compression is often used to reduce redundancy of long input. Therefore, this work should consider long prompts such as the NarrativeQA dataset. Moreover, the literature review is not thorough -- some baselines are missing (e.g., [1,2]). [1] Wingate, David, Mohammad Shoeybi, and Taylor Sorensen. "Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models." arXiv preprint arXiv:2210.03162 (2022). [2] Fei, Weizhi, et al. "Extending context window of large language models via semantic compression." arXiv preprint arXiv:2312.09571 (2023).

Questions

1. Why is the compression rate defined as the expected ratio between len(M) and len(X)? In information theory, the rate is usually measured by units such as bits. 2. In the query-aware setting, if query is provided, then why not compress the answer directly, so the optimal rate would be a constant? 3. In lines 85-86, you said "we assume that a query is provided in addition to the compressed prompt during the target LLM inference call", but later in lines 136-137, you then said "To simplify the presentation, we restrict our discussion here to the query-agnostic setting, and only briefly mention the analogous definitions and results for the query-aware setting." Is there any particular reason for this conflict? 4. In Proposition 1, what do you mean by $\mathbb{R}_+^{\mathcal{M}_x}?$ $\mathcal{M}_x\subseteq \mathcal{V}^*$ right? 5. The proof of Proposition 1 is unclear. In (LP), constants $D_x, R_x$ are used, but only $D_{x,m}, R_{x,m}$ are defined. 6. From the experimental results, it seems that the proposed method can bot achieve high compression rate/low distortion (lossless). Could you discuss this in your paper? 7. Could you provide comparisons to other baselines such as [1] and [2]? 8. Could you conduct experiments on other public datasets, such as the NarrativeQA dataset?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes.

Reviewer Jydp2024-08-09

response

I thank the authors for their response; my questions have been answered sufficiently. I maintain my score for acceptance.

Authorsrebuttal2024-08-09

We thank the reviewer for their timely response, and are happy that we were able to sufficiently answer all questions. We believe that including the table of timings for the compression methods is a useful benchmark to include, and we will add it to the revised paper.

Reviewer sxbS2024-08-09

The added discussion on "semantic distortion" is very satisfying. I'm happy with the response and have updated my score accordingly.

Authorsrebuttal2024-08-09

We thank the reviewer for their timely response, and are happy that the reviewer is satisfied with our rebuttal. We further extend our gratitude to the reviewer for increasing their score. As per the reviewer's suggestion, we will include a discussion on semantic distortion in the revised paper.

Reviewer cHot2024-08-12

Thank you for the detailed response. Given the modifications the authors promise, I have raised my rating to accept.

Authorsrebuttal2024-08-13

We appreciate the reviewer's timely response and sincerely thank the reviewer for raising their score.

Authorsrebuttal2024-08-14

Summary of review and discussion

Once again, we thank the reviewers for their feedback in the official review and their follow-up during the discussion period. We are delighted that both reviewer cHot and reviewer sxbS raised their scores from 5 to 7 in response to our rebuttal, and are appreciative of reviewer Jydp's initial score of 7, recognizing our contributions as listed in our global response above. The following list summarizes the additions that we will make to the final vision of our manuscript. These additions are based on the comments and questions of the reviewers during the official review, and are already made available in our rebuttal. 1. We will include our **new experiments** on the small natural language dataset and the large-scale NarrativeQA dataset. All three reviewers requested more experiments on either natural language data, a large-scale dataset, or both. We thank reviewer cHot in particular for the suggestion on using the NarrativeQA dataset. 2. We will include a table for the timings of each prompt compression method on the small natural language dataset and the NarrativeQA dataset, as a benchmark of **efficiency** of each method. We thank reviewer Jydp for this suggestion. 3. We will include a discussion on **semantic distortion** and our choice for the distortion functions in our experiments. We thank reviewer sxbS for this suggestion.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC