Large language model validity via enhanced conformal prediction methods

Summary

The paper proposes a new way to filter out LLM generations such that the resulting text only contains a set number of invalid claims with a certain probability. To do so, the authors adapt a new conformal mechanism to be level adaptive i.e. conditioned on the prompts rarity for example. They also propose a boosted version which additionally reduces the number of wrongly removed items in the generations set of claims. They then conduct experiments on synthetic and wikipedia data to confirm they claims.

Strengths

- The paper proposes a level adaptive CP algorithm which to the best of my knowledge has not been done before in the LLM setting. - The paper is clearly written (for someone who has published in CP before it was easy to follow) - The paper tackles an important problem of hallucinations in LLMs which is timely and crucial for trustworthy LLMs - The paper also proposes a boosted CP version and elegantly solves the new challenges albeit the main ideas were already introduced in Stutz. There is nonetheless novelty in the additional challenges to make it conditional. Prop 3.1 in particular is an interesting observation given the linearity - The experiments overall show the superiority of the proposed methods compared to the current baselines. (there seems to be only one)

Weaknesses

- The paper starts off unfairly in figure 1, where the authors fixed a level of 90% for the baseline and 76% for the proposed method. This is misleading as one could easily also pick a lower coverage rate for the baseline and it would potentially look much less bad. This is concerning as it misleads the reader quite drastically until one reads the caption. Could you please explain why you chose 90% for the baseline but then only 80% for yours? Maybe i misunderstood your intentions with this figure. - Secondly, I would like to note that the original conformal guarantees are finite sample however, if i understood correctly, this is no longer the case here eq (3), thm2.1. Hence my question is that are there any other baselines to compare against that are not conformal? I believe that this paper misses a substantial related works section. - Thirdly, the experiments in the wikipedia dataset were choses with at most 3 wrong claims. Why did you choose 3 and not 0, 1 or 2 ? The fact that these experiments werent added in the appendix as ablation makes me question that 3 was chosen specifically to show the benefits of this paper. Please add these additional experiments for completeness as this number 3 was chosen out of nowhere. This also leads me to my next concern which is that of, even if we have at most 3 wrong claims, we can still not point to which one is wrong ... Could the authors please elaborate on the usefulness of this in the first place? Shouldnt you set the threshold to 0 wrong claims? This is a major concern from my side as this guarantee doesnt allow me to do anything useful in practice. - Fourthly, I also agree with the authors that the exchangeable assumption is a big drawback of this method in practice. The method that needs to be trained with the conditional boosting, is fixed for a given type data. This can be quite limiting. - Lastly, there are not sufficient experimental results to support your claims. You have a single toy experiment with some Wiki dataset. Additional experiments or at least extensive ablation studies would be needed to convince me about the usefulness of this method.

Questions

Please see above. I am more than happy to raise my score if the above have been addressed.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper introduces two new conformal prediction methods aimed at enhancing the validity of large language models. The first method generalizes the conditional conformal procedure to issue weaker guarantees when necessary, thereby preserving utility. The second method improves the quality of the scoring function through a novel algorithm designed to differentiate the conditional conformal procedure. The proposed methods are validated using both synthetic and real-world datasets, addressing deficiencies in existing approaches by ensuring conditional validity and retaining valuable and accurate claims.

Strengths

- The introduction of generalized conditional conformal procedures and improved scoring functions addresses critical gaps in current approaches. - The paper is technically sound, with well-supported claims and robust experimental validation. The theoretical results, particularly Theorems 3.1 and 3.2, and Proposition 3.1, are well stated and proved. The methodology is well-developed, and the experimental design effectively demonstrates the efficacy of the proposed methods. - The methods are evaluated on both synthetic and real-world datasets, effectively showcasing their practical utility and robustness.

Weaknesses

- The methodology assumes the existence of an annotated calibration set of prompt-response-claim-annotation tuples, which may raise questions about the generalizability of the approach to different datasets or domains. - Some readers may not be familiar with the metrics used to evaluate the method. Providing additional explanations for these metrics would enhance clarity and understanding.

Questions

How sensitive is the methodology's performance to variations in the size and composition of the calibration set? How representative is this dataset of real-world large language model outputs?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors adequately address the limitations of their work in Section 5.

Summary

The paper presents two new conformal inference methods for obtaining validity guarantees on large language model (LLM) outputs. The authors consider the task of filtering invalid claims from LLM responses to ensure high probability factuality guarantees on the filtered output. They discuss two limitations of existing approaches and how their methods improve upon them – (i) previous methods provide a marginal guarantee over a random test prompt instead of conditional validity, and (ii) existing methods may remove too many claims as the score function is imperfect. To improve upon these, the authors generalize the conditional conformal framework of [1] to adapt the filtering threshold that controls monotone risks and adjust the error rate adaptively (level-adaptive conformal prediction). The second method introduced in the paper termed conditional boosting enables automated discovery of superior claim scoring functions via differentiation through the conditional conformal algorithm i.e., finding conformity scores that allow retaining more claims in the LLM output while also ensuring validity. The authors perform experiments on synthetic and real-world datasets to demonstrate the performance of their methods and improvement upon existing approaches.     [1] I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.

Strengths

**Originality**: The main novel contribution is the conditional boosting procedure that allows automatic discovery of new scoring functions that enable higher claim retention via differentiation through the conditional conformal algorithm. While the paper leverages the conditional conformal procedure in Gibbs et al [1] and conformal factuality framework from Mohri and Hashimoto [2], the generalization of [1] to incorporate general losses and adaptive error rate is a novel and important contribution. To add, the previous work is adequately cited in the paper. **Quality**: The submission is technically sound and the claims are well supported by theory. The experimental results empirically demonstrate the performance improvements of the methods introduced over previous approaches. **Clarity**: The submission is well-written and clear for the most part. I appreciate the introduction to conditional conformal procedure of Gibbs et al. [1] in section 2.1. It would be helpful to organize the empirical analysis better to understand the setup and make performance improvements offered by different methods more clear. **Significance**: The conformal inference methods proposed in the paper provide practically useful guarantees for LLM outputs. I believe the conditional validity formulation and evaluation are important contributions and the results will be of interest to the community. **References** [1] I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.   [2] C. Mohri and T. Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXiv:2402.10978, 2024.

Weaknesses

The main weaknesses and scope for improvement are wrt the empirical evaluation: 1. Lack of ablations - While the experiments include comparison with the split conformal calibration method of Mohri and Hashimoto [2] for the Wikipedia biographies dataset, I did not see comparison of the level-adaptive method with the fixed level conditional conformal procedure in [1] (where $\mathcal{F}$ can probably just be a linear combination of group indicators?). I believe this would also help understand the contribution of the level-adaptive method. - How do different sub-claim scoring functions compare to each other in practice? 2. Lack of justification behind the choice of score functions and function class $\mathcal{F}$ - Why is the conformity score chosen as naive absolute score value in Fig 4, 7? How would the results change if a better score function is selected? While the validity will always hold true, the empirical performance can give better practical insights (as is the intended goal of the paper). 3. Additional comment on organization of results: In its current form, it is hard to understand the experimental setup and variations of the method used for different figures. The experiments section can be organized better for greater clarity. **References** [1] I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.   [2] C. Mohri and T. Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXiv:2402.10978, 2024.

Questions

1. It seems that both methods require an additional step of data splitting. Would this introduce high variance if there are not enough samples in some subgroups? (it would also be helpful to mention the number of samples in each group e.g. in Fig 2, where there are just 381 total test points) 2. Is there any specific reason why experiments were not performed on other real datasets studied in Mohri and Hashimoto? 3. Typo: shouldn't it be $Y_{n+1}$ in Theorem 2.1?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss limitations in the Limitations section.

Summary

The paper provides a conformal prediction framework for the generation of hallucination risk-bound. The method can involve any loss function instead of the 0-1 loss as traditional conformal classification. They also provide a conditional (group) conformal prediction guarantee. They optimize the loss function with improved conformal prediction efficiency.

Strengths

1. The paper is well-structured and well-written. The background and related work in Sec 2 can be helpful for readers not familiar with the background. 2. Conformal prediction for LMs can provide statistical guarantee on the generation risk, which is important for universal deployment of diverse LLMs.

Weaknesses

1. [Generalization to alternative targets]: the rigorous proof is lacking about why the prediction set provided by lines 191 and 192 can lead to the conformal prediction guarantee. In line 181-187, the paper claims some assumptions on the loss function to be used. However, how these assumptions are used to derive the final valid conformal prediction guarantee is missing. 2. [Generalization to alternative targets]: conformal risk control methods [1] with a similar guarantee should be included in the discussion. [1] Angelopoulos A N, Bates S, Fisch A, et al. Conformal risk control[J]. arXiv preprint arXiv:2208.02814, 2022. 3. [Generalization to alternative targets]: missing related work [2,3] on risk control of LLMs with conformal prediction [2] Kang, Mintong, et al. "C-rag: Certified generation risks for retrieval-augmented language models." ICML 2024 [3] Quach, Victor, et al. "Conformal language modeling." ICLR 2024 4. [Level-adaptive conformal prediction]: Equation (8) use different $\alpha$ for the pinball loss (different $\alpha_i$), which can breaks the pinball loss formulation in Equation 4 and might also deteriorate the original conclusion of prediction coverage of $1-\alpha$. Do the authors have any justifications on it? 5. [Level-adaptive conformal prediction]: I think the designed level-adaptive conformal prediction is not that effective. If our goal is to use an adaptive level $\alpha$ for a particular example, why not directly do the conformal calibration with the particular $\alpha$ so that we can get the desired coverage? I am confused on the final goal of the method in Sec 3.2. What the adaptivity here different from what I am thinking? 6. [Conditional boosting]: The conformal training method (reference [18] in the paper) optimizes $\theta$ in an end-to-end way, but the paper claims that it is unclear how to backpropagate through the estimated cutoff in Line 236. What makes the difference in the scenario?

Questions

Please refer to the weakness part for concrete questions and concerns.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

discussed in Sec 5

Response

1. Thanks for the clarification, however in practice one could simply pick alpha based on a heuristic i.e. i want to retain x% of the sentence and hence i can simply pick that. This would simple require a few easy heuristic steps from the calibration data which is on par with your additional need for compute to do boosting. What i am saying is this is purposefully taken to make baseline methods look bad. In practice if someone were to take 90% and see that they only retain so little claims, they would simply lower alpha. From this perspective can the authors please explain how this is not a trivial heuristic to compare against? Happy to keep this discussion in case i missed something, but the fact that you simply use 90% and then the adaptive method uses mostly around 70-80 is just misleading to me. I could easily find a fixed heuristic that does pretty much the same. I want to see concrete experiments where this slight adaptation towards the test sample benefits the end user in a significant manner. 2. If it is truly finite sample, could the authors please tell me how the intervals scale with increased calibration data? The theorem has an expectation. 3. The authors did not address why 3 was chosen except to make the results look good. I see in the attached pdf that you have results on the medical data that have 0 errors. I agree in settings where the model is presumably decent, this is achievable. In the Wikipedia dataset i assume that using 0 errors will make the whole method somewhat useless at below 50% guarantees. My main question is, if you say you have at most 3 errors, and cant tell the user which ones are the errors, what can the user possibly do with that information? The only use case is the 0 errors. 4. Can the authors explain this covariate shift theory in more detail please. i might have missed that part. 5. I thank the reviewer for the additional experiments. However as mentioned previsouly a simple heuristic of trying to pick the desired alpha based on how many sentences to retain would be just as usable. and hence my main question here is what the concrete advantages of this adaptive nature is. Looking at the histograms the nominal values seems to be unimodal. I am not saying to use this as a guide but a separate computation (heuristic compared to your adaptive method) where given some calibration data, we go over alpha [10, 20, ... 90%] and then pick the alpha which fits the desired retention level seems like a basic baselines to me. Happy to keep this discussion going as there might be some parts i might have missed or misunderstood. but i believe the above is still valid and would like the hear the authors answers to them before increasing the score.