Library

Bayesian Low-rank Adaptation for Large Language Models

Low-rank adaptation (LoRA) has emerged as a new paradigm for cost-efficient fine-tuning of large language models (LLMs). However, fine-tuned LLMs often become overconfident especially when fine-tuned on small datasets. Bayesian methods, with their inherent ability to estimate uncertainty, serve as potent tools to mitigate overconfidence and enhance calibration. In this work, we introduce Laplace-LoRA, which applies a Bayesian approach to the LoRA parameters. Specifically, Laplace-LoRA applies a Laplace approximation to the posterior over the LoRA parameters, considerably improving the calibration of fine-tuned LLMs.

cs.LGSpeech Recognition and SynthesisTensor decomposition and applicationsSpeech and Audio ProcessingComputer Science
View graphSource

Paper

References (74)

01Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA EnsemblesYuanzhao Zhai, Han Zhang, Yu Lei et al.2023 · arXiv.org · 50 citations In Library
02Mistral 7BAlbert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch et al.2023 · arXiv.org · 3.7k citations In Library
03LoRA ensembles for large language model fine-tuningXi Wang, Laurence Aitchison, Maja R. Rudolph2023 · arXiv.org · 75 citations In Library
04DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningZhengxiang Shi, Aldo Lipani2023 · International Conference on Learning Representations · 51 citations In Library
05Llama 2: Open Foundation and Fine-Tuned Chat ModelsHugo Touvron, Louis Martin, Kevin R. Stone et al.2023 · arXiv.org · 18k citations In Library
06Preserving Pre-trained Features Helps Calibrate Fine-tuned Language ModelsGuande He, Jianfei Chen, Jun Zhu2023 · International Conference on Learning Representations · 30 citations In Library
07Passive learning of active causal strategies in agents and language modelsAndrew Kyle Lampinen, Stephanie C. Y. Chan, Ishita Dasgupta et al.2023 · Neural Information Processing Systems · 30 citations In Library
08Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human FeedbackKatherine Tian, E. Mitchell, Allan Zhou et al.2023 · Conference on Empirical Methods in Natural Language Processing · 829 citations In Library
09QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman et al.2023 · Neural Information Processing Systems · 5.2k citations In Library
10ASDL: A Unified Interface for Gradient Preconditioning in PyTorchKazuki Osawa, Satoki Ishikawa, Rio Yokota et al.2023 · arXiv.org · 23 citations In Library
11BloombergGPT: A Large Language Model for FinanceShijie Wu, Ozan Irsoy, Steven Lu et al.2023 · arXiv.org · 1.4k citations In Library
12Calibrating Transformers via Sparse Gaussian ProcessesWenlong Chen, Yingzhen Li2023 · International Conference on Learning Representations · 22 citations In Library

Scroll for more · 38 remaining

Similar papers

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language ModelsMoule Lin, Shuhao Guan, Andrea Patane et al.2026 · arXiv.org · 0 citations In Library
Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty EstimationJijie Zhang, Zhenjiang Ren, Quan Zhang et al.2026 · 0 citations In Library
BaLoRA: Bayesian Low-Rank Adaptation of Large Scale ModelsDario Coscia, Sindy Lowe, Max Welling2026 · 0 citations In Library
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language ModelsYibin Wang, Haizhou Shi, Ligong Han et al.2024 · arXiv (Cornell University) · 51 citations In Library
Bayesian Fine-tuning in Projected SubspacesV. Dubovik, Patryk Marszałek, Jacek Tabor et al.2026 · arXiv.org · 0 citations In Library
Low-Rank Adaptation for Scalable Large Language Models: A Comprehensive SurveyChia Xin Liang, Zhuming Bi, Tianyang Wang et al.2025 In Library
Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language ModelsEmre Onal, Klemens Flöge, Emma Caldwell et al.2024 · arXiv (Cornell University) · 19 citations In Library
Balanced LoRA: Removing Parameter Invariance to Accelerate ConvergenceVal'erie Castin, Kimia Nadjahi, Pierre Ablin et al.2026 · arXiv.org · 0 citations In Library
Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank AdaptationVaibhav Seth, A. Pathak, Ayan Sengupta et al.2024 · Trans. Mach. Learn. Res. · 6 citations In Library
Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRAPatryk Marszałek, Klaudia Balazy, Jacek Tabor et al.2025 · Conference on Empirical Methods in Natural Language Processing · 6 citations In Library

Peer review

Accept (poster)ICLR 2024 · via OpenReview
Reviewer CyBU8/10 · confidence 3/52023-10-30

Summary

This paper proposed a new fine-tuning procedure based on LoRA using Bayesian prospective.

Strengths

This paper is very well-written and the algorithm is presented in a very clean way. Although the algorithm is straightforward, as far as I know, nobody has made it work before. The empirical analysis is done carefully which is appreciated. Code is also linked.

Weaknesses

1. It looks like all the tasks using in this paper are still classification tasks. How it is different from standard Bayesian deep learning evaluation? I want to understand where the benefit is coming from on those eval metrics through a principled way. I feel it will be good to comment some potential on generation tasks that can reflect more of the autoregressive nature of transformers. 2. Where is the comparison of computational complexity and memory usage? How much additional memory you will use?

Questions

See above.

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Authorsrebuttal2023-11-22

Response

Thank you for your constructive feedback and positive remarks on our work. > It looks like all the tasks using in this paper are still classification tasks. How it is different from standard Bayesian deep learning evaluation? We use classification tasks precisely so we can use standard, well-understood methods for assessing the quality of uncertainty estimation. > I want to understand where the benefit is coming from on those eval metrics through a principled way. The improvement on calibration metrics is arising because we use approximate Bayesian inference to estimate our uncertainty in fine-tuning weights given a very small number of fine-tuning examples. **Autoregressive generation** Given that this is the first investigation of Laplace for LoRA fine-tuning, we chose to focus on multiple-choice QA because that allowed us to use robust, well-understood calibrations metrics like ECE and NLL. As a next step, we are indeed excited by the possibility of investigating free-text generation: Laplace-LoRA certainly could be applied in this setting. However, the development of robust, well-understood calibration metrics for free-text generation remains a challenging and open research question. Given the complexity of evaluating calibration in the free-text setting, this extension is out-of-scope here, and we leave it for future work. **Memory and runtime cost** We have added the memory and time cost comparison between LoRA fine-tuning and LoRA with post-hoc Laplace-LoRA in Section 5.4 and Table 5 in the main text. The upshot is that the dominant additional cost arises from accumulating the low-rank Kronecker factors; that takes about 10\% of the time, and only slightly more memory (actually about 1-5\% more memory) than the original finetuning run.

Reviewer sBHc6/10 · confidence 3/52023-11-01

Summary

This paper presented a Bayesian approach to improve the calibration of LoRA. It adopted Laplace approximation to the posterior of the low rank parameters. Empirical results on both in-distribution and out-distribution finetuning indicated the Bayesian approach does improve the calibration and the proposed Laplace approximation outperform a variety of baseline methods.

Strengths

1. This paper is the first to present a comprehensive result on using Laplace approximation to LoRA for LLMs. 2. This paper has clear presentation with visualization. 3. Claims are supported with sufficient amount of convincing experiment result. e.g., smaller datasets experience larger difference in ECE compares with larger datasets.

Weaknesses

1. Limited novelty. The Bayesian method part (as indicated in the paper) is well explored in the literatures listed in the software paper Laplace Redux [1]. This paper can be viewed as empirical results applying [1] to a specific model - LoRA for LLMs. 2. Majority of the benefits of Laplace-LoRA including ''post-hoc'' and ''scalable'' are from the existing method, which limits the contribution of this work. This one together with Weakness #1 above are the major concerns from my point of view. 3. There was some discussion on the cost to perform Laplace approximation when introducing Fasher. However, there is no empirical results that support it. 4. The ablation study (LLLA vs LA) is also from the [1]. Only comparing LA with Last Layer LA is not very convincing on where the uncertainty comes from. [1] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. NeurIPS, 2021.

Questions

1. Laplace Redux has discussion on relative wall-clock time cost and memory consumption comparison with other methods, is there similar result for Laplace-LoRA? 2. More ablation study is expected. For example, LA on different types of layers (attention, dense, etc.), LA on different layers and different number of layers (first k layers or random select layers), or some fusion of the two mentioned above.

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Authorsrebuttal2023-11-22

Response

Thanks for your constructive, thoughtful feedback. **Novelty** Bayesian deep learning is rarely if ever used by LLM practitioners. That's because there are currently no effective and efficient methods for Bayesian finetuning of modern LLMs. Here, we show comprehensively that highly efficient Bayesian methods can dramatically improve calibration in modern LLMs. That's the key novelty in this paper, and it opens the way to practical applications of Bayesian methods in modern LLMs, along with a far more thorough exploration of Bayesian methods for LLMs in future work. Regarding the novelty of our Laplace-LoRA approach, we believe its significance lies in the innovative integration of well-studied methods to create a novel framework for Bayesian fine-tuning on LLMs. This integration is not trivial and addresses a critical gap in current methodologies. In addition, we opensourced code that works with any publicly available LLMs on Huggingface. **Ablations** Laplace Redux only compares LLLA vs LA for ResNets for simple computer vision tasks such as CIFAR-10. This tells us very little about what to expect in modern LLM finetuning. Indeed, we come to the opposite conclusion from the Laplace Redux paper. They find that LLLA is sufficient in their tasks. While we find that retaining uncertainty at all layers is important in parameter-efficient LLM finetuning with low-rank adaptation. Additionally, our experiments weren't really directed at finding "where the uncertainty comes from". They were directed at discovering why, when optimizing the prior using the Laplace estimate of the marginal likelihood, the uncertainties for LLLA and LA were so different. Specifically, the uncertainty for LLLA was far too low, whereas the uncertainty for LA was about right. Our conclusion was that there is little uncertainty arising at the last layer. In response to your request, we expanded our analysis in Appendix G.3, by applying LA to different LoRA layers. Specifically, we experimented with applying LA on the first 8, 16, 24, and 32 (all) layers. While we observed only small differences, there does seem to be a small improvement in the NLL when LA is applied beyond the top 8 layers. **Memory and runtime cost** We have added the memory and time cost comparison between LoRA fine-tuning and LoRA with post-hoc Laplace-LoRA in Section 5.4 and Table 5 in the main text. The upshot is that the dominant additional cost arises from accumulating the low-rank Kronecker factors; that takes about 10\% of the time, and only slightly more memory memory (actually about 1-5\% more memory) than the original finetuning run.

Reviewer sBHc2023-11-23

I appreciate the author's rebuttal and their response to my question. I have increased my score.

Reviewer 5qzU6/10 · confidence 4/52023-11-06

Summary

This paper introduces Laplace-LoRA, a Bayesian inference method designed specifically for the LoRA parameters in LLM fine-tuning using a post-hoc Laplace approximation. They conducted extensive experiments on six commonsense reasoning tasks and show dramatic improvements in calibration of fine-tuned LLMs.

Strengths

1. This idea of combining Laplace inference with the fine-tuning LLMs using LoRA adapters is novel, which provides a new way of doing Bayesian fine-tuning on LLMs. 2. They conducted extensive experiments on six commonsense reasoning tasks under in/out-of-distribution settings and provided detailed analysis of the experiment results. 3. The writing is well-structured, clear and easy to understand.

Weaknesses

1. It has some novelty, but not dramatic, because both Laplace Approximation and LoRA method are well-studied. 2. It is quite weird that the Section 3 Background followed by Section 4 Results directly, without a Method section in between. Maybe it needs a better section name.

Questions

Does free-text generation tasks (e.g., free-from QA) still work under this framework? Why that kind of reasoning tasks are not considered in your experiments?

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Authorsrebuttal2023-11-22

Response

Thank you for your constructive feedback and positive remarks on our work. **Novelty** Bayesian deep learning is rarely if ever used by LLM practitioners. That's because there are currently no effective and efficient methods for Bayesian finetuning of modern LLMs. Here, we show comprehensively that highly efficient Bayesian methods can dramatically improve calibration in modern LLMs. That's the key novelty in this paper, and it opens the way to practical applications of Bayesian methods in modern LLMs, along with a far more thorough exploration of Bayesian methods for LLMs in future work. Regarding the novelty of our Laplace-LoRA approach, we believe its significance lies in the innovative integration of well-studied methods to create a novel framework for Bayesian fine-tuning on LLMs. This integration is not trivial and addresses a critical gap in current methodologies. In addition, we opensourced code that works with any publicly available LLMs on Huggingface. **Methods** We have added a Methods section describing the nontrivial methods required for the low-rank K-FAC approach that we designed for Laplace-LoRA, with detailed derivations in Appendix E. **Autoregressive generation** Given that this is the first investigation of Laplace for LoRA fine-tuning, we chose to focus on multiple-choice QA because that allowed us to use robust, well-understood calibrations metrics like ECE and NLL. As a next step, we are indeed excited by the possibility of investigating free-text generation: Laplace-LoRA certainly could be applied in this setting. However, the development of robust, well-understood calibration metrics for free-text generation remains a challenging and open research question. Given the complexity of evaluating calibration in the free-text setting, this extension is out-of-scope here, and we leave it for future work.

Reviewer 5qzU2023-11-23

Thanks for the clarification! My concerns have been addressed.

Reviewer EVj96/10 · confidence 4/52023-11-12

Summary

A Bayesian LoRA approach has been proposed to fine-tune large language models (LMMs) by estimating the weight posterior through layer-wise Laplace approximation. Experiment results on six QA benchmark datasets were provided in terms of accuracy, calibration, and OOD generalization, compared with three baseline methods.

Strengths

- **Timely research**: The proposed method focuses on improving the calibration performance when finetuning LLMs on small-scale datasets, which is an important and urgent research problem along with the rapid growth of large models. - **Clear Bayesian treatment**: The proposed method adopts well-established techniques from prior works of Bayesian neural networks and uncertainty reasoning, and successfully incorporates such a Bayesian treatment into parameter-efficient tuning approaches. The proposed Laplace-LoRA seems to be a scalable solution to enable uncertainty estimation for large models. - **Good experiment design**: Despite some practical issues (see *weakness*), the experiment was conducted well on six public datasets with several strong baseline methods, in terms of three settings -- 1) early stopping, 2) finetuning with validation, and 3) OOD generalization. The proposed method generally achieves better ECE scores across different cases.

Weaknesses

- **Unclear uncertainty estimation**: While the proposed Laplace-LoRA naturally estimates the weight posterior, it is unclear how to apply the proposed method to compute model uncertainties. Also, it remains unclear if the proposed method can handle the structured uncertainty estimation for next-token predictions (e.g., *Uncertainty estimation in autoregressive structured prediction, ICLR'21*). It would also be interesting to compare the proposed method with semantic uncertainty [Kuhn et al., ICLR'23]. - **Lack of LLM backbones**: The empirical evidence of the proposed method is somewhat weak due to the lack of more LLM backbones. Some concerns include: 1) can the proposed Laplace-LoRA work with larger model sizes (e.g., 13B, 70B)? 2) Can Laplace-LoRA be applied to models other than LLaMA2 (e.g., Mistral 7B, GPT-2, etc.)? - **Comparison to ensemble**: Besides the checkpoint ensemble, it is also expected to implement a baseline given by the LoRA ensemble (similar to the deep ensemble approach).

Questions

Please refer to the questions raised in *Weaknesses*.

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Authorsrebuttal2023-11-22

Response

Thanks for your careful and thoughtful response! **Model uncertainties** In common with with the standard Laplace approach, we compute a Gaussian approximation to the distribution over logits implied by the uncertainty in weights. We do this by linearising the model (Eq. 12, 13 and 15). **Autoregressive generation** Given that this is the first investigation of Laplace for LoRA fine-tuning, we chose to focus on multiple-choice QA because that allowed us to use robust, well-understood calibrations metrics like ECE and NLL. As a next step, we are indeed excited by the possibility of investigating free-text generation: Laplace-LoRA certainly could be applied in this setting. However, the development of robust, well-understood calibration metrics for free-text generation remains a challenging and open research question. Given the complexity of evaluating calibration in the free-text setting, this extension is out-of-scope here, and we leave it for future work. **Backbones** The approach can be applied to any setting where you would use a LoRA adapter. We had experiments in the original manuscript on RoBERTa-base and RoBERTa-large (Figs. 4-7, Tables 9-12 in Appendix G1). We have added additional experiments with Mistral-7B (Table 13 in Appendix G3). All these experiments show the same patterns as those in the main text. We will to add more extensive experiments on Mistral, as well as experiments on Llama 13B models for the camera-ready (we had intended to include them here, but we ran out of time in the discussion period). **Ensemble baseline** We have added LoRA ensemble to Fig. 1 (light green line) and Table 1 and 3. Ensembles give surprisingly little benefit in this setting, which is broadly in-line with another ICLR submission (number 3048) examining precisely this question.

Authorsrebuttal2023-11-22

Overall response

In this work, we introduce perhaps the first scalable Bayesian method for finetuning in modern LLMs: Laplace LoRA. We show that the resulting method offers dramatic benefits in terms of calibration. Our work thus opens the way to practical applications of Bayesian methods in modern LLMs, along with a far more thorough exploration of Bayesian methods for LLMs in future work. The reviewers recognised the timeliness, novelty and clarity of the contribution, along with the strength of our experimental results. **Timeliness:** * "The proposed method focuses on improving the calibration performance when finetuning LLMs on small-scale datasets, which is an important and urgent research problem along with the rapid growth of large models." (R1) **Novelty:** * "This idea of combining Laplace inference with the fine-tuning LLMs using LoRA adapters is novel" (R2) * "This paper is the first to present a comprehensive result on using Laplace approximation to LoRA for LLMs."(R3) * "nobody has made it work before."(R4) **Clarity:** * "The writing is well-structured, clear and easy to understand." (R2) * "This paper has clear presentation with visualization." (R3) * "very well-written" (R4) **Experiments:** * "Good experiment design" (R1) * "extensive experiments" (R2) * "convincing experiment result" (R3) * "The empirical analysis is done carefully" (R4)

Area Chair CGCFmeta-review2023-12-05

Meta-review

(a) A Bayesian LoRA approach has been proposed to fine-tune large language models (LMMs) by estimating the weight posterior through layer-wise Laplace approximation. Experiment results on six QA benchmark datasets were provided in terms of accuracy, calibration, and OOD generalization, compared with three baseline methods. (b) The idea of combining Laplace inference with the fine-tuning LLMs using LoRA adapters is novel, which provides a new way of doing Bayesian fine-tuning on LLMs. They conducted extensive experiments on six commonsense reasoning tasks under in/out-of-distribution settings and provided detailed analysis of the experiment results. The writing is well-structured, clear and easy to understand. Good rebuttals. (c) Should explore different text tasks. Some improvements to writeup suggested.

Why not a higher score

Only one glowing substantial review, others were marginal or short.

Why not a lower score

Unanimous decision.

© 2026 NYSGPT2525 LLC