QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation

Existing text-to-image models often rely on parameter fine-tuning techniques such as Low-Rank Adaptation (LoRA) to customize visual attributes. However, when combining multiple LoRA models for content-style fusion tasks, unstructured modifications of weight matrices often lead to undesired feature entanglement between content and style attributes. We propose QR-LoRA, a novel fine-tuning framework leveraging $Q R$ decomposition for structured parameter updates that effectively separate visual attributes. Our key insight is that the orthogonal $Q$ matrix naturally minimizes interference between different visual features, while the upper triangular $R$ matrix efficiently encodes attributespecific transformations. Our approach fixes both $Q$ and $R$ matrices while only training an additional task-specific $\Delta R$ matrix. This structured design reduces trainable parameters to half of conventional LoRA methods and supports effective merging of multiple adaptations without crosscontamination due to the strong disentanglement properties between $\Delta R$ matrices. Experiments demonstrate that $Q R$ LoRA achieves superior disentanglement in content-style fusion tasks, establishing a new paradigm for parameterefficient, disentangled fine-tuning in generative models. The project page is available at: https://luna-ai-lab.github.io/QR-LoRA/