Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation

Summary

The study introduces a data-driven variant of LoRA-like Parameter-Efficient Fine-Tuning (PEFT) for Vision Transformers (ViTs), which learns the optimal bottleneck dimension (the rank of the fine-tuning weight matrix) instead of preconfiguring it for each layer. The authors propose learning three additional vectors: $v_{\text{left}}$, $v_{\text{right}}$, and a vector of diagonal parameters $D$ that define the rank. Inspired by Singular Value Decomposition (SVD), they employ Householder transformations $I-vv^\intercal$ on $v_{\text{left}}$, $v_{\text{right}}$ to construct orthogonal matrices. These learned vectors and parameters $v_{\text{left}}$, $v_{\text{right}}$, and $D$ can then be fused into the frozen weights in a LoRA-like manner. Jointly training $v_{\text{left}}$, $v_{\text{right}}$, and $D$ alongside an extremely low-rank LoRA branch (r=1) results in improved performance on downstream tasks.

Strengths

The paper tackles the optimal rank determination problem for LoRA-based methods using a learning-based approach. Despite requiring an additional rank-1 LoRA branch to enhance accuracy, both experiments and the ablation study highlight the effectiveness of the proposed method, surpassing several state-of-the-art baselines.

Weaknesses

While the proposed method is parameter-efficient, the discussion overlooks the actual training cost and latency. For instance, it would be beneficial to include theoretical or actual memory and computational costs.

Questions

- What rank is applied to the LoRA-based baselines in Table 1? It would be beneficial to list these parameters in the appendix. - What are the settings used in Table 5? Are the LoRA branches also applied in the HTA row? - Could the authors include a section discussing theoretical and actual memory computation costs, such as FLOPS? - Is it feasible to extend the method to Large Language Models?

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

As the author discussed in the limitation section.

Summary

This paper proposes a new PEFT method by integrating Householder transformations, which enables the creation of adaptation matrices that have varying ranks across different layers, thereby offering more flexibility in adjusting pre-trained models.

Strengths

S1: This paper proposes a new PEFT method based on Householder transformations. S2: This method allows for layer-wise rank variations. S3: The authors validated the effectiveness of the proposed method on classification tasks.

Weaknesses

I am not an expert in the field of PEFT, it appears that the authors have proposed a new PEFT scheme. From an application perspective, we desire PEFT methods that can transfer the pre-trained models to downstream tasks with as few parameters as possible. In the experiments conducted by the authors, the proposed method achieved slight improvements with relatively fewer parameters. In fact, all methods utilize a minimal number of parameters. In addition, the authors did not make comparisons in more complex tasks, such as object detection and semantic segmentation. Currently, numerous related studies employ a backbone based on the ViT [A,B,C]. Validation solely on classification tasks is somewhat limiting. [A] Exploring Plain Vision Transformer Backbones for Object Detection. [B] SegViT: Semantic Segmentation with Plain Vision Transformers. [C] ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation

Questions

Is the proposed method effective in tasks such as object detection and semantic segmentation?

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The novelty, contribution and experimental comparison of this paper all have certain shortcomings.

Summary

The paper presents a novel method, HTA, for efficiently adapting pre-trained transformers by applying the Householder transformation to a single vector to approximate the SVD for representing the adaptation matrix. By combining this with an additional rank-1 adaptation matrix, HTA achieves a reduction in the number of parameters while maintaining strong performance across various benchmarks, including FGVC and VTAB-1k, and outperforms existing methods in terms of efficiency and performance.

Strengths

Originality: This work incorporates the Householder transformations to construct orthogonal matrices, replacing the previously proposed low-rank adaptation matrix, and avoids the hyperparameter choice of the rank. Quality: The submission is technically sound with well-supported claims backed by extensive experimental results. Clarity: The paper is well-written and organized, making it easy to follow the methodology and results. Significance: 1. This work addresses a challenging task and advances the state-of-the-art with high performance and reduced computational costs. 2. It may lead to further development of methodologies based on Householder transformations.

Weaknesses

Originality: Utilizing Householder transformations for matrix decomposition is not a brand-new idea. Quality: The improved performance of HTA over the state-of-the-art methods on several benchmarks is not very significant (<0.5%). Clarity: A scatter plot for accuracy versus the number of parameters tradeoff would give a clearer insight into HTA's performance. Significance: 1. The challenge of solely relying on Householder transformations for adapting pre-trained transformers remains. 2. The proposed HTA removes the rank choice in previous adaptation methods but introduces a new rank choice in the additional adaptation matrix, although rank 1 might be sufficient for a good performance.

Questions

1. Fig. 1c has a skip-connection, but Fig. 1d does not. Is it proposed to be like this? 2. The proposed HTA includes two feature paths, with both the Householder transformations path and the other down-up projection path being rank 1. It remains unclear why the additional down-up projection path would help significantly boost performance. 3. The results in Fig. 2 with dimension=0 are different from the ones in previous tables. What are the differences in settings between them? For Swin, it would be better to include which specific Swin model is applied. 4. For Swin, it would be better to include which specific Swin model is applied.

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The novelty and significance of HTA are somewhat limited. However, the extensive experimental results provide a robust foundation for the claims made in the paper. Overall, the work is well-executed and makes a valuable contribution to the field, justifying a recommendation for weak acceptance.

Summary

The paper presents a new parameter efficient fine tuning (PEFT) technique for vision transformer (ViT) models. The work utilizes the intuition of creating an adaptation matrix for fine-tuning from a popular dimensionality reduction technique named singular value decomposition (SVD) which is named Householder transformation based Adapter (HTA). HTA can be utilized to create adaptation matrices to accommodate layer wise variations. Extensive empirical evaluations have been performed over various popular vision benchmarks. Code has been made available for reproducibility.

Strengths

HTA presented in this work is a new PEFT technique to avoid full fine-tuning for large vision models. The empirical evaluations are extensive and rigorous and HTA is benchmarked against widely popular PEFT techniques. HTA achieves great performance similar to established previous state-of-the-art thresholds with gains in terms of trainable parameter efficiency. The readability of the work is great with great visualizations and empirical evaluations have been crisply presented. The writing structure and language are easily digestible for the reader.

Weaknesses

No weaknesses apart from the limitations mentioned in Section 5 by the authors

Questions

Do you think the HTA idea is adaptable to other modalities for PEFT?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

NA