Group and Shuffle: Efficient Structured Orthogonal Parametrization

Summary

In this paper, the authors proposed a new structured matrices, namely GS-matrices. It unifies and generalizes structured classes from previous works. Compared to the previous BOFT, it further improves the parameters and training efficiency. Experiments on language processing, text-image generation, and image classification have demonstrated the performance of GSOFT.

Strengths

1. The writing is easy to understand. 2. The proposed GSOFT has been applied to various networks on various downstream tasks.

Weaknesses

1. It's better to provide the training time comparison between BOFT, LoRA, and GSOFT. It seems that some results are lost in Table 2. 2. It's better to discuss more about single GSOFT and double GSOFT. It seems that the results in Table 1&3 are from the single version and the results in Table 2 are from the double version. 3. In Table 2, it seems that more learnable parameters result in lower CLIP scores. Maybe more discussions are required.

Questions

My main concerns are listed in the weaknesses. 1. It's better to demonstrate the performance of the proposed method on recent popular GPT series methods (e.g., Llama2, Llama3, Mixtral). This may expand the influence of this method in the field.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper follows the orthogonal fine-tuning paradigm that uses orthogonal matrices for adapting the weights of a pretrained model. It introduces a class of structure matrices, defined as GS-matrices, that can well be used to conduct orthogonal matrix. Based on GS-matrices, It proposes structured orthogonal parametrization and studies its performance in the orthogonal fine-tuning framework. Experiments are conducted on two standard setting ups: downstream task fine-tuning in language modeling and text-to-image diffusion tasks. It also claims the proposed method can be used to construct orthogonal convolutions for 1-Lipschitz neural networks.

Strengths

The description of this paper is very clear, and the overall presentation is good. It clearly describes how this paper motivates from orthogonal fine-turning. The proposed method using structured matrix to conduct orthogonal matrix is interesting.

Weaknesses

1.The experiments are not well conducted to support the claims/effectiveness of the proposed method: (1) This paper spends many words to claim the efficiency of the proposed methods. However, this paper does not provide the experiments to show the wall-clock time in the natural language understanding and subject-driven generation. I think this paper should provide the wall-clock time when comparing to baselines in Table 1 and Table 2, to show the efficiency of the proposed methods, compared to the baselines. What is the results of wall-clock time of these methods in Table 1 and 2? (2) This paper should conduct some ablation study to the Double GSOFT. It seems that Double GSOFT is the key to improve the performance, based on the description of this paper, but the double idea is apparently incremental. Besides, why Table 1 show the results using GSOFT, while Table 2 using Double GSOFT? Why not provide both in the experiments or using the best one (maybe Double GSOFT) with ablation study? I think this paper should address this point in the rebuttal. 2.This paper claims that “Nevertheless, parametrization of orthogonal matrices is a challenging task” In Line 19. I do not think so. There are many previous solutions to conduct orthogonal matrix in DNNs before 2019[1,2,3,4], even for conducting orthogonal convolutions [5,6]. And there are merits and drawbacks among these methods. I think this paper should well discuss these previous methods and take credits to them. 3.The original motivation of this paper is for parametrization efficient finetuning by using orthogonal fine-tuning. Meanwhile, it also wants to claim its contributions to the application orthogonal matrix benefits, E.g., conducting 1-Lipschitz network for robustness. But this claim is not well supported by experiments (There are many other orthogonal convolution methods(e.g., [5] and [6]), not only the baselines in the paper. E.g. SOC). Besides, this paper should pay more attention on discussing the orthogonal techniques in DNNS during the related work section. **Ref:** [1] On orthogonality and learning recurrent networks with long term dependencies, ICML 2017 [2] Learning unitary operators with help from u(n), AAAI 2017 [3] Orthogonal recurrent neural networks with scaled Cayley transform, ICML, 2018 [4] Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, AAAI 2018 [5] Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks, NeurIPS 2019 [6] LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness, NeurIPS 2022

Questions

The questions in weakness 1

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper introduces a new way of constructing orthogonal parametrization for efficient model fine-tuning. The development of the proposed method is motivated by: naive orthogonal fine-tuning, which can be too restrictive due to how it constructs the block-diagonal orthogonal matrix; and improved methods with dense orthogonal matrix can be too expensive in terms of both computation and parameters. The authors introduce a new way of constructing orthogonal matrix by an alternating product of block-diagonal matrices and several permutations, with advantages over previous methods justified theoretically. Experiments on language model, stable diffusion, and standard conv models are presented.

Strengths

- The proposed method is able to significantly reduce the parameter and computation compared to the previous method on dense orthogonal parameter tuning. - The overall writing of this paper is good. And the presentation is well organized. - The core advantages of the proposed methods are justified with theoretical supports. The development of GS Orthogonal Convolutions makes it more generalizable to more applications.

Weaknesses

My major concern with this paper is the empirical justifications. **Small experiment scale.** All experiments are conducted in very small scale. Specifically, the language models used in this paper are relatively old and outdated; Only few-shot personalization experiments are reported in the stable diffusion experiments; and the final conv application is only conducted on a tiny network on cifar100. This makes it very hard to objectively evaluate the proposed method's practical effectiveness. **Missing evaluations** If I'm not missing anything, the entire *training time* row of Table 2 is empty. Training time and memory are the core advantages compared to the previous methods. While it is true that the method is able to reduce the parameters and computations by numbers, significant speed-up and memory reduction in practice can still be a huge support to the paper's contribution. **Missing comparisons** The comparisons reported in this paper are mainly on LoRA and other orthogonal tuning methods. However, without the comparisons against other improved LoRA methods, such as DoRA, it can be hard to position the contribution and impact of this work.

Questions

- Does the SVD need to be performed in every optimization step?

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitation discussions presented in Section 9 focus mainly on how the model compares to LoRA and BOFT. The proposed method still requires more parameters than other improved LoRA methods.

Summary

This paper proposes a new parameterization for orthogonal matrices and applies this parameterization to orthogonal finetuning of foundation models. The new orthogonal parameterization is interesting and useful. The experimental results show some improvement over the baselines.

Strengths

- The study of parameter-efficient orthogonal finetuning is very much in need and is an important topic at the mement. I believe this new parameterization will be useful for orthogonal finetuning in general. - The GS parameterization seems novel to me, although I am not well knowledgeable about the related orthogonal parameterization literature. However, in the application of orthogonal finetuning, the paper, to my knowledge, is the first work to apply this parameterization. - Specifcially, the GS parameterization resembles the Monarch matrices, as mentioned in the paper. To some extend, it can be viewed as a generalization of the Monarch matrices. - The experimental results look promising and demonstrate that its parameter-efficiency outperforms both OFT and BOFT in some scenarios.

Weaknesses

- Although it is trivial to show that GS matrices can approximate arbitrary orthogonal matrices (as it generalizes the Monarch matrices), it is still important to discuss and (potentially re-use) the theoretical results from the Monarch matrices. - It can further strengthen the paper, if the authors can add the comparison of the Monarch parameterization in the orthogonal finetuning experiments. - The implementation should be released prior to its acceptance, otherwise it may be difficult for people to reproduce the results.

Questions

See the weakness section.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The major concern that I has is whether the results are easily reproducible. With that being said, the authors should consider to release the source code for the orthogonal finetuning experiments.

Thanks for the response

My concerns are properly addressed. I think this paper proposes an interesting parameterization for orthogonal matrices and is worth discussing at the conference. I will keep my current score and recommend for acceptance.

Thanks for Response

Thanks for the authors' response. My main concerns have been addressed. Thus, I will keep my score.