Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks

Summary

The paper provides a thorough investigation of the generalization behavior of t-NNs for the first time, which closes the gap between the practical success of t-NNs and their theoretical analysis. To be specific, the authors derived the upper bounds for the generalization gaps. The authors also propose that compressing t-NNs with a transformed low-rank structure will result in more efficient adversarial training and tighter bounds. In addition to that, the authors show that adversarial training with GF in highly over-parameterized settings results in t-NNs with approximately transformed low-rank weights, and derive the sharp adversarial generalization bounds in this scenario as well.

Strengths

- The paper provides a thorough investigation of the generalization behavior of t-NNs for the first time. - The authors provide the bound for t-NNs generalization gaps with firm proof. - The authors further explore the generalization bound of t-NNs with a transformed low-rank structure and show that there exists a training framework that will result in approximately transformed low-rank weights. - The work is well motivated and with firm theoretical evaluation.

Weaknesses

- Only theoretical evaluation is provided. Some empirical evaluations regarding the approximately transformed low-rank weights are expected.

Questions

- Instead of using adversarial training with GF in highly over-parameterized settings to train a approximately transformed low-rank t-NN, is it possible to apply some extra regularizations in training to achieve a better low-rank transformation, for example, LoRA?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

The paper "Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks" considers neural networks with t-product layers, that encode the weights in tensor format equipped with the t-product as the corresponding tensor operation and ReLU activations. The paper is an analytical work that derives upper bounds for the generalization error in t-NNs, using norm bound estimates. Further, the influence of low-rankedness of the Tensor structures on robust generalization is inspected in the setting of exact low-rankedness and approximately transformed low-rankedness. Results show that low-rank t-NNs improve robust generalization properties. In particular, the Adverserial generalization bound for low-rank tNNs is shown to scale with the tubal rank of the tensors, instead of their dimension as in the full-rank setting. Additional findings show that in an overparametrized setting t-NNs yield approximately low-rank weights during adversarial learning.

Strengths

- Although this is a rather technical paper, the authors manage to present the findings comprehensively by rigorously introducing the relevant concepts with a precise notation. - An extensive Appendix is provided to give an introduction to t-SVDs, and overview of previous work and technical details. - The theoretical results of the work are discussed and compared with previous works.

Weaknesses

- No code and no numerical examples are provided to support the presented results. Although this is a rather theoretical work, a small test case can improve the presentation of this paper tremendously. I think this is a strong paper that gives a multitude of insights. A brief explanation of the intuition behind t-NN layers would also be helpful.

Questions

- What is the intuition of the transformed t-product, i.e. what is the reason that the M matrix is needed? What is the special case of M being the identiy? - Theorem 6: Can this statement be related to "standard" matrix valued neural networks with low-rank factorizations? - Theorem 10: Are you able to validate the statement in a numerical example. It would be interesting to see, how the rank of a layer changes as J gets larger.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss the limitations of their work, which manifest in rather loose bounds on the generalization error. They provide a suggestion on how to mitigate this issue in future work.

Summary

The paper analyzes the generalization ability of t-product layers (t-NNs) by deriving upper bounds on generalization error in standard and adversarial settings.

Strengths

The paper advances the theoretical understanding of t-NNs and derives their generalization behavior in two practical settings.

Weaknesses

The paper focuses on and is written about t-product layers, referenced as [7, 28] in the first lines of the introduction. The first reference is a patent, and the second is a paper from 2018 (from the same authors as the patent), introducing the concept more or less academically. The submission assumes the reader's familiarity with these concepts right from the start, which is a mistake, and does not attempt to clarify or recap the importance of these concepts up until Sec. 3. From this standpoint, the writing should be improved. The importance of the usage of t-product layers in the Deep Learning community is not supported with sufficient evidence, leaving the question of the practical utility of the studied framework unanswered.

Questions

Is Definition-1 (t-product) equivalent to the batch matrix-multiplication operation, where `c` corresponds to the batch dimension? Is Definition-3 (t-SVD) equivalent to the SVD of a block-diagonal representation from Definition 2? Why do these concepts need special terms? What is the advantage of such complex structures over the traditional deep learning layers?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

1 poor

Limitations

Discussed in the Conclusion.

Summary

The paper studies the generalization error in the standard and the adversarial settings of t-NNs, neural networks with layers and features parametrized via t-vectors and t-products.

Strengths

- The provided generalization bounds for t-NNs show that neural networks with low-rank parameters have the potential for a better generalization. - This type of generalization analysis for tNNs seems new

Weaknesses

1. Although I appreciate this type of analysis for t-NN did not appear before, the t-product is essentially a composition of operations obtained through a suitable block reshaping of the parameter kernel, thus extending the results from standard fully connected linear layers seems like a natural and relatively direct result. In light of this, the novelty of this paper seems limited 2. I would have liked to see a few numerical experiments validating the theoretical findings and the tightness of upper bounds -- also to address the claimed limitations at the end of the paper

Questions

- the authors should clarify in the main text how the obtained results differentiate with respect the corresponding bounds for linear fully connected nets (for example the bound in Thm 6 and Thm 10) and should highlight what the main differences in the proof/analysis of these bounds are with respect to the linear case - I think the rank-adaptive approach for selecting the ranks of the network mentioned on line 186 deserves more attention - in particular, i do not really understand in what way rank-adaptivity can be exploited in the analysis provided by Thm 6. See also [x],[y] for alternative recent rank-adaptive strategies for linear layers. - The fact that adversarial training leads to low-rank weights and vice-versa reducing parameters has a positive/negative effect of robustness (as mentioned at line 250) is subject to ongoing active research and there is a variety of different observations/results on this subject (see eg [a-d]). I would rephrase here in the light of these papers. In any case, these empirical studies are done for networks in the standard form not for t-NNs. Using that evidence to comment on findings for t-NN seems to support my weakness no. 1 [x] H. Yang, M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification (IEEE/CVF 2020) [y] S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations (NeurIPS 2022) [a] https://openreview.net/pdf?id=SJGrAisIz [b] https://arxiv.org/abs/1912.02386 [c] https://arxiv.org/abs/2306.01485 [d] https://link.springer.com/content/pdf/10.1007/s10994-021-06049-9.pdf?pdf=button

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have added a "limitations" paragraph stating that the obtained generalization bounds may be somewhat conservative, which I agree with. I am not sure I understand the proposed solution to the limitations though.