Summary
In this paper, the authors propose an efficient fine-tuning method for deep learning models that learns the constraint for each layer more efficiently.
During neural network training, the proposed method optimizes the layer-wise constraint using the last batch of data, unlike a prior method (TPGM) that updates layer-wise constraints on a "validation" dataset. This new method is less computationally burdensome since the gradient of the constraint can be computed using the chain rule, which reuses the gradient stored for the last batch.
The authors also propose a notation to describe fine-tuning robustness and provide theoretical justification for the proposed method under this notation. To demonstrate the effectiveness of the new method, the authors conduct a comprehensive list of experiments and show that the proposed method is not only more efficient but also has better performance.
Strengths
-The writing in this paper is of high quality, as the authors have presented the methodology in a clear manner.
-The related work section is well-discussed. The authors have categorized the existing fine-tuning studies into three categories: when, where, and how much to fine-tune. This is an insightful way to approach the current studies.
-The authors have provided theoretical justification for the proposed method.
-The experimental results are comprehensive, as the authors have included a list of different datasets and model architectures. The proposed method has been shown to outperform all baseline methods.
Weaknesses
-My first concern is that the proposed method's novelty might be overshadowed due to its similarity to a previous method called TPGM. The major difference between the two seems to be the selection of batch data for updating the constraint parameter. The previous work used validation data, which may not be a common choice in the supervised learning setting, while this work uses the previous batch data. Although using the stored gradient indeed improves computational efficiency, the authors did not clearly state why this method has better performance than TPGM. The validation data and the other training data should have the same distribution.
-The proposed difference function's meaning requires in-depth justification. While (Lipschitz) smoothness is usually associated with better performance and (adversarial) robustness for deep neural networks, it is not directly implied that the proposed difference function will improve the downstream performance of a fine-tuned model. Additionally, this notation only describes the difference in feature space; how does it describe the linear-probing fine-tuning method, which only changes the linear last layer? Can it be shown that this proposed notation indicates fine-tuning downstream performance?
-The robustness notation here seems to be related to the OOD generalization capability. Are they the same thing here? Usually, the robustness is used to describe the model’s performance under some unwanted perturbation (adversarial samples or corruption)
-The Low-rank fine-tuning methods have been really popular recently. Is it possible to compare the proposed method with them?
Minor typo:
Line 52, Liptschitz → Lipschitz
Line 127, “Then FTP calculates the gradients”, is here FTP meant to be TPGM?
Questions
What’s the pattern of learned constraint for each layer, with regard to training epochs? Do they converge during the training?
I am really curious about the statement of discrepancies here. The author mentioned multiple times that the discrepancy between the validation data and training data or between the different batches of data enables the learning of meaningful projection constraints but without further elaborations. Typically, the training and validation data should be subject to the same distribution.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.