Summary
The paper proposes a new regularization technique, Constrained Parameter Regression (CPR), to replace weight decay for training deep learning models. Conventional weight decay penalizes significant weight deviation. However, this can be too restrictive since some layers may need a larger deviation. Instead of applying the same regularization strength for all layers as in weight decay, the paper uses different hyperparameters for each layer, resulting in a more flexible regularization strength. Intuitively, a model with a larger weight $||\theta||$ is allowed more deviation under CPR. The method is theoretically motivated by the augmented Lagrangian method and tested on multiple tasks and benchmarks.
Strengths
* The method is principledly motivated and intuitively explained. Overall, the paper is very well-written.
* The paper proposes multiple alternatives for setting the method's hyper-parameters, taking into account efficiency and tuning flexibility.
* The paper demonstrates superior performance on multiple tasks with modern deep learning models, including large language models.
Weaknesses
* **The regularization strength (upper bound $k$) relies on empirical observations and intuition**. While the overall method is theoretically motivated, the upper bound $k$ setting relies mainly on intuition. Specifically, the paper does not explain why $k\leftarrow R(\theta)$ is a good strategy. Intuitively, this strategy indicates that a larger weight $||\theta||$ should have less regularization.
* **Computer vision tasks are limited**. For classification, computer vision experiments are conducted on relatively small-scale datasets, such as CIFAR100. The paper should consider using moderate-to-large-scale datasets with well-established benchmarks, such as ImageNet, to make the empirical evidence more convincing. For segmentation, popular benchmarks, such as MSCOCO, could be a better choice to establish the superiority of this regularization.
Questions
* Why does the inflection point of the regularization function $\Delta\Delta R$ reflect a saturation of performance? Shouldn't the inflection point of the loss function $L$ be a better fit for this?
* Could the authors report results on ImageNet? If time is constrained, fine-tuning a pre-trained model, such as CLIP's pre-trained ViT-B [1], to ImageNet is a good choice.
* Is there any empirical evidence supporting the design choice $k\leftarrow R(\theta)$? It would be great if the authors could discuss this choice in detail.
* Prior works in hyper-optimization [2,3] show how to optimize hyper-parameters in optimizers differetiably. This could be a valid future direction for optimizing the upper bound $k$. A new work uses hyper-optimization in an optimizer for fine-tuning [4], which shares a similar spirit to this paper at a high level.
[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.
[2] Baydin, Atilim Gunes, et al. "Online learning rate adaptation with hypergradient descent." arXiv preprint arXiv:1703.04782 (2017).
[3] Chandra, Kartik, et al. "Gradient descent: The ultimate optimizer." Advances in Neural Information Processing Systems 35 (2022): 8214-8225.
[4] Tian, Junjiao, et al. "Fast trainable projection for robust fine-tuning." Advances in Neural Information Processing Systems 36 (2023).
Limitations
There is no potential negative societal impact.