We appreciate the reviewer's taking the time to examine the paper and the provided code closely. We clarified your concerns and added Infograph experiments as requested. If anything is not clear, we are happy to discuss it during the rebuttal period.
**The Himmelbau example.**
We appreciate the reviewer's effort in running our code! In this simple example, you are correct that regular L2-SP would generate a similar trajectory. However, to see the difference between SPD's selective regularization and L2-SP, we suggest that the reviewer print out which direction is regularized during the optimization process.
Add the following printing statement to line 40 in the code before the return.
* print(self.condition_buffer)
* return new_param_list
The condition buffer contains the selection condition $c_t$ for the X and Y directions. Whenever an entry is negative, regularization is applied. This demonstrates the underlying mechanism for SPD.
**Is there a way to disentangle the gains..?**
Sorry for the confusion! SPD introduces only one component that affects performance, selective projection $c_t$ (Section 3.3), which selectively applies regularization during fine-tuning. The selectivity is the only complement that provides the performance gain.
Section 3.4 introduces a re-parametrization of the regular L2 regularization as a projection for better hyper-parameter tuning (see a comment from the reviewer Ex5N). It's a different way of writing L2 regularization and does not introduce performance gain.
Selectivity kicks in very often. However, it is hard to quantify, and it changes depending on the task, model, data, optimization hyper-parameters, etc. An analysis of the selectivity's behavior will be insightful.
**Trends in the results on ImageNet.**
This is a very good question. Indeed, L2-SPD will pull the weights closer to the zero-shot model, as shown in Table 1 and Table 2 in the main paper, thus inheriting its robustness and biases.
However, the extent of performance gain has a more subtle answer. It depends on the interplay between the pre-trained model, the fine-tuning (in-distribution) dataset, and the OOD datasets. Specifically, the ID dataset can provide useful information to the OOD datasets. In this benchmark, the useful information is that all datasets share the same label space. If fine-tuning leads to more gain, it's possible that after fine-tuning (assuming with proper regularization), the fine-tuned model can outperform both zero-shot and vanilla FT.
**Can this technique of selective regularization be done at a more granular level beyond layers?**
Yes, theoretically, the method can be applied at a parameter level. However, the storage cost is significantly increased because we would need to store a different $c_t$ for each parameter in the model.
**In section 4.1, why are all domainnet domains not used, omitting the infograph domain?.**
Infograph was omitted because none of the methods achieved good ID performance on this domain. We added the domain back to answer the reviewer's question (Table 1 in the rebuttal PDF). The superiority of SPD remains unchanged with the addition of this domain.
**Larger regualrization and make the deviation very low? Will the same correlation with OOD accuracy still hold?**
We have tried larger hyper-parameters. At extremely large values, the model will eventually underfit and perform poorly on ID and OOD data like L2-SP. We intend to show that L2-SPD outperforms L2-SP for similar deviations as the reviewer suggested.
**Hyper-parameter selection is not clear for ImageNet.**
We used the ID validation dataset following prior works. ImageNet-V2 can be seen as the clean test set for ImageNet.
**On quickdraw, will more aggressive regularization yield better OOD accuracy?**
Yes, we conducted a similar experiment as in Table 2 in the main paper. More aggressive regularization yields better OOD on quickdraw. We swept hyper-parameters from 1.0 to 2.3 and calculated the correlation coefficient between the hyper-parameters and OOD performance. The correlation is 0.88, which means a robust positive correlation between large regularization and good OOD performance.
**Linear layer for For ImageNet**
Because we used a CLIP ViT-Base model, we used a zero-shot linear head to initialize and fine-tune the linear layer. This practice follows from prior works such as WISE-FT.
**Comparison with Wise-FT?**
Technically, WISE-FT is an orthogonal method. Therefore, SPD should further improve its performance. To validate this, we added experiments combining WISE-FT and SPD in our Imagenet experiments (Table 1 in the rebuttal PDF). Our results show that SPD further improves the performance of Wise-FT. It's also worth pointing out that SPD is more general than WISE-FT, which only applies to models capable of zero-shot. It does not apply to the segmentation experiment (Table 4 in the main paper), whereas SPD does not have this constraint.