Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

Summary

In this work, the phenomenon of asymmetric valleys in deep neural networks (DNNs) minima, first observed and described in [1], is systematically examined. By studying different types of (random) directions around a given minimum, the authors discovered that the degree of sign consistency between the chosen direction and the minimum point is a key factor for asymmetry. Besides thorough empirical evaluation, theoretical reasoning is provided to explain this phenomenon. The findings are also practically applied in model fusion. [1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.

Strengths

All in all, I like this work. Generally better loss landscape understanding is important for modern deep learning, and this work takes another valuable step in this direction. To my knowledge, the question of minima asymmetry was not systematically investigated after its first appearance in [1], so I consider the contributions of this study novel and significant. The paper is well structured. The narrative is easy to follow, the authors motivate their experiments (“Then, a fantastic idea motivates us to change the sign of the noise” – liked that) and consequently validate their conjectures. The claims are clearly formulated and well supported empirically. Empirical evaluation is solid, involving multiple datasets, architectures, and hyperparameter options. The authors tackled many important questions concerning the study of asymmetric values and the sign consistency conjecture. Although theoretical support is mostly intuitive, with no rigorous proofs (the authors mention that in the Limitations section, which is commendable), it paves the way for future work that will address this limitation. Demonstrated practical implications of the insights of this work for model fusion are also important and require further investigation. [1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.

Weaknesses

I see no concrete substantial weaknesses in this work. I list some of my questions and suggestions in the following section. To me, the main shortcoming is weak theoretical support of the claims made in this work. However, the authors acknowledge this, and I hope that future work will address this disadvantage in some way. Also, the experiment in Sec. 6.2 about practical implication in federated learning appears to me a little hastily conducted. Some more details would be appreciated (see specific questions below). The text could be slightly polished to eliminate typos and small inaccuracies.

Questions

1. The Investigation of 6 Special Noise Directions: Empirically, noise directions that have the same sign with the positive parameters and applies zero to negative parameters ($\epsilon_5$ and $\epsilon_6$) do not show asymmetry. I wonder, why so? Do the authors have any intuition on that? To me, that’s a little counterintuitive, since these directions specifically target the sensitive BN.weight parameters (according to the results of Sec. 4.2.1), don’t they? Also, does the same lack of asymmetry is exhibited by the negative-sign noise directions? 2. The Finding Holds for ImageNet and Various Parameter Groups: a) What type of noise is applied here? b) It is confusing that BN parameters seem like the most “robust to asymmetry” (according to Fig. 6) in light of other experimental results of this paper, underlining the specific role of BN parameters in asymmetry. Even the original work [1] claimed that “directions on BN parameters are more asymmetric”. Could the authors elaborate more on that? 3. Line 186: centered around 0.5? 4. Section 4.2.1: Is the noise applied to all parameters or to BN parameters only? If to all, it is indeed remarkable how the initial distribution of BN weights, which take a negligible part in total model parameters, may influence the asymmetry of certain directions (getting back to Question 2b)! 5. Section 5: Did the authors train a plain linear classifier or a neural network in the sklearn.digits experiment? 6. Section 6.2: Eq. (1): I might be wrong, but it seems that the regularizer is maximized when the signs of $\theta_t$ and $\theta_t^k$ are aligned, which is undesired, isn’t it? The experiments here require stds, as the results presented in Table 1 could lack statistical significance. Also, what are the hyperparameters of the baselines, how were they chosen? Were they validated just as $\gamma$ hyperparameter in FedSign? 7. I suppose that [2], where sign disagreement is also taken into account in model merging, could be a valuable related work. [1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019. [2] Yadav, Prateek, et al. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems 37 (2023).

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations in Section 7.

Summary

This paper explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. The major contribution is the observation that the degree of sign consistency between the perturbation and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function are provided to explain the asymmetry of DNN valleys. Imposing sign alignment in federated learning is proposed for model parameter alignment.

Strengths

The observation that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry is reasonable, and supported by the theoretical explaination based on ReLU activation and softmax function.

Weaknesses

1. The asymmetry of DNN valleys is not unexpected, and has been studied in [24]. 2. The observation that sign consistency between the noise and the convergence point affects the valley asymmetry, taking the ReLU activation and softmax function into account, is intuitive to a large degree and hence deos not emerge as a significant contribution. 3. Lack of theoretical analysis, such as the bound of degree of asymmetry caused by sign consistency. 4. Although the application of sign consistency in federated learning is provided, the readers may be more interested in the implications of valley asymmetry to local search based optimization methods, such as SGD, SAM etc.

Questions

1. Fig.1 is not clear enough. 2. There are two "then"s in line 177.

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Lack of theoretical analysis and the discussion on the implications of valley asymmetry to local search based optimization methods.

Summary

This paper investigates the characteristics of the asymmetric valley in deep neural networks (DNNs) for classification. The authors perform a perturbation analysis around the local minima, considering the direction of the injected noise. The asymmetric valley demonstrates that DNNs exhibit smaller fluctuations with sign-aligned noise, while opposite noise dramatically increases the test error. Other interesting findings include: (1) Batch normalization initialization and hyperparameter choices affect the occurrence of asymmetry. (2)The ReLU activation and softmax function might explain the phenomenon. The authors also leverage the findings to explain the success of model fusion and propose a sign-regularized method for federated learning.

Strengths

1. The paper is well-structured and clearly presents its interesting findings. The authors start with empirical findings and further apply their discoveries to practical applications, including the model fusion trick and federated learning. 2. The theoretical insights regarding the presence of ReLU and softmax are convincing. 3. The proposed direction-aligned noise method is simple and effective, potentially inspiring further research in areas like transfer learning and federated learning.

Weaknesses

1. In Section 6.1, the authors claim that pre-training is important for the success of model fusion. However, the compared models have different architectures (VGG16BN vs. ResNet18) and are trained on different datasets (CIFAR10 vs. Flowers). This experimental setting weakens the credibility of the conclusion. 2. Regarding the conclusion of Fig. 8, “The interpolation curve shows that the small batch training (i.e., λ = 0.0) lies in a sharper and nearly symmetric valley, while the large batch training (i.e., λ = 1.0) lies in a flatter but asymmetric valley,” I find this problematic. Fig. 9 already shows that the valley is asymmetric for BS=32, while Fig. 8 concludes that BS=32 is symmetric. For the possible explanation for the lower part of Fig. 8, my understanding is that BS=2048 has a more biased value distribution, while BS=32 is more zero-centered. Therefore, the same strength of interpolated noise affects BS=32 more. Besides, the degree of asymmetry is not well-defined in the paper, the conclusion based on this comparison might be problematic. 3. The conclusion from Fig. 8 also conflicts with the statement in related work that “large batch size training may lead to sharp minima with poor generalization.”

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The discussion doesn't include transformer architecture.

Reviewer's response

I thank the authors very much for the detailed and interesting commenting on my questions! I would like to remain my score unchanged and vote for acceptance.