The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof

Summary

This paper develops two types of modifications to neural networks to remove permutation symmetries: fixing certain weights, or using a non-elementwise activation function. The resulting "asymmetric" neural networks are found to improve on certain metrics that are observed in networks with permutation symmetries, and thought to be caused by such symmetries: lack of linear mode connectivity between randomly initialized/trained networks, reduced Bayesian network performance, and reduced metanetwork performance. The asymmetric networks also are more likely to have monotonic linear interpolation between their initial and trained weights.

Strengths

This paper is effectively an ablation study on permutation symmetries, which provides a much-needed tool and counterfactual perspective for investigations of permutation symmetries. The methods appear both novel and logical. A thorough survey of prior literature is given, which is extremely welcome. The paper is well organized, with well-motivated methodology and easy-to-follow experiments and hypotheses are easy to follow. The proofs are intuitive and comprehensive.

Weaknesses

The main issue is that the paper relies heavily on experimental rather than theoretical results, but the strongest results are for a single method ($W$-asymmetry) with settings much more different than "standard" training than seems at first glance - specifically, lower learning rates, longer warmup, and larger fixed weights that all differ from standard training by order(s) of magnitude. These settings, plus a lack of available code, seriously impact the relevance/reproducibility of the methods/results. There should be a comparison with standard models initialized to the same weights as the $W$-asymmetric networks. This would make certain metrics like $L^2$ weight distance comparable (currently the $W$-asymmetric networks are more likely to have extremely large magnitude weights which could increase the $L^2$ distance between independently initialized/trained networks). This would also eliminate some other variables such as reduced performance due to suboptimal weight initialization. Low learning rates and long warmup are likely to improve similarity and reduce barriers, particularly between identically initialized neural networks. An analysis of how learning rate/warmup affects the experimental results (particularly barriers after training) is needed to determine if the observed results are mainly due to asymmetry. Some additional details are needed to properly replicate the asymmetry methods, specifically in tables 5-7. - how are $n_{\text{fix}}$ and $\kappa$ determined for each architecture? - what do the names of the blocks refer to? There are not enough for a 1-to-1 mapping onto ResNet20 layers or permutations, so more explanation is needed.

Questions

Are the results in table 1 for networks with different random initializations, or the same identical initialization? If the former, are the masks and fixed weights different or the same between the two networks? I find it highly unlikely that networks with different fixed weights or different initializations would have low barrier after training without post-processing. The fixed weights in $W$-asymmetry are distributed very differently to the non-fixed weights. If the fixed linear transform $F$ in $\sigma$-asymmetry is similarly shifted in distribution, would that make its results closer to $W$-asymmetry? Why not fix biases to get asymmetry instead?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The reproducibility issues discussed previously, combined with the experimentally-focused framing, detract from an otherwise very strong and impactful paper. The results for $\sigma$-asymmetry are considerably weaker than $W$-asymmetry (and rely on even smaller learning rates). Another work has found that networks mainly use linear transformations [1]. Since $\sigma$-asymmetry is effectively a non-elementwise sigmoid activation function, I wonder if the network's activations mainly stay in the approximately linear region of the sigmoid, allowing many permutations to be approximately equivalent. Some other comments or experiments on why $\sigma$-asymmetry is less effective in practice would be helpful, especially since theoretically speaking, both methods are clearly asymmetric. [1] Mehmeti-Göpel & Disselhoff. Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think. ICML 2023. https://proceedings.mlr.press/v202/ali-mehmeti-gopel23a.html

Summary

The paper suggested asymmetric neural networks in terms of an asymmetric nonlinearity and weight. It demonstrated some tasks including linear mode connectivity without permutation alignment, Bayesian neural networks, training metanetworks, and monotonic linear interpolation to show the role of permutation symmetry inherited in the standard neural networks.

Strengths

1. Various evaluations to show the role of permutation symmetry. 1. Novel approaches to build neural networks that have no permutation symmetry.

Weaknesses

1. The number of learnable parameters of the standard NN and the asymmetric NN is reported only for the metanetwork task. It should also be reported for the LMC, BMA, and MLI tasks. 1. Comprehensible reason is required for the proposition “the posterior will have less modes” in the hypothesis of the BMA task in order to understand the purpose of the BMA task. No symmetry does not imply less modes. Thus, the improved performance of BMA may be induced from another reason other than asymmetry.

Questions

What do you mean by “loss” in BNN. Is it an ELBO? Why don’t you report the negative log-likelihood (NLL)?

Rating

5

Confidence

4

Soundness

1

Presentation

2

Contribution

2

Limitations

1. Fixing some weights or adding anisotropic activation function implies constraining not only symmetry but also distorted parameter manifold (like equivariant NNs) that also leads to increasing correlation between solutions. The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation. 1. The BMA and MLI tasks are based on the strong assumption, where its evidence is absent, that no symmetry induces the reduced number of modes. 1. Although the paper partially gives insights on the loss surface, it does not suggest a new practical method utilizing the insights.

Summary

This paper proposes to study the effect of parameter symmetries on the neural networks' training and final properties by analyzing the behavior of networks without such symmetries (or with fewer of them). To do so, the authors develop two methods of parameterizing neural network architectures without parameter symmetries: W-asymmetric parametrization fixes the different subsets of weights in each row of weight matrices to be constant and untrainable, and a $\sigma$-asymmetric network uses a new FiGLU nonlinearity. The paper analyzes the asymmetric properties of both proposed methods theoretically and demonstrates empirically that asymmetry improves the behavior of neural networks in several setups. Specifically, asymmetric neural networks have better linear mode connectivity after training and more stable monotonic linear interpolation between initialization and trained model, are more effective for Bayesian deep learning, and are easier to model with metanetworks.

Strengths

1. The paper proposes a new idea of analyzing the effect of parameter symmetries on neural networks through comparison with asymmetric networks in practical settings. I think this perspective is interesting and has potential for future research. 2. The authors develop easy and effective asymmetric parametrizations and analyze them theoretically. 3. The applicability of the proposed asymmetric perspective is demonstrated on a wide range of related problems. 4. The paper is clearly written and easy to follow.

Weaknesses

My main concerns are related to the limited discussion on the benefits of the proposed perspective in comparison to previous works, asymmetric properties of the proposed methods in practical setups, and poor results of $\sigma$-asymmetric networks: 1. As mentioned in the Related Work section, previous works examined the influence of parameter symmetries on neural network training by changing the optimization schemes instead of the network parametrization. Even though I find the idea of studying the behavior of asymmetric neural networks interesting, it is not clear to me how this perspective is beneficial compared to the constrained optimization one. Adding a more thorough discussion on that in the Related Work section would improve the paper. 2. It is not clear from the paper if W-asymmetric and $\sigma$-asymmetric networks are fully asymmetric or just have fewer symmetries in practical setups. A more accurate discussion on which symmetries are removed by the proposed methods in the experimental setups should be added to the paper. For example, the paper does not cover the normalization layer symmetries, even though layer and batch normalization are used in the experiments. It seems W-asymmetric networks remove normalization symmetries, while $\sigma$-asymmetric ones do not. It may be one of the reasons why $\sigma$-asymmetric networks show weaker results in linear mode connectivity and linear interpolation sections. The ReLU scale symmetry is also not adequately discussed. 3. The analysis of the $\sigma$-asymmetric networks is limited. There is no proof or discussion of the universal approximation for this method. The paper does not explain poor results on linear mode connectivity and linear interpolation and does not include the results on Bayesian neural networks and metanetworks. A more detailed analysis of the $\sigma$-asymmetric networks, or at least a discussion of its suboptimal results, would benefit the paper. Additionally, I have some minor concerns regarding the experiments: 1. In the Bayesian neural network experiment, the W-asymmetric networks have fewer parameters than standard ones, which may influence the optimal training hyperparameters. Hence, it may be the case that the difference in the results is due to, e.g., different effective learning rates and not the asymmetric network structure. 2. In the metanetwork experiments, standard and W-asymmetric networks have different numbers of parameters. Hence, it may be the case that the difference in the results is due to the different dimensionality of the input space and different optimal training hyperparameters for metanetworks and not the asymmetric network structure.

Questions

I would kindly ask the authors to address the concerns from the Weaknesses section and focus on the following questions: 1. Could you please elaborate on how the asymmetric network perspective differs from the constrained optimization one and in which cases it shows new insights, in your opinion? 2. Could you please clarify whether the W-asymmetric and $\sigma$-asymmetric networks are fully asymmetric in the experiments or not? Which of the known symmetries (neuron permutations, ReLU scaling, pre-normalization parameters scaling, etc.) does each method remove in practice? 3. Could you please comment on the poor results of $\sigma$-asymmetric networks in linear mode connectivity and linear interpolation experiments? Is there any specific reason why the Bayesian neural networks and metanetworks experiments are not conducted for $\sigma$-asymmetric networks? 4. Could you please comment on the technical differences between standard and W-asymmetric networks (optimal hyperparameters and parameter count) in Bayesian neural networks and metanetworks experiments and whether they can affect the conclusions? Minor questions: 1. Do I understand correctly that the standard deviation $\kappa$ used in W-asymmetric and $\sigma$-asymmetric networks is a hyperparameter tuned separately from the trainable weights? (lines 134 and 163). 2. Do the non-trainable weights of all W-asymmetric networks have the same values in the metanetwork experiment? Does a metanetwork take only the trainable parameters of W-asymmetric networks as input?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors adequately discuss the limitations of the paper in the appendix.

Summary

This paper studies how removing parameter symmetry of neural networks affects the loss landscape, Bayesian Neural Networks and meta-networks. The authors proposed two ways to remove the parameter symmetry: one is similar to pruning that making some parameters untrainable, the other is to adopt non-elementwise activations. Both ways can remove the permutation symmetry in parameter space. After that, the authors empirically demonstrate that removing permutation symmetry substantially 1) make LMC easier to satisfy 2) make BNN's training more efficient 3) improve the performance of meta-network (I am not quite familiar with meta-network or neural functionals.) 4) make training loss along the line segment between initialization and trained parameters more monotonic.

Strengths

1. The writing is clear and easy to follow. Especially, the structure of this paper is clear: front part is about the two ways to remove parameter symmetry, back part is to demonstrate the effect of removing parameter symmetry on loss landscape, BNN, meta-network and MLI. 2. This topic is pretty interesting. Permutation symmetry persists in most neural network architectures and impose structure beyond Euclidean structure to parameter space, however, the community has no deep understanding of how permutation symmetry relates to the success of Deep Learning. 3. The experiments on BNN, meta-networks and MLI are interesting.

Weaknesses

1. A major issue about this paper is that some findings are not new. Especially, some studies have already demonstrated that asymmetric networks are more easily to satisfy LMC. [Cite 1] showed that pruning solutions are within the same basin and pruned networks can be viewed as a special case of W-asymmetric networks. [Cite 1] Evci, Utku, Yani Ioannou, Cem Keskin, and Yann Dauphin. "Gradient flow in sparse neural networks and how lottery tickets win." In Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 6, pp. 6577-6586. 2022. 2. Another issue is that in experimental part, the four hypotheses seemingly do not relate to each other. There is not a unified conclusion drawn from the experiments. 3. The most important question about the permutation symmetry remains unresolved. As permutation symmetry holds in most neural network architectures, should we remove permutation symmetry or not? Is the success of deep learning related to the permutation symmetry? This paper seemingly cannot give an insightful answer. 3. In Sec. 5, despite each experiment are motivated from some simple intuition, there is no rigorous theoretical foundation for each hypothesis, which could potentially lower the value of this study.

Questions

N/A

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

N/A

Reviewer's response

Thank you for the detailed response and additional ablations! I find most of my concerns adequately addressed. After reading other reviews and responses, my evaluation of the paper remains very positive. Considering the clarifications on the asymmetric properties of the proposed methods in practice and the low performance of $\sigma$-asymmetric networks, as well as the new ablations, I am raising my score to 7. Generally, I really enjoyed reading this paper =)

Rebuttal

Thank you very much for the detailed clarification. While I understood some of them, I still have concerns regarding the proposition. As you and the cited papers mentioned, when we fix an architecture (both in width and depth) and then fix some weights, the total number of modes will obviously be reduced. However, in a fair comparison, where the standard NN and the asymmetric NN have the same number of learnable parameters, the architectures, and thus their parameter spaces, will be different. Since we have no idea about the total number of modes for each architecture, it cannot be definitively stated that the number of modes will be reduced. In fact, the modes could potentially increase due to the added nodes. Furthermore, **the number of modes can be uncountably many** when ReLU nonlinearity (e.g., ResNet) is used due to its scaling symmetry. For these reasons, I believe a comprehensive (theoretical) justification is necessary. Although you claimed that the experiment does not rely on this assumption, I believe it does. If the number of modes is not reduced, the improved BNN performance in the asymmetric NN could simply be a result of the increased number of learnable parameters, as you mentioned in the rebuttal table. If you intend to demonstrate the hypothesis regarding the number of modes, I would recommend using the exact same architecture for both the standard NN and the asymmetric NN, even if it causes fewer parameters.