Set-based Neural Network Encoding Without Weight Tying

Summary

This paper proposes SNE, or Set-based Network Encoding, a method of encoding the weights of arbitrary neural networks in order to predict properties such as performance, generalization gap, training loss, number of epochs, etc. Specifically, using a chunking mechanism as well as layer-wise and block-wise positional encodings to generalize across different types of neural networks. Transferability experiments are conducted where a predictor is trained on one network model zoo, then evaluate on another, demonstrating an ability few predictors have. Additional experiments and ablations are performed, producing convincing results. Finally, the paper is decently well-written, but has a few fixable presentation flaws.

Strengths

- The contribution of the paper, using a network's weights to determine attributes about it, are quite novel. - The Methodology section is quite clear and easy to follow. Specifically, the aggregation mechanism in Eq. 9 is similarly analogous to graph aggregation in GNNs. - The experimental results are generally convincing. - Some visualization of the network representations from different methods are provided. - There are ablation results included in the appendix. - A limitations section is provided, highlighting reasonable restrictions on the experiments.

Weaknesses

- First, the motivation of this paper, specifically lines 29-30, is quite weak and fails to really provide examples of the application of this research other than asserting that they must exist. - Overall, the presentation is a weak spot, and should be revised. Some examples: - E.g., in line 31, the authors mention "Implicit Neural Representations (INR)" - what are these? No citation is provided. By contrast, "b) the performance of CNNs" is very easy to understand. - "modelzoo", this should be "model-zoo" [1] or "model zoo" [2-4]. - Citations in the paper have issues. For NeurIPS its generally just [num], or if you want to be fancier, "Author et al., Year [num]" but not "Author et al. [num]" without the year. - Float captions need work, e.g., Tabs 2/3 need longer captions to fully explain the performance metric Kendall's Tau, etc. Should be longer than 1 sentence. - Figure 2 needs work. The plots should be bigger and legend is clipped. - NeurIPS checklist should be placed after the references and supplementary. - The evaluation is done on simpler datasets/networks, however, the limitations section highlights why.

Questions

I find the idea of predicting end-to-end neural network attributes from the weight values to be quite fascinating. It is very similar to the concept of a neural predictor [5] in Neural Architecture Search [6] but quite orthogonal in the approach, relying on the weights instead of the graph structure. Similarly, most neural predictors suffer from a lack of generalizability/transferability [7, 8], like most of the baselines in this paper. Can the authors provide some commentary/comparison between these two umbrella fields? Similarly, is it possible to leverage the encoding mechanism (Eqs. 1-9) to derive which types of weights/layers/values are responsible for certain end-to-end attributes, e.g., performance of epochs, e.g., as [9] do? Refs: [1] "Equivariant Architectures for Learning in Deep Weight Spaces" - ICML'23. [2] https://modelzoo.co/ [3] https://github.com/onnx/models [4] https://pytorch.org/serve/model_zoo.html [5] "How Powerful are Performance Predictors in Neural Architecture Search?" - NeurIPS'21. [6] "Efficient Neural Architecture Design via Capturing Architecture-Performance Joint Distribution" - AISTATS'24. [7] "GENNAPE: Towards Generalized Neural Architecture Performance Estimators" - AAAI-23. [8] "Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor" - NeurIPS'22. [9] "Building Optimal Neural Architectures using Interpretable Knowledge" - CVPR'24.

Rating

6

Confidence

4

Soundness

4

Presentation

2

Contribution

3

Limitations

Yes, limitations and future work is discussed in App. B. The authors explain that their experiments are on simpler/smaller/older networks as they have limited computational resources. I find this to be an acceptable justification.

Summary

This work introduces a set-based neural network encoding that processes each chunk of an input model independently to predict network properties. The proposed model, SNE, is trained using Logit Invariance instead of weight tying to maintain generalizability to unseen architectures. Unlike previous approaches, SNE can be trained and tested on different architectures and it demonstrated superior performance in out-of-distribution experiments involving both datasets and architectures.

Strengths

SNE is more general than its predecessors and can be applied to any input architecture. Results indicate that it outperforms other encoding methods.

Weaknesses

While SNE shows improvements in novel tasks such as cross-architecture and cross-dataset scenarios, its comparison on traditional tasks is limited to predicting INR frequencies. Expanding Table 1 to include SNE's performance on additional outputs, such as accuracy, would enhance the paper.

Questions

The writing is generally good but could be improved in certain areas. For example, the abstract contains a redundant sentence: “by utilizing a layer-wise encoding scheme that culminates to encoding all layer-wise encodings to obtain the neural network encoding vector,” which should be revised.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations were properly addressed in appendix.

Summary

This work tackles an original and interesting challenge: predicting neural networks properties from their trained weight values. To do so, the authors propose to leverage set to set and set to vector transformations in order to encode weight values. Furthermore, they propose to account for the deep neural network architecture and operation order through the addition of positional encoding at several stages of the property predicting model. Ultimately, they anticipate the problem of various model sizes with a padding/chinking mechanism. The proposed method is then evaluated on small scale datasets in a cross-dataset fashion against method of the field, published in recent years.

Strengths

Such research studies can have a significant in other fields. For instance, a good accuracy estimator from model weight values can be leveraged as a proxy objective in DNN quantization or weight pruning. The approach is well motivated and the solutions to the highlighted challenges are sound. The paper is well written and simple to follow. The authors provide a thorough evaluation and ablation study of the proposed method and its components.

Weaknesses

From my understanding of this work, I identified two major drawbacks: 1. the authors insist on the generalization across different model architectures and tasks. From my perspective, the former is the most important one. However, as it stands, there is little variation between Arch1 and Arch2 as presented in this study: both are simple CNNs. I wonder if the authors could provide numbers w.r.t transformer architectures against CNNs. 2. Since these experiments can be expensive (require multiple training of DNNs to create the predictor training set), it would be interesting to measure the impact of the dataset size w.r.t the predictor's performance, in order to estimate how far the community is from being to reasonably work on large scale datasets such as ImageNet. As it stands, I believe this work aims at providing empirical results which are not fully convincing yet. I look forward to the authors' response and other reviews and will update my rating accordingly.

Questions

I summarize the previous points: 1. could the authors provide results with a predictor trained on CNNs from arch1 and predict the performance of transformers? 2. could the author measure the importance of the training set size (pairs of DNNs and accuracy) w.r.t. the predictor's performance, in order to estimate the cost of working on ImageNet-like tasks?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors made a relevant attempt at highlighting the limitations of their work

Summary

The study introduces Set-based Neural Network Encoding (SNE) without weight tying, allowing for encoding network information into a compact form. SNE utilizes Logit Invariance Regularization to ensure the correct functional equivalence of network permutations, departing from traditional weight tying methods. The choice of Set-to-Set and Set-to-Vector functions, including attention mechanisms and pooling layers, are crucial in implementing the SNE framework. Experimental results demonstrate the effectiveness of SNE in predicting generalization performance for Convolutional Neural Networks (CNNs) and Incremental Neural Networks (INRs). SNE is compared with various baselines such as MLP, Deepsets, HyperRep, and others, showcasing its superiority in certain aspects.

Strengths

- New evaluation benchmarks of cross-dataset and cross-architecture evaluation. - Solid empirical performance. SNE significantly outperforms HyperRep by very large margins in the cross-architecture task. Also, SNE demonstrates true agnosticism to architectural choices compared to NeuralGraph, and shows robust performance without being trained on the training set of a specific model zoo.

Weaknesses

- The proposed method did not ensure weight-space equivariance by design but use a way that is more like an augmentation plus a consistency loss. The theoretic contribution is therefore a bit weak. - The padding and chunking designs look tricky and it is not clear for me how they help with the performance. - Fig 1 is not really clear. (1) the legend is not sufficient to understand the computation diagram presented in the figure. (2) the input is shown as neurons but from the method I think it should be the weight matrix that connecting neurons in adjacent layers.

Questions

- In table 1 NeuralGraph is not reported. Is it because the method is not applicable there? - Does the method need more optimization steps than baseline methods that ensure weight-space symmetry by design? I suppose optimization would be harder because it requires to optimize for the logit invariance.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The limitations of the method are discussed in Appendix B

Response to rebuttal

I would like to thank the authors for their response and the new experimental results they provided for this rebuttal. Overall, I consider that my concerns have been addressed. Thus, I am likely to increase my rating for this work after the end of the discussion period.