Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training

Summary

The paper conducts an extensive empirical study including several popular pruning criteria and analyzes their impact on the DST framework on diverse models. They found that within a stable DST hyperparameter setup, the majority of the studied criteria perform similarly, regardless of the model architecture and the selected growth criterion. In addition, in very sparse regimes, the simplest magnitude-based pruning methods surpass any more fancy choices.

Strengths

1. The paper is well-written and the key research questions examined are well-supported by extensive experiments and appendix results. 2. Experiments related to how the frequency of topology updates influences the effectiveness of different pruning methods are interesting.

Weaknesses

The primary concern related to the submission is the lack of novelty. Although some observations will be very beneficial to the sparsity community, the supremacy of magnitude-based weight removal in the high sparsity range is not novel. I believe extending this paper beyond small-scale models, to mid-to-large-scale models will make the paper more empirically strong and relevant in the current state of deep learning. In addition, some analysis of growth trajectory, connections which were never removed or always remained pruned across different pruning staretgies will add more value and uplift the quality of the work. Overall, it is still a strong empirical paper.

Questions

The paper's readability will benefit by including important hyperparameters like prune and growth ratios etc in tabular format in the main draft. When written with experimental discussion, it becomes difficult to follow up.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See above.

Summary

This paper performs a systematic study of pruning strategies for dynamic sparse training (DST) methods, comparing their performance and structural decisions across backbone architectures, datasets, structural change frequencies, connection densities, and batch sizes.

Strengths

Originality: This work is the first systematic study on DST methods, combining various pruning and growing strategies that have thus far only been studied in isolation. It is also the first work to structurally compare the results of various methods. It does not introduce any new DST strategies, although it adapts a well-known pruning metric (SNIP) for the DST context and makes novel combinations of known DST strategies. Quality: This paper backs up claims with adequate experimentation and analysis, although the scope is limited to mostly image classification tasks (plus one tabular dataset). Clarity: The paper is easy to understand overall, with a clear writing style. Significance: This paper provides a significant study on the interplay not only of growing and pruning strategies but also factors such as backbone architecture, dataset, structural change frequency, connection density, and batch size.

Weaknesses

I think random pruning (paired both with random or informed gradient-based growth) would be an interesting baseline in addition to dense and statically sparse architectures. As noted by the authors, expanding to tasks beyond image classification (and one tabular dataset) would broaden the scope of this work. You claim that growing is more thoroughly studied than pruning, but I’m not sure this is true. In Line 40, Dettmers and Zettlemoyer (2019) is cited as emphasizing the growth criterion but rather, to my understanding, uses random growth with informed pruning. There seems to be a roughly equal number of cited DST works that use simple pruning with more informed growing to those using simple growing with more informed pruning. This paper still has significance by focusing on the pruning method (although more growing methods could be included, but this would quickly increase the experimentation cost), but I think this language should be clarified. The analysis on update period in Section 5.3 is interesting, but I think should be qualified by noting how the pruning factor schedule and batch size may be confounding factors.

Questions

In addition to the batch size studies, I would be interested to see (in this or future work) how much the batch used for gradient evaluation affects the connections selected for pruning and/or growth in gradient-based pruning/growth methods. This could be studied by performing the training as normal until immediately before the 1st pruning step, then repeatedly resampling a batch and performing a backwards pass (without updating weights) in order to record which connections would be pruned and which would be grown. Is Figure 2 aggregated over multiple trials per protocol? If so, please add error bars. If not, please run multiple trials. Is there any way to add error intervals to Figure 6? I like the graphical presentation but would also appreciate error intervals to determine how noisy the metrics are and how significant the differences between strategies are. Typos and tips (not considered negatively against paper): * Line 13: “cautions”? * Footnotes referring to appendices are not necessary: placing the references directly in the main text may save space. * The paper layout and figure placement was a bit hard to follow. Figures that span across the whole width seem to be placed rather randomly, while they are conventionally placed at the top of a page. This helps the reader read the main text more continuously and find the appropriate figure when referenced. * Many captions ended with an analysis of the figure, like the last two sentences of the caption of Figure 1: I would prefer to see this in the main text rather than the caption. * The Jaccard index equation could be moved to an appendix to save space.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations and broader impacts are explicitly addressed.

Summary

This paper does a large scale study of dynamic sparse training, primarily on vision datasets. For a variety of models and hyperparameter settings, pruning using magnitude is found to be as or more effective when compared to pruning using more complex criteria proposed in the literature. The performance findings are reinforced by a study of the weights different pruning criteria select, and the impact of topology update frequency is studied.

Strengths

Originality: 1. This paper provides the most comprehensive study of DST pruning criteria that I am aware of, providing a valuable insight about the relevance of the magnitude criterion. Quality: 2. The experiments are designed well and support the contention that magnitude is a strong criterion in a variety of DST scenarios. Clarity: 3. The submission is well written. Significance: 4. The performance comparisons of criteria that have been shown to be effective in the literature (e.g. MEST and magnitude) are valuable for the community.

Weaknesses

This paper does not have any major weaknesses. I believe it's significance to the community (and correspondingly my score) would be best enhanced by ensuring the experiments cover a broad range of relevant architectures. I address this and some more minor issues, along with my suggestions, below.

Questions

1. In your related work, it is probably worth mentioning that Frankle et al. (2021) did an analysis of similar pruning criteria for static sparse training, also finding that magnitude was as good as more complex criteria. 2. Line 266: An alternative explanation might be that removing a smaller amount of weights leads to less disruption of the batch norm statistics with non-magnitude criteria. After pruning, do you update the batch normalization statistics (e.g., by running the model on the training data without updating its weights, see page 16 of [Zimmer et al., 2023](https://arxiv.org/pdf/2306.16788.pdf))? I ask because removing small magnitude weights may have little effect on these statistics, but removing larger weights (e.g., because their gradients were small) could have more substantial effects on the relevance of the old batch norm statistics to the new model. In which case, it would be difficult to conclude that the criterion is responsible for a certain performance level as that performance level could be attributable (at least in part) to incorrect batch statistics. It might be worth running an experiment to check this (e.g., on EfficientNet). 3. The networks considered cover a broad range but leave out architectures that have become more relevant in recent years. Also, the paper claims to cover large scale convolutional networks, but the largest network is a ResNet-50. Consider running on larger models and different architecture styles, e.g. ViTs. 4. Figures are too far from where they are discussed in the text. Consider adjusting their placement. 5. Lines 163-169 might benefit from being rephrased. As written, they do not provide a clear intuition for the expectation that a gradient based method would do better, which is what I think you are offering in these lines. In contrast, lines 184-185 (in conjunction with lines 177-178) give clear intuition for why I might expect a gradient based method to perform better. 6. Lines 188-189: wouldn't the mask solutions have to be "genuinely different" if the performances associated with the different criteria differed? I would say the prior experiments "do not necessarily indicate" instead of "do not indicate". 7. Line 270 and Figure 5 clash. You say that pruning adds regularization, but Figure 5 shows the dense model has the largest training loss - validation loss. Maybe you meant to define the loss gap as val - training instead of training - val.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The identified limitations are appropriate.

Summary

This empirical paper looks into the setting of adapting neural network architecture during training (dynamic sparse training), repeatedly pruning and growing the network. Authors try to draw conclusions about the performance and topology of various pruning criteria. They show that in high density regimes, most of criteria give similar results, with methods that incorporate gradients performing best, but in low density regime, simple magnitude based pruning performs the best

Strengths

- well written, easy to follow - author tackle the question that has not been investigated before - the emprical study is rigorous/well executed (modulo my questions/remarks) -assuming authors opensource their benchmark, it can serve as a test bed for new pruning and /or growth criteria

Weaknesses

- Limited novelty (empirical study of existing methods). Not very deep insights - mostly vision data and only 1 tabular dataset. No text data

Questions

This is comprehensive and well executed and I am willing to raise my score if the authors have good intuition for the questions below. 1) My conceptual question is related to line 218 - you are using the same hyper parameters for all pruning criteria. I wonder whether some pruning methods that are more crude for example can benefit from smaller learning rate, etc - basically I am worried that the reuse of all the same hyper parameters is what gives you the results that are all similar to each other. Because in reality, I assume when you train with DST and you set the pruning criteria, you tune the hyper parameters for that chosen criteria. Alternatively - any indication (for example, from earlier experiments) that the best hyper parameters don’t depend on the pruning criteria? 2) RE: conclusion that gradient information is useful (for pruning criteria) in high density and not so useful in low density regimes (e.g. magnitude based pruning wins in this case) - can’t it be also a function of the batch size? I would think that for very sparse networks, increasing the batch size will improve the quality of the gradient estimates => this might result in pruning that takes gradient into account to perform better 3) Are the time complexities for all the pruning criteria the same too (I don’t think so, e.g. you need to calculate gradients for some while e.g. magnitude based pruning uses just the weights). This would be nice to include somewhere - if they all perform similar, the cheapest to calculate is preferable Minor: Line 21 cautions-> careful or cautious Also I kinda find the title somewhat misleading - it suggestted to me that you will be coming up with a new criteria/method of "finding fantastic weights". Something that makes it clear that it is empirical study of pruning criteria is probably more appropriate (but I do appreciate snappy titles in general)

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

n/a

Summary

This paper provides a comparison of several dynamic sparse training methods and conclude that they are perform similar unless in the ultra-sparse regime, in which case the magnitude-based pruning performs best.

Strengths

The comparison is through and useful for new researchers in this field.

Weaknesses

This paper only provides empirical experimental results for known dynamic sparse training. No novelty is the main issue for this paper. I think it is more proper for it to be publish in a report than in a conference proceeding.

Questions

N/A

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Response to rebuttal

Thank you for your response. Regarding my question on how much the batch used for gradient evaluation affects the connections selected for pruning and/or growth in gradient-based pruning/growth methods, I meant the batch and not the batch size. Specifically, how sensitive is each method to the samples used for measurement? Your referenced experiment only partially studies this question, as we can assume a larger evaluation batch may give a more stable measurement, but only comparing two batch sizes isn't quite thorough enough to make any conclusions on this topic. I maintain my initial rating.

Acknowledgement of rebuttals and reviews

I have read the reviews and rebuttals, and I have updated my score. This submission's analysis of dynamic pruning criteria in various contexts is valuable, and my improved score reflects the addition of the ROBERTa analysis and the addressing of my more minor comments. Comments on your responses to my questions: 1. Yes, that paper. Sounds good. 2. Your response suggests there are enough iterations between the last mask update and the end of training for this to not be a concern, thanks! 3. Sounds good. 4. Sounds good. 5. Sounds good. 6. I agree that a large difference in performance does not imply a "large" difference in the mask, but I think a large (or a small) difference in performance does imply a "genuine" difference in the mask (at least at some point during training, assuming every training facet but the masking is held constant). If you agree, then you could address this concern by saying "significantly different" instead of "genuinely different". 7. Sounds good.

Rebuttal Response

I have read the author's rebuttal. Some of my concerns are addressed. I update my score to 6.

Thank you

Thank you for your response. RE: increasing the batch size experiments - I assume once you increased the batch size, you also had to retune the learning rate - or did you still kept it fixed? Fixed might be ok if you use some sort of adaptive optimizer (Adam?Adagrad), but if this is SGD, you will have to re-tune the lr and possibly the number of epochs/steps to get valid conclusions. In either case, I appreciate the responses and raise my score