Summary
The submission proposes a second-order method for unstructured pruning of neural net parameters, to leverage the efficiency gains of sparsity. It uses an optimization based on the empirical Fischer matrix to find saliency scores, that are used for the order in which weights are pruned. They present an algorithm based on solving the problem on individual sub-blocks, and then using that within a global pruning. They also present a schedule, across different hyperparameters, for training with pruning. This is demonstrated on ConvNeXt and DeiT models, along with other CNN and ViT models in the supplement.
Strengths
**i)** Achieves better accuracy & sparsity.
The proposed pruning gets pareto-dominant results relative to some similar prior work. The evaluation is done across multiple choices of the sparsity vs accuracy tradeoff.
Not all experiments consider some of the most similar previous methods, though, such WoodFisher in the "Gradual Pruning" results, though the original work did include gradual pruning during training. (See section 5.2 of [38])
**ii)** Considers good selection of recent models
Experiments demonstrate the pruning on both up-to-date ConvNet and vision transformer models. In the supplement and code, experiments on ResNet-50 and EfficientNet are also provided.
**iii)** Includes code
The submission includes the code used to run the experiments, based on SparseML, along with YAML files specifying the hyperparameters for a number of the experiments.
Weaknesses
**iv)** Largely superseded by Optimal BERT Surgeon in a lot of settings
OBS is a more scalable algorithm based on the same underlying principles. The core contribution seems to be a different method of approximating the solution to the original 2nd-order problem, plus better hyperparameter tuning. It does seem like a reasonable approach for relatively smaller models, to seek a tighter approximation to the original optimization.
**v)** Source of improvement in end metrics remains unclear
At its core, the starting point of the construction of the method in the submission is similar to [38]: namely, second-order pruning with the empirical Fisher matrix in place of the Hessian. The differences from [38] then include a) a different method of approximating the pruning on the full matrix, described toward the end of Section 3.1, and b) improved hyperparameters during gradual pruning.
The paper states that ablation studies are given in appendices B, I, and J. These appendices specify different sets of hyperparameters, and include results of sparsity vs accuracy for multiple different choices of some of these hyperparameters. I didn't find any "ablation" studies that evaluate parts of the proposed method with other parts "ablated" or removed: for example, the results of using Algorithm 1 *without* the changes to data augmentation, the sparsity and learning rate schedule, etc.
Experiments also don't compare against [38] for the gradual pruning case, so the effect of the hyperparameter tuning and schedules isn't measured by a comparison to baseline. It's possible that the improved choices in which weights to prune, as seen in the one-shot case, could also affecting the gradual pruning case, but it's also possible that the training can close a lot of this gap.
The experiments also use very recent ViT-based models, showing some more up-to-date results than prior work. It is plausible that better pruning is necessary to get good results for these models, as argued, but this is incompletely supported.
Questions
**Code**: Some miscellaneous minor comment from trying the included code
README.md marks the step to install wandb as "optional." Not entirely optional, though, given that it does import it:
```
File "CAP-code/research/one_shot_pruning.py", line 4, in <module>
import wandb
ModuleNotFoundError: No module named 'wandb'
```
Commenting out the import, then see:
```
AttributeError: module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`. To avoid this error in existing code, use `object` by itself. Doing this will not modify any behavior and is safe.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
```
within the import of onnx inside sparseml:
```
from sparseml.pytorch.utils import LOGGING_LEVELS, BaseLogger, LoggerManager
File "${HOME}/anaconda3/envs/CAP/lib/python3.9/site-packages/sparseml_nightly-0.13.0.20230704-py3.9.egg/sparseml/pytorch/utils/__init__.py", line 23, in <module>
from .exporter import *
File "${HOME}/anaconda3/envs/CAP/lib/python3.9/site-packages/sparseml_nightly-0.13.0.20230704-py3.9.egg/sparseml/pytorch/utils/exporter.py", line 26, in <module>
import onnx
File "${HOME}/anaconda3/envs/CAP/lib/python3.9/site-packages/onnx-1.10.1-py3.9-linux-x86_64.egg/onnx/__init__.py", line 20, in <module>
import onnx.helper # noqa
File "${HOME}/anaconda3/envs/CAP/lib/python3.9/site-packages/onnx-1.10.1-py3.9-linux-x86_64.egg/onnx/helper.py", line 17, in <module>
from onnx import mapping
File "${HOME}/anaconda3/envs/CAP/lib/python3.9/site-packages/onnx-1.10.1-py3.9-linux-x86_64.egg/onnx/mapping.py", line 27, in <module>
int(TensorProto.STRING): np.dtype(np.object)
File "${HOME}/anaconda3/envs/CAP/lib/python3.9/site-packages/numpy/__init__.py", line 305, in __getattr__
raise AttributeError(__former_attrs__[attr])
```
Following instructions in the README had given me `numpy==1.24.3` and `onnx==1.10.1`. It's likely that one needs to hold back numpy, likely an underlying issue with the requirements specified for SparseML. Presumably this codebase is using an old version of sparseml that specifies a requires.txt such that it's versions specified with '>=' no longer really work with the latest versions now available. Authors should check & perhaps update the install commands in their README accordingly, or update their sparseml version. I was able to resolve this with instead:
```
conda install numpy==1.20.3
```
Generally recommended practice for distributing this kind of research code is to explictly specify versions of the requisite packages with `==`, so we know exactly how the authors originally did all this.
Sample command for `one_shot_pruning.py` in README also includes the command-line arguments `--data-dir` and `--sparseml-recipe`, when the arguments actually added to argparse are `--data_dir` and `--sparseml_recipe`, and arguments `-b` and `--experiment` that don't seem to exist.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Cite computation cost (at training/pruning, not inference/deployment time) as the limitation. The Fischer matrix will still grow quadratically in the number of parameters. This is unfortunate given the likelihood that the largest models that would benefit the most from pruning, in absolute terms. (And it is an oft-observed principle in ML that simple methods that scale better are usually more impactful.)