Thank you for identifying the thoroughness in theory and experiments, and the value and soundness of the proposal. We have incorporated your suggestions into the updated version of the paper.
**Experimental verification for theorems**
Thanks for the great suggestion. We have added empirical verification for theorems in the appendix F. As you can see, the theoretical predictions and empirical computations match in all presented cases.
**Add parameters to the figures**
We have added parameter counts to both existing and newly added figures.
**Would it be possible to explore this method on a contemporary large-scale model?**
1. We added more datasets from diverse domains: We understand the concern from reviewers about restricted domains of evaluation. To provide more diverse evidence, we have added two more domains of "Graphs with Attention-based Networks" and "Deep learning Recommendation Models (DLRM) " in Appendix "MORE DOMAINS." The DLRM models (~540M parameters) and the associated dataset are relatively large (40M samples). The observations are consistent in that RPS outperforms other methods. In fact, pruning is especially bad in DLRM models due to the pruning of embedding tables which is expected since pruning of embedding tables at high compression (beyond d\times ) will give degenerate embeddings.
2. Why not train larger models/datasets such as vit on imagenet?
Another reviewer suggested training vit on imagenet. Creating a single data point by training a vit model on imagenet needs 300 GPU hours if using A100 GPUs. We do not have access to the kind of computing it would need to create Pareto curves for this experiment in a reasonable time.
**Limitation of random parameter sharing methods versus pruning methods, When would RPS really be competitive?**
We mention the current limitations of RPS in the section "Discussions and conclusion," Reducing the number of flops in RPS is currently an open problem. We will also mention it in the introduction to avoid any confusion.
There are many situations in which RPS can be potentially beneficial right away.
1. Embedding heavy models such as DLRMs. We added new results comparing pruning and RPS, and pruning is terrible when compressing embedding tables. As embedding tables do not have any flops, RPS is equally efficient in computation.
2. MoE models. A mixture of expert models is generally memory-heavy, and their computation is sparsified by design.
3. Other scenarios exist where higher compression can help us eliminate model parallel execution, which can have latency advantages.
Having said that, this paper focuses on memory-accuracy tradeoff compression techniques. By establishing this, we want to argue for further investigation of RPS to solve existing open problems.
**For the same parameter budget there are now many network architectures that perform much better. In other words, where would MobileNet or EfficientNet be placed on Figure 3?**
Given a memory budget, finding the best architecture is a challenging problem and comes under NAS. At this stage, we cannot compare RPS on one architecture with other architectures (alternatively, we cannot compare the RPS and NAS) . NAS finds the best architecture given standard modules. STABLE-RPS and pruning methods find the reduced parameters for a given architecture. From works on pruning, it is clear that once a model is decided, it can be sparsified further and still keep matching accuracies in some cases. However, we still need to choose a good model to begin with, which is where NAS is useful. In the future, NAS can be combined with methods such as STABLE-RPS to obtain even better architectures. But, this direction is out of the scope of the current paper.
**Unfortunately, as the compression ratio increases, the network quickly enters regions where the accuracy is not worthwhile.**
In practical scenarios, what tradeoff benefits a particular use case depends on the existing resource constraints and quality requirements. Thus, it is impossible to comment on whether RPS will solve the issue to satisfaction. But, in this paper, we want to rigorously compare RPS and pruning paradigms to see which method is better, given the resource constraints, and we find that RPS has better expressivity than pruned models at given memory budget.
**Writing Issues**
1. We have rewritten the background on parameter sharing to improve the readability of the section with a focus on the ROAST method. In the appendix, we can add details on other methods, such as ROBE and Hashednet, if needed.
2. The colors used in the full model denote the different ROAST chunks in the model, and the color in the linear array shows one instance of the mapping -- mapping of one chunk of ROAST or one fold of STABLE-RPS into the array showing which parameter goes where.