Need a Small Specialized Language Model? Plan Early!

Dear reviewer, We thank you for your review. We are happy to hear that you found the problem meaningful, and that you enjoyed the importance sampling method ! Regarding the weakness you mention: > It is unclear how important the design choice of using PN is for SLM-pn. Naturally, one could consider using an Adapter/LoRA/any possible PEFT module to adapt a large model to a specific domain. It would be better to show why taking PN is better, in terms of either performance or efficiency. A comparison with Lora is indeed interesting; we have conducted such experiment in Section F. We have made this comparison clearer in the text. Exploring how one could adapt the PN architecture to have Lora-like behavior would be interesting. We are not sure of how to design a hypernetwork in the spirit of Lora that has the same properties as PN. For instance, having the parameter of $\mathrm{SLM-pn}_i$ instantiated as $W + W^i$ where $W^i$ is low rank and $W$ is shared would be an interesting architecture to study. We are unsure whether this would improve upon PN, because we are in a context where we have plenty of data to train each expert. Could you clarify how you propose to use LoRa in order to build asymetric models that are pre-trained with many parameters, but are instantiated into small models? > In Section 6.1 of Scaling Expert Language Models with Unsupervised Domain Discovery, the author tried a method that is IMO similar to SLM-pn. Basically, there is a shared base model for all domains and each domain has a specific module. It is also relevant to the weakness mentioned above. What do you think? Thanks for this very relevant reference. The idea is indeed quite similar to PN, or even more to the hard mixture of experts, which trains one model per cluster. In our view, the main difference is that the inference procedure is quite different, since C-BTM uses an ensembling strategy, while our method only uses a single model for a task. We have added this reference to the text. We hope that we have adressed your concerns, and we thank you once again for your review.

Thanks for your question ! PN constitute an architectural change that requires some pre training: one first pre-trains a PN network with many parameters, and then at inference one can project the PN to obtain a small network with few parameters. This small network can then be fine-tuned. We are not sure how to obtain the same property using a peft method. Indeed, in the context of this paper, where we have a model size constraint, peft methods would adapt a small pretrained network during fine tuning. Alternatively, one could use peft to fine tune a large network, yielding fewer parameters to tune, but the resulting network would still be large, not respect the size constraint, and have a costly inference. In summary, combining PN and peft methods would be a very interesting topic, but we do not see how to do it.

Dear reviewer, We thank you for your comments, and pride ourselves that you have appreciated the comprehensiveness of our experiments. We now respond to the weaknesses you mentioned. > The paper is not well-written We are happy to clarify any points that are unclear besides those you mentioned. > Section 2.4 has many equations, but the symbols used in these equations are not defined We have done our best to clarify this. We indeed failed to mention that $\ell$ was the individual loss function. We could not find other undefined symbols. Note that $P(x | c)$ denotes a conditional probability, as is customary in most papers. > This makes it difficult to understand how to get the weight for importance sampling. We have now written more clearly the equation to estimate the weights, we hope that this dispells any confusion about the method. > Figure 4 is a bit misleading. The perplexity for SLM-is is a flat line, but it doesn't reflect the fact that SLM-is requires significantly more compute during specialization compared to other methods. This is an important remark, that we believe was already in the text: "These figures report the perplexity of SLM-is as a constant line. This method has no generic pretraining as its training starts only once the domain data are available; bearing all the training cost in the specialization phase. SML-is is the best method with a small inference model in terms of post-specialization perplexity. Interestingly, it even outperforms the much larger model when specialization data are scarce (ie the 1M tokens case), for a fraction of the overall training cost (<130 GPUh).". We have now also mentionned this in the figure caption to remove any confusion. > The training budget is defined as GPUh throughout the paper. Is it measured in a single GPU, multi-GPU, or multi-node setting? Because these different settings will have impacts on the training cost and actual throughput. Perhaps the standard FLOPs notation is clearer and less ambiguous. We believe that GPUh is a fairly common and widely recognized metric in the ML community. It is also easy to compute, looking only at the training run. On the other hand, quoting [1]: "It is important to highlight that FLOPS is never directly measured, but always estimated, with widely different practices across the paper"; FLOPs are an interesting metric, but we believe GPUh are of more interest to practitioners. Note that the number of GPUs used for each experiment is already reported in the paper, Table 8. [1]: Scao, Teven Le, et al. "What language model to train if you have one million gpu hours?." arXiv preprint arXiv:2210.15424 (2022). Thanks again for your comments, which help us improve the paper ! Please let us know if any concern remains, we are happy to address them.

@Reviewer fTrA, The review seems a bit too brief. Could you add more details, please? AC

Dear reviewer, We thank you for your review, and appreciate that you found our work relevant, and that you highlight the thoroughness of our experiments. We now address the weaknesses you mentioned. > There is lack of theoretical or heuristic justification behind why linear projections are a suitable operator for deriving SLMs from LLMs. Some exploratory work on, e.g., the activation patterns of the LLMs across different domains that justify the linear projections, would have been nice. This is indeed an interesting point. As mentioned in the text, PN can be seen as hypernetworks. We developed the PN architecture from a practical perspective: it is one of the simplest instantiations of a hyper-network, that is linear and that allows to scale gracefully the number of parameters in total and the number of experts. We were satisfied with its performance as-is; hence, we did not explore more convoluted, non-linear architectures. We agree that this would be an interesting avenue of research for future works. > A more direct comparison to other baselines such as LoRA would have made evaluations more informative. Appendix F does include a comparison, but a direct reference in the main text would make it even more helpful for further contextualization and comparison. In our view, our work is quite orthogonal to LoRA, as one can apply our methods with LoRA. Appendix F indeed alread provide the details of this ablation. Thanks to your suggestion, we now refer to this part in the main text. > It would have been nice to see actual applications of these methods to existing LLMs (e.g., Llama, Gemini) instead of an in-house LLM trained on C4. We are unsure how to apply our methods to already pre-trained LLMs, since our techniques are pre-training / continued pre-training techniques. Could you clarify ? We now turn to your questions: > Why do you use linear projections instead of more common hypernetwork formulations? What is gained or lost by doing this? See above > Do the linear projections learn anything meaningful, e.g., the singular values of the FFN layers? We want to clarify that the PN instantiates the weights of the FFN layers as a linear combination of a bank of weights of the same shape, the linear combinations learn to combine the weights for each cluster. We are therefore unsure how to explore what the projection coefficients learn. > How does this work relate to LoRA? It seems that there is an explicit connection to be made between LoRA and PNs as they both encourage low-rank behavior in the learned weights. Appendix F seems to address this to an extent, but I would appreciate a more direct account on this. We want to clarify that to the best of our understanding, PNs do not encourage low-rank behaviour. Indeed, the instantiated weight matrices of size $d\times d'$ are obtained as a linear combination of h full rank matrices of size $d\times d'$ stored in $T$. This yields a full-rank structure. Therefore, we believe that there is little connection between PN and LoRA. We hope that we have clarified the questions you had regarding our work. We thank you for these remarks which help improving the paper !