Thank you for your time and thoughtful comments. We appreciate your interest in the good performance of DARq and the additional insights our paper brings. Please see our responses to your questions below. We hope these address your concerns and look forward to your updated feedback! We are happy to address any additional questions that you may have.
--------W1: Exploratory and Recommendation--------
**Answer**: Thank you for your comment, and we would like to take this opportunity to clarify that we have indeed provided recommendations.
**Our recommendation** : We have made recommendations and presented a flowchart in Figure 4 to assist practitioners. Specifically, this flowchart delivers a clear message that one can customize the pruning strategies to meet different requirements and needs (e.g., scenarios with/without fine-tuning or validation data). Instead of a single solution, we believe our recommendations *better* align with the varied needs of practical applications. The utility and flexibility of this framework have been recognized in other reviewers’ comments (e.g., reviewers S9Bb and 3h3b), affirming its value in addressing the diverse needs of practical applications.
**Comprehensive analysis**: Also, we respectfully believe that ‘exploratory’ is not a bad thing, particularly when it provides a unifying/broader viewpoint on a topic that, as the reviewer says, is of interest and not yet well-addressed in the literature. We aimed to approach the problem with a wide perspective that kicks off from the performance of vanilla DARE and identifies its limitations. We address these limitations by considering the circumstances of application (e.g., whether fine-tuning is possible) while investigating other candidate approaches to Delta Parameter pruning (e.g., importance-based methods). The study provides concrete insights, which have also been acknowledged by other reviewers including S9Bb, xpsx, and 3h3b.
-------- W2: Tightness of our bound ---------
**Answer**: Thank you for your comment, we respectfully disagree with this assessment. The upper bound in Theorem 3.1 is indeed tight with respect to p, which can be seen as follows:
From Eqn (2), the output change can be expressed as:
$$h = \sum_{j\in[n]} c_j (1 - 1/(1-p) \delta_j)$$
where $\delta_j$ are iid $\text{Bernoulli}(1-p)$ RVs, we have dropped the subscript $i$ from Eqn (2) for simplicity and we have denoted $c_j=\Delta W_j x_j$. Our goal is to quantify the growth of the absolute output change $|h|$. Note that the random variables $Y_j= c_j (1 - 1/(1-p) \delta_j)$ have $E[Y_j]=0$ and $Var[Y_j]=c_i^2p/(1-p)$. Thus, by the central limit theorem, in the limit of large $n$, the (normalized) output change $(1/\sqrt{n})|h|$ is distributed as $\sqrt{p/(1-p)}\sqrt{\sum_i c_i^2}|G|$ where $G$ is a standard normal. Note the scaling $\sqrt{p/(1-p) \sum_i c_i^2}$ precisely matches the bound of Thm 3.1 for $p>1/2$, demonstrating that it captures the true behavior of the output change with respect to $p$. The exponential high-probability bound of Thm. 3.1 follows by a Hoeffding-type bound, but as explained in App. E.1, to achieve this optimal p-scaling, we use a bound by Berend and Kontorovich that improves upon Hoeffding’s bound on the moment generating function for generalized Bernoulli RVs.
We acknowledge the reviewer's point about precise language and have revised line 229 in revised submission to include "at most" for technical precision.
Finally, we emphasize that beyond the theoretical foundation of Thm 3.1, which successfully motivates our improved DARq rescaling, our extensive experiments on encoder- and decoder-based language models across multiple datasets demonstrate that DARq consistently outperforms state-of-the-art methods.
----------W3: Typo--------
**Answer** Thank you for pointing out this typo, we have corrected it in our revised version.
Thank you again for your time!