Weaknesses
The authors attempt to cover a broad scope in their analysis, but the exploration is often shallow, with certain aspects appearing incremental or inaccurate.
1. Assumptions weaken the generalizability and accuracy of their conclusions:
a. In Equation (4), the authors address the randomness in DARE [1] by asserting that “it is straightforward to deduce that,” leading to Equation (5). This is wrong as this need to rely on $\Delta W_{ij}$ and $\Delta \mathcal{L}_{ij}$ being uniformly distributed, which is generally not the case, making the equality invalid.
b. In BitDelta [2] , the authors state that it is difficult to conclude that Equation (9) equals zero due to the interaction between $\text{sign}(\Delta W_{ij}) \Delta \mathcal{L_{ij}}$. This approach is inconsistent, as they assume uniformity of $\Delta W_{ij}$ and $\Delta \mathcal{L}_{ij}$ when proving Equation (5) for DARE, but not for BitDelta in Equation (9). If the uniformity also assumed for BitDelta, the $\mathcal{L}=0$ should hold. Consequently, the conclusion that BitDelta performs worse than DARE is questionable.
2. Question on EXPO[3]:
a. Incremental Analysis and Use of EXPO Framework: The paper uses the similar framework as EXPO, particularly referencing Equation (2) in EXPO, where EXPO use first-order Taylor Expansion with an alignment objective (which functions similarly to the loss used in the current paper). This similarity can be seen as an incremental extension rather than a substantial innovation.
b. Claim on Gradient Correlation with Delta Parameter: In section 2.2 of EXPO, EXPO already established that the success of their approach depends on a positive correlation between the gradient and the delta parameter, highlighting a direct relationship with the approximation term. Given this established finding, it appears that the new paper is building on known results rather than offering a novel insight in this area.
c. Claims on EXPO Limitations: According to EXPO, extrapolation can improve performance when a positive correlation between the gradient and the delta parameter, a point that the new paper seems to question. However, if EXPO already addressed this with clear justification, the claim in the current paper may not hold strong novelty or accuracy. If they misunderstand EXPO’s stance on extrapolation, it could weaken their argument about EXPO’s limitations.
3. I feel the logic in DARE section is unclear. First, the extension of DARE lacks a clear connection to the theorem in Section 4.1, making the motivation for introducing k unclear. I though authors will give motivation in Section 4.3, but cannot found it. Also, in section 4.3, the authors claim that DARE overlooks delta loss. However, the original DARE analysis of random pruning considers both delta parameters and the input. Specifically, the authors’ analysis in Equation (4) focuses on delta parameters and delta loss, and due to the linear approximation, delta loss can be proportional to the input x. This resemblance to DARE makes it inappropriate to claim that DARE disregards delta loss (represented by x in DARE’s case).
4. I also find the logic in analyzing BitDelta [2] unclear. Similar to DARE, the motivation for introducing noise to mean magnitude lacks a clear connection to the theorem in Section 5.1. Additionally, the original BitDelta [2] already demonstrates that calibrating the scaling factors can improve performance as their contribution. This resemblance to BitDelta makes it inappropriate to claim that BitDelta overlooks this issue, which limits the novelty of the current approach.
[1] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit
[3] Weak-to-strong extrapolation expedites alignment.