Summary
This paper proposes a novel zeroth-order optimizer, named LeZO, which enhances the efficiency of the Memory-efficient Zeroth-Order (MeZO) optimizer by introducing sparsity in the parameter updates. The key innovation of LeZO is that it applies sparsification at the layer level, treating each layer as a unit of sparse updating. Through analysis, the authors observe that more than half of MeZO’s computational cost is spent on perturbation and updating stages. By selectively updating only a subset of layers in each iteration, LeZO significantly reduces this overhead, while still ensuring that all parameters are eventually updated. This strategy avoids additional memory costs, such as those incurred by masking. Experimental results demonstrate that LeZO achieves up to a 3.4× speedup in training wall-clock time compared to the original MeZO, without compromising optimization performance.
Strengths
1. The proposed method is straightforward and practical, making it easy to integrate into existing training frameworks with minimal engineering effort. By sparsifying updates at the layer level, it achieves efficiency without complex changes, making it highly usable.
2. Although writing clarity could be improved, the paper is organized logically, which helps convey the main ideas and findings in a generally understandable way.
3. Experimental results show clear benefits: the approach reduces computational overhead in perturbation and updating, achieving up to 3.4× faster wall-clock training times, highlighting its effectiveness in accelerating convergence.
Weaknesses
1. Inconsistent Terminology and Notation: The paper suffers from inconsistent use of terms and symbols. For instance, the term "sparse rate $\rho$" on Line 194 and "sparsity rate reaches $\rho = 1$" on Line 451 seem to refer to entirely different concepts, which may confuse readers.
2. Errors and Ambiguities in Mathematical Notation: The mathematical expressions and symbols lack rigor and contain errors. According to Line 195, $\theta' \in \mathbb{R}^{\rho d}$ while $\theta \in \mathbb{R}^{d}$, making the addition of $\theta$ and $z'$ in Equation (4) undefined, as these vectors lie in different spaces. Additionally, cases where $\rho d$ is not an integer are not addressed. Notation in Lemma 1 is also quite unclear, making it difficult to interpret.
3. Errors in Tables: Table 1 contains unexplained elements, such as the appearance of "SSZO," which is not introduced or defined, potentially causing confusion.
Questions
1. In Line 310, it is stated that as the sparsity ratio $\rho$ decreases, the upper bound on the convergence time to the expected value also decreases. If $\rho$ here is defined as on Line 194, then when $\rho = 0$, meaning no parameters are perturbed, why would the upper bound on the required convergence time be minimized in this case?
2. Your method appears quite similar to sparse MeZO, with convergence analysis and proofs that are almost identical. Sparse MeZO selects parameters for updates based on their magnitudes, while your approach randomly selects layers for updates. Why didn’t you include a comparison with sparse MeZO? Does your method offer any performance advantages, or is the primary benefit a reduction in memory overhead? Additionally, is there any theoretical insight for choosing layers as the basic unit for sparsification?
3. In Figure 3, there are notable performance drops between dropout numbers of 5 and 0 (MeZO), suggesting a significant performance gain by excluding updates for just 5 layers. There are also marked drops between dropout numbers 35 and 40. How do you explain these observations?