I thank the authors for their detailed response and for clearing out some of my confusion. Here are come comments:
- **Concerning isoleap:** The isoleap is defined in terms of the Hermite coefficients in the orthonormal decomposition of the target function, which always exist for squared integrable functions (which is always the case when using squared loss), so it is not limited to polynomials. (The isoleap is always defined and finite for any squared integrable function.)
- **Concerning Assumption (3.2):** Let's consider $S$ to be one direction $z_1$. Then $ h_* (z) = \sum_{k \geq 0} He_k (z_1) h_k (z_{-1}).$ Therefore the condition (3.2) shows that $h_1(z_{-1}) = 0$ for all $z_{-1}$. Hence, in this coordinate basis, $z_1$ only appears in $He_k$ with $k\geq2$, so its leap $\geq 2$ in that basis, and therefore the isoleap is $\geq 2$ (this generalizes straighforwardly to $S$ of any dimension). Hence Assumption (3.2) is equivalent to isoleap $\geq 2$ (if I am not missing anything!). Indeed, Assumption 3.1 takes into account the expressivity of the activation function. However, note that in your example, one cannot fit $h_*$ anyway (degree $n$ activation and degree $n+1$ activation). More generally, for any $\sigma$ degree $k$ polynomial, I would expect (didn't check) $\sigma (u + v^T z_{S^\perp})$ to span all polynomials of degree-$k$ and to be equivalent to Assumption (3.2) whenever $h_*$ is degree $\leq k$, which is anyway the only part of the target function that can be fitted by the neural network. Hence, for $\sigma$ degree-$k$ polynomial, we could imagine the following: decompose $h_* = h_{\leq k} + h_{>k}$, assume Assumption (3.2) on $h_{\leq k}$ and disregard $h_{>k}$, which does not contribute to the dynamics by orthogonality. (Note that $h_{\leq k}$ can only have higher leap than $h_*$.) All in all, I am still not convinced that Assumption (3.1) is significantly different from isoleap to claim ``we propose a basis-free generalization of the merged-staircase property in [1]''. I apologize for being nit-picking (and I do not consider it to be the most important part of your paper), but several other groups, including [2,7,9,10] cited in your paper, have worked on this basis-free leap complexity. Please let me know if I misunderstood anything.
- **Concerning Algorithm 1 and Assumption 4.1:** Thank you for the clarification, I mistakenly thought the initialization was a point mass. I acknowledge that the Gaussian case is much harder than the hypercube case, due to the need of an algebraic independence. Thank you for making the effort of verifying Assumption 4.1 in both these cases, I believe it already makes a strong case for Assumption 4.1!