Follow-up response
Thanks for the response and further questions, which we clarify below. We will also highlight the assumption of no representation error more clearly in the revised paper.
**(Are the experiments running generalized Lasso?)**
Please note that when we mentioned running CSGM, we meant using it to (approximately) solve Eq. (2.1), which is precisely the generalized Lasso. Eq. (2.1) is intractable to solve exactly, and the CSGM approach is to approximate it using gradient descent with random restarts. Previously CSGM was devised specifically for linear models, but generalized Lasso contains the exact same objective function and constraint (up to scaling by $T$; see Remark 5) as the original one for linear models, so CSGM remains applicable despite the non-linearity.
This idea of approximating the exact optimization by gradient methods is extremely standard in the literature on CS with generative priors, e.g., see Bora et al. (2017), Dhar et al. (2018), and Liu et al. (2020) in our paper’s reference list. Thus, up to very standard practical approximations, **we are running generalized Lasso**. We will re-word to make this clearer.
**(Do the experiments corroborate the theory?)**
We believe that the previous response partially addresses this, but provide further discussion as follows.
We chose the word “corroborate” to avoid overly strong language like “verify” or “confirm”, but we would be happy to tone this down further and replace it by a more precise statement, e.g., “to demonstrate that a practical variant of the generalized Lasso can be effective in recovering multiple signals with a single measurement matrix.”.
Confirming the scaling laws experimentally is a challenging task, and to our knowledge it has never been attempted in the literature on CS with generative priors. Our experiments are aligned with those performed in similar kinds of works that consider generative priors (e.g., those mentioned above), with the distinction that we take the *worst case* performance over *batches* of images to better align with our goal of uniform recovery. With the above-mentioned re-wording and the removal of any suggestion that we are verifying our theory, we believe that they are a useful addition to the paper.
Having said this, we emphasize that **by far** our main contributions are our theoretical results, so we hope that they will accordingly be the main factor in the final decision. (For comparison, other works on single index models (SIMs) typically have no experiments at all, e.g., Plan & Vershynin (2016), Genzel (2016), and the most relevant work Genzel & Stollenwerk (2023).)
**(How can unknown $f$ be possible?)**
From an experimental point of view, the fact that we run CSGM to approximate Eq. (2.1) and get good results supports the fact that this is possible.
From a mathematical point of view, existing works demonstrated that the SIM with unknown nonlinearity $f$ (as considered in Part C of our Section 2.3) can be “transformed” into a linear model with an “unconventional noise term”. Specifically, Plan & Vershynin (2016)'s Section 4 demonstrates that if $f$ satisfies the condition that $\mu := \mathbb{E}_{g \sim \mathcal{N}(0,1)}[f(g)g] \ne 0$, then $\mathbf{y} = f(\mathbf{A}\mathbf{x}^*)$ (where $f$ is applied element-wise and $\mathbf{A}$ has standard Gaussian entries) can be written as $\mathbf{y}=\mathbf{A}\mu\mathbf{x}^*+\mathbf{w}$, with $\mathbf{w}$ satisfying $\mathbb{E}[\mathbf{A}^\top\mathbf{w}]=\mathbf{0}$ thus acting as an unconventional noise vector. Although the generalized Lasso approach is most naturally suited to conventional noise such as Gaussian, it turns out to still work under this unconventional noise.
We omitted the above discussion because it is already well-documented in previous works, but we would be happy to use the extra available page (if accepted) to include an overview similar to the above paragraph.