Authors' response
We are very grateful to the reviewer for their careful reading of our paper, thoughtful feedback, and constuctive suggestions. Below we respond to the concerns and/or questions raised.
___
**Comment:**
"The stddev in table 2 are a bit odd to me, either a higher numerical precision is needed, or it needs to be explained why the are many cases of zero std."
**Response:**
Indeed most of the 0.00 standard deviations were simply because the results in the table were rounded to 2 decimal places. We have amended Table 2 to include higher precision to make the results clearer. 0.00 standard deviation was only accurate to higher precision as well in the cases when all of the seeds completely fail to train at all. Thank you for this suggestion.
___
**Comment:**
"Results on ST are only shown in the appendix. They should be shown and discussed in the main paper, given that they are an essential part in the rest of the manuscript."
**Response:**
Thank you for this suggestion. Putting $ReLU_\tau$, $ST_\tau$, and $CST_{\tau, m}$ experimental results in the Appendix was originally done purely due to space constraints, but we acknowledge and agree that it is important for all key experimental results to appear in the main paper. In order to make space, we simplified and combined Figures 2 and 3, and so we have now included all the important experimental results in Table 2.
___
**Comment:**
"The higher sparsity regime in table 2 (s=0.85) needs to be explored/explained more, there are interesting things happening."
**Response:**
We agree that the high sparsity regime is very interesting! But we suggest that our analysis of the impact of $s$ and $m$ on the shape of the corresponding variance maps and the related failure modes accounts for the results quite nicely. In particular, we see that initially increasing $m$ is necessary to achieve good accuracy at high sparsity, as expected. However, as predicted by the theory, increasing $m$ too much results in $V'(q^*)$ and $V''(q^*)$ together being too large, causing $q^*$ to fail to be stable in practice. This relates to your other question about Figure 4. The phenomenon described here is exactly what is illustrated in Figure 4, which shows the Variance maps of $CReLU_{\tau,m}$ and $CST_{\tau, m}$ in the high sparsity, large $m$ regime. When the variance map curves of to trace or even cross the $q=V(q)$ line, the result is that $q^l$ does not converge to $q^*$, and instead remains larger than $q^*$ layer on layer. Figure 4 shows that the corresponding $\chi_1$ value in this case is larger than one, which causes exploding gradients and training failure. We have tweaked the wording at the bottom of page 8 in Section 4 to make it clearer which point from the main text Figure 4 is illustrating.
If you still think something is not clear, could you perhaps explain what analysis you think is missing?
___
**Comment:**
No source Code provided, implementation details on experiments only in the appendix.
**Response:**
The implementation details were left to the appendix due to space constraints. We ask that this please not be considered a weakness of our paper. We are working to make the source code available and will alert you if this is possible before the discussion period concludes.