Thank you for increasing the score.
We followed the same setting as in [A] and [B], which did not regularize the skip connection. Previous works also considered other cases: without skip connection ([A] Theorem 2), and regularization on the bias [C].
Adding regularization over the skip connection is an interesting case to consider, which was not directly covered in previous works: [D] and [E] proved related results but in a different setting (constrained path/optimization path in classification vs. regularization path in regression in our case). From these works, it intuitively appears that a “large” target function is “cheaper” to implement without a skip connection (recovering the case without the skip connection), since we can distribute the function scale over two layers instead of one. However, for general target functions the situation is more complicated. We can prove (see below) that the cost of realizing the linear function $L(x) = Vx$ with the regularized skip unit is greater than the cost of realizing it with regularized ReLU's if and only if $2||V||\_* \leq ||V||\_F^2$, where $||V||\_*$ is the nuclear norm. This will generally hold for large norm matrices (following the intuition above), but it fails for matrices with sufficiently small norms. We are not sure yet what holds for general (nonlinear) target functions. This is an interesting open question to explore.
__Proof:__
Let $C\_{relu}$ be the minimum cost needed to realize $L(x) = Vx$ with ReLU units over some compact domain.
We can write every such realization as
$L(x) = A[W^Tx+b]\_+ -Ab$,
where $[W^Tx+b]\_+ = W^Tx + b$ on the domain, and $V = AW^T$.
This gives
$C\_{relu} = min\_{V = AW^T} \sum_k (||A||\_F^2 + ||W||\_F^2)$.
But by the variational characterization of the nuclear norm, we see that $C\_{relu} = 2||V||\_*$.
On the other hand, to realize $L(x)$ with a regularized skip connection costs $C\_{skip} = ||V||\_F^2$.
So as long as $2||V||\_* \leq ||V||\_F^2$, we have $C\_{relu} \leq C\_{skip}$.
[A] Ongie, G., Willett, R., Soudry, D. and Srebro, N., 2019. A function space view of bounded norm infinite width relu nets: The multivariate case.
[B] Hanin, B., 2021. Ridgeless Interpolation with Shallow ReLU Networks in $1 D $ is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions.
[C] Boursier, E. and Flammarion, N., 2023. Penalising the biases in norm regularisation enforces sparsity.
[D] Nacson, M.S., Gunasekar, S., Lee, J., Srebro, N. and Soudry, D., 2019, May. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models.
[E] Kunin, D., Yamamura, A., Ma, C. and Ganguli, S., 2022. The asymmetric maximum margin bias of quasi-homogeneous neural networks.