Reply to Reviewer otLk's comments
For the weakness:
Thanks the referee for additional comments. After reading the referee's additional comment, we conducted two more numerical experiments.
(a) The first additional experiment serves as a supplement to Section 3.3. In the original experiment, MGDL was configured with three grades, each consisting of two hidden layers of width 128. In the new experiment, MGDL was configured with six grades, each consisting of a single hidden layer with width 128. New experiment results demonstrate improvement in MGDL's performance compared to the results in the originally submitted manuscript.
(b) The second experiment is a totally new example on a color image. We compare performance of MGDL and SGDL when applying them to regress a two-dimensional color image. The input to the model is the pixel coordinates, and the output is the corresponding RGB values. The train dataset consists of a regularly spaced grid containing $1/4$ of the image pixels, while the test dataset contains the full image pixels.
The network structure for SGDL has 10 hidden layers, each with width 256, and the corresponding structure for MGDL has 5 grades, each grade has two hidden layers of width 256.
Both SGDL and MGDL use the ReLU function as the activation for hidden layers and the sigmoid function as activation function for the output layer.
The peak signal-to-noise ratio (PSNR) values are computed over the full image pixels for both images obtained from MGDL and SGDL. The PSNR value for the image generated by MGDL and SGDL is 24.07 db and 22.41 db, respectively. This result indicates that MGDL outperforms SGDL in terms of PSNR value. Notably, MGDL demonstrates the ability to learn high-frequency details more effectively than SGDL.
We hope these additional numerical experiments can alleviate the referee's doubts.
Thanks for pointing out the inconsistency of the commas used in Table 1. We have corrected them throughout the entire paper.
For question 2:
When the network is divided into multiple grades, the cost of forward propagation does not increase during training. This is because it is not necessary to reevaluate the forward propagation of the previous $n-1$ grades during the training of grade $n$. In fact, the computation cost remains relatively consistent across all grades. Below, we elaborate this point.
For
$$
\mathbf{x}\_{\ell}^{n} :=\mathcal{H}\_{D\_{n-1}}(\Theta\_{n-1}^*;\cdot)
\circ \mathcal{H}\_{D\_{n-2}}(\Theta\_{n-2}^*;\cdot)
\circ \ldots \circ \mathcal{H}\_{D\_{1}}(\Theta\_{1}^*;\cdot)(\mathbf{x}\_{\ell})
$$
we recursively let
$$
\mathbf{x}\_{\ell}^1:= \mathbf{x}\_{\ell}, \ \ \mathbf{x}\_{\ell}^{k} :=\mathcal{H}\_{D\_{k-1}}(\Theta\_{k-1}
^*;\cdot) \circ \mathbf{x}\_{\ell}^{k-1}, \ \ k=2, 3, \dots, n.
$$
When training grade $n$, we use the output of grade $n-1$, denoted as $\mathbf{x}\_{\ell}^{n}$ along the residual $\mathbf{e}\_{\ell}^{n}$, which are already obtained. The training dataset in grade $n$ consists of $\left\\{(\mathbf{x}\_{\ell}^{n}, \mathbf{e}\_{\ell}^{n})\right\\}\_{\ell=1}^{N}$. This data is used to train a new shallow network, which is independent of the previous $n-1$ grades. Therefore, there is no need to evaluate the forward propagation of the previous $n-1$ grades during the training of grade $n$. Moreover, $\mathbf{x}\_{\ell}^{n}$ can be computed recursively (as shown in the above equation), ensuring that the computation cost for each grade remains relatively consistent.
In the MNIST dataset example (Section 3.3), the network structures are $[784, 128, 128, 10]$ for grade 1 and $[128, 128, 128, 10]$ for both grades 2 and 3. With $\beta = 1$ and $\kappa = 1$ (as shown Table 3), we trained each grade for 2000 epochs. The total training time for MGDL was 3109 seconds, with grade 1 taking 1310 seconds, grade 2 taking 906 seconds, and grade 3 taking 893 seconds. While in the same setting, the training time for SGDL is 3298 seconds.