Response to Reviewer CcWa
We thank the reviewer for their valuable feedback. We provide additional clarifications with the hope of addressing the remaining concerns.
1,3 : We will include these results in the final revision of the paper.
2: We acknowledge the reviewer's point that if there are $N$ and $M$ candidate architectures for DeNetDM, selecting the optimal model could theoretically require up to $N \times M$ validations. However, the list of candidate architectures is manageable because we impose specific constraints on $N$ and $M$ based on our observations and assumptions. Specifically, we assume that the deep branch is either the same depth as the target network or one layer deeper, which limits the hyperparameter search space to $M=2$ across all cases. For the shallow network, we choose $N$ such that the depth of the shallow branch is significantly less than that of the deep branch, as our approach performs better when there is a substantial difference in depth between the shallow and deep networks. This further reduces the search space. For instance, in the case of CMNIST, where the target model is a 5-layer MLP, we consider the following pairs for hyperparameter search: (5,3), (5,4), (6,3), (6,4), and (6,5), which requires only five additional validations. Similarly in the case of C-CIFAR10, where the target model is a ResNet-18, we consider the following pairs for hyperparameter search: (ResNet-18, 3-layer CNN), (ResNet-18, 4-layer CNN), (ResNet-18, 5-layer CNN), (ResNet-20, 3-layer CNN), (ResNet-20, 4-layer CNN), (ResNet-20, 5-layer CNN) which requires only 6 additional validations.
A similar strategy is applied for hyperparameter tuning on other datasets. The best-performing pair from this limited search space is used to report results in Tables 1 and 2. More extensive hyperparameter search might yield better solutions.But, even within the restricted search space, we demonstrate that our approach consistently outperforms all the baselines. Moreover, methods like LfF and DFA also introduce their own set of hyperparameters, such as $q$ in LfF (e.g., $q \in (0.7, 0.8)$) and $(\lambda_{dis}, \lambda_{swapb}, \lambda_{swap})$ in DFA, necessitating method-specific hyperparameter tuning to obtain the best model.
The architectures used in Table 10 for C-CIFAR-10 were not originally included in the hyperparameter search space for C-CIFAR-10, as the deep network in question (ResNet-50) is significantly deeper than the target network (ResNet-18). The primary motivation for presenting the experiments in Table 10 is to demonstrate that DeNetDM can scale effectively with more advanced ResNet architectures. As discussed in the initial rebuttal, ResNet32 is more effective in capturing core features than ResNet8, which is a fundamental property of the architecture and not the depth. However, in both the cases, the inductive bias of depth modulation still holds since the deeper branch (ResNet-50) consistently captures bias irrespective of the depth of the shallow network. Hence, the inductive bias relying on depth modulation in DeNetDM is agnostic of such fundamental benefits / weaknesses stemming from architecture choices, and its sensitivity is a reflection of the best attainable accuracy from any given
architecture.