We are thankful for the detailed, careful, and thought-provoking
constructive feedback and comments. Addressing the raised questions:
**Weaknesses:**
> While the authors do empirically show that MIP benefits training, it
> is not clear whether the increased variance could also be controlled
> with, e.g., appropriately chosen (i.e., lower) learning rates and
> (i.e., higher) momentum, in the original parametrisation (which could
> attain similar performance, albeit slower).
This is an important point. In our experiments, we controlled for this
by testing a range of learning rates and reporting the best converging
models. We consistently found that MIP was less sensitive to
hyperparameter choices than the default formulation, and that there were
settings, like HyperMorph with momentum SGD, where we could not find any
suitable hyperparameter setting (learning rate & momentum factor) that
would allow the model to meaningfully train.
**Questions:**
The reviewer raises many interesting questions and suggests interesting
areas of future work. Regrettably, we do not have the time or space to
address many of these for this submission in sufficient detail, but we
are thankful for the suggestions as they will inform future work.
> It would be interesting to see whether the residual formulation of the
> hypernetwork closes meaningfully the gap between fully task specific
> parameters and the ones predicted by the hyper network, i.e.,
> $\theta^0 + h(E_{L2}(\gamma);\omega)$. For example, what is the
> performance if on a new task one starts from $\theta^0$ and just
> optimizes for a specific number of steps on that task to get
> $\theta^\ast_t$? Is $\theta^\ast_t - \theta^0$ related to
> $h(E_{L2}(\gamma);\omega)$. Is the performance of $\theta^\ast_t$
> similar to the performance of $h(E_{L2}(\gamma);\omega)$ ?
While not exactly the same as the setup described by the reviewer, we
did perform experiments where we individually trained classical networks
for each task $\theta_t$ for the HyperMorph and SSHN settings, to
understand how classical networks converged for these settings. When
comparing the task-specific networks to the performance of the
hypernetwork weights for the same task, we found that hypernetwork
weights performed no worse than task-specific weights.
> Does the update given by the hypernetwork in this residual formulation
> need to be dense? Is the hypernetwork only adapting a few dimensions
> of $\theta^0$?
In our experiments, the contribution of the hypernetwork $\Delta\theta$
was not particularly sparse, but perhaps this is because the learning
objective was not set up to promote this property in the first place. We
believe that adding a sparsity constraint either to the loss, or
structurally (such as with low rank decompositions) would be an
interesting avenue for future experiments. We will include this
discussion in the revised manuscript.
> It is not clear why for non-scalar inputs $\gamma$, i.e.,
> $\gamma \in \mathbb{R}^D$ with $D \geq 2$ a simple unit normalisation
> transformation, i.e., $\hat{\gamma} = 1/||\gamma||\gamma$ would not
> work for removing the dependence on the magnitude. It seems to me that
> in this case each $\hat{\gamma}$ would just correspond to a different
> point on the hypersphere and the output of the hypernetwork would not
> be independent of the input.
This is a good question that we should have addressed more directly.
Dividing by the norm of the input produces information loss in the
hypernetwork input space, even for multi-dimensional hypernetwork
inputs. As an example, we can consider inputs $\gamma_A = [0.1, 0.2]$
and $\gamma_B = [0.4, 0.8]$, dividing them by their respective norms
results in
$\gamma_A / ||\gamma_A|| = \gamma_B / ||\gamma_B|| = [0.447,0.894]$.
While $\gamma_A$ and $\gamma_B$ correspond to different points in the
input space, they are mapped to the same constant norm representation.
This is why our input encoding maps each dimension to a separate pair of
coordinates.
Since hypernetworks often feature low dimensional input spaces, removing
a degree of freedom in the input introduces substantial information
loss.