Principled Weight Initialisation for Input-Convex Neural Networks

Summary

The paper discusses a principled weight initialization strategy for Input-Convex Neural Networks (ICNNs). The authors propose a new theory that generalizes signal propagation theory to include weights without zero mean, and derive a principled initialization strategy for ICNNs from this theory. They demonstrate the effectiveness of their initialization strategy through empirical experiments and apply ICNNs in a real-world drug-discovery setting.

Strengths

1. The paper proposes a new theory that generalizes signal propagation theory to include weights without zero mean, which is a significant contribution to the field. 2. The authors derive a principled initialization strategy for ICNNs from their new theory, which improves learning and generalization in ICNNs.

Weaknesses

1. The proposed approach cannot be applied to networks with skip connections, limiting its usage in real-world models. 2. The experimental section is not comprehensive and there is a lack of ablation study

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The proposed method aims to solve the initialisation problem in an Input convex Neural Network (ICNN), where the weights are required to the non-negative. The commonly applied approach, setting the negative entries stamped from zero-mean Gaussian as zero, leads to varies the desired mean. By analysing the signal propagation in the Neural Networks, the authors are able to sample the weights with expected mean and variance. The method is evaluated on several tasks and compared with the ICNN without the proposed initialisation method with some promising results.

Strengths

1. The problem solved in the submission is important in the ICNN setting. 2. The derivation of the method is concrete.

Weaknesses

Some details are not well explained and are sometimes confusing. 1. $\rho_*$ is defined $\frac{1}{\sigma^2_*}Cov[s_1^-, s_2^-]$ in line 178 and the author claim $\rho_*$ is independed on $\frac{1}{\sigma^2}$ in line 184. And then \rho_* is set as $\frac{1}{2}$ heuristically. The experiments are not carefully designed to support the submission and I am not sure some experiments are sufficiently conducted. 1. In Figure 2, the loss of ICNN does not change during the training process in the MNIST setting but in CIFAR10 and CIFAR100 there is no such phenomenon. 2. In Figure 3, drawing the conclusion that with the proposed initialisation the training dynamic is more stable is not concrete via eyeball comparison. Some quantities are needed. 3. And according to my experience of training non-convex NN on cifar10, the average testing loss does not really show a dramatic increase after some training iterations and non-convex NN has the highest test loss and accuracy which are unusual. And the rest of the learning curve is more reasonable with low test loss and corresponding high accuracy. 4. It can be noticed that with skip on ICNN the performance of ICNN improves, I think applying the proposed initialisation to ICNN with skip will give good support to the submission. Additional question, which does not affect my score. Since in all the settings in the submission, the non-convex NN has the best performance compared with other ICNN-based methods, so is there any scenario for ICNN or why ICNN is essential in the first place? Notations: In some equations the $s_i$, $s_j$ are mixed used with $s_1$, $s_2$, for example, the one below line 172 and the one below 175.

Questions

1. In Figure 2, Non-convex and ICNN have lower training loss than other baselines at iteration 0 which, I believe, indicates the initialisation can the author explain why it happens? 2. Can we treat $\rho_*$ as a hyperparameter? If so how it affects training and generalisation of the ICNN models? these two questions are not discussed.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See the sections above.

Summary

This paper investigates the initialization for input-convex neural networks. They generalize the signal progagation theory by removing the assumption of centred weight distrubution. The experiments show that the proposed initiaization method is effective in a set of datasets.

Strengths

1 This paper generalizes signal propagation theory by removing the assumption that weights are sampled from a centred distribution. This generalization is necessary for ICNNs. 2 The experimetal results are solid and convincing. The experiment on the real-world drug discovery task is nice.

Weaknesses

1 The theoretical contribution of this paper is limited. The theoretical results are derived following the framework proposed in [1]. 2 It seems that the authors only consider the forward propagation of an initial ICNN. However, the backward propagation and the output diversity are also crucial for the initialization. [1] Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017

Questions

Please see weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors havve adequately addressed the limitations, and there is no potential negative societal impact of their work.

Summary

NOTE: edited after author rebuttal and score has been updated. This paper is related to input convex neural networks (ICNN). It analyzes the signal propagation through such a network and based on that proposes a new initialization scheme that allows the networks to be trained efficiently. It investigates the efficacy of the scheme in various ML tasks, including image benchmarks and drug discovery tasks.

Strengths

The initialization scheme is well grounded in theory (although with some stated simplified gaussian assumption). The method seems to work well in practice, at least in the tasks used in the paper. It is experimentally shown that in contrary to previous work claims, ICNNs do not require skip connections in order to be trainable to good results.

Weaknesses

The ICNN model family is intuitively less powerful than normal neural networks. The authors do not discuss this and the chosen experiments seem to be such that the performance degradation compared to normal neural networks does not happen. Could the authors discuss the limitations of the ICNN models, e.g., could imagenet level or GPT-style text understanding models be trained in the ICNN setting? The abstract says that the new initialization allows for more efficient latent space exploration, but as far as I see, there is no comparison to other methods in the paper. Figure 1 shows preactivation distributions on the top, but the scale of the x axis is missing, please add that to the figure. It also seems that the activations start to concentrate around 0 as one moves deeper in the network. Is this the case and could the authors discuss this phenomenon and how this will affect very deep networks (e.g., 20 layers). In the experimental sections, it seems that latent space exploration is the main use case enabled by the ICNN and the new init method? Could the authors discuss other use cases of ICNN and for example explain why they did not repeat the experiments from previous ICNN papers and show improvements stemming from their improved initialization? Also this use case seems very interesting, could the authors discuss pros and cons related to other methods that could be used to explore the latent space of this task? Also, are there some limitations of ICNN in this task or similar tasks, e.g., related to complexity or size of the network and the data?

Questions

Minor spelling mistake 3.2 ”sufficiently good to the derive” - delete ”the”?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As mentioned in the paper, the signal propagation model assumes gaussian distributed preactivations, which is not always what happens in reality. This is well noted in the paper, and at least in the practical tasks considered in the empirical section, the derived initialization scheme still works well. Although the authors say that it would be possible to analyze skip connections, they do not perform the analysis. The reasoning that skip connections are not needed might not hold for deeper networks.

Thank you

Thank you for the rebuttal. My main concerns have been addressed and this has been reflected in the edited score.

Rebuttal acknowledgment

Dear authors, Thank you for your efforts in writing a rebuttal. Unfortunately not all reviewers have acknowledged or responded to it, but rest assured that I have read it and will bring it up in further discussion with the reviewers, and will take it into account for the final recommendation. Best, AC