Smooth Min-Max Monotonic Networks

Initial reply to review

Thank you for your feedback. Corollary 1 is an existence proof, showing that for a certain beta value an SMM with a particular approximation accuracy exists. This network – and beta parameter – may not be the “best” one for the given task, there may be SMNs that have fewer parameters, that have smaller weights, are Lipschitz with a smaller constant, are faster to learn, etc. with the same or better approximation accuracy. Thus, Corollary 1 is not strong enough to establish a link between fitting error and the value of beta for practical experiments as considered in the paper. Thank you for the question regarding beta. In general, too extreme beta values may cause numerical trouble, which leads to vanishing gradients. If beta is too large, then the SMN gets stuck in undesired local minima as the original MM. To show this, we reran the experiments underlying Table C.4 with ln(beta)=2. This reduced the mean number of active SMM neurons from 35 to 29.3 (means for $f_\text{sq}$, $f_\text{sqrt}$, and $f_\text{sig}$ were 31.81, 30.71, 25.42, respectively). We are happy to add an extended analysis to the paper.

Initial reply to review

Thank you for your feedback. **Question 1:** "My understanding is that tuning it to ensure that the output network is monotone is not trivial and requires retraining the entire network.": *The network is monotone for all choices of beta.* Tuning beta is not an issue in practice. Note that beta is adapted during training. All experiments (except those in reported in Table C.7) were conducted with the same "default" initialization ln(beta)=-1. This shows the robustness of our method. We suggest to simply use the default initialization (much too large or too small initial beta values may impact the performance, see also reply to reviewer RbCP). **Weakness 1:** To the best of our knowledge, our idea is new in the context of monotonic neural networks. This is indeed remarkable, given the many papers building on Sill’s work coming up with complicated alternatives. **Weakness 2:** The results in Table 1 show statistically significant better performance of SMM compared to all other methods. The results in Table 1 may be regarded as the most relevant ones, because the experimental setup is very controlled and focussed on the monotonicity aspect. We are very sorry that we failed to make it clear in our text why one should prefer SSMs over LMNs or XGBoost in practice. XGBoost vs SMM: XGBoost models are piecewise constant. If you want a smooth model (e.g., in a natural science application), then you should prefer SMM or LMN over XGBoost. If you have a larger (deep) learning system and you need a monotonic module and would like to train end-to-end, then XGBoost is not a good option and you should pick a neural network such as LMN or SMM. If you do not have these constraints and tabular data, then XGBoost is a choice that is difficult to beat by any neural network approach - except perhaps SMMs as shown in Table 1. SMM vs LMN: “LMNs share many of the desirable properties of SMMs. Imposing an upper bound on the Lipschitz constant, which is required for LMNs and can be optionally added to SMMs, can act as a regularizer and supports theoretical analysis of the neural network. However, a wrongly chosen bound can limit the approximation capabilities. Our experiments show that there are no reasons to prefer LMNs over SMMs because of generalization performance and efficiency. Because of the high accuracies obtained without architecture tuning, the more general asymptotic approximation results, and the simplicity of the min-max approach, we prefer SMMs over LMNs.” Perhaps this was too modest, so we would like to add: Over all experimental results in our study, SMM performs (most often significantly) better than LMN. While SMMs are very robust w.r.t. to hyperparameter choices (as shown in the study), the Lipschitz bound to be chosen for LMNs might be a problem in practice. We see no reason to prefer LMN over SMM.

Initial reply to review

Thank you for your feedback. **I.** There are several smooth (approximations of the) maximum and minimum functions (e.g., computing the p-norm for large p). However, we need minimum and maximum relaxations which * are non-decreasing, * are smooth, * need to work for positive and negative arguments, * have a bounded approximation error which can be controlled (important for Corollary 1), and * can be computed efficiently without causing too much numerical trouble. The LogSumExp is the best choice we know fulfilling all these properties. If there was a better choice, this would make our idea work even better. **II.** Theorem 1 and Corollary 1 have the standard form of a universal approximation theorem. If one wants universal approximation capabilities, there are functions for which the construction used in the proofs will require many neurons to achieve a certain accuracy. This is inevitable. Still, we regard these results as meaningful because they show a fundamental theoretical property of the model, an asymptotic property such as unbiasedness and consistency of learning methods and convergence to an optimum of a gradient-based optimizer. From a statistical learning theory perspective, universal approximation theorems are important, for example, to understand which learning problems are realizable given the neural network hypothesis class. In practice we would argue that the properties – even if asymptotic – can help to choose one algorithm over another. Studying the theoretical properties of machine learning algorithms is an important area of machine learning research. One can of course argue about the practical relevance of each of the properties listed above, however, one should appreciate that there is a large community who is interested in studying these properties ( https://en.wikipedia.org/wiki/Universal_approximation_theorem ). **III.** The results in Table 3 are not statistically significant. The motivation for Table 3 was to reproduce *exactly* the experimental setting from Niklas Nolte et al. published at ICLR last year so that the results can be compared directly. The experimental setup is not ideal: Only *three* trials, early-stopping on the test set, too complex tasks where monotonicity only plays a minor role, varying hyperparameters for the LMNs. The results show that you can get at least as good (we would argue better) results with SMMs than LMNs if you use exactly the evaluation protocol as in the LMN paper from ICLR last year. For a statistical comparison, the more important results are those in Table 1, which come with proper statistical significance testing. Here SMMs show significantly better results.