Summary
Aiming to better condition the logistic loss function, this work proposed Rooted Logistic Objective (RLO), a very simple polynomial approximation to the log function with an additional power parameter k to control the approximation.
Such a loss can serve as a plug-in replacement for log-based loss functions for supervised classification and unsupervised generation tasks.
Theoretically, the RLO objective is shown to be strictly convex whenever k > 1, and guaranteed to be as conditioned as the Logistic objective function in terms of the Hessian term if k is not too large.
Empirically, the authors conducted a series of experiments on classification and StyleGAN training, involving both fully connected networks and transformers. The modified loss is shown to converge faster and can achieve better performance.
Strengths
The RLO objective seems novel to me.
Advertised to be a better alternative to logistic loss, the proposed RLO can potentially be widely applied to various supervised tasks.
In this work, the authors not only evaluated standard image classification but also extended to training the discriminator in GAN models. I like the inclusion of a toy data case to provide more visual clues.
Weaknesses
## Weak theoretical analysis
First, the scope of this work is on substituting the logistic loss, or more precisely, approximating the log function with polynomials. However, in classification, the logistic loss is only one of the many surrogate losses for the more fundamental 0-1 loss. The authors did not discuss any other surrogate loss functions and how they relate to the 0-1 loss. [1] is a related work.
Second, the theoretical statements are hand-wavy. For instance, the "better conditioning" concept is not clear in this work.
The authors wrote that "From lemma 1 and 2, we can conclude that there is a range of values of k that provides better conditioning for individual data points." I don't think this statement is well supported. The relationship between the $h(,)$ function and how conditioned is the objective function is not clear. The reason why a larger $h$ indicates that "the gradient directions may provide sufficient descent needed for fast convergence" should be made more rigorous.
Also, Lemma 2 only concerns the optimal solution $w^*$, but not the optimization trajectory. So I think the theoretical results are relatively weak.
## Insufficient experiments.
The empirical evaluation in this work is the biggest concern for me.
First, the experiments are not consistent enough. For example, in Table 1, the effect of the L2 regularization is not consistent. Ideally, if the weight decay strength is appropriate, shouldn't it always be better than that without weight decay? To be more specific, in Ionoshphere, L2 improves acc for LR but hurts acc for RLO (k=3). These inconsistencies cast doubt on the validity of the results. I wonder whether the hyperparameters are well-tuned.
Second, for the real-data experiments, the reported results are far from state-of-the-art. Take the CIFAR-10 classification for example, the reported acc for ResNet is only 87.67, which is significantly lower than other reported results. The red curve for CE in Figure 3(b) seems obviously not well-tuned.
The same can be said in the StyleGAN experiments, the FIDs are so high that it may not make much sense.
I strongly recommend the authors to first reproduce some SOTA results for image classification and GANs, and then try to improve upon them to make a more convincing case.
## Misc.
Many typos. For example $L_{RLO}(w)$ after lemma 1 should be $L_{LO}(w)$.
"$x$ is a smooth function, $f(z) \approx x$ are not appropriate. $x$ is a random variable that comes from a distribution.
**Overall**, I think this work can be significantly improved if either
* (1) Conduct a more thorough empirical evaluation and showcase that RLO can actually improve SOTA methods. [2] is an example.
* (2) Make a more significant theoretical contribution by providing improved convergence bounds for the proposed RLO. [3] is an example.
Reference:
[1] Bartlett, Peter L., Michael I. Jordan, and Jon D. McAuliffe. "Convexity, classification, and risk bounds." Journal of the American Statistical Association 101.473 (2006): 138-156.
[2] Hui, Like, and Mikhail Belkin. "Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks." ICLR 2021.
[3] Hu, Tianyang, et al. "Understanding square loss in training overparametrized neural network classifiers." Advances in Neural Information Processing Systems 35 (2022): 16495-16508.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.