Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines

Summary

The authors present an improved convolutional deep kernel machine (CDKM) that achieves state-of-the-art performance for kernel methods on the CIFAR-10 image classification task. They introduce a novel regularization technique where the learned inducing Gram matrices are randomly sampled from a Wishart distribution during training. This helps reduce overfitting and improves generalization.

Strengths

The experiments evaluating the performance of their methods are thorough and convincing in providing an improvement over previous work. They achieve 94.52% test accuracy on CIFAR-10, which is a significant improvement over the previous best kernel method result of 92.69% and comes close to matching comparable neural network performance (94.55% for an Adam-trained neural network with the same architecture). As shown in Table 2, their improvements in numerical stability allow for the use of TF32 arithmetic, making training about 5 times faster than previous implementations that used double-precision floating point arithmetic. In Table 1, the test-log likelihood improvements over previous work and traditional neural networks (without weight decay) is compelling as a result. But also surprised to see models with weight decay outperforming the other methods, if I understand correctly, CDKMs naturally provide uncertainty estimates for their predictions. Why would traditional models (with weight decay) be better at this?

Weaknesses

Disclaimer: I am not familiar with the space of CDKMs. It's unclear to me what the main scientific insight is provided in this work. The results appear to be state-of-the-art in this space. But aside from performance, it's unclear to me that this isn't just a variation of traditional neural networks with some aspects of Kernel Machines added. But this might be an issue with the general space of CDKMs and not this work specifically. Since the CDKM method is using some form of learning with backpropagation, what is the main point of showing this particularly models works as compared to conventional neural networks (like convolutional neural networks)? With the Gram Matrix itself being learnable and also patch-based, this CKDM architecture is reminiscent of an MLP-Mixer style architecture which under certain circumstances (particularly reasonably large dataset) also performs competitively with Convolutional Networks. Would it be unreasonable to consider the CDKM itself as just yet another neural network architecture?

Questions

Can the authors describe how this is more similar to a kernel method than a type of neural network since the Gram Matrices and mixup parameters are parameterized and trained with backpropagation?

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

Societal impact not applicable.

Summary

The paper explores how to improve the convolution deep nuclear machines (DKMs)to improve their generalization ability, especially on the CIFAR-10 dataset. The authors introduced several modifications, especially random kernel regularization (SKR), which involves adding noise to the learned Gram matrix during training. The technique is inspired by discarding in neural networks and aims to reduce overfitting. Moreover, they use single-precision floating-point arithmetic to accelerate the training, thus allowing for more training cycles with a fixed computational budget.

Strengths

Improving performance: The proposed modification greatly improves the test accuracy of DKMs on the CIFAR-10, from 92.7% to 94.5%. New regularization techniques: Introducing stochastic kernel regularization is an innovative way to reduce overfitting in DKM.

Weaknesses

Modifications involve complex changes to the DKM framework, which may be difficult to implement and understand for developers unfamiliar with these methods.

Questions

How does the stability of the single precision calculation results ensure, and whether the article has additional requirements on the data set

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

yes

Summary

This paper proposes a new method for Deep Kernel Machine, which achieves state-of-the-art results on kernel methods by using methods including regularization.

Strengths

Compared to previous kernel methods, the proposed method achieves better results in CIFAR-10 test accuracy. The use of stochastic regularization is straightforward and reasonable. Some ablation studies are provided, including results with different hyper-parameters.

Weaknesses

Though achieving state-of-the-art results for kernel methods, the proposed method is still relative time consuming and requires much resource. According to the experiments, the proposed method might be sensitive to hyper-parameters and can lead to failure. Given the fact that it is already a time-consuming method, obtain a successfully trained model with good performance on new dataset might be very expensive. CIFAR-10 is a relatively easy dataset, whether the proposed method works well on complicated cases are not investigated.

Questions

Is there a comparison between results trained with double precision and single precision?

Rating

5

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Summary

The paper reports numerical results in which the authors have achieved state-of-the-art performances for an image classification task, namely the CIFAR-10 dataset, with a ''convolutional deep kernel machine''. The performance of this kernel-based model is close to the ones of state-of-the-art neural network architectures on the same dataset. To achieve this result, the authors introduce a ''stochastic kernel regularization'' procedure consisting of adding random noise to the parameter during training. They also use an approximation of the training objective which improves the numerical stability of training and allows to leverage the performances of modern GPUs to speed up the training procedure.

Strengths

The paper is well-written, clearly introduces the ''convolutional deep kernel machine'' model it is studying, and clearly reports its methodology and its result. While I am not necessarily very familiar with the literature on deep kernel machines, the improvement in generalization with respect to previous works is significant and certainly contributes to closing a gap between kernel-based models and neural networks. Also the introduction of a stochastic regularization procedure for those kinds of kernel-based models seems like an interesting contribution.

Weaknesses

In my opinion, the main weakness of the paper is to only provide results for the CIFAR10 dataset. It would be very enlightening to perform experiment for more involved image classification tasks, for example on Imagenet. In addition, I think the paper sometimes lacks clarity in the way the training procedure is exposed. For example, in eq. 2 the kernel function $K$ is not defined. Also, the stochastic regularization term does not appear in eq. 2 while it is presented as the objective function used during training. To address this issue, the authors could add a precise description of the training algorithm (in the same spirit as algorithm 1 but in my understanding algorithm 1 describes the algorithm for inference on the training/test data, and the last line about training is quite elusive).

Questions

- Did you try to perform experiments on other datasets such as Imagenet ? Should we expect deep convolutional kernel machines with stochastic kernel regularization to have performances similar to ResNets or would they perform way worse ? - One of the encountered problem during training is that the condition number of the Gram matrices $G^\ell_{ii}$ tends to worsen over time. Isn't that behavior to be expected when the number of inducing points is larger than the number of classes ? In my understanding, if the features are sufficiently expressive, the rank of the Gram matrices should be equal to the number of classes. Do you agree with this intuition ? Did you try to compute the rank (or eigenvalue distribution) of the Gram matrices ? - It is written that ''we expect the Gram representations to be close to those of the NNGP'', could you provide a justification of this statement ? The rest of my questions are concerned with some incoherences which might simply be typos but hinder my comprehension of the paper. Please correct me if I simply misunderstood those equations. - At the end of section 4.2 you write several times $G^{-1} K G^{-1}$, shouldn't it be $G^{-1} K$ ? - In eq. 9b you write $K_{features}(F^\ell)$, shouldn't it be $K_{features}(F^{\ell-1})$ ? Also in this equation it is not really clear for me what the function $K_{features}$ is. - In eq. 14 shouldn't the divergence term be $D_{KL}(N(0, G^\ell)||N(0, K(G^{\ell-1}))$ ? - In eq. 15b shouldn't it be $K_{features}(F^{\ell-1})$ ? - In eq. 16 shouldn't it be $\mathbb{E} \left[ F^\ell_{ir,\mu} F^\ell_{js,\mu'} | H^\ell \right]$ ? - In eq. 21 shouldnt' it be $\Gamma(K(G^{\ell-1}))$ ? - In eq. 25 it should be $F^{\ell-1}$. - I do not understand eq. 29. The r.h.s. uses $\Gamma^\ell_{tt}$ which, if I understand correctly, is defined using the whole feature matrix $F^\ell$ whose probability density we are trying to compute. Shouldn't it be $\Gamma^{\ell-1}$ in eq. 29 ?

Rating

7

Confidence

2

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations of this work are discussed in the dedicated section. On top of that, I think that an important limitation is to only provide results for the CIFAR10 dataset. It could be discussed if one should expect convolutional deep kernel machines to have performances similar to the ones of neural networks for more involved image classification problems.

Thanks for the clarification

I have adjust my score based on the clarifications. To understand this better, What are the dimensions to the Gram Matrices? In Algorithm 1, how are the inducing points determined?