Summary
The paper presents a new deep Gaussian process (DGP) model, the thin and deep GP (TDGP), which does not suffer from a diminishing signal as the number of layers increases. The crux of the TDGP model is the covariance function that, for each layer $\ell$, acts on a linear combination of a (non-linear) transformation of the outputs of the previous layer and the inputs $\mathbf{x}$, $\mathbf{h}^\ell(\mathbf{x}) = \mathbf{W}^\ell(\mathbf{h}^{\ell-1}(\mathbf{x})) \mathbf{x}$. This hierarchical construction is shown to cover the usual DGP models while not suffering from diminishing signals as the depth increases. To perform inference in this model, the authors adopt a mean-field VI scheme, which is demonstrated to be effective through a number of experiments on both synthetic and real-world data.
Strengths
1. The paper presents an interesting and original approach to DGPs, an important field that should interest the NeurIPS community.
2. The paper is of high overall quality and very well-written.
3. The presented model has the potential to be of considerable significance, given that it generalises previous DGP models while not suffering from the same pathologies.
Weaknesses
1. While I think the paper presents a great idea with lots of potential, my main concern is the lack of a deeper analysis of the model. Given that the proposed model is essentially just a particular transformation of the inputs at each layer and a rather straightforward mean-field VI scheme, I would expect some insights into the behaviour of the model, even if these were just empirical. The current experiments do present some interesting insights, but, say, what happens to the model's performance as we increase the depth of the model? What if we change the width of the layers? What if we use different covariance functions than the squared exponential? How does the induced covariance matrix look at different layers of the model compared to, say, DNSGP? Why does the model encourage low-dimensional latent spaces, and does this happen too for models deeper than just two layers? A Gaussian prior shouldn't encourage sparsity, so this aspect of the model is particularly puzzling.
2. As mentioned, a standard mean-field VI scheme seems very rough given that this is known to perform poorly for many models, including DGPs. To my knowledge, the current state-of-the-art is still the doubly stochastic VI scheme by Salimbeni & Deisenroth (2017), so it would make sense to try something similar (and unless I misunderstand something, it seems fairly straightforward to do this for the proposed model).
3. One clarity issue (perhaps the only one) I find with the paper is that it is unclear to me which DGP model the authors refer to as "CDGP". Deep GPs have evolved dramatically from the original formulation of Damianou & Lawrence (2013), and the mentioned pathologies are not as pronounced anymore (to my knowledge). It is great that the authors compare against DNSGP, but comparing against the doubly stochastic DGP rather than the original formulation makes much more sense. As it is unclear if the authors actually compare to the current state-of-the-art, it is also unclear if the proposed model addresses actual problems.
4. Experimentally, the model seems to work well, but the authors use only two layers for all models. This is particularly strange as a key selling point of the proposed model is that it doesn't degenerate as the number of layers increases. But it is also an odd choice since deeper models should work better (and DNSGP demonstrably does for two of the UCI datasets). It is also strange that MRAE statistics are only reported for the GEBCO dataset; these should also be reported for the UCI experiments.
5. The model is claimed to be "interpretable", but this only seems to be the case for a two-layer model (i.e., one with a single latent space). For deeper models, the distance matrix is still a highly non-linear function of the inputs, which I don't think will be particularly interpretable.
References:
- Andreas Damianou & Neil Lawrence, "Deep Gaussian processes", AISTATS 2013.
- Hugh Salimbeni & Marc Peter Deisenroth. "Doubly Stochastic Variational Inference for Deep Gaussian Processes", NeurIPS 2017.
Questions
1. Which specific variation of the DGP model is referred to as "CDGP"?
2. In lines 138-139, you say that "By placing zero-mean priors on the entries of $\mathbf{W}$, we encode an inductive bias to maximally reduce the latent dimensionality of $\mathbf{h}$." Why is this? Intuitively, since the prior is Gaussian, it shouldn't encourage sparsity.
3. Your contribution seems to be an input transformation, which isn't tied to a specific covariance function, so is there a reason that you chose the squared exponential (or exponentiated quadratic) covariance function? The Matérn class is more often used in practice, I believe.
4. Is a model with more than two layers still interpretable? If so, how would you interpret the latent space?
5. In Figures 4 and 7, how is the inverse length scale computed for TDGP? Also, why can we not compute the inverse length scales for CDGP or DKL? And why can we not visualise the latent space for DNSGP?
6. In lines 51-52, you mention that TDGP is the only deep architecture that induces data embeddings. What do you mean by "data embedding" here, and why do other models not have this property? For instance, even for the original DGP model, we can easily visualise the distribution of a particular sample at a particular layer.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors have adequately addressed limitations in a dedicated paragraph.