A Metalearned Neural Circuit for Nonparametric Bayesian Inference

Summary

The authors present a recurrent neural net (RNN) that learns to mimic a Bayesian non-parametric (BNP) approach to classification with (potentially) heavy-tailed data and an a priori unknown number of classes. The method is evaluated on three experimental setups.

Strengths

The paper itself is well-written and motivated. It provides a clever approach to combining BNP with the straightforwardness of an RNN. Conditioned on the experimental weakness mentioned below, the improvements seem significant. However, as I am not familiar with the literature I cannot properly judge the significance, as I am not sure (but somewhat doubtful) whether the state of the art on this problem has not progressed in the last twenty years as the chosen baselines.

Weaknesses

The weaknesses lie primarily in the experimental evaluation. - As discussed, the only baselines are a publication by Fearnhead from 2004, a vanilla Chinese restaurant process, and a self-designed non-Bayesian model. This hardly provides a proper evaluation of the approach. - All results are reported from a single seed without any significance analysis. (And no, promising multiple runs in a future revision of the paper is not sufficient to answer _yes_ to this question in the checklist. _No_ would have been the correct answer.) ## Minor - The abstract claims better performance than _"particle filter-based method**s**"_ while only a single method is being compared against - Given that there is a single seed, reporting four decimals of significance seems unjustified.

Questions

- Only a single run is reported for each result. Was the seed to generate the same for all methods? E.g., has every model in Sec 4.1 access to the same training data? - In sec 4.2 the choice is to pick the representation space post-relu activations, placing an extra hurdle on the particle filter baselines (the hurdle model). Why was this choice made instead of sticking to common practice for taking the pre-activation representation features from pre-trained neural nets? Which outcome would the authors expect without this artificial introduction of sparsity?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations of the approach are adequately discussed by the authors.

Summary

This paper proposes a novel method for inference for Bayesian nonparametric models through an amortized approach. After simulating data from a Dirichlet Process Mixture Model, a Recurrent Neural Net is trained to characterise the relationship between the simulated data and the parameters, and consequently characterise the posterior distribution. Comparisons are made with a particle filter method for sequential inference for a DPMM, and experiments are run on simulated data, and image classification tasks using pre-trained neural net activations as inputs.

Strengths

This is an interesting article taking a novel approach to inference for DPMMs. The motivation and theory are adequately explained, and the application examples chosen are interesting. The use of amortized inference in this context is novel: there are some interesting links to make here with Likelihood Free Inference (LFI) methods, where using a neural net to model the relationship between simulated data and the parameter space is becoming more common. See https://www.pnas.org/doi/10.1073/pnas.1912789117 for a relevant review. The quality of the scientific writing is in general high and concepts are explained clearly.

Weaknesses

The experiments are not thorough: they each use one random seed per method, which gives no indication as to the uncertainty or robustness of the results. This is especially confusing, since the authors make a point of describing the new method as being faster than existing methods. The computational resources needed for training the neural net are quite expensive, so maybe tone down the enthusiasm about this method being so computationally efficient. At least make clear that this method makes effective use of a generous budget of compute resources. The examples used are interesting but quite limited in scope: the first example is (reasonably enough) a simple mixture of Gaussians, and the subsequent examples are practical but involve somewhat esoteric knowledge of using pre-trained neural nets as training data and similar. The article would benefit from a clustering example based on real observational data, as hinted at in the introduction. This would make the methods and article much more approachable for a more general audience Line 17/57: I wouldn’t call a mixture model a “classification model” exactly: that word is normally associated with supervised learning and known discrete labels. This is more generally considered “clustering” Line 76: “Chinese restaurant process” might need a little explaining for people who haven’t heard the analogy before. Otherwise good!

Questions

Can this method be adapted for non-streaming data? Would a more general class of neural network architecture work for a different data paradigm? Have you considered a comparison with finite parametric mixture models, or overfitted finite mixtures? These are being given greater consideration now after some inconsistency results for DP clustering models, and the inference is substantially easier. Would this inference method work with a finite parametric mixture model, or a truncated nonparametric model? High dimensional (in the observed covariate space) model-based clustering is in general hard. Would this method have an advantage here, as neural nets normally adapt pretty well to high-dimensional spaces? Line 32: “nonparametric Bayesian models have fallen out of favour in machine learning” Really? Give more justification to this statement, or provide some examples of how they have been replaced. If they have fallen out of favour, why are you working with them? Is it just the inference that is holding them back from flourishing, or is it something else? Line 203: For a more general audience: What exactly is a ResNet? What has it been trained on? What kind of properties do you expect its activations to have relevant to the task? Can you explain this more thoroughly before going further? Line 227/228: where are the computational efficiency results exactly? They are not visible where you mentioned them. Line 248: is there no way of tuning the alpha within the neural net framework? If not, please explain why. The additional analysis looking at the transferability of alpha (line 262) is good, however! Line 272: Can you explain more clearly why none of references [10,37,42,52] are appropriate comparisons? Is there nothing more usable than the fiddly particle filter model? Can you do better than MAP prediction? What kind of posterior uncertainty does this method provide, if any? Line 551: “One random seed was used for training each method”? Really? Is this why there is no errors on the results? If so, this is a major weakness of the results here hidden in the Appendix. From the code of conduct, it seems that this is the case, and will be “updated to multiple runs in the next version of the paper” - does that include the revised version for Neurips? Line 553: “Experiments were performed using NVIDIA A100 GPUs” why the plural here? How many GPUs per experiment? This is unclear and an important detail.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Table 1/2/3/4: there is no uncertainty/errors on the losses. This is because everything was done with a single random seed, so maybe you’re not in a place to add uncertainty easily. Line 227-228: claims are made about the computational efficiency of the method that aren’t really backed up, and are possibly undermined by the inability to use more than one random seed.

Summary

This paper presents an approach for nonparametric Bayesian inference using metalearning to train a recurrent neural network (RNN) to perform sequential inference over an unbounded number of classes. The key contributions are: (a) method to extract the inductive bias from a Dirichlet process mixture model (DPMM) and transfer it to an RNN through metalearning, (b) "neural circuit" architecture that can perform fast sequential inference over an open set of classes after training and (c) experimental results showing the neural circuit achieves comparable or better performance than particle filter methods while being faster and simpler to use. The proposed approach consists of first generating training sequences from a DPMM prior. These sequences are then used to train an RNN to predict class labels sequentially. When trained to completion the RNN captures the underlying structure from the sequences (DPMM) and consequently can be used for fast inference. The authors include empirical results on synthetic data from a known DPMM, as well as more realistic experiments on ImageNet features with classes sampled from a Chinese Restaurant Process and a long-tailed iNaturalist species classification dataset. The neural circuit outperforms particle filter baselines in predictive performance and computational efficiency across these tasks.

Strengths

* The proposed method is a neat way of combining the strengths of nonparametric Bayesian models and deep neural nets. By using a RNN to amortize inference in a DPMM, the paper leverages the flexibility of DPMM and RNNs. The paper falls in a line of recent work using NNs to amortize Bayesian inference, e.g. [1]. So while the idea of approximating inference with NNs is not completely new, this paper is the first to study it in the context of DPMMs. * While not perfect as I will discuss below, the empirical analysis is quite thorough. The authors present experiments on a relatively simple synthetic domain as well as more challenging image-based experiments. The amortized approach perform better than or at least on par with the particle filtering baseline while being much faster at inference. * The method also appears to demonstrate some robustness to distribution shifts as the experiment of tranferring the ImageNet-trained model to iNaturalist shows. * The authors also provide code to aid reproducibility. [1] Transformers Can Do Bayesian Inference. Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, Frank Hutter. ICLR 2022.

Weaknesses

* While the results are certainly significant, I find the experimental analysis lacking in several aspects. Firstly, the experiments in the more realistic open set image classification task are limited by fixing a pretrained featurizer to embed the image into a lower dimensional vector. This removes significant complexity from the inference task and also means the method already has access to some information about the underlying inference task. This also makes the experiments of transfer to the iNaturalist task a bit less impressive since the ResNet featurizer already embeds the images in a "similar" space. * The authors also perform all the experiments using a relatively small RNN (2-layer GRU). There isn't much of a discussion on how this particular architecture was chosen and more importantly there are no experiments to understand the effect of this choice. As this is a largely empirical paper, it is important to study these design choices. For instance, how does the performance vary if we use a larger RNN or if we use a transformer instead? Other such design choices are also not accounted for. * The comparison is somewhat limited to particle filter baseline, and despite the focus on open-set image classification, there is no consideration for other standard baselines for the task [2]. * Another limitation of the approach is robustness to model misspecifcation. Distilling the structure of a (fixed) DPMM into a RNN can make it harder to generalize to data coming from different (unknown) data generating process. [2] Large-Scale Open-Set Classification Protocols for ImageNet. Andres Palechor, Annesha Bhoumik, Manuel Günther. WACV 2023.

Questions

Please see weaknesses section above.

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors discuss the limitations of the method in the conclusion, but I would also encourage them to discuss some of the aspects I highlight in the Weaknesses section.

Summary

This paper proposes to use recurrent neural network (RNN) for classification with an open (non-fixed) number of class labels. This is motivated by non-parametric Bayesian models such as Dirichlet process mixture model and the iterative update on exponential family sufficient statistics in its particle filter methods. The latent state of RNN would resemble the iteratively updated sufficient statistic, and can be used for classifying data into previously seen or a first time occurring class label. Experiments on synthetic data and image data are provided, along with an extended study on robustness to distribution shift.

Strengths

1. The paper is well organized and written, with enough background and clear motivation. 2. The proposed method is interesting and innovative, bridging the two largely non-overlapping fields of Bayesian nonparametrics and deep learning.

Weaknesses

1. It seems that the proposed method only does the classification task, but looses the inference side of Bayesian nonparametric methods. 2. The ability to assign label to a class occurring for the first time is through the mask m_t in Eq.(10), whereas the dimensions of weight matrix W, mask m_t have to be pre-fixed. That is, we are not really assuming there can be infinitely many classes as in Bayesian nonparametric methods, but instead picking a large enough number and assuming that the number of classes is smaller (this is not necessarily a weakness, just some differences I noticed).

Questions

1. Have you tried visualizing the learned weight matrix W? It would be interesting to see if the probability of introducing a new cluster decays with the number of data and the number of observed clusters like Chinese restaurant process. 2. How does the expected number of clusters scale with the number of data? Does the curve resemble some Pitman-Yor process? 3. How could I use this approach as an efficient approximation of Bayesian nonparametric methods and make inference (e.g. getting posterior distributions or some summary statistics of it)? I could imagine that we repeatedly draw random samples from the softmax probabilities in Eq.(10) instead of taking argmax, but this would just become another version of particle filter methods. Is there some better ways of doing this? If so, then weakness 1 wouldn't really be a weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses and questions.

Response to rebuttal

Thanks for the response and additional experiments which are helpful. I will maintain my score.