Summary
This paper studies circuit algorithms for sampling-based Bayesian inference in continuous-time rate-based recurrent neural networks. Specifically, it argues that using a linear readout from a noisy reservoir provides a substantial expressivity benefit relative to representing the sampled distribution directly in the recurrent population.
Strengths
1. The question of the minimal recurrent circuit architectures that allow sampling from complex distributions is clearly relevant to the neuroscience community at NeurIPS, and I think the main result is interesting (albeit unsurprising).
2. The experiments show to a reasonable degree that the proposed method can sample from non-trivial non-Gaussian distributions (MNIST). This is an improvement on most comp neuro papers on sampling, which largely focus on Gaussian distributions. My enthusiasm for this point is, however, dampened a bit by the fact that the networks here are rate-based and the authors aren't trying to optimize functionals of the dynamics (e.g. as in work from the Lengyel group), so it's not very surprising that this can be made to work.
Weaknesses
1. How does Proposition 1 differ from the results of Ma, Chen, and Fox, "A Complete Recipe for Stochastic Gradient MCMC" (NeurIPS 2015)? Perhaps I've missed something, but I do not see a novel contribution in Section 3.1.
2. The present manuscript does not address the speed of relaxation to the stationary distribution. The ability to obtain accurate samples quickly is a necessary feature of any model of sampling-based inference in the brain, and indeed the question of how to achieve fast sampling in biologically-plausible implementations has been a major focus of past work (e.g., Hennequin et al. 2014, Aitchison and Lengyel 2016, Echeveste et al. 2020, Masset et al. 2022). In my view, this is a key shortcoming of the learning algorithm proposed in Section 3.4: no attention is paid to whether the resulting network enjoys favorable convergence properties. Of course, one could modify the resulting drift and diffusion terms (as in Hennequin et al. 2014 or Masset et al. 2022), but this would require an additional optimization step. The authors should at least discuss this point; adding more experiments to probe this issue would make for a stronger paper.
3. In Lines 293-295 of the Discussion, the authors state that "Our preliminary experiments show that one-hidden-layer RSNs cannot readily approximate high-dimensional heavy-tailed distributions (e.g., those of overcomplete sparse coding representations [36])." If the authors have such results, they should show them. In particular, related to point (2) above, they should show how the convergence speed and number of reservoir neurons required to obtain accurate samples scales with the dimension of the target distribution. As the authors themselves discuss, their approach is currently limited to approximating the score with a single-hidden-layer MLP, and such Cybenko-Hornik-type approximation results may require immensely wide networks. This is also a limitation of the MNIST experiment: MNIST is intrinsically not that high-dimensional, and already here the authors used a network with 20000 reservoir neurons (Line 638)!
4. I think it's important to mention that the striking result of Figure 2 depends critically on the choice of ReLU activation function, particularly as much of the paper is concerned with existence results. If instead one used $\tanh$, then the performance gap between the trained sampler-only firing rate and reservoir firing rate models should be small. In particular, a sampler-only firing rate network can implement exact Langevin sampling in this case. The score of $p(x) = \frac{1}{2} \mathcal{N}(\mu , \sigma^{2}) + \frac{1}{2} \mathcal{N}(-\mu,\sigma^2)$ is $\frac{d}{dx} \log p(x) = \frac{1}{\sigma^{2}} \left[ - x + \mu \tanh\left(\frac{\mu}{\sigma^{2}} x\right) \right]$. This gives the Langevin dynamics $dx(t) = \frac{d \log p}{dx} dt + \sqrt{2} dB(t) = \frac{1}{\sigma^{2}} \left[ - x + \mu \tanh\left(\frac{\mu}{\sigma^{2}} x\right) \right] dt + \sqrt{2} dB(t)$. With $\mu = 1$ as is used in Figure 2, this is exactly of the form of a single-neuron sampler-only firing rate network in eq (8) of the submitted manuscript, with $D = 1/\sigma^{2}$, $W_{rec} = 1/\sigma^{2}$, $I=0$, and $\phi(x) = \tanh(x)$. One can easily verify numerically that this network produces accurate samples from the desired bimodal distribution.
Questions
- The figure axis labels are too small, making them hard to read. Figure 4b is also too small. Please increase the font and panel sizes to improve legibility.
- Lines 32-33: Along with the other references on Gaussian sampling, Hennequin, Aitchison, and Lengyel, "Fast Sampling-Based Inference in Balanced Neuronal Networks" (NeurIPS 2014), should be cited.
- Line 225: I do not see how the proof of Theorem 3 is "constructive," since it is based on an existence-type universal approximation theorem.
- Lines 303-304: "This procedure is more aligned with the developmental processes involved in forming visual representations in the infant brain, where visual representations are thought to be noisier (less linearly separable) initially [3]" Please elaborate; I don't quite follow.
- Line 317-318: This is quite vague. Having a clearer proposal for what this model might help explain in neuroscience would make for a much stronger paper.
- It would be helpful to repeat the theorem statements in the Supplement.
- Lemmas C.2 and C.3 can be replaced by using standard results on the asymptotic expansion of the upper incomplete Gamma function for large argument, see https://dlmf.nist.gov/8.11.
- In Line 562, in the proof of Theorem 3, should $m$ be replaced by $n$?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
The authors provide a generally adequate discussion of the limitations of their work (modulo the weaknesses discussed above), but fail to show the data for some points (see point 3 under Weaknesses).