Gradient boosting is a sequential ensemble method that fits a new weaker learner to pseudo residuals at each iteration. We propose Wasserstein gradient boosting, a novel extension of gradient boosting that fits a new weak learner to alternative pseudo residuals that are Wasserstein gradients of loss functionals of probability distributions assigned at each input. It solves distribution-valued supervised learning, where the output values of the training dataset are probability distributions for each input. In classification and regression, a model typically returns, for each input, a point estimate of a parameter of a noise distribution specified for a response variable, such as the class probability parameter of a categorical distribution specified for a response label. A main application of Wasserstein gradient boosting in this paper is tree-based evidential learning, which returns a distributional estimate of the response parameter for each input. We empirically demonstrate the superior performance of the probabilistic prediction by Wasserstein gradient boosting in comparison with existing uncertainty quantification methods.
Paper
Similar papers
Peer review
Summary
This work proposes to perform boosting with base learner which are fitted to the Wasserstein gradient of a loss function on the space of probability distributions, which can be useful in particular to capture uncertainty of the models. Several variants of the algorithm are discussed (by adding a diagonal Hessian preconditioner and with different approximation of Wasserstein gradients for functionals not differentiable on discrete measures). The method is demonstrated on posterior regression tasks where the functionals are KL divergences with respect to some prior, and applied on different real datasets benchmarks.
Strengths
This paper is well written and proposes a new interesting boosting method to minimize loss over probability distributions. - The paper is well written - A new boosting algorithm guided by functionals on probability distributions - Application on real datasets outperforming baseline methods
Weaknesses
The paper is good overall in my opinion, but still has some weaknesses. - Experiments focus on KL divergence functional. - No theoretical analysis
Questions
Did you compare between the different algorithms (i.e. with and without Wasserstein Hessian preconditioner, and Kernel vs Langevin approximation) on the benchmark of Section 4.2? Are there other possible functionals which could be interesting besides the KL or other divregences between probability distributions? Could the method be used with input distributions, e.g. to do regression on probability distributions? Typos: - Line 145: "gradint" - Line 520: "Wasserstien"
Rating
7
Confidence
3
Soundness
4
Presentation
4
Contribution
4
Limitations
Yes
Summary
This paper proposed a new ensemble algorithm, called Wasserstein Gradient Boosting (WGBoost), which is a novel gradient boosting framework that leverages the wasserstein gradient for probabilistic prediction. Specifically, WGBoost fits a new base learner to the Wasserstein gradient of a loss functional on the space of probability distributions. Since we cannot access to the probability distribution at each iteration and we also cannot evaluate the wasserstein gradient at each iteration, the author proposed to implement it using the particle method and approximate the functional gradient by the kernel method. The algorithm returns a set of particles that approximate a target distribution for each input. The main application demonstrated is posterior regression, where WGBoost provides a distributional estimate of output-distribution parameters.
Strengths
- WGBoost's ability to approximate target distributions with particles offers a robust approach to posterior regression, capturing predictive uncertainty effectively. - The proposed method shows superior performance in empirical evaluations on real-world tabular datasets, both for regression and out-of-distribution detection tasks. - The implementation is seems easy by utilizing the particle method combined with kernel approximation.
Weaknesses
The approach seems interesting, but the paper lacks the comparison with existing work. Moreover, there is no analysis or discussion when this algorithm is useful as I will explain in the below. - It seems that the proposed method seems almost identical to the stein variational gradient descent (SVGD), but there is no qualitative and quantitative comparison with that. Please explain what is the fundamental difference compared to SVGD. - Since the proposed method is very similar to SVGD, I think the comparison with SVGD and its extended methods including [3, 5, 6, 7] (I think there are other many variants of extention in SVGD). - It has been known that the posterior approximation quality strongly depends on the choice of kernels in SVGD [1, 2, 3, 4]. Since the proposed algorithm is almost identical to SVGD, the quality of approximation by the WGBoost is strongly affected by the choice of the kernel function. However, there is no discussion about this point. - The numerical experiments are only conducted with respect to the final performance on benchmark dataset and I cannot understand when and what kind of problems the proposed algorithm is suitable to approximate the posterior distribution. It is known that SVGD suffers from collapse phenomena. - In addition to the above point, there is no discussion about the computational cost of the proposed algorithm. When I say the computational cost, I point about the computational cost at each iteration and convergence speed. How large computational cost is compared to existing method regarding the number of particles and training dataset size ? I think the proposed algorithm is based on the boosting method, so it suffers from large computational cost with respect to the training dataset size. - As for the convergence speed, it has been known that the convergence of the SVGD is slow, which does not show the linear convergence [4], and I suspect that the proposed method suffers similar problem. However, no discussion is present about the convergence speed or no numerical comparison exists with existing method. [1] Stein Points [2] Measuring Sample Quality with Kernels [3] Kernel Stein Discrepancy Descent [4] On the geometry of Stein variational gradient descent [5] FUNCTION SPACE PARTICLE OPTIMIZATION FOR BAYESIAN NEURAL NETWORKS [6] Feature Space Particle Inference for Neural Network Ensembles [7] Repulsive Deep Ensembles are Bayesian
Questions
I wrote questions in the Weakness.
Rating
3
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
The limitation is unclear. As far as I read no formal description is presented.
Thank you for the reply
Thank you for your response to my questions. However, I still have concerns. Specifically, the fundamental and mathematical differences between SVGD and your proposed methods remain unclear. Although you mention that the purpose of the methods differ—sampling versus boosting—the core principle in both cases involves minimizing the objective functional, such as KL, L2, within an RKHS framework [1, 2]. This suggests that, despite the different objectives, the algorithms and their mathematical foundations might be essentially similar. Therefore, I would like to keep the score as is. [1] Stein Variational Gradient Descent as Gradient Flow [2] On the geometry of Stein variational gradient descent
Response to Reviewer
Thank you for your response and the opportunity to clarify. > the algorithms and their mathematical foundations might be essentially similar (to SVGD) Please let us make our last attempt to clarify the difference. Here we will compare them from the viewpoint of sampling. While SVGD samples from one distribution, WGBoost samples from a **conditional** distribution as follows: - **SVGD** gets samples $(\theta_1, \dots, \theta_N)$ from a distribution $P(\theta)$, where let's say $P(\theta)$ is an arbitrary distribution over an arbitrary variable $\theta$. - **WGBoost** gets samples $(\theta_1(x), \dots, \theta_N(x))$ from a conditional distribution $P(\theta \mid x)$ for any input $x$. WGBoost learns the conditional distribution in the setting where we can know the form of $P(\theta \mid x_i)$ only at given data inputs $( x_i )\_{i=1}^{n}$. In the algorithm, WGBoost **predicts** the Wasserstein gradient used to sample $(\theta_1(x), \dots, \theta_N(x))$ for each input $x$ by tree-model. SVGD do not have input variable $x$ so do not have the prediction stage of the Wasserstein gradient unlike WGBoost. In this sense, WGBoost is a conditional version $P(\theta \mid x)$ of a Wasserstein gradient flow of $P(\theta)$, where the Wasserstein gradient of the particles $(\theta_1(x), \dots, \theta_N(x))$ will be predicted for each input $x$ because the form of $P(\theta \mid x)$ can be known only at $x=x_i$ in our setting. I hope this way of explanation concisely shows the difference. We are keen to hear if the reviewer's score remains unchanged.
Summary
This paper introduces a probabilistic boosting tree algorithm called Wasserstein boosting that uses a smoothed particle gradient to provide probabilistic predictions. Experiments are performed using UCI tabular regression, and out of distribution classification.
Strengths
Originality: - I like the application of Wasserstein gradients and gradient flow to gradient boosting trees, don’t think I’ve really seen any paper like this before. Quality: - I didn’t check especially carefully but the machinery for the algorithm seems to be well explained and correct. Clarity: - The relevant machinery of Wasserstein gradients and gradient boosting is mostly well explained. Significance: - Making trees more probabilistic with limited modifications to their tabular capabilities would be a quite nice advance.
Weaknesses
Originality: - Nothing really noted. Being a straightforward application of some machinery is totally fine. Quality: - For a paper on trees, I find the experiments quite small scale and limited. I would have expected NLL experiments on larger scale datasets, such as the xgboost and lightgbm papers. - Comparisons to (approximate) Bayesian neural networks are quite weak – many other methods such as sgmcmc, etc. tend to outperform on datasets of these scales. references: https://arxiv.org/abs/1902.03932, https://arxiv.org/abs/1907.07504, https://arxiv.org/abs/2002.03704, amongst others - The natural missing comparison here is to Gaussian processes and other kernel methods, which are naturally probabilistic and similarly nonparametric (like trees). - Another missing set of experiments is comparison to quantile regression / pinball loss using trees, which is implemented directly in lightgbm. Quantile regression itself is also naturally nonparametric in at least some sense. Clarity: - My understanding of Wasserstein particle flows is that the particles should interact in some manner during the gradient step. However, the writing of Algorithm 1 makes this quite unclear. I think that the kernel smoothing in the gradient step for the approximate flow is what makes the particles interact, but it’s overall quite unclear. o The code doesn’t seem to provide any clarity here. - Overall, it’s quite unclear which parameters in the loss are actually being estimated. If we’re only estimating uncertainty in regression parameters (or analogously classification), then there’s straightforward two stage approaches. o For example, one can easily fit a tree predicting the mean and then modify the loss function to predict its variance, or we can modify the loss in classification problems to do something analogous. - The writing is extremely passive and non-specific. Suggestions below. o L150: “procedure of exact or approximate” Please use the algorithm box to specifically write or point out which algorithm is used in the experiments. The current algorithm is so unspecific it’s very hard to follow. o L113-115: “Although [32] … originally suggested…” rephrase to something like “Although Friedman [32] originally proposed using a line search … , Buhlmann and Hothorn [34] recommend against the line search …” Significance: - Part of the strength of trees in my experience is that they scale pretty well to large tabular datasets (e.g. n = 10 million). You also tend to need strong uncertainty quantification on these types of datasets, which is part of the reason why Bayesian neural nets became popular for a while. Yet, these large scale uncertainty quantification experiments are lacking from the paper. o Bayesian neural nets: https://proceedings.mlr.press/v115/izmailov20a.html, https://proceedings.mlr.press/v130/immer21a/immer21a.pdf, o Gaussian processes: https://arxiv.org/abs/1809.11165,
Questions
Algorithm 1: - Is the Wasserstein gradient where the particles manage to interact? Otherwise, I see nowhere else the various particles / learners would interact. Table 1: where do the error bars in the baseline methods come from? - Was the same preprocessing done / e.g. same train / test split for these? - In general, I’m pretty sure that MC Dropout is a quite weak baseline for a Bayesian neural net, and many other BNN approaches are much stronger than it at this point (see references above) - the clear tree baselines here would be quantile regression (implemented in lightgbm) and fitting a mean model with MSE loss and then a second tree to predict the variance by optimizing $\max s = N(y | \hat y, \exp\{s\})$ which can be done in lightgbm with a manual loss function. Section 3.3: - so we’re just doing Wasserstein gradients on the mean and variance in a Gaussian regression problem? If so, then it seems like a mean / variance two stage fit model would be the natural “base” method to compare to (and could still plug in the “prior” as regularization”) Section 3.1: L190: “it depends only on the log gradient …”: yes, this is the requirement for most Bayesian (and probabilistic) inference, e.g. MCMC sampling, variational inference techniques. Table 1: what are the error bars? Table 2: I believe that MC Dropout is pretty well known for being poor at out of distribution detection [find reference], while the P Network is certainly a stronger baseline. What would entropies of a traditional multiclass classification tree perform like in terms of an out of distribution detector here?
Rating
3
Confidence
3
Soundness
2
Presentation
2
Contribution
2
Limitations
n/a
Response to Reviewer Comment
We appreciate the reviewer for providing further comments and opportunity for us to discuss about them. All your comments will be used and helpful to improve our manuscript. (PS: Please refresh this webpage in case the latex commands were not rendered as math equations properly.) ## Difference from NGBoost and Quantile Regressor To illustrate, we would like to consider a classification problem, where we have input $x$ and output label $y$. We assume a categorical distribution $C(y \mid q)$ over the label $y$ given the class probability vector $q$. What NGBoost (and two-stage fit model) will provide is a **point estimate of the class probability** $q$ at each input $x$, that is - $x \to \text{NGBoost} \to q(x) \to C(y \mid q(x))$ On the other hand, what WGBoost will provide is a particle-based **distributional estimate of the class probability** $q$: - $x \to \text{WGBoost} \to ( q_1(x), \dots, q_N(x) ) \to (1 / N) \sum_{i=1}^{N} C(y \mid q_i(x))$ What we show in the paper is that (i) taking the average over the distributional estimate leads to better performance (in Section 4.2 with better performance than NGBoost for regression) and (ii) uncertainty in the distributional estimate can be used for the OOD detection (in Section 4.3). These advantages are also different from quantile regressors since quantile regressors provide quantile of output variables $y$ only i.e. no distribution on $y$ nor distribution of $q$. (Such a distributional estimate of the class probability $q$ has been explored in Deep Evidential Learning (DEL) approaches. In this context, WGBoost is the first method that enables the tree version of DEL approaches.) ## Difference from SVGD Firstly, we would like to clarify difference in what WGBoost and SVGD can do: - **SVGD**: Given one target density $\pi$, we can obtain samples $(\theta_1, \dots, \theta_N)$ form $\pi$. - **WGBoost**: Given a set of several inputs and output-densities $( x_i, \pi\_{x_i} )\_{i=1}^{n}$, we can obtain **a map from input $x$ to particles $(\theta_1(x), \dots, \theta_N(x) )$** that approximates the given $\pi\_{x_i}$ for in-sample input $x_i$ and also predicts an unseen $\pi\_{x}$ for new out-of-sample input $x$. Next, we would like to clarify how WGBoost and SVGD is applied for clarification/regression. Let us consider the classification again, where we have input/label dataset $(x_i, y_i)\_{i=1}^{n}$: - **SVGD**: We have a model $f(x; w)$ with the parameter $w$ that outputs the class probability $q$. We use a **posterior $Pos(w \mid ( x_i, y_i )\_{i=1}^{n} )$ of the model $f(x; w)$ using all data points**. SVGD produces samples $(w_1, \dots w_N)$ from the posterior $Pos(w \mid ( x_i, y_i )\_{i=0}^{n} )$. - **WGBoost**: WGBoost is a learning model to estimate $q$ distributionally, so there is no other model $f(x; w)$ like the SVGD case. We use a **posterior $\pi\_{x_i}$ of the categorical distribution $C(y \mid q)$ for each single data point** $(x_i, y_i)$ like done in some DEL approaches: - $\pi\_{x_i}(q) \propto C(y_i \mid q) \times \nu\_{x_i}(q)$ for each $(x_i, y_i)$, where $\nu\_{x_i}$ is a prior over $q$ given at each $(x_i, y_i)$. WGBoost produces a map from input $x$ to particles s.t. $x \to \text{WGBoost} \to ( q_1(x), \dots, q_N(x) ) \approx \pi_x$. Finally, please let us elaborate the intuitive algorithmic idea of WGBoost. The challenge is how to produce particles for unseen input $x$: 1. We have the given set $( x_i, \pi\_{x_i} )\_{i=1}^{n}$. First let's sample from every given $\pi\_{x_i}$ using any user-choice of Wasserstein gradient flow (WGF) such as SVGD; 2. The WGF for each $\pi\_{x_i}$ uses the Wasserstein gradient $g\_{x_i}$, so we get a set of the computed Wasserstein gradients $(x_i, g\_{x_i})\_{i=1}^{n}$. Since $g\_{x_i}$ is finite-dimensional, we can train a ML model that predicts the gradient $g_x$ for new $x$ by fitting it to the set $(x_i, g\_{x_i})\_{i=1}^{n}$. 3. The trained ML model gives us a prediction of the Wasserstein gradient $g_x$ for unseen input $x$. So let's perform the WGF with the predicted gradient and have predictive particles even for unseen input $x$. This procedure uses a user-specified WGF (e.g. SVGD) for every $\pi\_{x_i}$ and train a ML predictive model with the computed Wasserstein gradient at $x_i$. So SVGD is simply used as an intermediate component of WGBoost. Rigorously formalising this idea requires academic work, for which we showed that it can be formalised as an extension of gradient boosting. ## Comment Would this clarify difference from NGBoost and SVGD? We have added these better clarifications in comparison with SVGD to the main text. To date, adapting Wasserstein gradient flows to other gradient-related ML methods is still underdeveloped. We truly believe this work can bring a new inspiration to the ML community, bridging Wasserstein gradient flows and gradient boosting for the first time. We are keen to hear any opinion from the reviewer.
Thanks for your further responses . > WGBoost is the first method that enables the tree version of DEL approaches. Indeed, this is a selling point (and a drawback as the inference should be more expensive..) of your approach as compared to other tree based methods from my understanding. However, it's not clear to my why a well specfied probabilistic model can't just be used for OOD, like the early deep net based approaches were - by entropy of the multi-class predictive distribution (which NGBoost can surely gather). > Difference between SVGD and WGBoost My (somewhat non-rigorous) understanding is that with WGBoost you're essentially non-parametrically estimating the integral $p(\hat y | y, \mathcal{M}) = \int p(\hat y | f(x)) p(f(x) | x) df$ where $f$ is the (non-parametric) functional form induced by the model class. SVGD ends up being quite similar, but (in its [original definition](https://arxiv.org/pdf/1608.04471)) operates on a _parametric_ form of model class, estimating $p(\hat y | y, \mathcal{M}) = \int p(\hat y | f_\theta(x) ) p(\theta | x) d\theta \approx p(\hat y | f_\theta(x) ) q(\theta) d\theta$, where $q(\theta) $ is the approximation distribution learnt by SVGD. However, as SVGD is a gradient flow [paper](https://arxiv.org/abs/1704.07520), it is also possible to express as a nonparametric algorithm in my understanding if we express it in terms of function space rather than parameter space. I guess practically they end up being different algorithms, but the underlying theory is quite similar, if not exactly the same. And there's several different function space algorithms that use stein flows as the other reviewer points out - training either neural networks or GPs with gradient flows like this is fairly well studied.
Response to Reviewer
Thank you for your response and the additional opportunity to discuss. > However, it's not clear to my why a well specfied probabilistic model can't just be used for OOD, like the early deep net based approaches were - by entropy of the multi-class predictive distribution (which NGBoost can surely gather). Thank you for the suggestion. As in the previous comment, we observed that entropy of multi-class classifier didn't perform well and decided to focus only on the uncertainty done in the current paper for the page limit. We will include entropy of multi-class classifier in the comparison. In our understanding, our experiment has already shown better performance of WGBoost on small to medium-large datasets, so adding large-scale dataset and other comparison would complement our experiment at sufficient level. > underlying theory is quite similar (with SVGD) Thank you for the discussion about this point. Infinite-dimensional SVGD has been studied and the algorithm/theory is different [1]. Please let us attempt to clarify the difference again. What WGBoost learn is a conditional distribution $P(y | x) = \int P(y | q) P(q | x) dq$, where let's say $P(y | q)$ is a categorical distribution and $P(q | x)$ is a conditional distribution of the class probability vector $q$ given $x$. In this view, WGBoost is a sampler from a conditional distribution with tree-model incorporated in the procedure as follows: - **SVGD**: Let's say $P(\theta)$ is an arbitrary distribution over an arbitrary variable $\theta$. SVGD gets samples $(\theta_1, \dots, \theta_N)$ from a distribution $P(\theta)$ (regardless of $\theta$ is finite-dimensional or replaced with infinite-dimensional variable $f$); - **WGBoost** gets samples $(\theta_1(x), \dots, \theta_N(x))$ from a conditional distribution $P(\theta \mid x)$ for any input $x$. WGBoost learns the conditional distribution in the setting where we can know the form of $P(\theta \mid x_i)$ only at given data inputs $( x_i )\_{i=1}^{n}$. In the algorithm, WGBoost **predicts** the Wasserstein gradient used to sample $(\theta_1(x), \dots, \theta_N(x))$ for each input $x$ by tree-model. SVGD do not have input variable $x$ so do not have the prediction of the Wasserstein gradient unlike WGBoost. In this sense, WGBoost is a conditional version $P(\theta \mid x)$ of a Wasserstein gradient flow of $P(\theta)$, where the Wasserstein gradient of the particles $(\theta_1(x), \dots, \theta_N(x))$ will be predicted for each input $x$ (because the form of $P(\theta \mid x)$ can be known only at $x=x_i$). We hope this way of explanation clarifies the difference better? We are keen to hear if the reviewers opinion still remains unchanged. [1] Stein variational gradient descent on infinite-dimensional space and applications to statistical inverse problems
Summary
The paper introduces a novel gradient boosting framework called Wasserstein Gradient Boosting (WGBoost). Unlike traditional gradient boosting methods that fit base learners to the gradient of the loss function, WGBoost fits them to the Wasserstein gradient of a loss functional defined over probability distributions. This approach is particularly useful for probabilistic prediction, where the goal is to approximate the uncertainty in the model prediction. The authors provide a general formulation of WGBoost, its algorithmic implementation, and empirical evaluations on various benchmarks. Paper's major contributions include, the introduction of the Wasserstein gradient flow framework into gradient boosting; the development of an approximate algorithm for posterior regression using the KL divergence; and the demonstration of WGBoost's performance on regression, classification, and out-of-distribution (OOD) detection tasks. Authors also propose a second-order WGBoost algorithm built on the approximate Wasserstein gradient and Hessian of the KL divergence.
Strengths
S1: The application of Wasserstein gradient flows to gradient boosting is a novel and promising direction, providing a new perspective on ensemble learning algorithms. S2: Solid theoretical concepts, with detailed derivations and explanations of the Wasserstein gradient and Hessian approximations.
Weaknesses
Authors did not specify details about the hyperparameter selection for Conditional Density Estimation, Classification and OOD Detection tasks. Misc: There is a typo in the line no 284, root is written as room.
Questions
I would be curious to know about the scalability and the computational complexity of the proposed algorithm.
Rating
8
Confidence
3
Soundness
4
Presentation
4
Contribution
4
Limitations
NA
I thank the authors for their clarifications. However, without further experimental comparisons, my opinions on the paper are unchanged. In terms of experimental benefits, I don't think there's a compelling application as described in the paper to use Wasserstein boosting for tree methods as compared to something like NGBoost (or other probabilistic baselines like 2 stage fits, quantile regression etc.). Wasserstein boosting provides non parametric uncertainty estimates, which NG Boost and quantile regression on trees also provide. I understand that Wasserstein boosting could be used on top of a quantile regressor or the two stage fitter, but these experiments just aren't done presently so there's, to my understanding, not as much of a compelling story here. re " , Subspace Inference for Bayesian Deep Learning. " I believe the pre-processing may be slightly different (you can see by comparing their "SGD" baseline to your deep ensemble baseline), but this is good to understand that the Wasserstein boosting algorithm performs quite sensibly. I also think that, even after the rebuttal, the comparison with SVGD is a bit confusing to me (as mentioned by reviwer q9Jy), and would like to see the authors further clarify here. The algorithm seems to be different primarily in its applications to boosting, but in some sense this is quite similar to the application in parametric models as well.
Final Author Comment
We would like to express our gratitudes for all the discussions. While Reviewer jx13 & DY4u agree on the novelty of our work, the remaining concern of Reviewer UM7J & q9Jy is their interpretation that WGBoost is essentially same as SVGD. In our view WGBoost is clearly different from SVGD as attempted to clarify in the rebuttal, but some of our explanation made from the viewpoint of regression so far may sound differently to some reviewer e.g. expert in sampling contexts. In this comment, we wanted to summarise our last explanation about the difference of WGBoost and SVGD from the viewpoint of sampling. ## Difference of WGBoost and SVGD from Viewpoint of Sampling While SVGD samples from one distribution, WGBoost samples from a **conditional** distribution as follows: - **SVGD** gets samples $(\theta_1, \dots, \theta_N)$ from a distribution $P(\theta)$, where let's say $P(\theta)$ is an arbitrary distribution over an arbitrary variable $\theta$. - **WGBoost** gets samples $(\theta_1(x), \dots, \theta_N(x))$ from a conditional distribution $P(\theta \mid x)$ for any input $x$. WGBoost learns the conditional distribution in the setting where we can know the form of $P(\theta \mid x_i)$ only at given data inputs $( x_i )\_{i=1}^{n}$. In the algorithm, WGBoost **predicts** the Wasserstein gradient used to sample $(\theta_1(x), \dots, \theta_N(x))$ for each input $x$ by tree-model. SVGD do not have input variable $x$ so do not have the prediction stage of the Wasserstein gradient unlike WGBoost. In this sense, WGBoost is a conditional version $P(\theta \mid x)$ of a Wasserstein gradient flow of $P(\theta)$, where the Wasserstein gradient of the particles $(\theta_1(x), \dots, \theta_N(x))$ will be predicted for each input $x$ because the form of $P(\theta \mid x)$ can be known only at $x=x_i$. We hope the contrast from SVGD is clear in this explanation. As the author-reviewer discussion period comes to end, we will leave the final decision to the AC and all the reviewers. Thank you once again for all your effort.
Decision
Accept (poster)