Answer to comments of reviewers tj82 and B5U1
We thank the reviewers for their prompt answers. Reviewer B5U1 is correct in interpreting Figure 6 and in saying that our answer is connected to [A] "Winner-takes-all learners are geometry-aware conditional density estimators". But we understand reviewer tj82's concern. The reason why we say that **oracle MPJPE** is one way of assessing diversity is because it writes as
$$\frac{1}{N} \sum_{i} \min_k \ell(f_k(x_i), \mathrm{p}\_i),$$
which **is an approximation when $N$ is large of the quantization error**, also known as distortion:
$$\int_{\mathcal{X} \times \mathcal{P}} \min_k \ell(f_k(x), \mathrm{p}) \rho(x, \mathrm{p}) \mathrm{d} x \mathrm{d} \mathrm{p},$$
where $\ell$ is the average joint-wise $L2$ distance in our case. The latter is traditionally used to measure the efficiency of an estimator in summarizing a distribution with few representatives, commonly used to study the K-means estimator for example [C].
So the fact that we achieve better oracle MPJPE than methods requiring a large number of hypotheses (GFPose, D3DP, Wehrbein et al., …) shows that **ManiPose has better quantization properties** than the latter, i.e., it is more efficient in summarizing the diversity of the conditional distribution with fewer representatives/hypotheses.
More practically:
1. the extreme case of a model with no diversity at all (e.g., predicting $K$ times the same pose) would lead to an oracle MPJPE = vanilla MPJPE of its single-hypothesis version. This is not what we obtain in our ablation study of Table 4 (1st vs last rows).
2. In the opposite extreme case, a naive way to obtain $K$ very diverse hypotheses would be to use a regular grid of the pose space $\mathcal{P}=\mathbb{R}^{3 \times J}$. The latter learns nothing and is uninformative, but could still achieve better oracle MPJPE if given an unrealistically large number of hypotheses (cf [A] equation 16).
3. As shown in [A], the winner-takes-all learning scheme, used in ManiPose, allows it to sit between these extreme cases by learning an adaptive “grid” made of a few hypotheses, capturing the geometry of the underlying conditional distribution. Results on its quantization optimality can be found in section 5.2 of [A] for example.
Of course, oracle MPJPE is not the only way of assessing diversity, which is why **we have provided in our rebuttal additional results** measuring the coverage of ManiPose, which corresponds to the ratio of ground-truth poses whose neighborhood contains at least one generated pose. It is a common metric used in the literature on generative models to analyze diversity.
[C] Pages, Gilles, and Jacques Printems. "Optimal quadratic quantization for numerics: the Gaussian case." (2003).