Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation

Summary

This paper proposes Neural Localizer Field, a continuous field of point localizers, for localizing any point of the human body in 3D from a single RGB image. The method enables mixed-dataset training using various skeleton or mesh annotation formats. The method has three main parts: a point localizer network, a neural localizer field, and a body model fitting algorithm. Trained on a mix of datasets with different annotations, the model achieves good performance across various benchmarks.

Strengths

+ Novelty. The idea of utilizing a neural field to unite different data sources is interesting and novel. The paper does a good job in explaining the motivation as well as laying out the technical details. + Impressive performance on extensive benchmarks and experiments. The method enables training with multiple annotation sources, and it achieves better performance as compared to SoTA. Notably, it achieves good performance on shape prediction, which is a hard problem in human mesh recovery due to the lack of training data with shape annotations.

Weaknesses

- Lack of discussion on inference speed. Since this method would require on the fly point inference, I wonder what is the inference speed and whether this method is suitable for real-time inference.

Questions

How does the model do on small details such as fingers?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

Limitations (lack of temporal cues) and are explained.

Summary

The authors proposes a Neural Localizer Field (NLF) to learn a continuous representation of the canonical human pose by learning to predict a set of functions that map a query point in the canonical human volume to a point in the human posed space, given a single rgb image. By introducing a meta-learning architecture, they are able to train on diverse datasets with different annotations in both 2D and 3D. The authors claim that this scaling from a large number of datasets allows for a better pose predictor than prior work and show relevant results.

Strengths

* Clear Insight/Idea: The insight is simple, clearly explained and well motivated. Having a single architecture that ingests all sorts of human pose, shape annotation would certainly benefit from the diversity if handled correctly during training time. Although, the current architecture might not be the *best* design choice, the paper does show that a simple architecture + large datasets boosts metrics. * Impressive results: The quantitative metrics in Table 2-6 are quite impressive and perform better on most comparison axes. The shape estimation results also are convincing and show benefits from better pose prediction. * Although not trained for temporal stability, the method does show some temporal stability in the supp. video.

Weaknesses

* Better data inspection: The core contribution is that simple architecture + more data gives better results. Since data is the main focus here, a thorough ablation on the data sources is missing. Its not clear if the performance is derived from just a few data sources or all of them i.e. how each dataset affects results. Without this understanding, its hard to argue that more diverse data improves metrics while a few datasets might have the biggest quality impact. * Extent of generalizability: Usual suspects for human pose&shape estimation failures/limitations are loose clothing, occluded views, unique poses. It would be nice to see how the method works on such cases and if the method generalizes well to such cases. Additionally, points that are often not annotated in pose estimation datasets might be prone to failures. The lower accuracy for face and hands in Table 5 makes me believe that this could be the case. It would be great if the authors could comment on the performance of the method in such cases. * 3D Loss weighting: Given the 3D loss for 3D datasets, the network would have to account for the different dataset scales which might be widely off. This can affect training and test time results.

Questions

* Its not clear what the canonical space representation is. Since the query points share the same domain across multiple datasets, I presume all of them are sampled from a single canonical space. But Fig. 4 column 1 shows query points that are with respect to each dataset. * Corresponding to the previous comment, how are 3D losses weighted across datasets such that scale is handled appropriately?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes, authors have adequately addressed limitations and societal impact of the work.

Summary

This paper focuses on 3D human pose and shape estimation from a single RGB image. The main insight is, to avoid the influence of a fact that different human pose dataset defines different skeleton in their annotations, this paper proposes a point-based representation to ensure the model can learn from many datasets without suffering the skeleton mis-alignment problem between existing datasets. The idea follows the mechanism of dynamic convolution, encoding a point (inside the 3D human body) in canonical coordinates as the weight of the dynamic convolution, and converting the image features into a heat map that estimates the 3D position of the point in the target 3D human body mesh. The proposed model is trained with nearly 50 datasets with different annotations, including SMPL parameters, 3D / 2D keypoints, densepose, etc. Then they compare the performance on multiple benchmarks.

Strengths

1. Extensive ablation studies to exploit the effects of many important settings, e.g. different way of encoding canonical position, uncertainty estimation. These results would be helpful for HMR developer to make better decision on model designs. 2. Great qualititive results on InterNet videos, especially the 2D alignment seems pretty well. 3. Video demo of sampling random canonical points proves the effectiveness of position encoding.

Weaknesses

1. The quantitative comparison is not fair and cannot verify the superiority of the proposed point-based representation over previous ones.The model is trained with nearly 50 datasets, while none of the compared methods are using the same experiment settings. Without fair experiment settings, readers can't tell wether the proposed point-based representation helps or not. See the following questions section for details. 2. Some typos. For example, a period is missing between "process" and "I" at L#213. At L#215, "We use EfficientNetV2-S (256 px) and EfficientNetV2-L (384 px) [98]".

Questions

1. About fair comparisons. If we use the same training dataset but remove the point-based representation, will the results be similar? To what extent does this new point-based representation help? However, the current paper does not answer this fundamental question very well. In rebuttal, this question doesn't get well answered. The concern about what is one solid technical contribution of this paper is still there.

Rating

4

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

No.

Summary

This paper deals with the task of 3d human pose estimation. It contains three main contributions: 1. A hypernetwork that takes as input a point in a 3d body volume (in a canonical pose) and outputs the weights of a network (a single layer, really) that, when applied to the features of a vision backbone, is able to localize said 3d point in R^3 given an image (plus a 2d point and 2d uncertainty). 2. An application of this approach to train on multiple datasets with SMPL and SMPL-like annotations, 3d, and 2d annotations. 3. An algorithm to fit SMPL parameters given joints and vertices. These approaches, combined, (plus a series of engineering tricks, such as creating a synthetic dataset and treating some annotations as themselves learnable) result in network that yields state-of-the-art results on several 3d pose estimation benchmarks.

Strengths

## Originality Using a hypernetwork to predict arbitrary points in a human volume is a novel idea. Putting together a super dataset for this task is very nice and novel as well as far as I am aware. ## Quality The results, whether they come from a novel architecture or a novel super-dataset, are strong across the board. ## Significance Regardless of the soundness of the contributions, the fact that the paper promises to make all the contributions easily reproducible is a big plus. The field could really benefit from a way of sourcing multiple datasets together, and I can see multiple people building upon the ideas presented here if everything is released in decent shape.

Weaknesses

## Soundness The main weakness of this paper is the lack of experiments that independently test the importance of each of the contributions. The paper proposes two main ideas: a hypernetwork for 3d human modelling, and a superset of datasets used to train this system; the former being primarily a methodological contribution, and the latter being primarily an engineering contribution. Unfortunately, there is no experiment or ablation distilling the importance of each contribution. Concretely, this could be achieved by, for example * Training the novel architecture on a single dataset * Training the novel architecture on a subset of the compiled datasets (eg, on the datasets with SMPL annotations), or * Training a baseline architecture on the superset of datasets (or a subset thereof, such as the ones with SMPL annotations) These results would help the readers understand whether and to what extent the access to more data or the novel architecture make a difference in the SOTA results reported. As is, this crucial question remains unfortunately unanswered, and takes away from what would otherwise be a very, very strong paper. I think these experiments are extra important because the paper is implicitly making a very bold and counterintuitive claim: that by posing the task of 3d human pose estimation as 3d registration (a more complicated task), it is possible to achieve better 3d poses than SOTA. Furthermore, this is achieved by exploiting data that is not annotated for 3d registration; this is very counterintuitive and, in my opinion, likely to be untrue. Therefore, I am inclined to think that it is the extra data that helps the most towards the strong results. ## Clarity In my opinion, the treatment of the "localizer field" is overly convoluted. While yes, it is true that the localizer field technically defines a neural field of functions, the paper makes it sounds like this is a very new idea (L163-164 "Although neural fields are typically used to predict points or vectors, here we use them to predict localizer functions"). This is not the case; at the end of the day this is a hypernetwork, which has been a staple of work in human modelling for a long time (eg [a, b]). The authors seem to be aware of this connection, since the paper mentions that the localize field "modulates" (L731) the convolutional layer of the point localization network, which is the terminology used in [a] for hypernets. I believe S3.2 could benefit from rewriting to make this part clearer and more in line with previous notation and descriptions. Re: Efficient body model fitting. The method is described as really fast, compared to the official code which is said to take 33 minutes and achieves a slightly lower error. Most optimization methods have exponential error decreases, so it is not uncommon to see exponentially longer times for slightly lower errors. I think it would be clearer to plot the error as a function of time for both the official and new methods. Re: Using 2d and 3d annotations. I am unable to understand how datasets annotated with only 3d poses are used to supervise an approach to volumetric registration -- the description in the paper is very terse (1 line). Is this done by fitting SMPL to the 3d points and obtaining an approximate place in the human volume? If so, it seems like training with these fitted SMPL meshes would be another baseline worth trying; ie, bring all the datasets to SMPL, then train on it. This would further disambiguate whether the architecture or the use of extra data is the main contribution. [a] Karras et al, A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR'19 [b] Chen et al, Authentic Volumetric Avatars from a Phone Scan, SIGGRAPH'22

Questions

0. Could the authors elaborate on how datasets with 2d and 3d annotations are used for training? How does the "approximate initialization" work? Is this some approximate initialization to 3d registration (via SMPL fitting)? 1. The supplementary material discusses the creation of a large synthetic dataset using SMPL fittings of the DFAUST dataset, which is not mentioned in the abstract or the paper -- how important is this for the overall results? 2. What is the dimensionality of the volumetric heatmap? Is this depth defined over the entire scene or only over the depth of the human body? If so, is the range of the function over the entire $\mathbb{R}^3$ in the human body, or a discretized subset? 3. Why does the architecture predict a 2d and a 3d heatmap? Is it possible for the 2d heatmap to disagree with the projection of the 3d prediction? 4. The last two layers of Fig 6 show FC layers going from 1024 to 384 dimensions, and later going from 1024 to 384 again. Is this a typo? If so, what does the actual architecture look like? 5. The paper uses the number 384 several times in seemingly unrelated areas * The size of the images used in the larger network variant * The number of channels predicted by the localization field (or is it both the size of the input plus output?) * The number of points sampled from the interior of the human volume Is this a coincidence? 6. What is the time it takes the official SMPL fitting code to achieve an error comparable to the one achieved by the proposed method?

Rating

7

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

Limitations are addressed adequately.

Still unclear how initialization work

Thanks for clarifications and discussion on my questions. > Training can be started with an approximate placement of e.g. the Human3.6M shoulder point in the canonical human volume in the general shoulder area, and we let the gradient-based optimization update all parameters jointly My question is about how exactly this initialization is done, and this paragraph does not provide an answer. When the authors say that training "can be started with an approximate placement" on a canonical human volume, how exactly is this done? Is it manual? automatic? via optimization? I am more interested to hear how the initialization was done in this paper, rather than how it can be done in the abstract.