Referring Human Pose and Mask Estimation in the Wild

Summary

This paper introduces a new task named as Referring Human Pose and Mask Estimation (R-HPM), which adopt text/point/scribble to represent a specific person and estimate its pose and segmentation mask. To achieve this goal, this paper proposes a new R-HPM dataset named RefHuman and a new method UniPHD to perform R-HPM. Experiments on the proposed dataset and MSCOCO demonstrate the effectiveness of the proposed method.

Strengths

1. The proposed R-HPM task is useful and complementary to existing HPM tasks, which can bring new insights into this area. 2. The proposed dataset RefHuman is large and can support the following research in R-HPM. The provided text/point/scribble annotation is complete and can accurately describe a specific person. 3. Experiments on the proposed dataset and MSCOCO demonstrate the effectiveness of the proposed method in R-HPM task.

Weaknesses

1. The definition of point and scribble in Sec.3.1 is not clear. Does point prompt only contains only one point and scribble contains 12 points? How to define the point in point prompt? This paper should give a form definition, not textual description. 2. Some confusion in experiments. First, what is the meaning of dagger in Table 2. If I understand correct, dagger denotes to adopt all images from MSCOCO (~60K) into training, but RefHuman only contains 20K images, how UniHPD can utilize the rest 40K images without ref annotation? Second, I think * should be the default evaluation configuration (namely choose the top-1), so what is the evaluation configuration without * and why adopt it as default configuration? 3. How about adopting multiple types of prompts to perform R-HPM? For example, using both text and point simultaneously to refer a specific person. 4. Evaluation metric is not suitable to R-HPM. This paper adopts AP to evaluate performance, which is designed to evaluate multiple objects in one image. However, in referring setting, only single instance is involved, therefore single instance evaluation metric such as PCKh@0.5 is more suitable for R-HPM, or just evaluating the keypoint error/segmentation IoU is ok. 5. To establish a comprehensive benchmark, the authors are expected to test some existing methods so that the following work can take as reference. None of the compared methods in Table 2 are referring-based methods, so this paper should reimplement some referring-based methods to test their performance on RefHuman, e.g., some referring segmentation methods.

Questions

See Weakness. Overall, I think the motivation and contribution of this work is pretty good. But there are still some questions that should be clarified in the revised paper.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper already discusses limitation in Sec.6.

Summary

In this paper the authors tackle the problem of in-the-wild human pose estimation in a “referring” setting where the goal is to determine the pose of a person referred to using either a text prompt or positional prompt. To achieve this, the authors annotate MS COCO dataset with over 50K annotated instances for 2D keypoints, mask and prompt (either as text, points or scribbles). They use this dataset to train a model called UniPHD which consists of several submodules. The results show the paper is able to train a strong baseline model which will be useful for future research.

Strengths

S1. This paper introduces a new task of referring human pose estimation by releasing a large dataset of 50K annotations (as an extension to MS COCO) enabling researchers to train models that can interact with the model using text and points/scribble. S2. The paper also releases a baseline method for the same task with an aim to learn end-to-end R-HPM.

Weaknesses

W1. The motivation for this obtaining pose in a “referring” manner is unclear. For instance, it might be more valuable to focus a text based human detection as opposed to pose estimation because the former can open up many avenues such as human tracking for research. W2. Multimodal encoder is shallow potentially limiting the interaction of the different encoded features. It might be helpful increase the capacity of the the multimodal encoder model. W3. Parts of the paper not are fully clear. I understand that the paper aims to be “single-stage” but on L205-L214, there seem to be multiple stages where candidates are detected and then used for subsequent pose estimation. (Which makes it two-stage?) Minor comments * Table. 1 does not require last column * Table. 3 typo “Scibble" —> “Scribble”

Questions

Q1. How does Eq. 4 fit into mask prediction task? Is that applied only to the pose prediction branch?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have briefly acknowledged limitations of their method.

Summary

This paper proposes a new task called Referring Human Pose and Mask Estimation and introduces the corresponding RefHuman dataset, which is beneficial for research on human behavior comprehension. Additionally, the authors present a model that leverages three types of prompts for this task. The proposed UniPHD model achieves promising performance in this area.

Strengths

1. The proposed R-HPM task and the RefHuman dataset are beneficial to related research. 2. The proposed method shows promising performance on both the new task and traditional human pose estimation tasks.

Weaknesses

1. The descriptions of the method, especially in Section 4.3, is confusing. For example, what is the relationship between F^(vl) and P’? How do you enhance the template based on P’? And why do “the keypoint queries struggle to perceive the prompt information and lacks interactions with each other, challenging the target-awareness and instance coherence” ? It seems that these keypoint queries can interact with each other in existing decoders. More detailed descriptions and explanations are needed to understand this work. 2. The author proposed that existing research overlooks joint human pose and mask estimation, which provides comprehensive human representations. However, the paper lacks comparisons of the proposed model with other variants, such as UniPHD without the mask head and UniPHD without the pose head. Such comparisons could help clarify the relationship and benefit of the pose estimation and mask estimation tasks. 3. The proposed method needs further validation. Providing an ablation study of the proposed query initialization method and comparing model parameters and computational complexity would be helpful.

Questions

See the weaknesses

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Post-rebuttal

I appreciate the authors' response. The additional explanations and ablation studies improve the quality of the paper and effectively demonstrate the design's effectiveness. As a result, I have raised the score.