KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension

Summary

The paper introduces and studies the problem of Semantic Keypoint Comprehension to evaluate the capability of Multimodal Large Language Models (MLLMs) in tackling fine-grained perception and comprehension. It does so by defining three related tasks: (1) keypoint semantic understanding, (2) Visual prompt-based keypoint detection and (3) Textual Prompt-based Keypoint Detection. The first task targets the semantic interpretation capabilities of objects and keypoints, while the other two tasks involve position detection and semantic comprehension from visual or textual descriptions. In the context of this paper, keypoint detection (a.k.a object pose estimation) is the task of localizing the semantic keypoints of objects. The authors further present KptLLM, an MLLM for Semantic Keypoint Comprehension based on LLaVA. The model employs an identify-then-detect strategy, where the model first captures the required semantics and then detects the precise keypoint position. The model employs attention to fuse features from image and keypoint encoded prompts. The fused encoding and the encoding of visual inputs are used as additional latent contexts for an LLM, which is finetuned with LORA to generate the prompt. The keypoint position is further regressed with an MLP head. The authors evaluate their proposed model on the MP-100 (diverse and category agnostic) dataset and the AP-10K dataset and compare its performance to state-of-the-art methods for keypoint detection and to pretrained and finetuned LLaVA models.

Strengths

- The paper introduces new tasks for studying the semantic comprehension of MLLMs at a fine-grained level - The paper presents an MLLM-based method for tackling the introduced tasks, which outperforms current keypoint detection methods across datasets and tasks by notable margins - The proposed method also outperforms state-of-the-art MLLM (pre trained and finetuned) on the target tasks - The authors ablate their proposed detection strategy and other architectural choices.

Weaknesses

- The properties (and potential limitations) of the MLLM-based approaches in terms of runtime, memory, and scalability to multiple key points, compared to existing state-of-the-art methods, are not discussed - Lack of details on the methods chosen for cross-comparison and their key properties/assumptions makes it harder to evaluate the contribution While the paper is overall well structured, the evaluation protocol and details are not always easy to follow

Questions

- Can you please provide more details on runtime, memory, and scalability? - Is the visual encoder trained/fine-tuned or frozen? The textual description mentions a pretrained frozen encoder, while the image shows it to be fine-tuned

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

It is worth elaborating on memory, runtime and scalability aspects of the proposed method.

Summary

This paper proposes an LLM-based keypoint localization method that can detect keypoint by visual or text prompt and classify the keypoint category by given its coordinates. Experiments on several public benchmarks demonstrate the effectiveness of the proposed method.

Strengths

Adopting LLM to perform in-context learning of keypoint localization is interesting and this paper demonstrates that this pipeline is feasible and can get superior performance.

Weaknesses

1. It seems that the Visual Prompt-based Keypoint Detection and Textual Prompt-based Keypoint Detection are trained separately. However, it is more useful if we can complete these tasks using a single model. So how about jointly training above two tasks? Can they benefit from each other or not? 2. What’s the meaning of keypoint identifiers in Table 2 and 3? It would be confusing if these is no specific illustration. 3. Some typos. “Texual”->Textual in line217, 234, 254, etc. 4. Some claims are not proper. For instance, in line 57-58 “To the best of our knowledge, this is the first work that equip LLMs with the point-level visual perception capabilities.” [A] already introduces LLM into keypoint localization and this paper should emphasize the different aspect such as visual prompt-based keypoint localization. [A] D. Wang, S. Xuan, S. Zhang. LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model. CVPR 2024.

Questions

In Weakness.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper discusses the limitation in A.3.

Summary

The authors aim to enhance multi-modal LLMs with semantic keypoint comprehension. They introduce a hybrid visual prompting approach using a query and a support input image. In their pipeline, visual features from both images are extracted via a vision encoder model. Additionally, a support keypoint prompt, indicating keypoint positions on the support image, is used. The embeddings from this prompt are combined with the support image embeddings and fed into the LLM. This mixed embedding helps the model detect and understand the keypoint in the input query image, generating the desired keypoint location information.

Strengths

The paper is well written and easy to follow. The proposed method shows promising results albeit its scope of usability and also generalization should be better examined.

Weaknesses

The proposed support image and support keypoint prompt essentially act as few-shot examples for the model. It would be beneficial to explicitly acknowledge this in the paper. Another limitation of the proposed method is the requirement for a visually similar support image with keypoint locations for each input query image. Have the authors tested the model with support images that are visually different from the query images? How much visual difference can the model handle without a significant drop in accuracy? Conducting an ablation study on this would be crucial to determine the extent of semantic (high-level) or texture (low-level) distribution shifts the method can tolerate without a notable accuracy decline.

Questions

The authors should compare their approach to GPT-4V as a baseline to better understand the benefits of the additional steps they have added to their proposed pipeline. It is unclear where the support image and support keypoint prompts come from during inference. Are these data available to other methods in the experiments? Figure 2 is somewhat confusing. (Z_q) is fed to the LoRA section, while the rest of the information is fed to the rest of the LLM. Is the figure designed this way on purpose? If so, could you please clarify this?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations of their work.

Summary

The paper proposes to build a MLLM for keypoint detection. They propose three main points to achieve this. First, the paper introduces a new benchmark that measures various types of keypoint detection including both text and visual-prompt-based, as well as traditional keypoint understanding. Next, they propose to build a multimodal LLM, which is designed specifically for keypoint detection that is shown to perform quite well on this task.

Strengths

[S1] Good results: The paper boasts quite good results for the task at hand. The experiments are thorough and clearly outline the contribution of various components of the model. [S2] New benchmark: The paper proposes a new benchmark for keypoint understanding, which goes beyond the traditionally available keypoint understanding benchmarks by bringing visual and text prompting. While this is mainly in service to the current paper, it could be independently useful for posterity.

Weaknesses

[W1] Not a lot of novel insights: This is my main, and mostly only, weakness for the paper, but it take a few different forms. Firstly, this is a straightforward "building" of MLLM formula; i.e., convert the task at hand into "instruction format" -> train a vision to LLM connector --> fine-tune LLM; that is used by a lot of recent works. As such, this would be mostly of interest to the domain performance, Keypoint comprehension in this case. This already puts some doubt as to whether this is of interest enough to a broad community. Next, there are also not a lot of modular components that might be broadly useful that hasn't already been studied by other works. Some details below: W1.1 LLava does almost as good (for supported tasks): The paper shows that, once fine-tuned, the popular LLAVA model does almost as good as the proposed method. The significance of this finding is not thoroughly discussed. Why build a new model at all? Furthermore, there are more capable MLLMs baseline such as R1, R2, R3 that might be more suitable targets for fine-tuning. See 1.2 below as well W1.2 The proposed method of generating a special token that can be decoded to a key point is also explored before (R1). Besides, there are also various MLLMs that are capable of doing region comprehension (R2, R3...). Those are not sufficiently discussed. If fine-tuned, like LLAVA, how would they do on this task? W1.3 Is visual prompting a good paradigm for keypoint detection? What is the importance of visual prompting for this task? [EDIT: Clarified in post-rebuttal] References: R1: https://arxiv.org/abs/2311.06612 R2: https://arxiv.org/abs/2306.15195 R3: https://openreview.net/forum?id=2msbbX3ydD

Questions

Please address the weaknesses above, which also contain questions.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Official Comment by the Authors of Submission 9290

Dear AC and Reviewers, Thank you for taking the time to review our paper. We are delighted to see that the reviewers recognized the strengths of our work. Specifically, all the reviewers (j2Jb, Hwow, wNmk, 7RyA) acknowledged the great results we achieved. Moreover, Reviewers wNmk and 7RyA appreciated our contributions to advancing the semantic comprehension of MLLMs at a fine-grained level and the adoption of LLMs for in-context learning of keypoint localization. However, the reviewers also raised some concerns. For instance, Reviewer j2Jb questioned the broader impact of our work, while Reviewer Hwow requested more detail on the visual prompt-based keypoint detection. We have provided a detailed rebuttal addressing these concerns and kindly ask the reviewers to consider our responses when convenient. Thank you once again for your valuable feedback. Sincerely, The Authors of Submission 9290

Post rebuttal comments

Thank you for your detailed reply. > Response to Q1 I agree with some of the points and am unclear on others. Even the points that sounds good. E.g., "For instance, semantic keypoint understanding equips MLLMs with enhanced visual comprehension and analytical capabilities, essential for tasks such as structural analysis, action recognition, and medical image interpretation." is not shown in the paper. A stronger submission would indeed do a generic eval to highlight these properties as well. > Response to Q2 I kinda disagree that existing MLLMs cannot be fine-tuned for this task. However, I do not think that is a weakness by itself. MLLMs are very versatile and they could be fine-tuned to a broad range of tasks, that does not, by itself pose a weakness. However, I am worried about a slight overclaim in treating the current work as more than what it does, i.e., enabling a new decoding to take place similar to perception GPT. > Response to Q3 Thank you for your response. I understand that it is indeed an established benchmark and I was able to verify that. So, I stand corrected from my earlier comments. Overall, some of my concerns have been clarified but I think a major rewrite with additional experiments and discussion is needed. So, I will slightly raise my score (also in light of other reviews and responses) but I am still leaning towards a major revision is needed.