Tree of Attributes Prompt Learning for Vision-Language Models

Summary

This paper proposes Tree of Attributes Prompt learning (TAP). Unlike previous works that rely on unstructured class descriptions, this approach distillates structured knowledge graphs associated with class names from LLMs. Text/vision prompts and vision-conditional pooling module are designed to extract instance-specific text features. Extensive experimental results demosntrate its improved performances.

Strengths

- Overall, the idea of distillating structured knowledge from LLMs in the task of prompt learning is new and interesting. - The paper designed an effective prompt learning framework to capture fine-grained attributes, using vision expert tokens and vision-conditional pooling layer. - The illustrated way to generate structure tree of attribute from LLMs can also be used in other tasks. - From the experiments, using structured knowledge leads to better performances than unstructured descriptions in base-to-novel and few-shot classification tasks. - The visualization of class activation maps and attention weights look good. The paper is well written and easy to follow.

Weaknesses

- Apart from the new framework, the method highly relies on the quality of tree of attribute generated with GPT-3.5-turbo. There is no study on the robustness aganist different LLMs, different generation prompts, or varying attribute sets. - The loss includes a model regularization and its effectiveness is not discussed. - In Figure 2, it is not too clear to me about $I_1 T_1$, $I_2 T_2$,etc. They seem not be discussed in the text parts.

Questions

- In Table 6, what is the difference between Attn. Max Pooling and VPC? Is that the former selects one most similar while the latter uses a soft weighted sum? - In Table 5, why using an adaptive number of attributes is better than a fixed 8 number of experts. When increasing the number of experts from 1 to 8, did the authors observe some patterns in the order of added attributes? Say, at the beginning some general attributes, then finer ones, and later irrelevant ones? - In Ln 189, could the authors clarify 'deep prompting'. Does it mean, in additional to vision prompt tokens at the input layer, there are other vision prompt tokens inside vision encoder not plotted in Figure 2? - In Eq. 7, could the authors clarify how $\alpha=0.4$ is chosen, and compare the performances of using CLS token only $\alpha=1.0$? - In Ln 262, how to revert to unstructured set of descriptions? Are they converted from the same tree of attributes by keeping all attributes and descriptions?

Rating

6

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

One limitation is its reliance on LLMs (GPT) to generate the tree of attribute. When generating more complex responses, it is challening to ensure the quality and variances. How to keep a balance between the diversity of attribute sets and relevancy of attributes to classification is important.

Summary

This paper proposes a new method called "Attribute Prompt Learning Tree (TAP)" to improve the performance of CLIP on zero-shot and few-shot classification tasks. The authors leverage large language models (LLMs) to generate more descriptive text prompts and introduce a hierarchical tree-like structure to systematically generate and integrate these descriptions, ensuring a layered and comprehensive understanding of the visual content. The method also learns specialized "domain expert" prompt tokens that focus on different visual attributes and uses a vision-based pooling module to extract text features for specific instances. Extensive experiments show that TAP outperforms state-of-the-art methods on zero-shot and few-shot classification tasks across multiple datasets

Strengths

1), The idea that utilizing LLM to generate tree-like prompts makes sense. This structured description approach is significantly different from the existing simple text prompt methods and provides an efficient way to improve VLMs. 2), The image-conditional pooling module looks like good for capturing instance-specific features. 3), Experiments and visualization demonstrate the effectiveness of the proposed model.

Weaknesses

1), TAP introduces many textual and visual prompts, which leads to high computing and time costs. This may limit its applications. 2), TAP first generates hierarchical token prompts, while it seems like TAP does not use such a hierarchical structure to integrate the output of the text encoder. It only uses a pooling strategy to update the text encoder output with the visual feature. That is, TAP also does not utilize these relationships in the prompt graph. 3), TAP can be viewed as a multimodal prompt tuning method. What is the main difference between TAP and MAPLE, ALIGN.

Questions

Please see above

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

None

Summary

This paper propose a method that aiming to align the vision modality with not only the category name but also the whole concept subgraph the noun represents in the knowledge graph. This is achieved by adding a bunch of attributes branches attached to this concept. The authors argue that this integration of attribute knowledge will make the alignment more transferrable and thus result in a good performance boost in terms of zero/few shot results. Basically, this work focusing on the topic of textual prompt enrichment task that is investigated before but implement in a different manner. Additionally, the proposed method use seperate tokens to learn different aspectrs of attributes of given images, working as 'domain expert'.

Strengths

1. Might be the first work trying to align the vision image with structured data. It is quite interesting considering that most text prompts now are less organized and noisy. And structured data, as pointed out in the recent research of LLM, may lead to better reasoning skill for a foundation model. 2. The proposed vision-conditional pooling can help the model filter out descriptions that are not direct appeared in the image. 3. Recieve good results on different classification datasets with the model trained with this method.

Weaknesses

1. The attributes description is generated by the LLM, which could contain hallucinated content. While there are many reliable sources of knowledge such as wikipedia or conceptNet, this paper seems skip these sources to obtain some accurate attributes. 2. Though this paper decide to use a tree structure to represent the concept. The built tree is not encoded in a structure-awared manner. They are still feed as langauge tokens to the LLMs. 3. in equation (5), what is $v_y^a$ stands for? 4. The author argued that the vision-conditional pooling, which is bascially a cross attention layer between the visual and language modal. The authors believe this this design will make the model filter out non-exisiting material in the text description. However, we know that due to the quirk of softmax function. You can never make some tokens attention to be '0'. Thus, the model is learning some spurious correlation aftertall.

Questions

1. Why structured description is so important in your presumption? Given that only the text prompts are structured but visual data are not, will this fact hinder the model to learn a structured in-detail alignment? 2. How do you make sure one expert token will only learn from one attribute? 3. Can the model trained this way also work well on the downstream tasks?

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

Not applicable.

Summary

The TAP method structures textual descriptions in a hierarchical “concept-attribute-description” format, effectively creating a knowledge graph from large language models (LLMs) for each category name. This structure allows for a more comprehensive and detailed understanding of the visual content. The paper reimagines learnable prompt tokens as "domain experts," each specializing in different aspects of the image, supplemented by a global perspective provided by the CLS token. To address potential misalignment between general descriptions and specific image content, the paper introduces a vision-conditional pooling module. This module extracts instance-specific text features, ensuring optimal image-text alignment.

Strengths

The proposed method incorporates structured tree of attribute into prompt tuning that provide richer supervisory information compared to unstructured attribute information. A set of experiments has been conducted, and the results look promising.

Weaknesses

One major limitation of the method is that it requires human review to "ensure the quality of the example" (L175). Recall that one major advantage of prompt tuning is that it can adapt large models quickly to specific tasks. However, the requirement of human reviewing in the proposed method is not consistent with this goal. In addition, it is not clear how many human efforts are needed here, and how to handle the potential human bias in quality evaluation. The paper lacks cross-dataset experiments, which is typically provided in existing PT papers. The results are important to examine the domain generalization capability of the method. For training details, different learning rates were used for different datasets, however, existing methods typically use a same LR for all datasets. From this point, the comparison is somewhat unfair.

Questions

In Section 3.3, for attribute generation, what type of dataset information is given to the large model? In Figure 4, each image is accompanied by only two descriptions. Are all images described using two sentences each? In this paper, it mentions that the method can capture subtle differences between attributes. Could you provide a relevant example?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

yes

Summary

This paper proposes a new prompt tuning method for adapting the vision-language model. The authors design the tree of attribute prompt learning to substitute the categorical description for adapting the vision-language model. A vision-conditional pooling module is proposed to extract instance-specific text features. Extensive experimental results demonstrate the effectiveness of the proposed method.

Strengths

1. A tree of attribute prompt learning method is proposed to guide the adatpation of VLM with the hierarchical semantic information. 2. This paper is well-written and easy to follow.

Weaknesses

1. According to the experiment, the performance improvement of TAP is marginal, e.g., the few-shot performance on most of datasets. Although the visualization results of VCP layer are impressive, the improvement of this module is also very slight compared to average pooling. 2. The core motivation of this method is learning fine-grained attributes to adapt VLMs. However, similar ideas have been explored in previous works , e.g., APPL[1], MAP[2]. Please discuss the differences. [1] AAPL: Adding Attributes to Prompt Learning for Vision-Language Models [2] Multi-modal Attribute Prompting for Vision-Language Models 3. The construction of ToA depends heavily on the prior information on the category of attributes suitable for the dataset. However, one of the most capability of VLM is its zero-shot ability in the open-vocabulary context. What's the performance of the proposed method in the domain generalization setting? 4. The model details in Figure 2 are not presented very clear, especially the input & output streams. This figure should be refined for better clarity. 5. The mechanism behind Equation (5) and the function of VCP needs more clarification. Why conduct constrastive learning between expert token P_a^v and attribute embedding v_c^a generated from P_a^v itself, instead of P_a^v and the embedding of attribute descriptions D?

Questions

Please refer to Weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes.

Reminder -- please reply to rebuttal

Dear Reviewers, Thank you for your comments. Please read through all the reviews and the rebuttal and see if authors' responses have addressed your and others' concerns. Best, AC

Reply to rebuttal

The responses have addressed all my questions.

Please respond to the authors' rebuttal

Dear Reviewer ZE3g, Thank you again for reviewing this paper. Since the reviewer-author discussion phase is closing soon, could you please respond to the authors' comments? Best, AC

Thank you

Thank you for your recognition of our work and for increasing your rating. Regarding your question about the efficiency of the iterative training approach: To improve efficiency, we used einops rearrange to manage multiple classes and batch sizes, along with zero-padding and attention masks to handle varying numbers of descriptions across attributes. We note that these implementations can already maintain a high training effiency. Thank you for your nice suggestion. We will certainly further explore more strategies to improve training efficiency. Thank you again for your valuable feedback and for the increase in your rating.

Reply to rebuttal

Thanks for the rebuttal. It addresses some of my concerns but one major concern still exists. One limitation of the method is that it requires many tunings specific to each dataset in order to achieve good performance. The tuning process involves two key aspects: __Human Intervention (W1)__: thought the required human effort is minimal, its impact on model performance remains unclear after the rebuttal. __Learning Rate Variability (W3)__: The method employs different learning rates for different datasets. While TaskRes is trained in a similar manner, the experiments lack comparative analysis with this approach. Additionally, further discussion is needed on how to determine the most appropriate learning rate for a given dataset. I expect authors provide additional insights regarding this issue.