TransHP: Image Classification with Hierarchical Prompting

Summary

This paper studies the problem of image classification leveraging the idea of hierarchical image classification (HIC), which exploits semantic relations across target classes to learn meaningful distinctive features. The core idea of hierarchical classification is that if a model should classify plants and it knows that two species of flowers are from the “rose” family, it can focus on determining features that characterize the two classes instead of looking for more general features that help it associate the classes to the flowers category. The authors consider visual transformers as models and introduce a new “prompting block” that aims at modeling the coarse class to which target classes belong so that the features extracted by the transformer are dynamic with respect to this task-and-image-specific information. At an intermediate layer of the transformer, a set of learnable vectors, i.e., prompt tokens which correspond to the number of coarse classes, is prepended to the feature tokens of an image (composed of the class token and the patch tokens). The transformed version of prompt tokens is used to compute the similarity between the tokens and learnable prototypes of the coarse classes via a cross-entropy loss. In this setting, the prompting block explicitly learns the coarse-class prompts while learning to predict the coarse class of the input image. Experiments demonstrate that training transformers with the prompting block(s) improve performances on tasks where a hierarchy can be defined (e.g., Imagenet, iNaturalist). An ablation study shows that embedding the information of the coarse classes without coarse class prompts already helps to improve the results, but prompting does it even more. Compared to the baselines, the methods are more data-efficient. Moreover, a qualitative study shows that the methods put attention to more distinctive features of the images.

Strengths

1. The paper is easy to follow and supports the method explanation with good visualizations. 2. The experiments support the validity of the method and more importantly the claims of the authors 3. The results confirm the intuition of the authors about the capability of focusing on more discriminant features

Weaknesses

1. Although the method consistently works on the tested tasks, what are the characteristics of the tasks that let us understand that the method generalizes? Number of classes? Number of added prompt blocks (determined by the number of hierarchical levels)? Or other characteristics? 2. The application of the method is limited to the cases where we have access to a hierarchy among target tasks. In the question section, I ask if we can show that the method allows us to learn better pre-training embeddings that lead to improvements on other downstream tasks.

Questions

1. The authors explain that for tasks with multiple hierarchical levels, they add a prompting block for each of them. Do you think it is necessary to exploit all the hierarchical levels? Or maybe the more granular coarse levels are enough for better conditioning? 2. By construction, the method it’s limited to cases where one is able to build a hierarchical relation across target tasks. However, training ViT on Imagenet with the hierarchical information added via prompting block might help in learning a space that generalizes better than standard ViT. Did you try to use TransHP pre-trained on Imagenet as the backbone to finetune with a few labels on other downstream tasks? Maybe fine-grained?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations were mentioned and I asked follow-up questions in a previous section.

Summary

This paper aims to improve image classification accuracy by introducing hierarchical prompts. For a dataset with L hierarchy with M prompt at each level, the author proposed to inserting M prompt at each of the L randomly selected transformer layers. Each prompt is tasked to predict the intermediate coarse label in the hierarchy using a supervised manner. While the proposed idea is interesting, its scalability and novelty are limited. See weakness for more details.

Strengths

1. Figure 4 highlights the difference between the proposed method and the baselines. 2. Figure 2 clearly summarizes the high-level idea of the paper. 3. The paper is simple to follow.

Weaknesses

Major 1. The proposed method inserted M prompt in the lth prompt block (See Eq5 and Eq6). Since each prompt represents a coarse class (L129-L131) at the lth layer of the hierarchy, the underlying assumption is that the hierarchy is a balanced tree (e.g. each level of the hierarchy has M coarse classes). What will happen if the hierarchy is imbalanced? Will the proposed method still work? The author is suggested to conduct experiments on a dataset that has an imbalanced hierarchy. 2. L103-104 is confusing. Does the model optimize the entire backbone model (e.g. model parameter other than the prompt) + prompt? Or just the prompt? 3. Caption of Figure 2. As shown in Eq2, the prompt is appended after the class and patch token. This contradicts the caption of Figure 2, which claims “the prompting block pre-pends the whole prompt pool consisting of M prompts”. 4. It is unclear how the proposed method TransHP selects an intermediate transformer block for inserting the prompt (L127). As mentioned by the author in L159-L160, there is no explicit guidance on how to select the block for prompt insertion. Furthermore, there is no discussion for the case where the level of dataset hierarchy exceeds that of the transformer blocks. This indicates that the proposed method is not applicable to deep hierarchy. 5. Figure 4 is a bit misleading and there are some missing baselines (L221). For example, one could insert prompts (purple dots in figure 4) and do coarse classification using the [CLS] token. Is the proposed method better than this baseline? 6. The caption of Figure 3 mentioned that the prompt is autonomously selected. However, the Eq5 is nothing but a cross-entropy loss. There is nothing to be selected. 7. How is the proposed method different from a simple baseline that combines HD-CNN [a] into the transformer architecture? 8. The author is suggested to compare with VPT. [a] HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition

Questions

The author is suggested to address the concerns in the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitation is mentioned in the end of the paper.

Summary

The paper proposes to use coarse token prompting for the task of image classification. The basic idea is to add coarse label tokens at intermediate layers of ViT and add additional coarse classification loss at intermediate level. It in some way correlates to convolutional networks based Hierarchical classification which incorporate additional branches for coarse level classification at intermediate levels. However, the idea is applied through prompting and the authors claim that this is the first paper to apply prompting for the task of hierarchical classification.

Strengths

- The idea of coarse label prompting appears novel - Apart from some missing details, the paper is well written and the general idea is easy to comprehend - The results show that proposed hierarchical promoting leads to consistent gains across ImageNet, iNaturalist2018/19, CIFAR 100 datasets.

Weaknesses

- Related work is not well covered. For instance, there are ideas from prior art which could be applied without much difficulty. For instance, [A] uses marginalization over the predicted fine grained labels to obtain coarse level predictions and applies an additional loss there. [B] simply trains two networks (one for each hierarchy and makes a post hoc correction). Comparison with atleast peer reviewer papers like [A] is warranted. Also a flavour of how these different papers have tackled the problem is worthwhile. Authors do mention that do not compare again convolutional methods, however, giving a general idea on branching and contrasting it with prompting will add value to the paper. - Many details are either not clear or missing. (a) For example, where exactly do they add the prompting, the exact location has to be given. Especially when you use multiple hierarchies, the locations of prompting blocks need to be clarified. Line 159, mentions that they do not have an exact position scheme and lower levels are better. However, they would have used some schema and the authors should mention that? (b) How many hierarchies are used in the main paper results. It should be clearly mentioned. I could gather that for iNaturalist only two levels are used, however, there are no detail regarding other datasets. (c) iNaturalist19 in my knowledge has 7 levels of hierarchy (3-kingdom, 4 phylum, 9-class, 34-order, 57-family, 72-genus, 1010 -species). In supplementary the authors mention to use 6 levels for genus and 1010 for species. Where did the 6 come from? (d) In Table5, supplementary material, how did you get level 11 for iNaturalist dataset? Is the the block number or level number in the Table? - Is it meaningful to do Ablation 2? Thats like sending some random vectors and checking if that helps. Not sure, if that is worthwhile to show. - I could not fully understand the use of absorption weights in equation 7. Is it just used for explainability and statistics (Figure 2) for illustrating how the model is behaving or is it used while training? Please clarify. - Another variation of your method could be to predict something like P^hat_{cls} and use that for coarse prediction. Did authors try this variation, reducing the need of the learnable prototypes w_i. Please clarify. - Other ablations like changing the position of the prompting level or varying the number of hierarchies used, would be valuable. What if only the bottom two hierarchies are used (leaf and just above). Compare that against full hierarchy. [A] Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. arXiv:2111.11595, 2021 [B] Kanishk Jain, Shyamgopal Karthik and Vineet Gandhi. Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification arXiv 2023

Questions

- Please provide clarifications on missing details, as mentioned in limitations sections. Clarify on other points raised in the previous section as well.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Task level limitations are mentioned. However, there is nothing which reflects on the functioning of the proposed framework.

Summary

This paper introduces a new approach called hierarchical prompting for hierarchical image classification (HIC). It's incorporated into a model named TransHP, which uses broader class 'prompts' to better distinguish between similar classes. The process improves image classification accuracy, data training efficiency, and model explainability. For example, it enhanced ViT-B/16's ImageNet classification accuracy by 2.83%.

Strengths

1. The innovative concept of utilizing additive tokens to garner coarse class data, aimed at enhancing fine visual recognition, holds great potential. 2. The methodology proposed in this paper appears to be a valuable tool in understanding and interpreting the ViT backbone's functioning and structure.

Weaknesses

1. The foundation established in this paper seems to lack strength. I recommend utilizing a more robust baseline within the High Interaction Control (HIC) domain. The chosen backbone, a lightweight Vision Transformer (ViT), does not exhibit wide applicability. It would enhance the argument's validity and strength if the authors considered employing the standard ViT-B as a baseline. 2. The diagram in Figure 4 implies that the bulk of the performance improvement arises from the coarse loss. This renders the necessity of prompt introductions potentially insignificant, as they contribute to less than a single point in improvements. Therefore, the authors should provide a deeper insight into the rationale behind incorporating prompts.

Questions

1. In the method part, what does "soft weighting" mean? If I understand correctly, soft weighting happens without learning any special weighting parameters, instead it means the attention value toward "correct" course prompt increases along the training more epochs.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. There appear to be certain limitations to the methods employed, particularly for datasets devoid of HIC structures. Understanding the process of implementing these methods on such datasets would be enlightening.

Summary

This works presents a novel hierarchical prompting mechanism for hierarchical image classification, named TransHP. In TransHP, a set of prompt tokens are learnt to represent coarse classes and was injected in the prompting block for coarse class prediction, and the injected prompt token can strengthen the feature. The proposed method was evaluated in several benchmarks and show consistently improvements compared with baseline model.

Strengths

- This work is well-written and well-organized, and the proposed method is simple and easy to implement. - Good performance compared to previous methods and baseline model. - Visualization in Fig. 5 gives intuition to understand the proposed method. - Studies on data-scarce scenario is good.

Weaknesses

- The ImageNet performance of HiMulConE used in this work is even worse than the official paper that adopt ResNet-50. - Lacking intuitive explanation on the data efficiency of the proposed method. - Table 4 shows that in data-scarce senario, the performance of all methods decreases significantly, and the results suggest that TransHP has better performance. However, I wonder if the settings used in this work is too weak? This is because recent semi-supervised work has shown that very high accuracy can be achieved with only 10% of ImageNet data, and some self-supervised pre-training methods (\eg MIM) can also alleviate the problem of data scarcity. - The main experiments are conducts on a lightweight ViT, which is a weak baseline. It would be best to conduct more experiments on a stronger backbone & pre-train models.

Questions

I hope the author could address my questions in the weakness.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes. The author adequately addressed the limitations.

Post rebuttal comment

I have carefully read the author rebuttal. The authors do address some of my comments in a satisfactory manner, hence I am increasing my rating to borderline accept. It appears that the method gives similar performance without the learnable prototypes, that does raise a concern about unnecessary complexity. I would expect that the authors will compute and update all tables without learnable prototypes (directly predicting P^hat_{cls} and using that for coarse prediction). I also suggest the authors to add discussion with other methods like [A] and [B]. Also requesting for some other clarifications: - Are you using only two hierarchies in the iNaturalist dataset? "Amphibians, birds, fungi, insects, plants, and reptiles" appear to be coarse labels not hierarchy. - The prompting with multiple Hierarchies is not entirely clear. Can you expand on that aspect a bit, while clearly mentioning the promoting positions.