Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification

Summary

The paper considers the problem of fine-grained classification and proposes to use label hierarchy information at test time to improve the performance of the fine-grained classifier. The overarching goal is to improve the top-1 accuracy while at the same time reducing the severity of the mistakes (e.g., misclassifying species from the same kind or family is more acceptable than making a totally unrelated prediction). The paper presents a post-hoc correction technique called Hierarchical Ensembles (HiE). The key idea is to train 2 classifiers - 1 fine-grained and 1 coarse-grained. At inference time, the predictions of the coarse-grained classifier are used to re-weight the probabilities of the fine-grained one. The authors perform experiments on 2 standard benchmark datasets: iNaturalist-19 and tieredImageNet-H and claim to achieve new state-of-the-art results on both of them. They also show the promise of their method in a semi supervised learning setup where the coarse grained labels are known for all samples in the dataset but the fine-grained (being more expensive to collect) only for few samples per class.

Strengths

* The paper is written well and is easy to read. * The paper extends and builds up on prior works and seems to exceed their results. * The Semi Supervised Learning setup is interesting. * The conducted experiments and ablations are sensible.

Weaknesses

My main concern is with the methodology (Section 3). I understand the final decision rule (Eqs. (2) and (3)) but its derivation and motivation is confusing to me. L118: "Assuming conditional independence between the estimated logits...": $i ⫫ i_{parent} \mid x$ means $P(i, i_{parent} \mid x) = P(i \mid x)P(i_{parent} \mid x)$. Thus, to the best of my knowledge, the denominator in Eq. (1) should not be there. Moreover, I believe that the conditional independence assumption is unrealistic for the considered setup. It does not feel right that the fine-grained prediction/probability is independent of its parent. This affects the derivations in the rest of the section. Minor: * The following related, in my opinion, citation is missing:\ Ridnik et al., ImageNet-21K Pretraining for the Masses, NeurIPS 2021 (Datasets and Benchmarks)\ They train ImageNet-21K classifiers using the WordNet hierarchy. * For deep hierarchies, it may still be costly to collect labels from the penultimate hierarchy level. It would be useful if the authors include the set of labels from the last two levels for each of the datasets and refer to them from the main paper.

Questions

Q1: Could you please clarify how you formally derive the intuition for the decision rule defined in Eqs. (2) and (3)? Q2: Would it help if you use the logits from $\hat{y}_{L}$ and $\hat{y}_{L-1}$ directly instead of the softmaxed probabilities in $Q$ and $R$? Q3: To complete the discussion from L143-147, you can also consider the case when the fine-grained prediction is correct but the coarse-level prediction is wrong. Q4: I am curious how often there is a mismatch between the fine-grained and the coarse classifier. And what is the distribution / the average of the depth of their LCA nodes. Q5: L220-221: "... training a separate network at the coarse label allows explicit disentanglement of features at the coarse and fine-grained levels ...". I don't believe that the current set of experiments is sufficient to make such a strong claim. Q6: Does the $\pm$ in the tables refer to standard deviation? How many runs were executed? Q7: Sec. 4.4: Following the comments in the Weaknesses section above, it might be useful to formally clarify what is meant by "cascade the predictions top-down" (L279-280). Also, do you need to incorporate consecutive hierarchical levels (e.g., would it be possible and sufficient to take levels 5 and 7, omitting level 6)? Q8: Sec 5: What do preliminary experiments suggest if you unify the architecture of the fine-grained and the coarse-grained classifier (and, e.g., only fine-tune separate classification heads for the two classifiers)?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This method introduces a novel approach to achieve state-of-the-art fine-grained image classification. It addresses the issue of mistake severity by developing the Hierarchical Ensemble (HiE) loss, which effectively penalizes incorrect predictions of both course and fine-grained labels. The HiE loss combines the probabilities of predicting the course label and the fine-grained label, providing a joint probability measure. The authors of this method provide a proof that accurately predicting the course label probability enhances overall accuracy. To validate their approach, they conducted extensive experimentation on two hierarchical datasets, comparing their method against various existing related works and three additional baselines. The evaluation metrics used include top-1 error, mistake severity, and hierarchical distance. The results clearly demonstrate that their method outperforms comparable approaches across all these measures. Moreover, the authors conducted further experiments in a semi-supervised setting to assess the performance of their method when only 10% of labels are available. Remarkably, their approach exhibits significant performance improvements even under such limited label availability, highlighting its effectiveness and robustness. In addition to the aforementioned experiments, the authors explored the impact of using different pretrained backbone models in their method. They conducted experiments with various hierarchical depths to evaluate the method's adaptability and generalization across different classification hierarchies. The results of these additional experiments further support the superiority and versatility of their proposed method. Overall, this research presents a compelling method for fine-grained image classification, showcasing its state-of-the-art performance, reduced mistake severity, and its ability to improve accuracy even with limited labeled data. The comprehensive experimentation and analysis conducted by the authors demonstrate the effectiveness and versatility of their approach in various scenarios, making it a valuable contribution to the field of image classification.

Strengths

The text is very well written and easy to understand. It’s very clear what the problem is and how their method can be used to increase performance. The authors conducted a comprehensive range of experiments to thoroughly evaluate the performance of their model. The method demonstrates robustness by effectively utilizing any pretrained backbone model.

Weaknesses

Since I’m not as familiar with this method, it would have been helpful to have a clearer description of these metrics within the paper, as I had to refer to the related works to gain a better understanding.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I believe the authors appropriately discussed the limitations of their method. The collection of course-grained labels in addition to fine-grained labels seems very costly but this is outside of the scope of this method.

Summary

This paper proposes a novel approach called Hierarchical Ensembles (HiE) to improve the performance of fine-grained classification by utilizing a label hierarchy and coarse-grained predictions at test-time. The method significantly reduces mistake severity while improving top-1 accuracy on benchmark datasets, achieving state-of-the-art results. The approach is also effective in the semi-supervised setting, bringing notable gains in accuracy and reducing mistake severity as training data decreases for fine-grained classes.

Strengths

 Originality: The paper introduces a novel approach called Hierarchical Ensembles (HiE) that combines coarse-grained predictions and label hierarchy to improve the performance of fine-grained classification. This approach is unique and addresses the challenge of reducing mistake severity while improving accuracy in fine-grained classification. Therefore, the paper demonstrates originality in its proposed methodology.  Quality: The paper achieves state-of-the-art results on benchmark datasets by significantly reducing mistake severity and improving top-1 accuracy. The approach is effective not only in the supervised setting but also in the semi-supervised setting, bringing notable gains in accuracy. The paper also compares its approach with previous baselines and demonstrates superior performance. These factors indicate the high quality of the research presented in the paper.  Clarity: The paper provides a clear and concise explanation of the proposed approach, including the motivation, methodology, and experimental results. The authors effectively communicate the problem statement, the significance of their approach, and the experimental setup. The paper also includes figures and examples to enhance clarity. Overall, the paper is well-written and easy to understand.  Significance: The paper addresses the challenge of fine-grained classification, which requires domain expertise and large amounts of labeled data. By utilizing coarse-grained predictions and label hierarchy, the proposed approach significantly reduces mistake severity and improves accuracy. This has practical implications in various domains where fine-grained classification is important, such as image recognition and object detection. The paper's state-of-the-art results and compatibility with existing semi-supervised methods further highlight its significance.

Weaknesses

There are some problems, which must be solved before it is considered for publication. If the following problems are well-addressed, this reviewer believes that the essential contribution of this paper are important for fine-grained classification. The paper has some context inconsistency errors, for example, it refers to these semi-supervised models on line 233, but then refers to them as self-supervised models on lines 242 and 244. In addition, although the complementary method proposed in this paper can be applied to the off-the-shelf model, the overall innovation of the paper is insufficient.

Questions

This paper makes extensive and comprehensive experiments on the new method proposed by the author, which fully proves the effectiveness of the method. The innovations of the paper can be applied to many off-the-shelf models, but there are not enough of them. If more general structures could be proposed, this paper would be able to make a greater contribution to the field dealt with.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately address their proposed limitations in previous models, as their proposed method is a complementary structure that can be applied to any existing model and can effectively improve the performance of the model on the dataset.

Summary

This paper focuses on label hierarchy problems and proposes using Hierarchical Ensembles (HiE) of independently trained networks over coarse and fine-grained levels. The reported experimental results show that the proposed method can achieve comparable performance to a fully supervised baseline, even using merely 10 annotations for each fine-grained class on a large fine-grained image classification dataset encompassing 1010 classes.

Strengths

1. The topic of this paper is well-introduced and the proposed approach is straightforward. 2. The proposed HiE utilizes label hierarchy to improve the performance of fine-grained classification at test time using coarse-grained predictions. 3. The proposed methods and experimental results are well presented and the manuscript is overall well organized. 4. Performance of the proposed method is promising according to the comparison experiments with other state-of-the-art studies.

Weaknesses

1. The paper lacks a general investigation of hierarchical architectures and hierarchical embeddings. 2. The proof of the Theorem 3.1 is not convincing, thus making the overall scheme lack theoretical support. 3. The experimental setup of semi-supervised learning is not shown clearly, so it is not possible to judge whether the comparison of the proposed method with other methods is fair. 4. Some errors, such as why Figure 2 is 4 levels.

Questions

Please give a detailed motivation for this paper and a detailed derivation of Theorem 3.1

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The subject of the paper is Test-Time Amendment, thus limiting the potential performance improvement.