Summary
The paper introduces a approach called Dual-stage Post-hoc Logit Adjustments (DPLA) to tackle the challenges in Realistic Open-world Long-tailed Semi-supervised Learning (ROLSSL). DPLA is a two-stage logit adjustment technique designed to improve model performance on both frequent and infrequent (tail) classes. In the first stage, logits are adjusted based on the sample frequency and class count, which helps balance predictions across known classes. In the second stage, adjustments are dynamically refined using the predicted class distribution from unlabeled data, with a particular focus on emphasizing tail classes to counter bias towards dominant classes. This dual-stage approach enables better recognition in long-tailed, semi-supervised, and open-world settings, especially where labeled data is sparse and imbalanced. Key experimental results highlight that the DPLA approach consistently surpasses the OpenLDN baseline in both known and novel class recognition, achieving up to 50.1% improvement experiments on datasets such as CIFAR100 and ImageNet100. They also show the robustness and adaptability of the proposed method across different data imbalance conditions.
Strengths
1. The problem addressed by this paper is highly practical, as determining the distributional relationships between known and novel categories is indeed nearly impossible in real-world scenarios.
2. The paper is well-written and easy to follow.
3. Key experimental results demonstrate that the DPLA approach consistently outperforms the OpenLDN baseline in both known and novel class recognition, achieving up to a 50.1% improvement on datasets like CIFAR-100 and ImageNet-100. Authors further showcase the robustness and adaptability of the proposed method across varying levels of data imbalance.
Weaknesses
1. A key concern is the complexity of the proposed pipeline, which involves multiple stages and numerous hyper-parameters. With additional stages and hyper-parameters to tune, it becomes more challenging to generalize this pipeline to new problem settings and datasets. For instance, as shown in Tables 3 and 4, using non-optimal scaling factors can significantly impact performance on CIFAR-100 and ImageNet-100, despite both datasets having the same number of classes. Have you explored methods to automatically tune the hyperparameters or analyzed which parameters are most sensitive?
2. Technical contributions. A primary contribution of this paper is the proposed logit adjustment method for pseudo-labels, which often exhibit class imbalance. However, the issue of naturally imbalanced pseudo-labels has been previously addressed by DebiasPL [1], which also applies a logit adjustment based technique to mitigate class imbalance in pseudo-labels, resulting in improved classification performance on long-tailed datasets. A discussion on the technical distinctions between this approach and DebiasPL would be beneficial.
3. While this paper presents experiments on several benchmarks to demonstrate performance gains, most datasets used are small to medium-scale, such as ImageNet-100 or CIFAR. I believe it is essential to include experiments on larger datasets, such as iNaturalist or ImageNet-1k or ImageNet-22k, to better assess the generalizability of the proposed method. It will be nice if you could provide any potential challenges or modifications needed to apply your method to larger datasets.
4. In addition to presenting results on novel and known subsets, I believe it is essential to include performance metrics for many-shot, medium-shot, and few-shot subsets. This would help clarify whether the current logit adjustment methods compromise performance on many-shot classes to improve results on few-shot classes—a common issue with most logit adjustment techniques. It would be valuable for the authors to discuss whether their proposed method faces this challenge and, if not, to explain why it avoids this problem. If it does, providing insights into potential strategies to address this limitation in future work would be beneficial.
5. I may suggest authors to include a specific breakdown of performance metrics for these subsets in their results tables. Add a discussion section that explicitly addresses the trade-offs between performance on different shot categories and potential strategies to balance these trade-offs. In my opinion, while improving performance on few-shot classes is important, it is equally crucial to avoid jeopardizing performance on many-shot classes, as they are more prevalent in real-world applications.
[1] Wang, Xudong, Zhirong Wu, Long Lian, and Stella X. Yu. "Debiased learning from naturally imbalanced pseudo-labels." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657. 2022.