Summary
This paper proposes to hierarchically generate the label-specific stochastic prompts using generative modules from sampled noisy latent vector. Then, a conditional transport framework is employed to establish a relationship between visual patches and textual prompts. Several experimens are performed in few-shot, transfer learning, domain generalization, and base-2-new manners using ViT-B/16 and RN50 as the backbones.
Weaknesses
1. In Fig.3, the reported PLOT using ViT-B/16 is run by this submission. However, this result is much different from the one reported by PLOT on github (https://github.com/CHENGY12/PLOT/tree/main/plot-pp). According to these results, PLOT achieves better few-shot performance when using the ViT-B/16 as the visual backbone.
2. The PLOT base to new experiment using ViT-B/16 reproduced in this paper also lacks credibility, considering that the performance of the proposed method and PLOT are comparable when similar experiments using RN50 as the visual backbone are performed.
3. In my view, the primary reference for this paper is PLOT, and therefore it needs to be compared to PLOT as exhaustively as possible. However, this paper lacks some important comparisons. For example, PLOT mainly employs RN50 as the visual backbone, although this paper has added experiments in few-shot and base-2-new manners using RN50 as the backbones, the domain generalization experiments using RN50 are missing.
4. I appreciate the proposed Stochastic Prompts Generation, however, the conditional transport seems not meaningful. In the main text, the author only claims that OT needs two stages. However, the first stage of OT is not time-costly in the CoOp-related experiments. So, what is the main contribution of using CT instead of OT?
5. The paper lacks an ablation study on the proposed conditional transport and optimal transport (OT). We need to compare experiments using SPG and CT with experiments using SPG and OT to determine whether the proposed CT is meaningful.
6. In PLOT, the number of prompts is set to 4. However, this paper only uses C for the number in Eq. (4) without stating the exact value in the experimental details, which may result in unfair comparisons. I also find that the Moter Carlo sampling number is set to 20 as the default setting. Does this Moter Carlo sampling number correspond to the number of PLOT prompts? If yes, this is unfair, please conduct fair experiments and explain the reason.
7. The learnable parameters shown in Table C.9 indicate that the proposed approach uses much more parmas compared with CoCoOp and PLOT. I would like to know the composition of these parameters and whether the additional parameters rather than the suggested method are the reason for the performance improvement.
8. Multi-modal approaches such as CoPrompt[1], MaPLE[2], and VioLET[3] can achieve much better base-to-new performacne using ViT-B/16 as the visual backbone. I understand that the proposed approach only tunes the language branch, however, I wonder whether the proposed approach can further improve the multi-modal approaches.
9. From the ablation studies in Table.1, P-Prompt shows better performance compared to B-Prompt, while in Figure.7(a), B-Prompt exhibits better few-shot performance. These results indicate that the proposed CT is useful for generalization while SPG accounts for better supervised performance. This may not be intuitive, as the proposed Bayesian approach is more capable of introducing uncertainty thereby enhancing generalization performance and reducing overfitting (also mentioned in Sec. 2.2).
[1] Roy, Shuvendu, and Ali Etemad. "Consistency-guided Prompt Learning for Vision-Language Models." arXiv preprint arXiv:2306.01195 (2023).
[2] Khattak, Muhammad Uzair, et al. "Maple: Multi-modal prompt learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[3] Wang Y, Liu Y, Zhang X, et al. VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 4595-4605.
Questions
The proposed method claims to be generalizable and can solve the overfitting problem well. Then I am curious, when we increase the number of training epochs to 50 or even 200 in base-to-new experiments, does it affect the generalization performance?