Summary
The paper addresses the challenge of optimizing text prompts for vision-language models, specifically focusing on the interpretability of these prompts. Traditional methods for prompt optimization rely on gradient descent, which often results in overfitting and produces prompts that are not human-readable. This paper introduces a new approach called Interpretable Prompt Optimizer (IPO), which leverages large language models (LLMs) to generate and optimize prompts in a dynamic and interpretable manner. The paper details the design of the IPO framework and provides extensive experimental results across 11 datasets. The findings demonstrate that IPO not only improves the accuracy of vision-language models compared to traditional gradient-based methods but also significantly enhances the interpretability of the generated prompts.
Strengths
1. The paper introduces IPO, a novel method that uses LLMs to optimize prompts in a way that maintains human readability and interpretability. This approach contrasts with traditional gradient-based methods that often produce opaque prompts.
2. The POP system stores past prompts and their performance metrics, allowing LLMs to generate more effective prompts through iterative refinement. This system enhances the contextual understanding and effectiveness of the generated prompts.
3. The IPO method incorporates LMMs to generate image descriptions, which improves the synergy between textual and visual data. This leads to more accurate and contextually relevant prompts.
4. The paper validates the effectiveness of IPO across 11 different datasets, demonstrating its superiority over traditional methods in terms of both accuracy and interpretability.
Weaknesses
The main weaknesses of this paper lies in the experiment can not support the effectiveness of the LLM optimizer, which is the most important contribution of the authods. IPO incorporates the Large Language Model (LLM) as an optimizer to learn an interpretable prompt pool, and everytime a new instance comes, prompts will be extracted from this prompt. This mainstream is similari to knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. Therefore, the experiments should contains:
1) **A fair comparitive with some of the knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. To exclude the effectivness of the memory retrieval mechanism of IPO.**
2) **The current comparitive experiment merely involves the 1-shot setting in Table 5. Please provides a more comprehensive experiment settings, such as 16-shots and full-sized tuning setting. Besides, the IPO methods do not draw an obvious performance improvement in benmark datasets in the case that IPO has an obviously longer prompt that the learnable method.**
3) What's more, the IPO does not perform well in the large-scale generic datasets such as ImageNet. This is not intuitive, because the LLM should show a better performance in tghe generic datasets. Lastly, I am not agree with your average metric, because the data size of these 11 benchmarks have significant difference.
4) Table 6 are not referred in the author's paper, and the detailed benchmark is also not elaborated.
[1] Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149.
[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663.
[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.
Questions
Authors explor using LLM as an interpretable optimizer for prompt learning, I think this is a contribution worth a weekly accept score, i.e., 6 scores in NeurIPS rating. However, this score should be build upon all below concers are addressed:
1) I can tolerate that the current performance are not that remarkable, but this tolerance should be built upon the experiments are fair. To make a fair comparison in the framework, authors should make comparison with some of the knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. So that we can exculde the effectivness of the memory retrieval mechanism of IPO.
2) Why Table 5 only contains the 1-shot comparive experiments ? Detailed 16-shots or training with more samples setting are needed.
I will confirm the effectivness of the your IPO if you draw a remarkable performance. If not, just take a breath and provide enough evidence that LLM optimizer is effectiveness, such as sound losses, sound improvement in the training set, sound improvement in the test sets in different settings. I will agrre with the effectivness of the your IPO if the evidence are sound.
[1] Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149.
[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663.
[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.
Limitations
Experiments now are not sound and the authors do not provide enough evidnce to prove the effectiveness of IPO.