Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

Summary

The paper presents an approach to optimize hard text prompts for generative models through efficient gradient-based optimization. The method automatically generates hard text-based prompts for both text-to-image and text-to-text applications, allowing users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model. The paper highlights the advantages of hard prompts over soft prompts and demonstrates the effectiveness of the approach in tuning language models for classification.

Strengths

1. The paper's approach is efficient and can be optimized on smaller language models and then transferred to other, much larger models. 2. The method performs consistently across all four datasets and outperforms other gradient-based optimization baselines. It can achieve similar performance to CLIP Interrogator. However, the proposed method only uses the CLIP model for prompt discovery and 8 tokens in total demonstrating its simultaneous simplicity and strength. 3. The proposed approach can also be easily adapted to style transfer. Given several examples that share the same style, the method can extract their shared style characteristics into a single hard prompt and use this prompt to apply the style to new objects or scenes (Page 5).

Weaknesses

1. While the method can generate relevant tokens and phrases, the prompts are not always coherent English. This suggests that while the optimization process is finding relevant words to the task, it lacks the ability to create full sentences. 2. Longer prompts do not necessarily produce better results when generating with Stable Diffusion, even though they strictly reduce loss on the CLIP image encoder. Long prompts thus overfit and are less transferable. Thus, it may require hyperparameter tuning, which may heavy and tedious.

Questions

You mentioned that longer prompts do not necessarily produce better results and can lead to overfitting. Could you elaborate on the mechanisms behind this phenomenon and suggest potential strategies to mitigate this issue while maintaining the quality of the generated prompts?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

As the author has discussed, the generated prompts may still contain several un-interpretable tokens. Also, it may lack the ability to create full sentences.

Summary

The paper presents an easy-to-use approach to automatically obtain hard prompts for images. The work introduces PEZ, a gradient-based approach to obtain hard prompts for images. The experiments comparing a popular baseline show improved CLIP score. Finally, the qualitative results show that the method can distill prompts into shorter sequences and concatenates

Strengths

The paper is easy to follow. The algorithm is well explained and the results are quite promising. The results for prompt inversion appear to be fair and outperform the baseline. The analysis is quite clear and thorough. Overall, the work presents a straightforward method to get prompt tokens for images.

Weaknesses

The contribution of the work is limited: the work is mostly an application of existing work that discretizes soft prompts to tokens [a,b]. There is no significant difference between the proposed work and existing related work. The authors should clarify, in-depth, the main differences between their work and other work. Further, like AutoPrompt, the work suffers from gibberish prompts according to the authors (see Figure 2, “uuuu romantic canvas impressionist …”). This limitation makes the hard prompts harder to interpret. The explanation regarding the emoji appearing in the hard prompt (Figure 2 and line 174) is not entirely convincing. It would be really helpful to also know how often the model produces gibberish. Limited Evaluation: the experiments do not overwhelming support that PEZ is better than CLIP interrogator. On two out of the four datasets under consideration, CLIP interrogator achieves a higher CLIPScore. It is also unclear why AutoPromptSGD is not included in Table 1. Wouldn’t this be a good baseline? Negative Qualitative Examples: The paper could benefit from showing negative results where the method doesn’t perform as well. Nit: It would be helpful to look at the results along with Tables/Figures on the same page. At the moment, style transfer, prompt concatenation, and prompt distillation do not appear on the same page. [a] Gradient-Based Constrained Sampling from Language Models. EMNLP 2022. [b] Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? ArXiv 2022.

Questions

Please address the questions in the weakness section. Minor: since you are using cosine distance as a loss function, does it run into instability issues? Is there a particular reason you are using cosine distance as a loss function?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The method requires the model weights to produce hard prompts. Given that recent methods use APIs, it would be useful to highlight this as a potential limitation.

Summary

This paper works on hard prompt optimization with gradient methods, especially a discrete text prompt is discovered using CLIP, and optimized to prompt stable diffusion.

Strengths

1. Without hand-crafted design of hard prompt, the proposed solution directly discover and optimize prompt with gradient descent, leading to very efficient prompt engineering. 2. The learned prompt works great on stable diffusion and achieve effective style transfer, further explains the superiority of the proposed solution.

Weaknesses

The proposed solution works on discrete text prompt optimization without constraints on meaningfulness of the text, making it hard to directly understand to the effectiveness of prompt optimization.

Questions

1. Ablation study shows performance saturation with middle number of prompt length. More analysis is needed to explain this result. 2. Prompt distillation seems interesting to explain the prompt length issue, however, how the distillation ratio is correlated with the quality of the generated samples should also be explained further.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This work proposes a new paradigm which optimizes discrete prompts simply by gradient projection to bridge the gap between relatively “easy-to-optimize” soft prompts and their hard counterparts. Their methodology is straightforward and effective in downstream text-to-image task requirements. And more specifically, comparing to existing gradient-based approaches, such as AutoPrompt, their approach is less hyper-parameter sensitive equipping with their gradient projection. All these advantages offer a new lens on how to optimize discrete prompts for text-to-image generation models, and LLMs likewise.

Strengths

1. This framework is easy to implement, by simply gradient projection in the soft embedding space, and the performance seems promising compared to popular caption-based "CLIP interrogator". It does contribute another approach for such optimization problem. 2. The experiments do show the advantages of such framework, such as "the insensitiveness to hyper-parameters". 3. The paper organization is good, and the writing is well-polished.

Weaknesses

1. I think adding more baselines in your main table would definitely be a plus! Currently, you mainly compare this to a heuristic-based yet popular method called "CLIP interrogator". Can you compare your approach with more diversified baselines, e.g., "AutoPrompt with different LRs", "RLPrompt", etc.. These can let readers gain a basic intuitive understandings of your method's performance against others. 2. In your main table, the improvements of yours compared to CLIP interrogator seems to be marginal, where your performance is almost the same level of performance with CLIP interrogator, even though you only include fewer tokens. 3. Besides, can you show some quantitative and qualitative examples on directly showing on the same set of images, what are the optimized prompts by CLIP interrogator, and what are the optimized prompts by yours (and perhaps, their word overlapping by distillation), such that we can enhance our understandings that your approach identify a "secret" language, like initial work [1] (in addition to possible application contributions, the secret language could also contribute to scientific research if your pipeline is good at identifying such phenomenon). Otherwise, to one of your contributions that your identified prompt is shorter, we may speculate that the prompts in CLIP interrogator can also be distilled into the same short length, and even with the same key tokens as your pipeline. Therefore, the secret language observation would be weaken. Besides, the CLIP interrogator even does not require any training with similar level of performance and even similar level of short lengths through distillation, why not we use their easier pipeline? --- Currently, figure 6 further enhances my confidence on such point as well, as your prompts include a lot of natural tokens in accordance with our common-sense understandings, such as cat images should use "cat" tokens. 4. Additionally, practitioners would be interested in your optimization efficiency as well for such algorithm to identify well-crafted prompts, comparing to many other soft/hard prompt optimization work. In terms of this, if you have better efficiency, that is also another good point, where I am also interested. [1] Discovering the hidden vocabulary of DALLE-2, arxiv 2022 So right now, I am more viewing this work's contribution as a different perspective/approach to prompt optimization for text-to-image generation models, and even more models. Of course, that is interesting, and useful/insightful to the community. However, a key limitation would be such utilities are not quite clear alongside all current prompt optimization approaches. And the performance improvement seems to somewhat limited compared to CLIP interrogator. Hopefully, my suggestions would be useful for you to improve your content.

Questions

See weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See weaknesses.

Reviewer Lu7o Replies

<1. baselines> - From your rebuttal, I have understood what is your key advantage you want to show now. Precisely, that is the **efficiency** compared to other baselines (w/ your similar level of performance as many baselines), such as AutoPrompt, RLPrompt, etc. It would be much better for you to consider adding efficiency comparisons visualizations/discussions. Additionally, I am not sure whether RLPrompt would fail in this case, as image-level/instance-level optimization takes fewer time, which deserves some experiments as well. If it is weaker/much slower than your approach in a few cases, it would be good to include this and highlight several times in your paper, to make it more clear about your key advantages. Moreover, from your new AutoPrompt experiments, it would be good for you to add one **rigorous** figure on the efficiency comparison results, e.g., using the best setup and also the less good setup (such as other LR, prompt length in AutoPrompt), to position your approach. This would definitely add some values to your paper. <2. CLIP interrogator> - Thanks for your clarification. It does seem that interrogator performs much weaker with fewer tokens. But as you said, it would be good for you to incorporate some efficiency studies as well. That can address my concerns on your approach vs. CLIP interrogator. <3. Secret Language> - This part is just for one of your claims. You said you identify a secret language of text-to-image diffusion models [1]. But in the initial paper, the lexical items that they used are purely gibberish tokens, w/o any significant real-world meanings, such as "Apoploe vesrreaitais" for birds. But in your results, and also your new pdf results, you always include some natural tokens, for instance, "butterfly emoji + users', 'cruise + green render lights', "balloon relationships", “cat” for cat image, and many more shown in your paper figures. I am curious about more fine-grained studies on your generated prompts. For instance, others might suspect that when deleting your natural meaningful tokens, your generated results would fail completely, which may indicate a weaker claim of "secret language". Or in other words, perhaps, most of your used nonsense tokens are mainly search artifacts. Therefore, my comments are to say it would be better for you to provide some rigorous studies. I believe this point is also mentioned by reviewer DXf6, for which I agree with him/her. [1] Discovering the Hidden Vocabulary of DALLE-2 <4. efficiency results> - See 1, it would be better for you to provide rigorous studies to position your approach. For current rating, I am saying that this work should be further improved to be an interesting paper published in this conference. So I select the borderline rating, in which I would like to wait for other reviewers or ACs for final judgements.

Acknowledgement of rebuttal

Thank you for the rebuttal. My questions were addressed . Thus I keep my score. Good luck!

Reply to the Authors

Thank you for the detailed response. I really appreciate you uploading additional negative results in the PDF. I also liked your response to Reviewer LSBN where you have included prompts with more tokens (8 to 64). It would be awesome if you could include this ablation in the paper as well. I do have a few concerns: 1. The additional table included in the rebuttal shows that PEZ converges faster than AutoPrompt. This is a positive result. I would include this detail in the paper. But, as pointed out in my review, Table 1 in the paper is incomplete without AutoPrompt. It might be the case that AutoPrompt performs better than PEZ on all the other datasets. If the work is about efficiency compared to AutoPrompt, then you might have to highlight the tradeoffs between PEZ and AutoPrompt in terms of efficiency vs. performance. 2. Minor: could you clarify what you mean by the following statement? > It's important to note, however, that these two scenarios can be mitigated by optimizing with various initializations. Overall, I think the authors have put in a lot of effort to improve their paper. However, I feel the paper could be positioned better to highlight the practical benefits in terms of efficiency compared to existing work. I think the work would benefit from another round of submission. I will gladly increase my scores if other reviewers feel strongly about the paper.