Summary
The paper describes work on evaluating ChatGPT for word importance with respect to prompts. The author/s motivate the study by claiming that “recognizing the impact of specific words or linguistic structures on LLM outputs can offer a granular understanding of model behavior, providing valuable insights into how information is processed and weighted across different layers of the model.” For the experiments, the author/s propose a simple method for approximating a word’s importance value fro the prompt that is “inspired by permutation importance” in tabular data analysis. The method requires iterating through the prompts while masking each word and evaluating the resulting response from the model to approximate the masked word’s importance. There are no mentions or discussions whatsoever of the limitations and adaptability of the proposed method. The author/s use readability, embedding similarity, and simple word count for scoring. The prompt choice used for the experiment setup has not been properly discussed, which is confusing. Overall, the task presented itself is framed as explainability but is more closely similar to prompt engineering as the method itself optimizes for word importance in prompts.
Strengths
The paper explores and interesting concept of word importance which I do find essential in further understanding how large language models like ChatGPT works. The proposed method has some potential provided that it carefully addresses some of the very obvious limitations discussed below and further improve its algorithmic features to consider scale, flexibility, and efficiency.
Weaknesses
The depth of the experiments conducted in the study is extremely limited as only three metrics which cover Flesch Ease, word count, and topic similarity (cosine embedding) have been explored. The model variation is also very limited, with only one model used for experimentation, GPT-3.5-Turbo (ChatGPT), despite the diverse publicly available models in Hugginface such as Llama, FlanT5, BLOOMZ. This implies that the study essentially optimizes for OpenAI products instead of prioritizing diverse results from open-sourced models. There is no ablation or in-depth exploration. This form of limitation needs to be addressed for inclusion to ICLR.
There are several obvious limitations of the proposed methodology involving masking each word in the prompt. The method seems to be not practical for prompts that are considerably long, which is realistically common in most interdisciplinary fields. This should be discussed thoroughly in the paper. Moreover, there are given words that are obviously non-important (ex. stopwords), it would be computationally expensive and impractical to still iterate and and compute the importance of these words in the prompt. The proposed methodology seems to have no workaround for optimization and compression.
While the authors are correct that the proposed method is text score agnostic, it is worth exploring what linguistic scoring features are better than others. This begs more in-depth exploration/ablation of an extensive set of features (which is expected for an ICLR paper).
The paper is basically prompt engineering as it optimizes the quality of generation based on some measure of word importance. The author should explicitly mention this as it directly aligns with the task covered by the paper. It would also help other researchers discover similarities with works on optimizing prompts / explainable prompts in general.
Minor comments:
1. The aesthetics of the paper, including figure quality, structure of sections, proper captioning, and layout, should be greatly improved for readability. The algorithm figure has no number, the tables are too wide instead of compact.
2. The tables are confusing and are not presented properly. For example, Table 2 could have been represented much better as it is confusing what the author/s mean in parallel with the discussion on suffixes. In terms of the suffix configuration, the examples on bullet points are not well presented. Instead, show an actual diagram instead of how the suffixes are added with respect to each evaluation metric used.
Questions
1. Is there even a need to mask all words, including stop words (ex. “and”, “is”)? These words might already be obviously unimportant for the user, and the proposed methodology seems to be static and not adaptive.
2. How does using embeddings capture topics? The method only captures semantic relatedness as it only uses cosine similarity. Also, why the FlagEmbedding model? What’s the justification for using this specifically?
3. One thing that is very confusing is that the choice of the prompts used, as evidenced by some instances shown in the paper in the Appendix, for querying responses is unusual and unmotivated. Why should the prompts look like these? If word importance is being measured, I would have expected prompts in qualitative question form (with an absolute gold standard answer on hand) where important entities in a sentence are iteratively being masked, and the goal of the language model is to answer the question. The author/s can then evaluate the correctness of the generated responses by the model with the gold standard to see if there are some negative effects with some entities removed or masked in the prompts. In the paper, I do not understand the motivation and importance of using phrases like “You answer like David Attenborough.” or “You are a surgeon.” in the prompts.
Rating
3: reject, not good enough
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.