Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting

repsonse for reviewer

Thank you for your reviews. The definition of batch prompting is not clear in the intro Lines 43-48; We will definite the detail of "batch prompting" and introduce the advantage of it. The efficiency experiments will be added. **Motivations:** Our main idea is to revisit the detail of the autoregressive inference of Batch prompting and amplify the demonstration role of batch data. Previous batch prompting works( BatchPrompt: accomplish more with less, Batch Prompting: Efficient inference..) have claimed that some previous data in batch may play a role of demonstration. However，there are not enough evidence and explanation on this. After we revisit the inference process of batch prompting, we find that the advantage of mutiple data is neglected. Previous works do not make most of the demonstration role of multiple data in batch prompting, which can be used to improve the performance. **Novelty** (1) We are the first to construct the squential demonstrations in the output of large language models. And We show its effectiveness by the experiments. Especially, the obivious improvement at GSM8k and SVAMP, almost %5 higher than “batch Prompting”。 (2) Further, we find that "similar batch data" leading "simialr demonstrations in the output" will be quite beneficial to the perfromance on condition with Auto-demo promting. (3) what we want to argue is that The best strength of Our method is that we get higher accuracy than conventional single prompting with less token cost and lower average time cost per data. As figure 4 shows, 3 results are better than single data inference with less cost. ***the question*** How does this compare to methods like Hydragen (https://arxiv.org/abs/2402.05099), which do not increase the context length, but rather share the KV-cache on the shared prefix? They are two kinds of efficient inference techniques. I will list some shortages of prefix cache to prove the value of "batch prompting". (1) Cache Pollution: prefix cache can cause the cache to be filled with a large amount of infrequently accessed data, a phenomenon known as cache pollution. When the cache is filled with irrelevant data, the data that is really needed may be expelled from the cache, thus reducing the cache efficiency. (2) Cache Coherence Issues: In multi-core systems, prefix cache may need to maintain cache coherence, which can add additional complexity and overhead, especially when it is necessary to ensure that multiple cores or processors are seeing the most recent data (3) Performance Variability: The performance of the prefix cache can vary widely from application to application and workload. In some cases, the prefix cache may not provide the expected performance improvement or may even degrade performance due to interaction with the hardware prefetch mechanism (4) Hardware Support Requirements: prefix cache may require specific hardware support, such as Content Addressable Memory (CAM), which can add cost and design complexity (5) Adaptability Issues: The prefix cache may not adapt to the environment that dynamically changes, for example, the routing table changes frequently. This may cause the prefixes in the cache to quickly become outdated, requiring frequent updates ***response*** We think Auto-demo Prompting can work well with many prefix techniques. Further, Auto-demo prompting do not require some cache room to enhance the efficiency. Obiviously, Batch prompting and AutoDemo Prompting have less hardware requirements.

Response for few-shot prompting and efficiency

Thank your reviews for our work. Some clarity issues will be addressed and the efficiency experiments will be added. **Motivations:** Our main idea is to revisit the detail of the autoregressive inference of Batch prompting and amplify the demonstration role of batch data. Previous batch prompting works( BatchPrompt: accomplish more with less, Batch Prompting: Efficient inference..) have claimed that some previous data in batch may play a role of demonstration. However，there are not enough evidence and explanation on this. After we revisit the inference process of batch prompting, we find that the advantage of mutiple data is neglected. Previous works do not make most of the demonstration role of multiple data in batch prompting, which can be used to improve the performance. **Novelty** The advantage of Autoregressive decoder is that it can generate the smooth text from "left to right". Previous Batch prompting answers each question in such a way that each question is independent and unrelated, which waste the natural advantage of large models. Our method prompts llm to generate "(question, answer)" pair to address this issue. The "(question, answer)" format of demosntration is significant as "Rethinking the Role of Demonstrations: What Makes In-Context Learning Work https://arxiv.org/pdf/2202.12837”. Also this paper still provide a conclusion that "Using demonstration is better than not using it" and “only labels” is not working. We are the first to provide how to construct the demonstrations in the output of batch prompting. And we use the method to cluster the similar data into batches to make most part of results of batch size >=32 better than single data prompting. Our method can provide a more effective batch prompting method with a minimal cost. **Extra token cost discussion** There are new discussion of the tokens cost of our method. The mainly extra token cost comes from the "question" regeneration. However, we can insert each question instead of the generation of LLM. As the following example shows, we add "#" as the mark of the start position of each answer in the prompt. when llm inference "#", which is a mark, we can insert one question from the input question list orderly. By this means, we save the extra token cost of our method, and the positive effect of demonstration -(question, answer) - generated by llm can still remain. Therefore, our method has the same "output tokens" as the conventional batch prompting. The following is our prompt example to show. ``` You are a professional NLP expert at sentence pair relationship annotation. Each data includes a sentence pair, ”Premise” and ”Hypothesis”. Your goal is to classify the sentence pair into two classes as below: [class 1]: the given Hypothesis and Premise are logical and following (entailment) to each other. [class 0]: the given Hypothesis and Premise are NOT following (entailment) to each other. You will be given sentence pairs each time, and the below is the format of sentence pairs which will be given: ============ @Sentence pair: Premise: xxxxx Hypothesis: xxxxx ...... ============ Below are the outputs you need to generate. ”X” can be ’1’ or ’0’. ============ #Sentence pair: Premise: xxxxx Hypothesis: xxxxx Label: [class X] ..... ============ ``` ***embedding role*** We use embedding model to encoding all the data, and cluster them into batches with the similarities. In this way, we hope to amplify the role of output demonstration in our method. And the result of experiments show the large improvement. We will explore some other metrics, and some other embedding models, such as "sentence-bert" and "BM-25". ***few-shot prompting*** We think that conventional few-shot prompting and our method are totally different. We just want to show the format relation between them. conventional few-shot prompting has only one question to predict, however our "few-shot prompting" is a sequential behavior, each question will work as the demonstraion after being answered. We think they have no necessity to compare their performances.