Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models

Summary

The paper presents Promptriever, an innovative retrieval model that incorporates instruction-following capabilities akin to LLMs. By curating a new instruction training set from MS MARCO, the authors developed a bi-encoder retriever that adjusts its responses based on natural language instructions, demonstrating substantial improvements in retrieval tasks. Specifically, Promptriever achieves state-of-the-art results on instruction-following retrieval tasks, showing significant robustness to query phrasing and the ability to perform hyperparameter search via prompting. This model sets a new precedent for future work in aligning LLM prompting techniques with IR systems.

Strengths

- The focus on enabling retrievers to understand human instructions aligns well with the current demand for more intuitive and adaptable search technologies, marking a significant advancement in retrieval systems. - The methodological approach of not requiring human annotations is notable, offering a scalable and cost-effective solution for training retrieval systems. - The clarity of the paper and its comprehensive details support replicability and transparency, which are crucial for advancing research in this area.

Weaknesses

My main concern is about the **experiments**. - The t-test is not conducted on the experimental results. In IR, it is very important to validate whether the improvement is significant. - The use of varying amounts of training data for different models (Promptriever vs. RepLLaMA) raises concerns about the fairness and validity of the comparative performance analysis. - The performance improvement on in-domain retrieval is very marginal. I guess the instruction-tuning process may affect the original retrieval performance. - In Table 5, the selected baselines are not strong on BEIR benchmark. For example, E5-mistral has achieved 56.9 on BEIR. I think the authors should compare their methods with more advanced baselines. - A minor suggestion: please use `` and ‘’ (two single quotes) to generate double quotes in LaTeX.

Questions

Please consider to address my concerns in Weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Summary

The paper introduces Promptriever, a bi-encoder model that enriches the per-instance query context within prompts. To train this model, per-query instructions and instruction negatives are synthetically generated. The authors evaluate Promptriever across various scenarios, including instruction-following datasets, in-domain retrieval, and out-of-domain retrieval. The model not only surpasses previous bi-encoders but also achieves performance comparable to state-of-the-art cross-encoder retrievers, showcasing its effectiveness in handling free-form language prompts.

Strengths

- The paper introduces a novel idea by challenging the assumption that the same instruction can be applied uniformly across queries. - The authors curate a training dataset that serves as a solid resource for providing detailed, per-instance instructions. - I particularly liked the analysis in Section 5, which addressed several questions I had while reading the methodology and experimental results sections.

Weaknesses

- The generated instructions are only used during model training. It would have been insightful to hold out a dev or test split to evaluate how the model performs on this dataset. - For BeIR, the use of generic instructions feels too broad. Task-specific instructions, such as those in TART, might have been more effective and better aligned with the goals of the paper.

Questions

- For out-of-domain retrieval, what is the rationale behind selecting the best score out of 10 prompts, rather than fixing a prompt or reporting an average score? - Could the authors provide some qualitative examples illustrating how Promptriever performs across different categories in Table 6? - In Table 11, simpler prompts like “Think carefully about relevance” seem to yield strong performance. Why might these simpler prompts outperform other more detailed ones that closely match the training data?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Summary

In this paper, the authors introduce an instruction-aware retriever named Promptriever, which can be prompted like an LM. They build a dataset for training such a retriever based on MS MARCO. They conduct experiments based on the dataset, and reveal several interesting conclusions. Pros: - The problem discussed in the paper, i.e., building instruction-following IR models, is interesting. - The problem definition is clear. The proposed method is easy to follow. - Experimental results verify the effectiveness of the proposed method. Cons: - The dataset is generated solely based on MS MARCO. It would be great if more datasets could be considered, especially for the test. BEIR should also be considered for the zero-shot scenario. - It would be great if the definition of instructions in IR could be clearly defined. For example, given the example "Which type of volcano eruption has not been seen?", the authors claim that the volcano types and formation can be added as additional instructions. I wonder why these are treated as additional instructions but not part of the original information need (i.e., the original queries). - More details about the experimental setup should be provided. For example, are the baseline models trained using similar assessment data (with instructions in the queries)? - Better baseline models should be added. For example, we can generate query rewrites via LLMs and then retrieve documents using the rewritten query (converting the NLP-like instruction into keyword-like queries).

Strengths

Pros: - The problem discussed in the paper, i.e., building instruction-following IR models, is interesting. - The problem definition is clear. The proposed method is easy to follow. - Experimental results verify the effectiveness of the proposed method.

Weaknesses

Cons: - The dataset is generated solely based on MS MARCO. It would be great if more datasets could be considered, especially for the test. BEIR should also be considered for the zero-shot scenario. - It would be great if the definition of instructions in IR could be clearly defined. For example, given the example "Which type of volcano eruption has not been seen?", the authors claim that the volcano types and formations can be added as additional instructions. I wonder why these are treated as additional instructions but not part of the original information need (i.e., the original queries). - More details about the experimental setup should be provided. For example, are the baseline models trained using similar assessment data (with instructions in the queries)? - Better baseline models should be added. For example, we can generate query rewrites via LLMs and then retrieve results using the rewritten query (converting the NLP-like instruction into keyword-like queries).

Questions

- How are the baseline models trained? Are they using similar assessment data (with instructions in the queries)?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Ethics concerns

na

Summary

Currently, retrievers are only able to retrieve texts similar to input queries, mostly with text similarity. In this paper, the authors present Promptriever, the first retrieval model able to be prompted with textual instructions. For example, the users can pass in complex instructions to filter the passages to be relevant to a specific topic or exclude certain categories of passages. To do so, the authors create and release a synthetic dataset of query-passage relevance pairs augmented with instructions. They use the MS MACRO dataset and generate instructions using Llama-3-70B, which includes diverse length formats and styles. Then, they use GPT-4o to generate the instruction negative passages. The Promptriever is then trained on the augmented data. As a result, the Promptriever maintains strong retrieval scores in standard settings. Compared to the original RepLlaMA, it also follows instructions better. The authors also perform multiple ablations such as the instruction-negatives.

Strengths

1. The motivation is strong and the story is convincing. The authors identify the retrievers' current lack of instruction-following abilities to motivate the method. They help to bridge the gap by introducing Promptriever. 2. The experiment performance is promising and shows improvements in various datasets. 3. The ablation studies are sufficient and necessitate the needs of the different training components.

Weaknesses

See questions.

Questions

1. Can you list out some use cases where the promptriever is used in scenarios such as RAG? How would it perform? 2. It would be nice to see some qualitative examples of Promptriever compared to traditional retrievers, besides the example in the intro. 3. You mentioned in L240 that cross-encoders perform best due to their significant compute advantage. Could you list out the compute resources required by different baselines and Promptriever?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Qualitative Example from FollowIR

Several reviewers requested additional qualitative examples. We will include these in the appendix of our paper. --------- ## FollowIR Example **Query:** Identify documents that discuss the reasons for automobile recalls. **Instruction:** A relevant document will specify major or minor reasons for automobile recalls by car manufacturers. Documents that discuss truck recalls are not relevant. **Retrieved Passages comparing Promptriever and RepLLaMA** ### Promptriever: Rank 1 (relevance=True): YOUR WHEELS: DEFECT INVESTIGATIONS AT RECORD HIGH IN '89 \n\n February 1, 1990, Thursday, Home Edition The auto safety tally for 1989 has come in, and the news isn't particularly good for consumers. Last year, 6.7 million vehicles, including cars made by virtually every major producer, were recalled by manufacturers because of safety defects that needed correcting. The recalls included 1.7 million General Motors cars with sticking throttles, 1.4 million Ford Escorts with defective seats, 500,000 Chryslers with rusty front-suspension parts, and 408,000 Toyota Camrys with problem seat belts. The National Highway Traffic Safety Administration boasted that it opened 213 defect investigations during 1989, a record in the 20-year history of the agency. Of the 6.7 million recalls, the agency claims that it influenced 77%. The statistics beg the question: Are cars safer because more investigations are being run, or do cars have more defects than ever? The numbers, critics say, show the public is at risk…. Rank 2 (relevance=False): JAPAN WARNING ITS CAR MAKERS AFTER RASH OF DEFECTS, RECALLS \n\n December 5, 1989, Tuesday, P.M. Final Transport Ministry officials will ask Japanese car makers to pay more attention to design and quality control because of defects found in new and remodeled cars recently, a ministry official said today. The officials will issue the request at a regularly scheduled meeting with the heads of the Japan Automobile Manufacturers Assn. on Thursday. Since April 1, 11 Japanese models and 14 foreign models have been recalled because of defects. This compares with 15 Japanese models and 24 foreign models in the year to March 31, 1989 … ### RepLLaMA: Rank 1 (relevance=False): full-sized GM pickup trucks and cab-chassis equipped with fuel tanks mounted outboard of the frame rails contain a defect related to motor vehicle safety. At the same time, he released a comprehensive Engineering Analysis (EA) Report that explains in detail the basis for the decision. A summary of that Report is set out in the Federal Register notice announcing that decision. 59 Fed. Reg. 54025 (October 27, 1994). The EA Report makes two fundamental findings. First, on the basis of statistical analysis, laboratory crash tests, and a review of available information concerning specific real- world crashes, the EA Report demonstrates that the GM pickups at issue exhibit an increased risk of fire in side-impact crashes compared to similar vehicles (specifically, to contemporary Ford and Dodge full-sized pickups). Second, primarily on the … Rank 2 (relevance=False): or type of crash? If so, how relevant? 4. Are the extent of a manufacturer's awareness of a potential or ongoing safety risk, and the extent of a manufacturer's efforts to avoid that risk, relevant to the issue of whether an unreasonable risk exists? Is a manufacturer's failure to implement measures to mitigate or eliminate an increased safety risk relevant to that issue? 5. What weight should be given to the foregoing three factors, and any other relevant factors, in deciding whether a vehicle contains a defect related to motor vehicle safety? Other Information Sought 1. Additional information concerning post-crash fires in real-world crashes involving the GM vehicles covered by this investigation and comparable vehicles. 2. Additional information regarding GM's awareness of an increased risk of post-crash fires due to the placement of the fuel tanks outside of the frame rails of these trucks, both at the time of their design and afterward. ... ### **Our Comment** You can see that RepLLaMA incorrectly focuses on trucks, despite the instruction to avoid them. Whereas Promptriever correctly focused on sedan recalls.