The increasing use of retrieval-augmented applications, where large language models (LLMs) are instructed to generate queries, assess relevance, and synthesise responses, has introduced new challenges in Information Retrieval (IR).The lack of transparency in LLMs means that even subtle variations in instructions can significantly impact the quality, consistency, and reliability of their responses.To address this issue, we propose Instruction-Response Study, an experimental framework for systematically analysing how task instructions influence LLM-generated responses in IR tasks.This paper presents the core components of the framework and demonstrates its utility through four case studies, examining 1) the effect of IR tasks on query formulation, 2) the impact of topic information size on retrieval effectiveness, 3) the reproducibility of LLM-generated queries, and 4) the role of meta-instructions in diversifying instruction design.The findings highlight how the proposed framework enables controlled experimentation on instruction design and its effects, offering a foundation for optimising prompt engineering and enhancing retrieval-augmented applications.
Paper
Full text
An Instruction-Response Perspective on Large Language Models in Information Retrieval Tasks
OpenAlex · Topic Modeling · 2025
Abstract
The increasing use of retrieval-augmented applications, where large language models (LLMs) are instructed to generate queries, assess relevance, and synthesise responses, has introduced new challenges in Information Retrieval (IR). The lack of transparency in LLMs means that even subtle variations in instructions can significantly impact the quality, consistency, and reliability of their responses. To address this issue, we propose Instruction-Response Study, an experimental framework for systematically analysing how task instructions influence LLM-generated responses in IR tasks. This paper presents the core components of the framework and demonstrates its utility through four case studies, examining 1) the effect of IR tasks on query formulation, 2) the impact of topic information size on retrieval effectiveness, 3) the reproducibility of LLM-generated queries, and 4) the role of meta-instructions in diversifying instruction design. The findings highlight how the proposed framework enables controlled experimentation on instruction design and its effects, offering a foundation for optimising prompt engineering and enhancing retrieval-augmented applications.