Retrieval-Augmented Generation (RAG) systems integrate large language models with information retrieval components. These systems mark a shift from structured user interfaces to natural language interfaces (NLIs). This transition eliminates the predictable interaction patterns that traditional performance testing tools rely on to control content access distributions. In RAG systems, the unstructured nature of user queries hampers the ability to simulate required content access distributions; nevertheless, content access distribution remains a critical determinant of performance due to caching effects and the uneven popularity of content. To address this challenge, we make two primary contributions: (1) an optimization-driven query issuance mechanism that enables the simulation of required content access distributions during performance testing, and (2) query generation strategies that provide broad retrieval coverage of the knowledge base while reducing query generation costs by at least $\mathbf{2 0. 9 \%}$ compared to straightforward approaches. Constructing a synthetic test query set with wide coverage is essential for shaping the desired access distribution and for identifying performance bottlenecks across both frequently and infrequently retrieved content. Empirical evaluations also demonstrate that content access distributions, particularly in the presence of caching, have a substantial impact on system's performance metrics. By enabling testing across diverse operational scenarios, our framework supports robust, access distribution-aware performance evaluation. This capability is essential for performance testing of enterprisegrade RAG applications that must meet stringent service-level agreement (SLA) requirements.
Paper
Full text
QueRA: Query Synthesis and Resource Access Modelling in RAG
Semantic Scholar · 2025
Abstract
Retrieval-Augmented Generation (RAG) systems integrate large language models with information retrieval components. These systems mark a shift from structured user interfaces to natural language interfaces (NLIs). This transition eliminates the predictable interaction patterns that traditional performance testing tools rely on to control content access distributions. In RAG systems, the unstructured nature of user queries hampers the ability to simulate required content access distributions; nevertheless, content access distribution remains a critical determinant of performance due to caching effects and the uneven popularity of content. To address this challenge, we make two primary contributions: (1) an optimization-driven query issuance mechanism that enables the simulation of required content access distributions during performance testing, and (2) query generation strategies that provide broad retrieval coverage of the knowledge base while reducing query generation costs by at least $\mathbf{2 0. 9 %}$ compared to straightforward approaches. Constructing a synthetic test query set with wide coverage is essential for shaping the desired access distribution and for identifying performance bottlenecks across both frequently and infrequently retrieved content. Empirical evaluations also demonstrate that content access distributions, particularly in the presence of caching, have a substantial impact on system's performance metrics. By enabling testing across diverse operational scenarios, our framework supports robust, access distribution-aware performance evaluation. This capability is essential for performance testing of enterprisegrade RAG applications that must meet stringent service-level agreement (SLA) requirements.