Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data

Author Response

Thank you for recognizing our comprehensive experimental analysis and clear design principles. We address your main concerns: 1. Context quality: Unlike model-based systems, we **ensure faithful and natural text generation** through manually crafted templates for each table (examples in Table 4). This template-based approach **guarantees complete preservation of the original tabular data** while maintaining fluent text, preventing hallucination or information loss that could occur with neural generation methods. 2. RAG Comparison: We acknowledge that our RAG setup in the original paper could have been more comprehensive. To address this concern, we conducted additional extensive experiments that provide deeper insights into the relationship between retrieval size and model performance. ## Additional Experiments with Varying Retrieval Sizes First, we investigated scenarios where half of the 64k context (i.e., 32k tokens) contains query-relevant information, examining performance trends across different retrieval sizes (2k, 4k, 8k, 16k, 32k). We used Llama-3.1-405b for these experiments. The results demonstrate that performance peaks when the retrieval size (32k) matches the actual amount of relevant information in the context. This supports our paper's argument that reducing unnecessary information can enhance reasoning capabilities. ### Table 1: Accuracy with information density = 0.5 | # Retrieved Tokens | 2k | 4k | 8k | 16k | 32k | 64k (LCLMs with full context) | |-------------------|------|------|------|-------|-------|---------------------------| | RAG | 21.70 | 25.28 | 27.02 | 34.91 | 37.34 | 30.18 | ## Impact of Information Density However, in real-world applications, the density of relevant information isn't known a priori. To investigate this, we conducted experiments with contexts where all 64k tokens are query-relevant. Note that due to different answer distributions, the absolute values between information density = 0.5 and 1.0 scenarios aren't directly comparable. The results show that while using all 64k tokens achieved the best performance, the margin of improvement over 32k retrieval was surprisingly small. This suggests that current LCLMs still face limitations in processing ultra-long contexts, indicating significant room for improvement in their long-context reasoning capabilities. ### Table 2: Accuracy with information density = 1.0 | # Retrieved Tokens | 2k | 4k | 8k | 16k | 32k | 64k (LCLMs with full context) | |-------------------|------|------|------|-------|-------|---------------------------| | RAG | 14.85 | 19.18 | 21.79 | 29.36 | 38.69 | 39.06 | ## Retrieval Quality Analysis We further investigated whether RAG would be optimal if we knew the distribution of relevant information beforehand. In scenarios with 32k relevant information within 64k context, our analysis using bge-large-en-v1.5 (a state-of-the-art retrieval system) achieved: - **Precision**: 83.92% - **Recall**: 80.89% These results, while strong, highlight that even in ideal settings, retrieval errors remain significant. ## Key findings: 1. While RAG can outperform LCLMs when the optimal retrieval size is known (matching the amount of relevant information), this information is rarely available in practical settings 2. Even with state-of-the-art retrieval models (bge-large-en-v1.5), substantial retrieval errors persist (Precision: 83.92%, Recall: 80.89%), highlighting the inherent challenges in information selection 3. The practical deployment of long-context reasoning systems requires either: - Development of more adaptive and robust retrieval mechanisms that can dynamically adjust to varying information densities, or - Enhancement of LCLMs' capabilities to process and reason over ultra-long contexts more effectively These findings suggest that the challenges in long-context reasoning cannot be solved by simply increasing retrieval size or context length, but rather require fundamental improvements in both retrieval quality and model capabilities. Thank you for your valuable feedback!

Author Response (1/2)

Thank you for your detailed review. We address each of your concerns: 1. Benchmark Size: Although HoloBench contains only 90 questions, each question is evaluated across numerous configurations, such as context length, information position, and information amount. For instance, testing 5 context lengths, 5 information positions, and 2 information amounts results in 4,500 test cases (90 × 5 × 5 × 2) per model—a scale unmatched by existing benchmarks mentioned. Moreover, we meticulously crafted our questions to encompass diverse query types and difficulty levels, with each configuration validated for correctness via SQL execution. This comprehensive coverage significantly amplifies the effective size of our evaluation, providing robust insights into model behavior across varying conditions, far beyond what the raw question count implies. 2 and 3. Diversity of databases and difficulty of benchmark: To create HoloBench for evaluating LCLMs, we required large tables capable of generating long contexts. Considering this, we applied the following criteria to ensure diversity in both domains and answers: Databases must have more than two tables and at least 500 rows in total. Each selected database must belong to a distinct domain. Among the databases in the SPIDER dataset, only 15 meet the row count requirement: ['soccer_1', 'wta_1', 'baseball_1', 'sakila_1', 'flight_4', 'formula_1', 'college_2', 'bike_1', 'store_1', 'chinook_1', 'world_1', 'csu_1', 'flight_2', 'wine_1', 'car_1’]. Then, we chose five databases in different domains. From these, we selected five databases spanning different domains while also diversifying the number of tables and rows. The selected databases are summarized below: | Base DB | # Tables | # Total Rows | |------------|----------:|--------------:| | Wine_1 | 3 | 555 | | Store_1 | 4 | 2,548 | | College_2 | 9 | 3,554 | | Flight_4 | 3 | 20,989 | | Soccer_1 | 5 | 185,608 | This selection balances domain variety, table count, and row diversity to create a robust evaluation benchmark. Note that we did consider alternatives such as using external corpora (e.g., Wikipedia) to increase diversity and complexity in the context. However, ensuring control over the relevance of external information proved challenging, making it difficult to guarantee the correctness of the ground truth. To maintain a systematic and automated evaluation process, we chose to limit the context to the content within the databases. Although one might assume that LCLMs could easily distinguish between relevant and irrelevant contexts when they are drawn from different tables within a database, our results show that the models still face difficulties in making these distinctions. 3. Evaluation fairness and reliability: While we rely on GPT-4o-mini for automatic evaluation, we find the evaluations were aligned with human judgements (93.8% agreement). 4. Potential information leakage due to publicly released text-to-SQL dataset: While this is a valid concern, our benchmark is constructed by dynamically sampling table rows and generating contexts with relevant and irrelevant information. We derive the golden answers for different settings by executing SQL queries on the subtables. This makes it highly unlikely that LCLMs would have any parametric knowledge about the golden answers.

Author Response (2/2)

5. RAG comparison: We acknowledge that our RAG setup in the original paper could have been more comprehensive. To address this concern, we conducted additional extensive experiments that provide deeper insights into the relationship between retrieval size and model performance. ## Additional Experiments with Varying Retrieval Sizes First, we investigated scenarios where half of the 64k context (i.e., 32k tokens) contains query-relevant information, examining performance trends across different retrieval sizes (2k, 4k, 8k, 16k, 32k). We used Llama-3.1-405b for these experiments. The results demonstrate that performance peaks when the retrieval size (32k) matches the actual amount of relevant information in the context. This supports our paper's argument that reducing unnecessary information can enhance reasoning capabilities. ### Table 1: Accuracy with information density = 0.5 | # Retrieved Tokens | 2k | 4k | 8k | 16k | 32k | 64k (LCLMs with full context) | |-------------------|------|------|------|-------|-------|---------------------------| | RAG | 21.70 | 25.28 | 27.02 | 34.91 | 37.34 | 30.18 | ## Impact of Information Density However, in real-world applications, the density of relevant information isn't known a priori. To investigate this, we conducted experiments with contexts where all 64k tokens are query-relevant. Note that due to different answer distributions, the absolute values between information density = 0.5 and 1.0 scenarios aren't directly comparable. The results show that while using all 64k tokens achieved the best performance, the margin of improvement over 32k retrieval was surprisingly small. This suggests that current LCLMs still face limitations in processing ultra-long contexts, indicating significant room for improvement in their long-context reasoning capabilities. ### Table 2: Accuracy with information density = 1.0 | # Retrieved Tokens | 2k | 4k | 8k | 16k | 32k | 64k (LCLMs with full context) | |-------------------|------|------|------|-------|-------|---------------------------| | RAG | 14.85 | 19.18 | 21.79 | 29.36 | 38.69 | 39.06 | ## Retrieval Quality Analysis We further investigated whether RAG would be optimal if we knew the distribution of relevant information beforehand. In scenarios with 32k relevant information within 64k context, our analysis using bge-large-en-v1.5 (a state-of-the-art retrieval system) achieved: - **Precision**: 83.92% - **Recall**: 80.89% These results, while strong, highlight that even in ideal settings, retrieval errors remain significant. ## Key findings: 1. While RAG can outperform LCLMs when the optimal retrieval size is known (matching the amount of relevant information), this information is rarely available in practical settings 2. Even with state-of-the-art retrieval models (bge-large-en-v1.5), substantial retrieval errors persist (Precision: 83.92%, Recall: 80.89%), highlighting the inherent challenges in information selection 3. The practical deployment of long-context reasoning systems requires either: - Development of more adaptive and robust retrieval mechanisms that can dynamically adjust to varying information densities, or - Enhancement of LCLMs' capabilities to process and reason over ultra-long contexts more effectively These findings suggest that the challenges in long-context reasoning cannot be solved by simply increasing retrieval size or context length, but rather require fundamental improvements in both retrieval quality and model capabilities. Thank you for your valuable feedback!

Author Response

Thank you for your positive feedback about our benchmark's potential value to the community and the reproducibility of our work. > How to differentiate b/w long but simple questions and genuinely complex ones Since we manually create questions based on SQL queries, the more complex the SQL query, the more conditions it contains, leading to longer questions. So, longer questions generally tend to be more complex questions. To quantify this, we calculated the average number of words per question for each level of SQL query difficulty. The results are as follows: | SQL Difficulty | Average Question Length (words) | |:--------------:|--------------------------------:| | Easy | 10.8 | | Medium | 13.8 | | Hard | 17.0 | This confirms a trend within our dataset: questions derived from more complex queries tend to be longer. If we have misunderstood your point, please let us know.