TableRAG: Million-Token Table Understanding with Language Models

Summary

This paper primarily concentrates on the scalability challenges associated with encoding entire tables as input for LLM reasoning. It introduces a retrieval-augmented generation (RAG) framework, named TableRAG, which utilizes query expansion along with schema and cell retrieval to identify essential information effectively. Furthermore, this study contributes to establishing new benchmarks for million-token-sized tables, significantly enhancing the understanding of LLM performance in large-scale table analysis.

Strengths

- the idea of schema-cell retrieval seems reasonable and shows better performance compared to baselines on some types of questions. - this paper seems contribute to new benchmarks concerning million-token sized tables and enhances the understanding of LLM performance in large-scale table comprehension.

Weaknesses

- while the idea of schema-cell retrieval seems reasonable, the generalizability of the method is not very clear, especially for some questions that cannot be directly addressed by manipulating the tables (also noted in the limitation section) - the experiments only consider GPT-3.5-turbo and Gemini-Pro as LLMs which are limited, further testing on more open-sourced and close-sourced models is necessary to prove the consistency of the proposed method. - the performance in Table 4 appears effective solely for the ArcadeQA dataset, with the retrieval of schema and cells proving useful. However, for BirdQA, the enhancements seem limited. I'm little concerned about the method consistency when adding more datasets.

Questions

- for figure 8, I am slightly confused by the results obtained from the solver. It appears the query intends to request the average price for all types of wallets. Yet, in the provided example, the third action only yields the average price of leather wallets (leather_wallet_prices.means()), rather than the average for all wallets (wallet_prices.means()). I'm uncertain if this discrepancy stems from a misrepresentation or if there's potential confusion in the example. - for controlling the budget in cell retrieval, how does the paper manage a column like "description," which consists of long sentences that might exhaust the entire budget, preventing other cells from being processed? - for schema/cell retrieval, if the query expansion driven by LLMs fails to generate keywords that align with the schema/cells in the table, what will the outcomes be, or are there any solutions proposed in the paper? Additionally, I am curious about how the paper attempts to align the schema mentioned in the table with the knowledge from LLMs themselves, or does it simply rely on the generalizability of LLMs?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

- the proposed method seems to primarily support the questions that can be answered by manipulating the tables, such as "What is the average price of the wallet?" Here, queries can be transformed into specific table operations (e.g., df['item_total'] = df['item_total'].str.replace('$','').str.replace(',','').astype(float)). However, it may not perform as well with more complex reasoning tasks. For instance, in the Tabfact dataset, where one must determine from a Wikipedia table and its caption which statements are supported or contradicted, the method might not show promising results for such reasoning tasks.

Summary

The paper introduces TableRAG, a framework that enhances LM-based table understanding by incorporating query expansion and retrieval mechanisms. TableRAG aims to improve the performance of large-scale table understanding tasks by efficiently encoding data and utilizing precise retrieval techniques. The framework achieves state-of-the-art results on various benchmarks and datasets.

Strengths

1. The overall method is relatively simple and intuitive. 2. Experimental results on three datasets show the effectiveness of the proposed method.

Weaknesses

1. The contribution of the paper is relatively small. The author mainly builds upon the original Row Column Retrieval and further reduces the retrieval cost by Scheme Cell Retrieval. The original work has included column selection, but this paper only further reduces the returned row to only return specific cells. 2. The baseline of the experimental comparison is relatively weak, and the advantage of finer grained retrieval is obvious.

Questions

N/A

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper introduces a novel approach to retrieving cell values and database schema within the table question answering domain. The method operates as follows: 1) It expands the initial question by identifying potential queries for column retrieval or cell value retrieval. 2) For each query generated in the first step, the method identifies the top-K columns using their embeddings. 3) Within a given encoding budget for cell retrieval, it proposes embedding the most frequent B distinct values for each column and retrieves them based on their similarity to each query. Additionally, the authors have developed two new datasets for large-scale table question answering. Their approach demonstrates improved cost efficiency and effectiveness on these large-scale datasets compared to previous methods.

Strengths

1) The proposed approach significantly outperforms the baseline models, particularly in retrieving cell values, showing a substantial improvement in recall metric. 2) Unlike previous works in the table question answering domain, which focused on small-scale tables that do not accurately represent real-world databases, this approach tackles a more challenging setting. It addresses scenarios where tables can contain millions of rows and numerous columns. The authors also developed two new datasets specifically designed for large-scale table question answering. 3) The paper includes well-conducted ablation studies and experiments that compare their method against their baseline methods. These studies demonstrate the effectiveness of each component of their approach.

Weaknesses

1) My primary concern pertains to the comprehensiveness of the baselines used. While the paper correctly mentions that the table schema is predominantly considered in the text-to-SQL domain, cell values also play a crucial role in SQL generation. Notably, papers like CodeS [1] have proposed using BM25 retrieval to find cell values, which would serve as an excellent baseline for comparison. Additionally, schema linking—the process of identifying the correct rows and columns—is well-explored. Approaches proposed in studies like TaBERT [2] could also be utilized as baselines. 2) Given an encoding budget, the strategy of retaining only the most frequent distinct cell values seems problematic, especially for queries that require data from columns containing names, addresses, etc. Heuristic-based methods, such as those using syntactic similarity with edit distance, can effectively filter most of the cell values and might be more appropriate. [1]: CodeS: Towards Building Open-source Language Models for Text-to-SQL [2]: TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Questions

Not applicable

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Not applicable

Summary

The paper introduces TableRAG, a novel framework that improves LM-based table understanding by incorporating advanced query expansion and retrieval mechanisms. TableRAG addresses the critical scalability challenges associated with large-scale tables by efficiently encoding data and employing precise retrieval techniques. Through a comprehensive evaluation on self-constructed benchmarks sourced from real-world datasets and synthetic data from TabFact, TableRAG demonstrates superior performance and reduced token consumption across various table sizes.

Strengths

1. **Scalability**: The framework is scalable for larger tables, as demonstrated by new benchmarks sourced from real-world datasets and synthetic data from TabFact. 2. **State-of-the-Art Performance**: TableRAG achieves the highest retrieval quality and new state-of-the-art performance on large-scale table understanding tasks.

Weaknesses

1. **Lack of Novelty** Many papers about RAG reveals that the accuracy of the evidence is crucially important for the performance, and the idea of this paper is to is another practice in the TableQA domain. 2. **Lack of evaluation robustness** The paper only evaluates the model on closed models gpt-3.5 and gemini, but fails to evaluate on open models like Mistral, Llama etc.

Questions

1. Why not evaluate on more models? 2. Why not evaluation on commonly used TableQA benchmarks like WikiTableQA?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes, the paper discusses the limitations.

TabFact Example

To help reviewers better understand how TableRAG works and its capabilities, we provide a real example from TabFact for reference. When asked to verify, **“the game against New Orleans didn't have the highest attendance,”** TableRAG successfully answered the question with the following procedure: > Schema Retrieval Queries: **opponent, attendance** > Schema Retrieval Results: > *{"column_name": "location attendance", "dtype": "object", "cell_examples": ['us airways center 18422', 'american airlines center 20301', 'new orleans arena 17781']}* > … (omit other retrieval results) > > Cell Retrieval Queries: **2008-09 phoenix suns season, game against new orleans, highest attendance** > Cell Retrieval Results: > *{"column_name": "team", "cell_value": "new orleans"}* > … (omit other retrieval results) > > Thought: **I need to find the attendance for the game against New Orleans and compare it with the attendance for other games.** > Action: `df[df['team']=='new orleans']['location attendance']` > *Observation:* > *3 new orleans arena 17781* > *Name: location attendance, dtype: object* > Thought: **The attendance for the game against New Orleans was 17,781.** > Action: `df['location attendance'].str.split(' ').str[-1].astype(int).max()` > *Observation: ValueError: invalid literal for int() with base 10: 'center'* > Thought: **I need to adjust the approach to extract and compare attendance values correctly.** > Action: `df['location attendance'].str.split(' ').str[-1].str.replace('center', '').astype(int).max()` > *Observation: ValueError: invalid literal for int() with base 10: ''* > Thought: **I need to handle the case where the attendance value is not a valid integer.** > Action: `df['location attendance'].str.extract('(\d+)').astype(float).max()` > *Observation:* > *0 20301.0* > *dtype: float64* > Thought: **The highest attendance in the table is 20,301, which is not for the game against New Orleans with an attendance of 17,781. > Final Answer: Yes** In this example, TableRAG performs the following steps: 1. Successfully retrieve relevant schema and cells. 2. Write the correct program to obtain the score against Team New Orleans. 3. Recognizes that the cell values are compound strings rather than numerical values. 4. Writes and corrects the program to parse the numerical values from the column and find the maximum. 5. Compare the two numbers and conclude the final answer. This example demonstrates TableRAG’s capability in understanding, reasoning, and planning, which goes beyond simple table manipulation and Text2SQL.