DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction

Summary

In their study on the Text2SQL task, the authors demonstrate the efficacy of breaking down the generation problem into sub-problems when utilizing LLMs to improve performance on the Spider dataset. The approach is simple yet effective on the Spider dataset. It yielded consistent improvements across three LLMs: GPT-4, Codex Davinci, and Cushman. Notably, the paper achieves a new SOTA performance on the Spider holdout test set.

Strengths

- SOTA on Spider holdout test set - Approach consistently improve results on three OpenAI LLMs

Weaknesses

- The analysis and experiments conducted in the study were solely focused on a single dataset, namely Spider. - The study does not provide sufficient evidence to support the generalizability of the proposed approach to Text2SQL tasks in general. - It is important to highlight the existence of several large-scale and robust Text2SQL datasets that could have strengthened the results in terms of generalizability. Notably, the BIRD-bench dataset (https://bird-bench.github.io/) provides an additional resource for evaluating the proposed approach. Furthermore, the Spider dataset has various variants, such as Spider-SYN, Dr.Spider, Spider-realistic, and Spider-DK, which offer opportunities to assess the effectiveness of the approach in different contexts and scenarios. Conducting experiments on these datasets would have enhanced the overall robustness and applicability of the findings. - The foundation models utilized in the study are exclusively based on OpenAI models. - While the classification step for SQL difficulty proves effective on the Spider dataset, it is unlikely to exhibit strong generalization capabilities to other real-world Text2SQL data. The classification of SQL difficulty may be influenced by annotation artifacts specific to the Spider dataset.

Questions

1. How did the authors come up with the self-correction prompts? Was it through a lot of trial and error from the training and validation set? 2. How did the author decide on this set of in-context learning exemplars?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

No negative societal impacts

Summary

This paper addresses the task of text-to-SQL prediction using large language model (LLM) prompting. The authors propose DIN-SQL, a chain-of-thought (CoT) prompting method which decomposes the SQL prediction process into four substeps: schema linking, complexity classification, SQL prediction, and self-correction. For each step, a manually designed prompt with few-shot in-context learning samples are provided to the LLM to obtain the prediction. Especially, after complexity classification, samples with different complexity levels are addressed with different prompts. Experiments on the Spider dataset demonstrate the effectiveness of the proposed method.

Strengths

- The overall pipeline is conceptually simple but practically effective. It appears to successfully leveraged the power of LLMs and achieved promising results. - The idea of applying different prompts for samples with different complexity is justifiable. It is based on prior observations that CoT prompts are effective on harder samples but can hurt the performance on easy samples. The ablation studies also echoed this observation, and verified the usefulness of this proposed mechanism.

Weaknesses

- The proposed method does not carry much techinical innovation, as CoT is already a widely studied and applied technique, and the intermediate representation is imported from previous work NatSQL. - The complexity categorization and corresponding prompt designs are more or less tailored for the Spider dataset. It might be unclear whether the method is directly applicable in real-world scenarios, where the variety of target SQL might be much larger, and may include additional mechanisms such as self-join. - Based on my observations on the "sub-questions" generated for nested complex samples, it seems that many of them (though not all) are identical to the original questions. This might not be the desired behavior and leaves room for further improvements. For example: - "Find the title, credit, and department name of courses that have more than one prerequisites?" can be solved by knowing the answer to the following sub-question "What is the title, credit value, and department name for courses with more than one prerequisite?".

Questions

- Around L231: The SQL-prediction step prompt for nested complex questions are designed in a comprehensive way, allowing multiple sub-questions to be included. However, looking at the samples in the appendix, it seems that at most one sub-question is predicted. Is the method actually able to handle very complex queries by generating >1 sub-questions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Several limitations are discussed, although I think there are additional points as I mention above.

Summary

The paper proposed to decompose the text-to-SQL reasoning into multiple steps and solve them with large language models. The authors began by conducting an error study of LLMs with few-shot learning and identified the common errors, such as "schema linking" and "JOIN". To address these common errors, they proposed a new method called DIN-SQL which breaks the text-to-SQL task down into four modules: (1) schema linking, (2) query classification and decomposition, (3) SQL generation, and (4) self-correction. In the schema linking and query classification modules, examples from the Spider dataset's training set are used to guide LLMs in identifying the relevant database schema and potential SQL structure. The SQL generation modules leverage different examples to handle various SQL structures. For more complex SQL structures, such as non-nested complex and nested-complex queries, demonstration examples with hand-written intermediate reasoning steps are utilized. At last, the self-correction module asks the LLMs to correct the predicted SQL if they realize that their initial generation was incorrect. The proposed framework demonstrates remarkable performance, surpassing state-of-the-art models on the Spider test set.

Strengths

1. The proposed method DIN-SQL demonstrates very impressive results on the hold-out test set in the Spider dataset, showing the effectiveness of DIN-SQL on Spider. 2. The error analysis reveals the drawbacks of LLMs with standard few-shot demonstrations and the proposed method are designed to address these drawbacks.

Weaknesses

1. The proposed methods seem highly tailored to the examples in the Spider dataset. Previous work [1-2] has highlighted the spurious correlations in the Spider dataset, such as the strong lexical matching between the database and the question. It seems important to evaluate the proposed method on other datasets such as Spider-syn [1], Spider-DK [2], Dr. Spider [3], or KaggleDBQA [4]. 2. The proposed SQL generation modules require hand-written intermediate reasoning steps. I don't think it is a problem given the limited number of examples requiring annotation but my concern is whether these examples will still serve as good demonstrations for the datasets other than Spider. I hope this can be addressed along with 1. [1] Gan, Yujian, et al. "Towards robustness of text-to-SQL models against synonym substitution." arXiv preprint arXiv:2106.01065 (2021). [2] Deng, Xiang, et al. "Structure-grounded pretraining for text-to-sql." arXiv preprint arXiv:2010.12773 (2020). [3] Chang, Shuaichen, et al. "Dr. Spider: A diagnostic evaluation benchmark towards text-to-SQL robustness." arXiv preprint arXiv:2301.08881 (2023). [4] Lee, Chia-Hsuan, Oleksandr Polozov, and Matthew Richardson. "KaggleDBQA: Realistic evaluation of text-to-SQL parsers." arXiv preprint arXiv:2106.11455 (2021).

Questions

I am somewhat confused by the disparity between the results of DIN-SQL in the Spider development set and the Spider test set. On the test set, DIN-SQL demonstrates superior performance compared to the state-of-the-art model [1], achieving a margin of 5.4 points higher (85.3 vs. 79.9). However, on the development set, it falls short by 9.9 points (74.2 vs. 84.1) and is outperformed by other in-context learning methods [2]. Do you know what factors contribute to DIN-SQL's high performance on the test set but comparatively low performance on the development set? [1] Li, Haoyang, et al. "Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023. [2] Ni, Ansong, et al. "Lever: Learning to verify language-to-code generation with execution." arXiv preprint arXiv:2302.08468 (2023).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Looks good to me.

Summary

This paper proposes to improve few-shot prompting LLMs for text-to-SQL task. It first provides detailed error analysis on existing few-shot prompting LLM approaches, into six categories. Then the paper proposes a new approach to decompose the task into a few sub-tasks, solve each task individually, and compose the subtask for the final answer. Experiments show the proposed approach achieve state-of-the-art on the Spider benchmark dataset.

Strengths

1. The result is solid, as it achieves state-of-the-art on the challenging Spider benchmark dataset. In addition, this paper provides detailed ablation study and analysis, therefore it's straightforward to understand the strength and weakness of the proposed approach. 2. The paper is well-written, easy to follow the motivation and details.

Weaknesses

Overall, my biggest concern is there's little additional novelty beyond "just another prompt engineering paper". It's expected that with larger LM, few-shot approaches could outperform existing fine-tuning based approaches. However, after reading this paper, it's still unclear whether scaling LLM can solve this text-to-SQL task and if not, where is the bottleneck. More specific concerns: 1. The manual error analysis (Section 3) only applies to Codex, it's unclear whether these errors still exist for larger language models such as GPT4. 2. There's only marginal improvement compared with chain-of-thought prompting (Table 4). And I guess the margin would be even smaller if we use GPT4 as LLM.

Questions

1. What's the difference on errors for Codex and GPT4 2. What's the result on chain-of-thought promoting for GPT4. 3. What do you think is the upper bound for this dataset, and do you think scaling LLM can achieve the upper bound?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None

Summary

The paper's contributions are the following: - The authors examined the common failure modes of doing text-to-SQL with few-shot prompting of LLMs: schema linking, JOIN, GROUP BY, nested queries and set operations, invalid SQL and miscellaneous. - The authors propose a method to do text-to-SQL by decomposing the task into 4 sub-problems, and solving each with a few-shot / zero-shot prompt to an LLM: schema linking, classification and decomposition, SQL generation, and self-correction. Their in-context learning method is reported to attain the highest execution accuracy on the test set of the Spider dataset, without making use of database content. - The work shows that LLMs can be used for text-to-SQL via prompting with performance equivalent to or better than methods that make use of fine-tuning.

Strengths

- The proposed method is a good application of chain-of-thought style problem decomposition to in-context learning techniques for text-to-SQL. - The method is well motivated by first conducting an examination of common failure modes when using LLMs for text-to-SQL in the few-shot setting. Investigating the error improvements using their method in Figure 4 is a very clear way to show how their method helps with better in-context learning. - The paper is written in a largely clear manner that makes it easy to follow. - The paper's results are a relevant contribution to the text-to-SQL community, in that their performance is significantly better than other LLM techniques like those in Rajkumar et al., 2022 and Liu et al., 2023.

Weaknesses

- There could have been more discussion about how the proposed four modules have been implemented without using prompting, to better situate the work in the literature. - The paper should have been clearer about how the intermediate representation of NatSQL bridges the gap between queries and SQL statements. An example each for the non-nested complex class and nested complex class would have been helpful in the main paper, instead of leaving it to the Appendix. In particular, it is not clear in the paper 1) how the intermediate representation is obtained, 2) how the removal of operators like GROUP BY or the WHERE clause from the syntax of the intermediate representation can help the LLM still generate the right SQL statements and 3) how the LLM is induced to solve sub-queries for the nested complex class. - The paper could have been clear about the latency of its proposed method. While high-performing, it is likely that making several sequential calls to an LLM will be high latency.

Questions

- How is the intermediate representation obtained? I refer to Appendix A.5.2 to see that it is inserted into the prompt, but it is not clear where this is generated. - How do the authors think the removal of operators like GROUP BY or the WHERE clause from the syntax of the intermediate representation help the LLM still generate the right SQL statements? - Can the authors illustrate how a query is decomposed into sub-queries for the nested complex class? It is not clear to me still in Appendix A.5.3. - What is the latency of the proposed decomposed in-context learning method, and how does it compare to other methods (like RESDSQL-3B + NatSQL or Graphix-3B+PICARD)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, limitations are adequately addressed

Spider variant or other text2sql datasets

Thank you for your response. > 1. Many of these benchmarks share a similar SQL query structure with the Spider dataset but extend the benchmark in one direction. > 2. Cost was the main factor in evaluating our model. We had to use the OpenAI API calls in our evaluation, and the cost of making those API calls with the large number of queries in the Spider dataset was not cheap. Running our model on other variants would have doubled or tripled the cost. Given the significant cost implications associated with evaluating a few thousands of examples, I remain skeptical about the contribution and practicality of this method for real-world natural language interfaces to databases. I think both BIRD and Dr. Spider evaluate systems from multiple directions. Also, there are non-API open-source LLMs such as Llama and GPT-J-6B.

Evaluation on other datasets

Thank you for providing the response. I think that the disparity between the results of DIN-SQL in the Spider dev set and the test set, when compared to other approaches, raises a crucial concern about the robustness/stability of DIN-SQL. Given that, it is essential to evaluate the proposed method on another text-to-SQL dataset.

DIN-SQL's triumph on BIRD benchmark

As a follow-up on the applicability of our approach to other text-to-SQL benchmarks, we now have the results on BIRD, another challenging benchmark that was released after our paper and was cited by reviewers. Based on a comprehensive evaluation conducted jointly with the authors of BIRD, we are happy to announce that our approach not only demonstrates outstanding performance on their benchmark, but it also sets a new state-of-the-art performance. The final performance metrics for our approach are as follows: 50.72 in Execution Accuracy (EX) and 58.79 in Valid Efficiency Accuracy (VES) on the development set; and 55.90 in EX and 59.44 in VES on the test set. These results are remarkable, truly confirming the robustness and effectiveness of our method across different databases and benchmarks. For comparison, the prior state-of-the-art methodologies achieved scores of 46.35 in EX and 49.77 in VES on the development set, along with 54.89 in EX and 60.77 in VES on the test set. It should be noted that Spider queries cover 200 different databases and BIRD queries cover 95 different databases. These outcomes not only validate the efficacy of our approach but also highlight its potential for generalization to diverse text-to-SQL scenarios.

Please Reply to Author Rebuttal

Dear reviewer, Thanks a lot for your efforts and valuable reviews. Would you please check this author rebuttal and see how they address your concerns on more discussion on implementation without prompting, intermediate representation of NatSQL, and latency of its proposed method? Please reply to authors by adding your following comments below this author rebuttal. As the author-reviewer discussion is closed soon, we would appreciate if you could submit your reply to authors by Aug 21st 1pm EDT. Thanks! Best, AC

Please Reply to Author Rebuttal

Dear reviewer, Thanks a lot for your efforts and valuable reviews. Would you please check this author rebuttal and see how they address your concerns on novelty, manual error analysis, and marginal improvement? Please reply to authors by adding your following comments below this author rebuttal. As the author-reviewer discussion is closed soon, we would appreciate if you could submit your reply to authors by Aug 21st 1pm EDT. Thanks! Best, AC