UQE: A Query Engine for Unstructured Databases

Summary

This paper proposed a Universal Query Engine to directly draw insights from unstructured data. For aggregation queries, this paper designed an unbiased sampling algorithm to solve the problem that it’s hard to apply index on virtual columns. Inspired by the workflow of the C++ compiler, this paper also proposed a workflow of optimizing and generating prompt to achieve more efficient query.

Strengths

1. This paper suited an interesting and important problem, by using LLM, more complex, semantics -related SQL queries can be achieved. 2. This paper took efficient into consideration and proposed corresponding algorithm and workflow to improve the query speed. 3. Provided theory proof of the unbiased sampling algorithm. 4. The idea of introducing compiler’s workflow into such problem to accelerating the LLM’s inference is interesting.

Weaknesses

1. UQE seems to lack scalability on the Semantic Retrieval task. The latency reported in Table 5 is as high as 38.08 seconds. Also, the paper didn’t state that whether these numbers are the total latency or average latency per query. If they are latency per query, using UQE on the Semantic Retrieval task still has a far distance from practice. 2. The cost of embedding operation of Algorithm 1 and 2, and the cost of Algorithm 2 are not discussed in the paper. Although the authors wrote in the appendix that the g function in Algorithm 2 caused the high latency problem, no quantitative analysis was provided. 3. Lack of comparison with important baselines. I noticed that the references [9] and [33] mentioned in related work are not included in the baselines. Using LLM as UDF may significantly slow down the queries, however, they may have better accuracy on aggregation queries. Thus, I think it’s necessary to include them in the baselines to further evaluate the performance of the trade-off made by Algorithm 1 between accuracy and latency. 4. Lack of evaluation of UQE’s time cost in the main text of the paper. I noticed those results are reported in the appendix. However, in the database area, the latency and cost are significant metrics that should be reported in the main text of the paper.

Questions

I noticed that lc-gpt-4-turbo also has quite a high latency on Semantic Retrieval tasks, is it possible to do more optimization on the model itself to improve the performance to make such a method practice?

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Yes, the authors have mentioned the limitations of this work, but I suggest to provide more comprehensive evaluations, detailed discussions about UQE’s scalability and cost. After all, these are critical problems about the practicality.

Summary

This paper propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections. Further, a Universal Query Language (UQL) is proposed as a dialect of SQL that provides full natural language flexibility in specifying conditions and operators. UQE leverages the ability of LLMs to conduct analysis of unstructured data and achieve efficient and accurate query execution.

Strengths

1. The idea of designing new query language and query engine for unstructured databases leveraging the power of LLMs is interesting. 2. The paper is also very clear with thorough experiments and analysis. 3. Well-organized presentation of the proposed method and its components.

Weaknesses

1. Many details about the design and implementation of UQE are missing. For example, the description of the generation of machine-specific code (assembly instructions) during the compilation stage (L251) is not clear. Please clarify the details of this strategy. Additionally, the authors claim that calling the LLM dominates the overall cost per query (L239). Providing detailed information about the LLM-calling stage would be beneficial. 2. The indexing technique introduced by UQE is related to the sampling strategy based on specific queries. However, database indexing is traditionally meant to facilitate quick searches. If each query requires sampling, will this impact efficiency? Additionally, doesn't this compromise accuracy? 3. This paper primarily addresses unstructured datasets, including video and audio. However, the experiments only involve text and images, lacking validation for broader generalizability. 4. Furthermore, there are plenty of multi-modal vector databases, yet the authors only compare the RAG-based query. Can UQE outperform recent multi-modal query databases, such as Milvus and MillenniumDB? If so, demonstrating this would make the evaluation more convincing. 5. Does the query support join operations, such as joining across multiple different modal files? 6. It's hard to validate the results without supplying code/models.

Questions

Please see the weakness section.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

No discussion on limitations, I suggest the authors to add one.

Summary

This paper proposes an ambitious new framework for analytics on unstructured databases, the Universal Query Engine (UQE). The authors first present a list of semantics and clauses for querying unstructured databases and propose methods for implementing these functionalities, including indexing and compiling. Experiments on multimodal unstructured data analytics tasks show that UQE outperforms other methods, demonstrating the promise of utilizing UQE in database analytics.

Strengths

- The motivation is quite interesting and necessary to broaden the functionalities of databases. - The paper is clearly written and well-presented. - The authors presented the exact scope of the task and proposed methods accordingly. - The proposed approach is reasonable and performs well within the scope of the work.

Weaknesses

- Could you elaborate more on the difference between BINDER and this approach (line 39)? Is it due to the lack of indexing and compatibility with unstructured information? - Similar to the first point, the baselines seem a bit weak. Why have the authors not used BINDER as their baseline? - I want to hear the authors' opinions on how advancements in LLMs and vision models might affect the guarantee that databases consistently return the same results. Traditional database engines, which are widely used, provide this consistency over time. Could these technological advancements affect the reliability of database analytics and, subsequently, compromise their reliability?

Questions

I have listed my questions in the Weaknesses section.

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations of the work are properly stated.

Summary

This paper proposes an unstructured query language based on a small segment of SQL. Its key feature is that it further supports unstructured texts and images because the authors assume that LLMs work on intra-row texts. During the query, the authors propose using online learning on LLM outputs over batches to improve the sample quality in the upcoming batches. Empirical results convey important information; that is, feeding longer context into larger LLMs does not work as well as using smaller LLMs in a more organized way.

Strengths

This paper discusses several possibilities for using LLMs in relational databases with text columns. It reveals a new possibility that one could do something between 1. using LLMs as a black box and letting them handle everything in the context. and 2. using LLMs to parse the question into SQL and let the database engine do the job with much smaller LLMs. For queries requiring sampling, a simple online learning algorithm is used to improve the sample efficiency.

Weaknesses

- This paper is more like a technical report or a guideline for building certain products. The overall product presented is simplified from the introduction's proposed question. Realizing this system under the strong assumption that LLMs work on intra-row semantic understanding is not too difficult: the virtual column, although it seems to be filled by LLMs, can be realized in several prompts, such as closed-domain classification for WHERE and open-domain classification for GROUPBY.

Questions

- Given that the output in the virtual column will work with other parts of the SQL, what is the failure rate in output formatting in the experiments?

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes,

last reminder on adding your thoughts on NeurIPS authors' feedback

Dear reviewers, We sincerely appreciate your effort during the review process. As the deadline for reviewer-author discussion is approaching in several hours, we would like to send out a last reminder to see if you have any quick follow up questions to our rebuttal, so that we might still be able to respond. Thank you so much for your understanding! Best, Authors