AutoSurvey: Large Language Models Can Automatically Write Surveys

Summary

This paper introduces a framework AutoSurvey that uses LLMs to automatically write scientific literature surveys. The process contains mainly four steps with retrieval, content generation and evaluation. The authors compare the framework with human writing and naive rag based LLM generation, in terms of speed, citation quality (recall and precision of citations) and content quality (5 scale scoring by different LLMs as judges). Empirical results show the performance gain of the proposed method. The authors also did additional ablation studies on the robustness of the framework and the variations between different LLMs als base models for the framework. Those ablation studies show that the framework could be sensitive to the components of the framework.

Strengths

1. Good paper writing. The scopes are made clear and very easy to follow and reader-friendly. 2. Solid experiments. Additional ablation studies also provide valuable insights to the framework. 3. Very interesting usecase to employ LLMs to write literature surveys. The novelty should be acknowledged.

Weaknesses

1. As far as I understand, the authors only experiment with the Claude-3-Haiku model in the framework for paper writing. Although the authors employ different models in the evaluation process, the main findings could be biased based on only one model. 2. In evaluation of the citation quality, the metric used an NLI model but didn't report the details of the performance of the NLI model.

Questions

1. In Table 2, you also include the speed of human writing. Where does the result come from? 2. As far as I understand, the framework is conducted by a single LLM, or is it possible to have multiple LLMs involved in the process?

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are already presented in sec5.

Summary

This paper presents a methodology for generating automatically surveys called AutoSurvey. AutoSurvey leverages the power of Large Language Models (LLM) and a Retrieval Augmented Generation (RAG) approach using as external resource a database of publications. Based on those publications, an outline is generated which is used as a guide for generating the sections of the paper. After the section generation the generated survey is refined. This process is repeated for several iterations. AutoSurvey is compared with two other baselines. The first baseline involves human-written surveys while the second one is a RAG-based LLM. An evaluation technique is also defined called Multi-LLM-as-Judge which is a combination of various LLMs accessing the generated responses. The surveys are evaluated based on speed, citations and content quality. Experiments have been conducted for comparing AutoSurvey with the baselines and the results are presented in the paper. Moreover, Multi-LLM-as-Judge evaluation results are also compared with evaluation from human experts.

Strengths

• The paper is well-written and easy to follow. • The presented results support the claims of the paper. • Code is provided.

Weaknesses

- Database used for the retrieval is not provided. - The collection of publications seems specific to the field of computer science and related to Large Language Models. - Based on the ablation study, the retrieval technique has more impact on AutoSurvey. It seems that the reflection part does not really influence the results. - Details about the naïve RAG-based LLM used as baseline is limited. - More baseline techniques could have been used for comparison to better support the paper. - Line 66: Muti-LLM-as-judge -> Multi-LLM-as-judge

Questions

• In Figure 1 you mention that it requires 3 minutes for generating a survey, which according to your speed calculation is 20 while in Table 2 AutoSurvey has speed more than 73. How do you explain that? • In Figure 1 you mention a cost of $1.2 for AutoSurvey, does this mean that one has to pay this to use AutoSurvey? • The publications are basically in the computer science domain and more specifically related to LLMs. Have you tried to use publications for other domains? • It is not clear what do you mean in lines 257-258 “ …(4) can refer to 20 papers (30k tokens in total) retrieved using the options provided (Upper-bound, directly retrieving the answers).” • In the ablation study, which parts of AutoSurvey does the reflection study include?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Publications are limited to computer science domain and more specifically about Large Language Models.

Summary

In this paper, the authors propose an automated system based on LLMs to draft literature surveys on a given topic. The core idea behind the approach involves decomposing the task of writing a survey into multiple smaller subtasks in 4 stages. The first stage focuses on retrieving relevant papers from a database using embedding-based retrieval, leveraging the topic and abstract of papers for selection. An LLM is then prompted given the papers to generate a plan / outline for drafting the survey paper. To address the limited context window, the papers are randomly divided and processed across multiple LLM calls to generate several outlines. These outlines are then merged in a using another LLM call. In the next stage the subsections identified in the outline are populated by retrieving the relevant papers specified in the outline and providing it to an LLM to extract information relevant to the subsection. This is followed by an iteration and refinement stage where the goal is improve the overall coherence and readability of the drafted survey. Finally this is followed by an Evaluation phase where multiple LLMs are used to rate the overall generated content and the generated survey with the highest rating is picked. The authors conduct comparison experiments of their methods against Human and Naive RAG based LLM approaches and show that their proposed approach is faster and leads to better citation and content quality in some settings.

Strengths

The paper attempts to tackle a new problem of automatically writing academic surveys and provides metrics and evaluation criteria that can be used to compare different techniques on this task. The authors show that their technique can generate academic surveys much faster than humans / naive RAG based LLM generation. The authors perform experiments that depict that the proposed technique outperforms naive RAG based LLM generation in content & citation quality.

Weaknesses

The paper lacks a significant novelty component. The concepts of decomposing tasks into smaller subtasks for LLMs, using Retrieval-Augmented Generation (RAG), iteratively refining generated content with LLMs, and employing multiple LLMs as evaluators are well-established in the literature. The extension of these ideas to the application of automatically writing surveys seems to be the only novel contribution of this work. The evaluation criteria seem to overly rely on the ability of ML models (NLI and LLMs) to judge the quality of the survey. A well-written human survey, for instance, cannot be assessed merely based on the number of relevant papers cited or the presentation of content. Subtle nuances, such as comparing and weighing the pros and cons of different methods proposed in the literature through the use of statistical tools and analysis, are often distinguishing elements of a good survey. The human evaluations proposed in the paper do not appear to account for these qualities. The evaluation metric requires a closer examination. The experimental analysis is weak. Conclusions are drawn too early without much deliberation and analysis. For instance the authors do not provide a deeper analysis on why the Auto-survey performs better/at-par than human-writing for shorter context lengths while being significantly worse at longer context lengths.

Questions

Are there any experiments that show how useful these generated academic surveys are to people working in the field? Which aspect of the generated survey is the most helpful for humans? How well does this technique handle multimodal information (i.e figures/graphs) or information from structured sources such as tables?How often do these appear in the generated surveys?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The technique appears to be more effective at summarizing works in the literature for a survey, rather than performing the detailed analysis and comparison of works that characterize a high-quality survey. The technique seems heavily biased by the retrieval stage. This could limit its ability to reference and use relevant works whose topics/abstracts do not semantically match well with the topic of the survey.

Summary

This paper introduces a fast automating way to write literature surveys based on LLM.It aims to solve the challenges of large volume, complexity, context window limitations, parametric knowledge constraints, and lack of evaluation benchmark.The AutoSurvey pipeline contains initial retrieval & outline generation, subsection drafting, integration and refinement, rigorous evaluation and iteration. The experiments are conducted compared to human experts and naive RAG-based LLM and evaluated on survey creation speed, citation quality, content quality. The experiments show that AutoSurvey is much faster than human writing and RAG, matches human writing and outperforms RAG in citation and content quality.

Strengths

This paper is relatively novel and meaningful to adopt LLM effectively in automatic creation of survey papers. The paper is well structured, and clearly written. The general pipeline of AuthoSurvy is logical, from outline generation to integration and refinement, which ensembles the human writing process. The experiments and analysis are clear.

Weaknesses

- About evaluation metric, it is unclear how the citation quality and content quality are obtained, although the metrics and scores are defined by formulas and words. If you use an LLM-based procedure to get the scores for citation quality and content quality, what are the prompts you give to the LLM? Please give a clearer introduction on this. - The algorithm and methodology this paper introduces are too straightforward. Although it is a good application of LLM, I doubt the value of sharing this knowledge with the scientific society, especially for Neurips. - Regarding experiment comparison, I highly recommend comparing Autosurvey to not only naive RAG, but more advanced methods, which could make the results more convincing. - In table 2, apart from the speed, the other benefits and improvements that AuthoSurvey brings are not significant enough. - Minor issues: typo in line 66, page 2: Muti -> Multii

Questions

In Experiments, how do you obtain the citation and content quality score? What is the novelty of this method and the bottlenecks this paper breaks through? Does this method address the limitations of automatic survey generation? I would suggest a more thorough summary on it. Minor questions: Where do the forecasted numbers of publication in 2024 in Figure 1 come?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes. The limitations are discussed.

Acknowledgement of rebuttal

Thank you for providing the clarifications. Based on the comments provided by the authors and their intent to revise the manuscript with additional information I am raising my score.

Kindly Request for Reviewer's Feedback (deadline is coming)

Dear Reviewer, Since the end of author/reviewer discussions is coming in one day, may we know if our response addresses your main concerns? If so, we kindly ask for your reconsideration of the score. Should you have any further advice on the paper and/or our rebuttal, please let us know and we will be more than happy to engage in more discussion and paper improvements. Thank you so much for devoting time to improving our paper!

Thank you for the response

Dear authors, Thank you for the response. I think my initial score properly reflects the quality of this work. And I would like to see this overall exciting work included to the proceedings. Only one final remark: As for the scores for human writing, I would like to suggest the authors to include details of the human writing (e.g. in the appendix), as a few researchers would be very interested in this part.