A Language Agent for Autonomous Driving

Summary

This paper introduces an LLM-supported agent named Agent-Driver, and the proposed system consists of a multifunctional tool library, a cognitive memory, and a reasoning engine. The cognitive memory contains commonsense memory and experience memory. The reasoning engine contains four parts: chain-of-thought reasoning, task planning, motion planning, and self-reflection. It conducted experiments on open-loop and close-loop driving challenges, and achieved SoTA results on the nuScenes benchmark.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

- Agent-Drivers are redefining the traditional framework of autonomous driving by embedding human-like reasoning and decision-making capabilities. The system uniquely integrates a dynamic perception and prediction tool library and cognitive memory to achieve a more humane and anthropomorphic driving process. - This paper is well structured and highly readable. - It surpasses existing autonomous driving methods, showing over 30% collision improvements in motion planning, and excels in few-shot learning on the nuScenes benchmark.

Reasons to reject

- Using LLMs as agents to achieve reasoning and memory functions is not a unique design of this work. Previous research, such as the DiLu [1] framework, has also implemented similar concepts. [1] Wen L, Fu D, Li X, et al. Dilu: A knowledge-driven approach to autonomous driving with large language models. ICLR, 2024. - Although the paper proposes a novel paradigm for autonomous driving, the inherent latency issue due to reliance on API calls to LLM is still a concern. The author also mentioned this in the limitation, which is worthy of recognition. The delay issue is directly related to the real-time applicability of this method in actual driving scenarios.

Questions to authors

Have the authors considered choosing a loadable LLM model such as Vicuna instead of calling the LLM Api?

Summary

This paper presents Agent-Driver, an LLM-based agent designed for autonomous driving. Agent-Driver operates other perceptual neural models as tools and is equipped with a reasoning engine that facilitates chain-of-thought reasoning, task planning, motion planning, and self-reflection. The model's promising performance is evidenced by both open-loop and closed-loop evaluations. Overall, this paper is well-written and features robust experimental work, with some minor concerns over the setups and reproducibility.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The Agent-Driver: The proposed Agent-Driver framework applies an LLM agent to the autonomous driving domain. This agent operates other perceptual neural models as tools and includes a sophisticated reasoning engine. The technical execution regarding the LLM agent is thoroughly solid. 2. Experiments: The experimentation is comprehensive. It includes both closed-loop and open-loop evaluations, alongside various ablation studies that test different LLM variants and neural module variants, providing a thorough examination of the system's capabilities.

Reasons to reject

1. Multimodal Domain Integration: Given that autonomous driving is a highly multimodal domain, this paper utilizes perception modules from prior work, such as UniAD. However, considering the availability of multimodal LLM agents like DriveGPT-4, the paper should justify why it opts to use other modality processors as tools instead of leveraging a unified multimodal approach. 2. Language-related Benchmarks: It would be beneficial for the experiments to be conducted using the BDD-X dataset, which focuses on the language capabilities and interpretability of driving agents. This would provide a clearer assessment of how well the Agent-Driver can handle real-world language and driving scenarios. 3. Clarity in Experimental Setup: The descriptions of the closed-loop experiments lack clarity. It is not specified whether the same neural networks are employed across different test setups, which is crucial for assessing the consistency and reliability of the results.

Questions to authors

Questions: 1. Are the same neural networks employed in the CARLA evaluation? 2. As an LLM agent, is Agent-Driver capable of giving justifications to actions, e.g., as defined in the BDD-X benchmark? It might be interesting (though not required) to compare with effective baselines on BDD-X (ADAPT, DriveGPT-4, DriVLMe, etc). Suggestions: 1. I was confused by the terms “commonsense memory and experience memory”, I think in the memory modeling community these are close to semantic and episodic memories.

Summary

The paper presents Agent-Driver, a system utilizing an LLM as a driving agent. The system consists of several modules: tool library for detection, prediction, occupancy, and map modules, cognitive memory which allows search in past driving scenarios, reasoning engine which prompts an LLM for chain-of-thought reasoning, task planning, motion planning and self-reflection (which are all steps of giving prompts in a specified format). The system is evaluated with experiments on open-loop driivng on the nuScenes dataset and closed-loop driving in the CARLA simulator. The system is compared to several other recent works with LLMs for autonomous driving. The results show improvement in route completion for closed-loop driving and improvement on most metrics for open-loop driving.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

The presented system builds on top of recent line of research that uses LLMs for autonomous driving. In contrast to some recent work, the system uses components which specialize indifferent aspects of the driving task, and according to the reported results, this setup seems to be beneficial. The components are described in a good detail and the architecture of the system is mostly understandable.

Reasons to reject

Not really a reason to reject, but maybe something that should be explained. The text prompts to the LLM in the reasoning engine resemble the one in GPT-Driver. Maybe this could be commented. What is the difference between the reasoning engine and the implementation in GPT-Driver? Is the current paper an extension that adds memory and tool library (this does not mean this is bad, but just that maybe it could be explained for clarity)? In the current paper, the prompt is given step-by-step, how is this information passed to the LLM? As separate messages? Does it make any difference in comparison to having one big message, or does this separation make it easier for the model to learn?

Summary

The paper presents a novel framework using LLMs for autonomous driving planning tasks. The proposed method integrates 1) a tool library for perception and prediction, 2) a memory module of commonsense and experience, and 3) LLM-based reasoning module that performs chain-of-thoughts reasoning, and motion planning. The paper conducts comprehensive experiments to demonstrate 1) the effectiveness of the methods in terms of L2 distance to ground truth and collision rates, 2) few shot learning ability by training on a small percentage of data, and 3) performance difference for in-context learning and fine-tuning. The paper also includes comprehensive details of the method and implementation in the appendix (20+ pages), which is laudable. The paper is well written, and easy to follow. The diagrams are illustrative. Experiments are reasonable. Overall, it's a forward-outlook novel paper that showcases the effectiveness of LLMs applied to autonomous driving. This work is a valuable addition to the field. The results are worth sharing. I hesitate between top 5% and top 15% accept papers, as this paper is already pretty well-known in the field of autonomous driving, and is considered one of the early notable papers that apply end-to-end LLM agents to autonomous driving.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

- The paper presents a novel framework using LLMs for autonomous driving planning tasks. The proposed method integrates 1) a tool library for perception and prediction, 2) a memory module of commonsense and experience, and 3) LLM-based reasoning module that performs chain-of-thoughts reasoning, and motion planning. - The paper conducts comprehensive experiments to demonstrate 1) the effectiveness of the methods in terms of L2 distance to ground truth and collision rates, 2) few shot learning ability by training on a small percentage of data, and 3) performance difference for in-context learning and fine-tuning. - The paper also includes comprehensive details of the method and implementation in the appendix. - The paper is well written, and easy to follow. The diagrams are illustrative.

Reasons to reject

Some minor comments and questions (not necessarily reasons to reject the paper): I would be interested in learning - Training and inference costs. The paper only mentions it in Appendix G as "unable to obtain the inference time". It'd be helpful to provide a rough estimate of at least the magnitude of inference time compared to non-LLM based state-of-the-art-ish baselines. - Where does the method struggle? The paper provides a couple of failure examples but does not provide a description on where the LLM-based driver likely to fail. The limitation paragraph in the appendix only mentions inference time. - How much does CoT help in long-tail / challenging driving scenarios? The paper does not demonstrate it despite the opening example in the Intro section. Experiments - The paper reports L2 distance to ground truth and the collision metrics. Showing additional metrics such as progress, and comfort would strength the results. - What does "invalid" mean in sec3.6? - The main text does not specify the "in-context learning" method. From the appendix, in-context learning learns from "four human-annotated examples". Does fine-tuning also use such small number of samples? Would be better to clarify in either the main text or appendix to establish fair comparisons.

Comments from Reviewer

Thanks for the reply. I would like to keep my positive evaluation of this work.