ScenicNL: Generating Probabilistic Scenario Programs from Natural Language

Summary

This paper describes a system to generate a 3D animated scene of vehicle accidents from written descriptions. The processing pipeline take a description as input, passes it to a large large language model that generates a program to control the CARLA visualization system that produces the animation. The authors collected a corpus of accident narratives that they classified according to their interpretation difficulty. Given a text input, the authors used a LLM and a sequence of prompting strategies to identify the actors in the text, infer some unknown conditions like the weather if not described in the text, and generate the code to create the scene and animate the actors. The authors used an SQL-like LLM language to create this code from a set of constraints on the actors. Finally, the code serves as input to the CARLA API to produce the animation. The authors evaluated their system on the easy part of their dataset with a few metrics, but not the final visualization

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* Interesting and ambitious application * Smart aggregation of tools: LLMs and CARLA visualization to produce a 3D generation * Partial evaluation that hints at a promising performance

Reasons to reject

* The system seems to be at an early development stage * The part describing the generation of code with LMQL and HyDE would be impossible to reproduce from the description in the article * The authors contribution is sometimes difficult to evaluate as they merely glue together existing programs and resources * The system is only partially evaluated. The authors notably do not address the validity of the end visualization * The authors used a curious wording in the introduction to describe the problem they solve. See the paragraph starting with Given the critical need for simulation based testing...

Questions to authors

* Did you think of an evaluation of the final generation? * Will you release your code or instructions, especially the part involving LMQL? * How did you create the example scenic programs from Fig. 1?

Summary

Due to the scarcity of real-world recorded car crash videos and sensor data, there is a heavy demand for simulation-based data. This paper proposes to use LLMs to generate Scenic programs based on natural language descriptions from car crash reports. These programs can then be simulated and used to train more robust autonomous driving systems. They combine many prompt engineering techniques and propose ScenarioNL that achieves 90% program syntactic correctness and gets high human ratings (4.3 out of 5 on average).

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. They apply LLMs in an interesting, novel, and challenging scenario. 2. They thoroughly explore and combine many prompt engineering techniques. 3. Their results show that their final method (ScenarioNL) works very well, compared to single prompt engineering method. 4. They show that through careful prompt engineering, LLMs can be used to generate domain-specific languages that LLMs have not seen and little data exists for finetuning.

Reasons to reject

1. The paper is more like a technical report for solving a domain-specific code generation task via prompt engineering. I find limited research interest out of it. 2. Though the paper tries to claim that their methods are not unique to Scenic, the generalizability is concerning. ScenarioNL is more like a task-specific recipe. I don't see how it can easily generalize to other domain specific languages. 3. As an application oriented paper, the discussion or evaluation on how useful the LLM-generated programs are for downstream training of autonomous driving systems is quite thin.

Questions to authors

1. Do you plan to release your code and the generated programs? Because the whole system involves many steps, without code releasing, there might be reproducibility issues.

Summary

This paper introduces ScenarioNL, an AI system that creates scenario programs from natural language in police crash reports. It uses a Probabilistic Programming Language called Scenic to handle uncertainties about the scenarios. The system, which combines LLMs, prompting strategies, a compiler, and a simulator, has been tested on autonomous vehicle crash reports from California.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- Very clear and compelling motivation that helps the reader understand the problem of creating scenario programs to model CPS. - The processed AV DMV crash reports datasets is a useful contribution for the research community. - The methods underlying ScenarioNL highlight some limitations of LLMs in out-of-domain settings and how to circumvent them.

Reasons to reject

- There are few details of the evaluation setup. For example, how many experts were used to evaluate the outputs? Were the outputs anonymized? - The system behind ScenarioNL is interesting but has several components that are not presented in detail. It would be beneficial to have a more thorough description of exactly which components are used in ScenarioNL and which ones contribute the most to the overall performance. In particular, are all the components needed? - The RAG/RAG+HyDE performance is very high compared to the rest. Why is that? How did you ensure that the data store used does not contain examples that are similar to the solution?

Questions to authors

- Why is expressiveness interesting to track and how is it correlated with desirable properties? Wouldn’t accuracy or semantic similarity to the language prompt be a more interesting metric? - See questions regarding the evaluation setup - See questions about RAG results - Have you tried to provide Scenic documentation in a prompt? It would be interesting to know the performance of this simple approach to understand the degree to which LLMs are capable to rely on a description of a DSL