LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation

Evaluating autonomous vehicles with controllability allows for scalable testing in counterfactual or structured settings, improving both efficiency and safety. We introduce LangTraj, a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenar-ios. By conditioning on natural language inputs, LANGTraj enables flexible and intuitive control over interactive behaviors, generating nuanced and realistic scenarios. Unlike prior approaches that rely on domain-specific guidance functions, LangTraj incorporates language conditioning during training for more intuitive traffic simulation control. In addition, we propose a novel closed-loop training strategy for diffusion models to enhance realism in closed-loop simulation. To support language-conditioned simulation, we develop a scalable pipeline for annotating agent-agent interactions and single-agent behaviors, which we use to develop InterDrive, a large-scale dataset offering diverse and interactive labels for training languageconditioned diffusion models. Validated on the Waymo Motion Dataset, LangTraj demonstrates strong performance in both realism, language controllability, and languageconditioned safety-critical simulation, establishing a new paradigm for flexible and scalable autonomous vehicle testing. Project website: https://langtraj.github.io/.