TuRTLe: A Unified Evaluation of LLMs for RTL Generation

Rapid advancements in LLMs have driven the adoption of generative AI in domains like Electronic Design Automation (EDA). Within the field of software development, EDA presents unique challenges derived from specific requirements of generated RTL code; RTL code must not only be syntactically correct and functionally accurate, but also synthesizable by hardware generators, while matching performance, power and area (PPA) constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. To address this gap, we propose TURTLE, a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks. TURTLE integrates multiple existing benchmarks and automates the evaluation process, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion. Using this framework, a diverse set of forty open LLMs are assessed, tracking their strengths and weaknesses in EDA-specific tasks. Our results identify the best match for specific tasks (e.g., base models are better in module completion tasks, instruct-tuned models are better in specification-to-RTL tasks), while finding that recent models with autoregressive reasoning chain perform the best overall. Some benchmarks, particularly within syntax correctness, show signs of saturation, while others remain as open problems for LLMs.