Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

RTL generation demands more than code synthesis: designs must be syntactically valid, synthesizable, functionally correct, and hardware-efficient. Existing evaluations often stop at functional correctness, leaving synthesizability and implementation quality unmeasured. This paper evaluates 32 language models on 202 Verilog tasks from VerilogEval and RTLLM (with five attempts each), scoring via the Hardware Quality Index (HQI), a 0–100 metric integrating post-synthesis area, delay, and warnings related to expert references under a Nangate45 45 nm flow. Three performance regimes emerge: 14 frontier models achieve HQI above 66, led by Gemini-3-Pro at 87.5% coverage and 85.1 HQI; 15 mid-tier models cluster at 43–66 HQI; 3 fall below 43 HQI. The gap between best-of-five capability and single-attempt quality spans 3.7–22.1 HQI points, which reveals a deployment deficit that limits integration into agentic pipelines. A tool-adjudicated taxonomy of 195 genuine synthesis failures reveals systematic divergence: proprietary models fail late through elaboration errors and synthesis timeout; open models fail early often due to missing module wrappers and non-synthesizable constructs, a pattern consistent with training corpora skewed toward simulation over synthesis-grade RTL. Rankings hold across three technology libraries at Spearman ρ > 0.99.