How Prompt Design Affects LLM-Based Password Strength Evaluation: A Comparison Against zxcvbn

Abstract

Password-strength meters are widely deployed in modern authentication systems, with zxcvbn representing one of the most accurate and commonly used heuristic-based estimators. Recent advances in large language models (LLMs) raise the question of whether these models and their sensitivity to prompt design can reliably assess password strength. However, no prior work has systematically examined how prompt structure affects small, locally deployable LLM-based password-strength evaluation. In this study, we evaluate the impact of five prompt strategies (structured JSON, minimal instruction, step-by-step reasoning, expert persona, and zxcvbn-emulation) on the scoring behavior of a small LLM (Llama-3 8B). Using a controlled dataset of 100 passwords distributed across zxcvbn strength levels (0-4), we measure exact agreement, soft agreement, and hallucination rates relative to zxcvbn as ground truth. Our results demonstrate substantial prompt-dependent variability, with structured prompts yielding the highest stability and chain-of-thought prompts producing the highest hallucination rates. Across conditions, the LLM performs best on mid-strength passwords and consistently misjudges very weak and very strong passwords, forming a U-shaped error pattern. These findings highlight the importance of prompt engineering in security-sensitive tasks and show that small LLMs, while capable of limited alignment with heuristic meters, remain insufficiently robust for reliable password-strength scoring in real-world authentication systems.