Developers implement safeguards for large language models (LLMs) to restrict specific responses and prevent the generation of unsafe content. However, a growing number of jailbreak approaches aim to circumvent these safeguards by crafting prompts that elicit prohibited content. The success of these approaches is typically evaluated using techniques such as string matching and natural language understanding, which yield an outcome indicating whether the response violates the safeguard. Our analysis reveals two key limitations in existing jailbreak evaluation methodologies: (1) their evaluation objectives lack clarity and do not directly assess the safety of generated responses, and (2) they conflate different jailbreak response outcomes, oversimplifying the jailbreak result into a binary success/failure outcome. In this paper, we propose a novel set of metrics for evaluating LLM jailbreaks: safeguard violation, informativeness, and relative truthfulness. These metrics can differentiate between attackers with varying goals and capture the nuances of their motivations. To automate the computation of these metrics, we introduce an automated multi-metric jailbreak evaluation framework (amJE) comprising two components: (1) LLM response preprocessing, which tokenizes the response at the paragraph and sentence levels and removes invalid segments to enhance metric accuracy, and (2) natural language generation, which evaluates each metric from the processed responses. We evaluate amJE on a benchmark dataset containing 250 malicious intents constructed from recent jailbreak approaches. amJE outperforms three existing jailbreak evaluation methods, achieving average F1 score improvements of 45%, 322%, and 12% over three baselines. These findings highlight the need for sound and precise jailbreak evaluation metrics to ensure language model safety and accurately assess the goals of malicious actors.