High-integrity systems, such as autonomous vehicle fleets or large-scale energy infrastructures, rely on structured assurance cases, represented as rooted directed acyclic graphs (DAGs), to justify safety claims. To remain valid in the face of change, these cases must be robust against potential challenges, known as ‘defeaters’. While large language models (LLMs) have recently enabled scalable generation of such defeaters, validating their quality remains a predominantly manual and subjective process. This paper presents an automated method for assessing LLM-generated defeaters using natural language processing (NLP) techniques. Our approach combines structural analysis of assurance case graphs with vector-based semantic embeddings and meta-classifiers trained on expert-assessed consensus defeaters. We evaluate our method through two case studies in the automotive and energy domains, quantifying human reviewer dissensus using Cohen’s kappa ($\kappa \lt 0.442$), which indicates low inter-rater agreement. Our automated approach achieves greater consistency with individual raters, improving ($\kappa \approx 40 \%$). The method delivers an average F1-score of 0.84 across validation, reducing subjective variance through scalable, objective assessment. Our approach aims to advance the tool support for automation of assurance case synthesis.