Large Language Models (LLMs) have demonstrated impressive capabilities across natural language tasks, yet critical concerns persist regarding their factual reliability, societal bias, and alignment with regulatory norms. Central to addressing these challenges is the ability to systematically extract, normalize, and rank the claims made by LLMs-whether factual, normative, or policy-relevant. However, existing approaches often assume that claims are self-contained within individual sentences, overlooking the reality that many important claims emerge only through multi-sentence context. This leads to fragmented analysis and underestimates the complexity of model behavior. Furthermore, current methods are limited in scope, often relying on narrow domains and fixed knowledge sources, and struggle to identify or prioritize claims with potential for social harm or legal noncompliance. By advancing methods for context-aware claim extraction, standardization across sensitive attributes and regulatory categories, and risk-informed ranking, this research aims to provide a more comprehensive foundation for evaluating and auditing LLM outputs. Such a framework is essential for building systems that are not only factually grounded, but also fair, transparent, and compliant in high-stakes applications.
Paper
Full text
Towards Trustworthy AI: Enhancing Factuality, Bias, and Compliance in LLMs
Semantic Scholar · Computer Science · 2025
Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities across natural language tasks, yet critical concerns persist regarding their factual reliability, societal bias, and alignment with regulatory norms. Central to addressing these challenges is the ability to systematically extract, normalize, and rank the claims made by LLMs-whether factual, normative, or policy-relevant. However, existing approaches often assume that claims are self-contained within individual sentences, overlooking the reality that many important claims emerge only through multi-sentence context. This leads to fragmented analysis and underestimates the complexity of model behavior. Furthermore, current methods are limited in scope, often relying on narrow domains and fixed knowledge sources, and struggle to identify or prioritize claims with potential for social harm or legal noncompliance. By advancing methods for context-aware claim extraction, standardization across sensitive attributes and regulatory categories, and risk-informed ranking, this research aims to provide a more comprehensive foundation for evaluating and auditing LLM outputs. Such a framework is essential for building systems that are not only factually grounded, but also fair, transparent, and compliant in high-stakes applications.