Evaluating Fallback Strategies for Reliability in Large Language Model Systems

Large language models (LLMs) power conversational agents, search assistants and recommendation engines, but nondeterminism and third-party dependencies create timeouts and incorrect outputs. Production systems mitigate these failures with caching, fallback policies and retry logic, yet there is limited empirical guidance on correctness-latency-cost trade-offs. This paper presents a reproducible evaluation framework and harness, and applies it to cache, small-model, rule-based, and multi-stage fallback policies. We show that combining a semantic cache with a smaller fallback model improves accuracy from 85% to 96% while keeping cost within 4% of baseline, with a 10% increase in mean latency.