It is 03:47 on a Tuesday at a Berlin AI-native SaaS. The on-call SRE gets the page. The company's main product — an AI-first document processor — is returning 500 errors to about 6,000 customers per minute. She opens the observability dashboard. The reason is not in the application. OpenAI's US-East-1 region has been degraded for twelve minutes, every request to gpt-4o is timing out, and the application's naive retry logic is turning each timeout into three more, all also timing out. The provider outage is real. The reason it is her outage tonight is that the application had no strategy for provider failure.
Every LLM provider will fail eventually. The question is not whether it happens; the question is whether users notice. Part 1 of this two-part series covers the six resilience patterns that separate a demo from a production LLM application — what each pattern is, when to reach for it, what it costs, and the minimum viable stack. Part 2 walks through the implementation with LiteLLM Router, Redis-backed circuit-breaker state, Tenacity with jitter, and the Prometheus alerts that actually fire.
Key takeaways
- Reliability is not about preventing failures. It is about recovering faster than users notice. Circuit breakers, fallback chains, and graceful degradation together give you a system that self-heals on any provider outage.
- The six patterns compose into three functional pairs — don't cascade (Circuit Breaker + Fallback Chain), manage concurrency (Retry with Backoff + Bulkhead), and operate in production (Graceful Degradation + Observability). Missing either half of any pair leaves a real gap.
- Circuit breakers must hold state across all replicas — a per-pod breaker still hammers the failing provider from every other pod. Redis-backed shared state is the discipline.
- Fallback chains earn their place when the workload is model-agnostic — summarisation, translation, generic Q&A. They earn less on tasks that depend on model-specific behaviour (long-context reasoning, tool-use fidelity, jailbreak resistance).
- The minimum viable resilience stack is smaller than the six-pattern list suggests — LiteLLM Router plus one Redis-backed circuit breaker plus one Prometheus alert on circuit-open covers roughly 80% of what breaks at 3 AM.
What Actually Fails
Before naming the patterns, name the failure modes. Rate-limit errors (HTTP 429) during traffic spikes, especially when a competing product on the same account is also spiking. Provider outages that ranged from twelve minutes to nine hours across the major providers over the last twelve months. Timeout errors on long generations where the tail latency crosses the 60-second mark and downstream clients retry aggressively. Quota exhaustion when a monthly limit trips silently mid-month. Model deprecations announced with 30 days' notice that the platform team missed.
Each of these turns into a user-visible 500 if the application has no strategy. Each of them turns into a background alert with no user impact if it does.
The Don't-Cascade Pair — Circuit Breaker + Fallback Chain
The two patterns that stop one provider's bad hour from becoming your bad night.
Circuit Breaker. Stops sending requests to a failing provider after a threshold, waits, sends one probe, resumes if the probe succeeds. Three states — CLOSED (normal), OPEN (failing, drop requests fast), HALF-OPEN (one probe in flight). Reach for it on every external LLM dependency. Trade-off — the breaker adds one Redis read per LLM call (or in-process state that does not survive across replicas — the former is what production looks like). Poorly tuned thresholds cause false positives during a transient hiccup, or false negatives during a slow degradation. Cost and effort — a few hundred lines of Python plus Redis. The tuning work — threshold, timeout, half-open probe cadence — is the ongoing tax.
Fallback Chain. OpenAI unavailable, route to Anthropic; Anthropic unavailable, route to Google; all unavailable, degrade gracefully. Reach for it on model-agnostic workloads where a competent-enough answer from a fallback is better than a 500. Trade-off — the chain assumes the workload is model-agnostic. Long-context reasoning, tool-use fidelity, and jailbreak resistance vary meaningfully by provider; a chain that treats them as interchangeable ships subtle regressions to production. Cost and effort — LiteLLM Router covers the integration cost; the semantic testing to prove the fallback is safe on your workload is the real cost, and it is where most teams under-invest.
The Manage-Concurrency Pair — Retry with Backoff + Bulkhead
The two patterns that stop one bad request from bringing down twenty good ones.
Exponential Backoff with Jitter. Do not retry immediately. Space retries with growing delays. Add jitter to prevent every client retrying at exactly the same moment. Reach for it on every transient error — 429, connection timeout, 5xx from the provider. Trade-off — smart retries reduce outages; blind retries amplify them. The rule that catches most teams is not to retry on errors that will never succeed on retry — AuthenticationError, ContentFilterError, InvalidRequestError. Retrying those wastes budget and slows the failure signal. Cost and effort — Tenacity in Python, retry middleware in Node, resilience4j in JVM. Minutes to add the decorator; hours to tune the delay curve for your specific traffic.
Bulkhead. Separate concurrency limits per feature. A traffic spike in document summarisation should not exhaust the concurrency slots that keep the chatbot answering. Reach for it whenever multiple features share an LLM pool. Trade-off — bulkheads add complexity to the pool management and require knowing your per-feature concurrency ceiling ahead of time. Set the limits too tight and legitimate traffic queues; too loose and the noisy-neighbour case reappears. Cost and effort — a semaphore per feature. Near-zero runtime overhead. The design work is picking the limits based on real traffic patterns, not on whiteboard guesses.
The Operate-in-Production Pair — Graceful Degradation + Observability
The two patterns that decide whether users notice the outage and whether the operator can see it.
Graceful Degradation. Every provider failed, all three fallback tiers returned an error — do not return a 500. Return a cached response for the top-100 frequent queries. Return a simplified answer computed without LLM. Return a helpful message that acknowledges the situation. Reach for it on every user-facing path. Trade-off — a degraded response is not the answer the user came for; a 500 is worse. The design work is knowing which paths tolerate a cache and which tolerate a static message. Cost and effort — implementation is trivial. Deciding the fallback content per feature is the work, and it belongs to product as much as to engineering.
Observability. Traces, logs, metrics that make circuit state, fallback rate, and degradation rate visible. Reach for it as infrastructure alongside every one of the five patterns above. Trade-off — observability data grows fast; retention policy and cost of storage are real ongoing concerns. Under-instrument and you cannot debug; over-instrument and the storage bill becomes the largest line item. Cost and effort — Prometheus counters, gauges, histograms. Modest instrumentation cost. The Prometheus alert rules that actually fire before the user tickets arrive are where the value shows up.
The Minimum Viable Resilience Stack
Six patterns is the complete list. The minimum viable stack is smaller. If a Berlin AI-native SaaS goes to production tomorrow with a single-provider LLM dependency and no resilience infrastructure, the smallest change that removes the 3 AM page is roughly three components — LiteLLM Router with two providers configured as fallback, one Redis-backed circuit breaker shared across replicas, and one Prometheus alert firing when any circuit stays OPEN for more than 30 seconds. That stack absorbs the vast majority of what actually breaks in production — provider outages, rate-limit bursts, transient timeout spikes.
The full six-pattern stack becomes justified once the application has multiple features sharing a pool (add bulkhead), user-facing paths where any error is unacceptable (add graceful degradation), and enough traffic that per-feature observability is worth the operational tax. The order to build them, roughly, is the order the operational pain shows up.
What This Doesn't Solve
The six patterns do not solve model-quality drift — the model returning worse answers to the same questions over time as the provider updates their pipeline. They do not solve semantic regression between providers — the chain returning technically-successful answers that miss the intent of the original prompt. They do not solve capacity planning — the resilience stack keeps you online during a spike; it does not scale your provider budget to support the spike being sustained.
They do solve the question of whether one provider's bad hour becomes your bad night. That contest is winnable.
If the outage that pages your on-call SRE at 3 AM this Tuesday is a provider outage, which of the six patterns above would have kept you asleep?
FAQs
Do we really need Redis, or can the circuit breaker hold state in-process?
In-process breakers work for single-replica deployments and fail silently in every other case. Each API pod holds its own breaker; Pod A opens its breaker but Pod B keeps hammering the failing provider, and the outage continues from Pod B's perspective. Redis makes the breaker cluster-wide, which is what matches the actual failure mode. The Redis dependency is worth the operational cost the moment you have more than one replica in production.
Does the fallback chain work for every workload, or are there tasks where it makes things worse?
It works well for model-agnostic workloads — generic Q&A, summarisation, translation, most retrieval-augmented answers. It works badly for workloads that depend on model-specific behaviour — long-context reasoning where the context-window advantage of one provider is load-bearing, tool-use fidelity where different providers have different function-calling implementations, jailbreak resistance where the guardrails vary by provider. Semantically test the fallback path on your actual workload before shipping it; do not assume the chain is transparent.
How do we set the circuit-breaker threshold without guessing?
Start with a broad default — five failures in sixty seconds. Let it run in staging with realistic traffic. If it opens on legitimate transient bursts, raise the threshold; if it stays closed through a real degradation, lower the recovery timeout or shorten the window. Most teams end up with different thresholds per provider based on that provider's actual reliability profile. The recovery timeout, typically 30 seconds, is where most of the tuning attention should go, not the threshold itself.
What is the single most under-invested pattern on this list?
Observability, followed closely by graceful degradation. Both are easy to add later and hard to add well after the first outage has already trained the team to assume the pattern is not there. Instrument the circuit state, the fallback rate, and the degradation rate as first-class metrics; set the alert that fires when the circuit stays OPEN longer than the recovery timeout; and decide the degraded response content per feature before you need it. All three are cheap; none are exciting; each pays for itself on the first bad night.
Companion content
- LLM Resilience Patterns, Part 2: Implementation — the LiteLLM Router honest read, Redis-backed circuit breaker code, Prometheus alerts that actually fire, six anti-patterns
- Building the Agent — The Working Vocabulary, Part 1: Foundations — the loop and context terms this piece assumes
- Governing the Agent — The Working Vocabulary, Part 3 — the observability + evals discipline this piece sits inside
- Agent Action Approval Gates — the human-in-the-loop pattern adjacent to graceful degradation
How to engage
If your team's on-call rotation includes provider-outage pages you would rather stop getting, talk to us at creativeminds.dev/contact. The cmdev resilience diagnostic maps the six patterns above against your existing LLM stack, names the minimum viable investment before the next outage, and produces a ninety-day plan that lands the missing pieces in the order the operational pain justifies.