Mitigating Non-Deterministic AI Failures: The Day 2 Problem for Enterprise LLMs

An operator-grade pattern from the CreativeMinds Development (cmdev) AI engineering practice. Companion to Custom Evaluation Frameworks for Enterprise LLMs and the Amazon Bedrock for Production AI series.

Week Eleven, When Nothing Broke

For ten weeks the deployment runs like a metronome. Customer support is quiet. The compliance officer reads the monthly audit-evidence pack and signs off without notes. The on-call rota is uneventful enough that the senior engineer who designed the system has rotated off to a different initiative. Week eleven, the support queue starts filling with tickets that share a strange family resemblance — answers that are not wrong exactly, but not the answers they used to be. Week twelve, the compliance officer notices the same monthly query now reads differently than it did in March. Week thirteen, the on-call gets paged at 02:14 because P99 latency has crept to 22 seconds where it used to be 4. Nothing broke. Everything drifted.

This is the Day 2 problem. The model did not change, the prompts did not change, the corpus did not change — but the system behaviour did. Enterprise LLMs rarely die in a single catastrophic event the way a database does when a disk fills. They die the way a tyre dies: slowly, by a thousand small losses of pressure, until one morning the car is dragging on the rim. The day-one architecture from the air-gapped Bedrock article is what makes the workload deployable. The day-two monitoring architecture is what keeps it deployable for the eighteen months that justify the original investment. Most teams build the first and not the second, the same way most homeowners install a smoke alarm and then never change the battery.

What follows is what we ship for Day 2: the three classes of non-deterministic failure, the detection patterns that surface each one, the recovery playbooks that close the loop, and the alarm-fatigue trap that turns expensive monitoring into expensive decoration.

Three Ways a System Loses Pressure

Drift does not have one shape. It has three, and each has its own signal, its own root cause, and its own recovery.

The first is structural hallucination drift. The model starts producing answers that are confidently wrong in a slightly different way each week — not the same hallucination on repeat but new ones, plausible enough to pass a casual review. The cause is almost always upstream and silent: a Bedrock model-version update that the team did not notice, a Knowledge Base ingestion that pulled in stale or contradictory content, a Guardrails policy tightening that pushed the model toward less anchored answers, or a slow degradation in calibration as Anthropic or Meta or Cohere update training weights between minor versions. The signal that catches it is the faithfulness-metric trend from the eval harness in the prior article. Hallucination drift looks like a faithfulness score moving from 0.94 in week one to 0.91 in week eight to 0.87 in week twelve. Each week's drop is within noise. The line through the points is the alarm. Think of it as the slow downward creep of a barometer — one reading is weather, the slope is climate.

The second is semantic variance. Two customers ask the same question on the same day. They get different answers. Both answers are technically correct. But the inconsistency erodes trust faster than a single wrong answer ever could, because it whispers to the user that the system is non-deterministic in a way they cannot reason about. The cause is the combination of temperature settings — even at temperature zero there is residual non-determinism in transformer inference at scale — and the retrieval layer, where slightly different top-K chunks come back for semantically equivalent queries and the model synthesises slightly different summaries from them. Multi-model routing compounds the problem: a query classified as "simple" on Monday and "standard" on Tuesday flows through different model tiers and emerges with visibly different answers. The detection signal is answer consistency, measured by running the same canonical golden-set queries through the production pipeline N times per day and comparing the outputs via embedding distance. Variance above the configured threshold trips the alarm.

The third is latency degradation, and it is the one most monitoring systems miss because the median lies. P95 and P99 latencies drift upward over weeks while P50 stays flat — which is why median-based dashboards see green while the long tail rots. The causes are several at once: KB vector indices growing past their tuned size, Bedrock regional load shifts, prompt templates that accrete context as the team adds features, retry storms from upstream rate limits, AgentCore session memory that does not get released, response-token counts creeping upward as tool descriptions grow. The signal that surfaces it is P95 and P99 tracked per query class (per Bedrock Part 6 observability), because aggregate P95 can hide a single query-type degradation behind the volume of other classes. CloudWatch metrics tagged with query_class show the shape the aggregate flattens.

Three Detection Pipelines in Parallel

The architecture runs three parallel detection pipelines into a single alarm-and-recovery layer. Each pipeline is its own deployable subsystem, but they share a metric vocabulary so the on-call sees one coherent picture rather than three competing ones — closer to a hospital's vitals monitor, where pulse and oxygen and blood pressure feed one screen than to three machines beeping at different rhythms in three corners of the room.

The continuous-evaluation pipeline is the one that catches faithfulness drift. A scheduled job runs the golden set from the evaluation article against the production pipeline daily. The judge model — pinned Claude Opus, calibrated against human SME labels — scores each item on the metrics that matter for the workload, and the harness emits per-metric CloudWatch entries tagged with query_class, judge_model_version, pipeline_version, kb_snapshot_id. Alarms fire on three patterns: a 2σ regression on any metric against the 30-day rolling baseline; a sustained 1σ regression over a 7-day window, which catches slow drift that does not trip the 2σ gate; and a calibration delta between judge model version N and version N+1 when a judge upgrade is in flight. The same harness runs against a 1–5 per cent sample of live production traffic for the online-eval pipeline, with stratified sampling so rare query classes are not silently absent. Cost runs $3 to $8 per day on typical workloads, the kind of money a regional team spends on coffee.

The consistency pipeline catches variance. A standing job picks a 50-item subset of the golden set and runs each query through production N times — typically five, spread across the day to capture intra-day variance. Pairwise embedding similarity is computed for each pair of answers, the minimum similarity per query becomes the per-query consistency score, and the aggregated semantic_consistency_score metric tracks per query class. For multi-model routing pipelines, the consistency pipeline runs per tier — Haiku, Sonnet, and cross-tier — and cross-tier inconsistency is the signal that the router itself is the variance source rather than the underlying models.

The latency profile catches the third class. Every Bedrock invocation emits a CloudWatch metric tagged with query_class. Dashboards track P50, P95, and P99 per dimension. The alarm condition is sustained P99 above the 30-day P99 baseline by 50 per cent for any single query class — sustained meaning a six-hour window, so transient spikes do not page the on-call at 02:00. For agent workloads, latency is also profiled per tool with tool_call_latency tagged by tool_name. A single tool's degradation, usually a downstream service's degradation, gets flagged independently of the agent's overall response time, the way a body in fever can be localised to one infected joint before the whole patient runs hot.

What Makes an Alarm Worth Reading

A drift alarm without causal attribution is an alarm that gets snoozed by week three and ignored by week six. The pattern that closes the loop is to make the alarm carry its own forensic context. When the alarm fires, the harness reports which recent changes correlate with the regression window: recent deploys to the workload account, Knowledge Base ingestion events, Bedrock catalogue model-version updates, Guardrails policy changes, query-distribution shifts, downstream service health. The alarm body no longer reads "faithfulness regressed by 0.07 on compliance-policy-query." It reads "faithfulness regressed by 0.07 on compliance-policy-query — correlated with KB ingestion job ing-3af2 completed 2026-06-02 14:30 UTC and Guardrails version 4 deployed 2026-06-03 08:15 UTC."

The on-call at 02:00 no longer has a search problem. They have a verification problem with three candidate hypotheses already lined up. The difference is the difference between a doctor handed a chart of vitals and a doctor handed a body with no notes. The causal-attribution table lives in a CloudWatch Logs Insights query that runs at alarm fire time and embeds in the PagerDuty incident at creation. No human is correlating timelines at 02:00, because the system has already done it.

Four Playbooks for the Four Things That Break

Recovery routes by detected failure class, and four playbooks cover most of the production ground.

The first is model-version rollback. When the causal attribution points to a Bedrock model-version update as the trigger, the recovery is to re-pin the previous version. The agent's foundation model ID is held as a CloudFormation or Terraform parameter; rollback is a single parameter change and a redeploy, five to ten minutes end to end. The discipline that backs it: always run at least one minor version behind the catalogue's current. The version Bedrock just published has not been tested against your eval harness yet. The version you are currently running has. When the catalogue advances, the team runs the eval against the new version in shadow, confirms equivalence or better, and only then advances the pin. Skipping this discipline is the most common way teams get caught by quiet catalogue updates — the model "got worse overnight" without anything on their side having changed.

The second is Knowledge Base rollback. When the causal attribution points to a KB ingestion as the trigger, the recovery is to roll back to the prior snapshot. Bedrock Knowledge Bases support data-source version management; the rollback is a single API call. The discipline: snapshot before every ingestion job. The KB version graph becomes the rollback target list. Ingest, run the eval against the new snapshot, confirm equivalence, then advance the production pointer. If the eval regresses, the production pointer never advances. The pattern is the same as a database migration that runs against a staging instance before touching production — the change is reversible until it is verified, and only then irreversible.

The third is Guardrails tuning. When the causal attribution points to a Guardrails policy change, the recovery is to roll back the guardrail version. Bedrock Guardrails are versioned by design; the agent references the guardrail by ARN with version, and rollback is a version-reference change. The discipline: stage policy changes in DRAFT, evaluate, then publish. The published version becomes the production reference. The DRAFT remains mutable. A bad publish rolls back to the previous published version while the team iterates on a new DRAFT.

The fourth is retrieval re-tuning, and it is the slowest of the four because re-ingestion of the corpus may be required. When the causal attribution points to retrieval-layer drift — KB hit rate dropping, chunk relevance scoring lower — the recovery is to re-tune top-K, chunking strategy, hybrid-search weights, reranking thresholds. This is also the playbook most likely to be the actual cause when the failure mode is slow drift rather than sudden regression. The discipline that makes it tractable: keep the retrieval harness independent of the synthesis harness. Retrieval-quality regressions surface against retrieval-specific metrics — hit rate, MRR, context precision — before they show up downstream as faithfulness regressions. Catching the drift early is much cheaper than catching it after the synthesis layer has already produced wrong answers based on it.

How an Alarm Becomes a Snooze Button

The hardest problem in production AI monitoring is not building the system. It is keeping the team responsive to the alarms over months. Alarm fatigue is the slow erosion that turns a working monitoring system into expensive decoration, and the erosion has five patterns we have watched play out across deployments.

The first is the 2 AM noise alarm. A drift alarm fires at 02:00 on a Saturday. The on-call snoozes it. The pattern repeats. Within six weeks, drift alarms are routinely snoozed and the system might as well not exist. The counter is to route drift alarms to a daytime queue rather than PagerDuty Sev-1 — the signal is slow drift, so the response can be slow too. Reserve Sev-1 paging for hard failures, where the agent is throwing 5xx errors or the harness is producing nothing, and for customer-impacting incidents.

The second is the alarm with no runbook. An alarm fires, the on-call improvises, the next on-call improvises differently, and the responses are uneven across engineers. Each alarm class needs a linked runbook with the playbook, the queries to run, and the parameters to check, version-controlled in the same repository as the agent itself.

The third is the metric nobody owns. Five engineers built the harness; ten months in, three of them have moved teams, and the remaining two are uncertain whether a faithfulness regression of 0.04 is a real problem. A named metric owner per signal closes the gap. The owner reviews the trend weekly, makes the calls on threshold changes, and signs off on alarm acknowledgements.

The fourth is judge-model drift — the harness's own scoring drifting silently, covered in detail in the eval article. When the judge becomes more lenient, faithfulness metrics rise without the pipeline improving. Version-pin the judge, run human-SME calibration quarterly, and treat judge drift as its own monitoring class. The judge cannot grade itself.

The fifth is the threshold-tuning trap. A team's response to a noisy alarm is to raise the threshold until the alarm stops, rather than to fix the underlying signal. Threshold changes have to require sign-off from the metric owner with a rationale entered into a change log. Cosmetic tuning becomes visible the moment the discipline exists.

What Bites Once the System Is Running

Five operational frictions surface in real deployments, and we have engineered past each.

The first is the cost of continuous judging. Running the full golden-set harness daily with claim-decomposition on Claude Opus costs $20 to $50 per run, which is $140 to $350 a week, or $560 to $1,400 a month. Real money, but well below the cost of one undetected production incident. The mitigation is tiered judging: Haiku as a first-pass classifier to skip detailed judging on obvious-correct cases that score highly against the golden answer, and Opus only on the ambiguous middle band. The pattern drops judge cost by roughly 70 per cent.

The second is drift-detection latency. A 30-day rolling baseline means a regression takes 14 to 21 days to clearly trip the 2σ alarm. Slower drift takes correspondingly longer, and for workloads where regulatory exposure is high, that lag is operationally significant. The mitigation is to add a 7-day baseline alongside the 30-day, with a tighter threshold — catches faster drifts at the cost of more false positives on legitimate week-over-week noise.

The third is the harness becoming the pipeline's flakiest dependency. The harness runs against Bedrock; Bedrock throttles; the harness fails; the team disables the harness "until tomorrow"; tomorrow never arrives. We have watched this pattern repeat. The mitigation is to treat the harness as production code itself, with its own retry policy, exponential jitter, and Bedrock-aware error classification (per the Step Functions article) — and its own oncall and runbook for when the harness itself is failing.

The fourth is per-tenant drift hidden by aggregate metrics. A multi-tenant deployment may have a single tenant whose retrieval has drifted while the aggregate metric stays healthy. The fix is per-tenant metric dimensioning with alarms per high-value tenant, because for heterogeneous tenants the aggregate is decoration and the per-tenant signal is the actionable one.

The fifth is monitoring without postmortems. A team builds the monitoring, the alarms fire, the on-call acts, the system recovers — and nobody writes a postmortem. Institutional learning never compounds. Six months later the same failure mode reappears, fresh, as if it had never been seen before. The mitigation is to make the postmortem a mandatory step in alarm closure. The alarm is not closed until a brief cause/response/prevention note is filed in the runbook directory. The body of postmortems becomes the team's memory across staff rotations.

The Signal That Earns the Next Deployment

Five observations hold up across the Day 2 deployments we ship.

The architecture that makes a workload deployable is not the architecture that makes it operable. Day 1 is what the reference series documents — secure perimeter, identity, RBAC, audit-evidence, model invocation, retrieval. Day 2 is what this article documents. Teams that ship one without the other are shipping an experiment, not a system, and the experiment runs out at the end of week thirteen.

Drift is the dominant failure mode at 12-month timescales. Catastrophic failures attract attention precisely because they are loud. Slow drift is the failure mode that quietly does the actual damage to enterprise AI's reputation in the year after deployment, while everyone is busy looking at the explosion that never came.

The eval harness becomes the operations team's source of truth. Once the harness exists and produces trusted metrics, the conversation shifts from "I think the model got worse" to "faithfulness is down 0.07 on policy-query since the KB refresh." The change in conversation quality is itself the return on the investment in the harness.

The causal-attribution layer is the difference between an alarm system and a working alarm system. Alarms without attribution get snoozed. Attribution turns the on-call's job from a search problem into a verification problem, and that change determines whether the system survives its first six months of staff rotation.

The team that maintains the Day 2 architecture is the team that has earned trust to deploy the next AI workload. Customers, executives, and regulators all read this signal correctly. Teams that show up with a working day-2 story get the next greenlight. Teams that show up with only day-1 do not.

What did the deployment look like on week eleven, before the support tickets started arriving — and what would it have taken to notice the silence had already turned strange?

Engaging with cmdev

CreativeMinds Development (cmdev) ships the day-2 monitoring architecture as a standing part of every production AI engagement. Our pattern works across banking under CBN CSAT, energy operators under NMDPRA and NIS2, fintechs under NDPA, and healthcare networks under HIPAA-equivalent regional regimes. The eval harness, the drift detection, and the recovery playbooks compose as the operability layer the day-1 architecture assumes.

• Email: [email protected]
• Cloud security services: /services/cloud-security
• Companion architecture series: Amazon Bedrock for Production AI, Air-Gapped LLM Deployments, Custom Evaluation Frameworks, Compliance Automator case study

Mayowa A. is CTO of CreativeMinds Development. He leads cmdev's AI engineering practice for regulated enterprises across Africa and the EU, with deployments in production for banking, energy, and critical-infrastructure customers.