Cold-Start Latency and Cost for Multimodal RAG Pipelines

An operator-grade pattern from the CreativeMinds Development (cmdev) AI engineering practice. Companion to Cost Optimization on Amazon Bedrock and RAG with Bedrock Knowledge Bases.

The two questions every CFO asks

Every customer engagement we run reaches the cost conversation by the second meeting. The CFO or the budget-owner sits in. The question is always one of two shapes:

• "How long will the first user wait?" — the latency question, almost always asked first.
• "What does this cost per query at our volume?" — the unit-economics question, asked second.

The two are not independent. The decisions that bring cold-start latency under two seconds — connection pooling, model warming, prompt caching, semantic caching — also drive per-query cost down by 60-80% in the typical case. The decisions that drive cost down — model-tier routing, batch inference, smaller fine-tuned models for narrow tasks — also keep the latency envelope predictable. Optimising for one is optimising for both, if the architecture is right.

This piece documents the architecture we ship for multimodal RAG workloads at enterprise scale. The same patterns apply to text-only RAG; the multimodal extensions (image embeddings, OCR pre-processing, mixed-modality retrieval) add specific cost surfaces that the text-only version does not.

The cold-start cascade

The first query of the day in a typical Bedrock-backed RAG pipeline hits an unwarmed cascade of dependencies. The latency we have measured across deployments:

Cold-start dominates the user's first experience. Warm-state dominates the rest of their session. The architectural moves that cap cold-start under two seconds are the ones that matter most for buying-audience trust — the demo where the answer takes seven seconds to start streaming is the demo that does not advance to procurement.

The architecture

Every layer has its own cost-and-latency dial. The discipline is to tune each independently against the eval harness from the eval frameworks article, not all at once.

Layer 1 — Semantic caching, the highest-leverage single move

Most user queries are not unique. In customer-support workloads we have profiled, 40-65% of incoming queries are semantically equivalent to a query asked in the prior 30 days. The exact wording differs; the answer is the same. Caching at the query level is the move that drops both average cost and average latency materially.

The semantic cache architecture:

• Cache key — the query embedding (Cohere Embed v3 or Titan v2). Stored in Redis (Redis Cluster on ElastiCache, or Redis Enterprise Cloud for higher-scale workloads), Pinecone, or pgvector on Aurora.
• Lookup — for an incoming query, embed it, search the cache for the nearest-neighbour with similarity above a threshold (typically 0.92-0.95). If a hit, return the cached answer.
• TTL — cache entries expire after a configured window (24-72 hours for support workloads; 5-15 minutes for time-sensitive workloads like incident triage where answers go stale fast).
• Invalidation — explicit invalidation when the underlying corpus changes. KB ingestion event flushes the cache.

Cost per cache hit: $0.0001 (Redis lookup + embedding generation). Cost per cache miss: full pipeline cost ($0.02-0.10 per query depending on model tier). The break-even is hit at ~5% hit rate; typical workloads see 40-65% hit rates and recover the cache infrastructure cost within hours.

The trick — and this is where most teams get it wrong — is scoping the cache per user-context, not globally. A query from User A and the same query from User B may have different correct answers because of row-level security or per-tenant filters. The cache key is (query_embedding, tenant_id, user_role) not just (query_embedding). Global caching leaks data across tenants; scoped caching does not. The cache also drops in hit rate because of the additional dimensions — typically to 25-45% — which is still material.

Layer 2 — Prompt caching at the model layer

Bedrock prompt caching (from Part 7 of the Bedrock series) caches the prefix of the prompt across invocations. The cached portion is charged at ~10% of the standard input rate.

For a multimodal RAG pipeline, the cached prefix is typically:

• The system prompt (3,000-8,000 tokens of instruction)
• The tool descriptions for the agent (1,500-4,000 tokens)
• The static context (company facts, current date, persona definition — 500-2,000 tokens)

This adds to ~5,000-14,000 tokens of recurring prefix per invocation. Without caching, those tokens get billed at full input rate every call. With caching, they cost ~10% of input rate after the first call within the TTL window.

Operational rules:

• Stable cached prefixes only. Any change to the prefix invalidates the cache. Version-control the prefix; treat changes as deliberate events.
• TTL of 5 minutes by default. High-volume workloads keep the cache warm naturally; low-volume workloads see frequent cache misses and lower realized savings.
• Measure the cache hit rate — Bedrock surfaces it via CloudWatch. A workload running at <60% cache hit rate is usually a prefix-stability problem worth investigating.

For a workload running 100,000 queries/day with a 7,000-token cached prefix, the recurring input-token savings are roughly 300-400 USD/day at Sonnet pricing. Real money.

Layer 3 — Model-tier routing for RAG specifically

The cascade pattern from Part 7 of the Bedrock series applies cleanly to multimodal RAG. The routing table for a typical workload:

The router itself runs on Haiku (~$0.001 per routing decision) and classifies in under 200ms. The savings on routed-to-cheap-model queries pay for the router thousands of times per day on any workload with the typical query distribution.

For multimodal extensions, the routing decision tree includes a modality classifier: is the input text-only, image-only, or mixed? Image-only and mixed queries route through Amazon Textract (for documents) or Titan Multimodal (for natural images) before reaching the reasoning model. The pre-processing step is cheaper than sending the image directly to Claude for full multimodal reasoning, when the question is well-defined.

Layer 4 — Small fine-tuned models for narrow recurring tasks

The customisation patterns from the Model Customization article apply specifically here. A workload that runs 50,000 daily queries through a narrow extraction task — invoice line items, contract clause classification, compliance-flag identification — should not pay Sonnet rates for each. A distilled Llama 4 Scout or fine-tuned Titan Text Lite achieves 85-95% of Sonnet's quality on the narrow task at 15-25% of the per-call cost.

The combined-models topology: Claude Sonnet for the open-ended reasoning step; custom-tuned smaller model for the recurring extraction step. The router decides which model gets each call. The economics improve materially at scale; below ~10,000 daily queries the customisation overhead is not yet justified.

Cold-start mitigation — beyond the cache

Beyond the semantic cache (which addresses warm-state cost more than cold-start latency), the cold-start moves we ship:

• Provisioned concurrency on Lambda for the agent runtime. Eliminates the Lambda cold init. Cost: ~$10-30/month per provisioned concurrency unit; eliminates 1.2-3.0s of cold-start.
• Connection pool reuse on the Bedrock client. The botocore.config.Config with tcp_keepalive=True and pooled connections shaves 100-200ms per call after the first.
• OpenSearch Serverless provisioned OCUs (rather than serverless-only) avoid the cold-start scale-out latency on the vector store. Cost: $700-2000/month baseline; eliminates KB-side cold-start variance.
• Pre-warming the system prompt cache via a synthetic warm-up query issued every 4 minutes. Keeps the Bedrock prompt cache warm even during low-traffic windows.
• Streaming the response to the user from the first token. Cuts perceived latency dramatically even when actual end-to-end latency hasn't changed.

For a deployment where the first-impression latency budget is two seconds, these moves combined reliably hit ~1.4-1.8s cold-start, ~600-900ms warm-state.

Multimodal-specific cost surfaces

The pure-text RAG pipeline has predictable cost structure. Adding image inputs surfaces new cost vectors that need explicit instrumentation:

• Amazon Textract for document images (PDFs, scanned forms) — $0.0015 per page for standard text extraction, more for forms-and-tables. Cache the Textract output per document hash to avoid re-running.
• Titan Multimodal Embeddings for natural-image retrieval — $0.0006 per 1k tokens for image+text inputs. Bulk image-ingestion runs should use Bedrock Batch Inference for ~50% savings.
• Claude vision input — Claude Sonnet accepts image input directly. Pricing is per-image with image-size-dependent token consumption. Useful when the question requires reasoning over the image content, not just OCR.

The cost-tag dimensioning becomes critical here. Without per-modality tagging, the bill arrives as "Bedrock — $X" and the team has no idea which feature drove the spend. With per-modality tagging, the cost dashboard attributes spend to text vs image vs OCR per feature.

The friction points — what bites in real deployments

Five frictions we have engineered past:

1. Cache poisoning from a single bad query

A query that returns a wrong answer gets cached; subsequent semantically-similar queries return the wrong cached answer. The cache becomes a poison vector.

The mitigation: cache invalidation on negative feedback. If a user flags an answer as wrong, the cache entry for that query embedding is invalidated immediately. Quality-score-aware caching where low-faithfulness answers are not cached at all is a more aggressive variant for high-stakes workloads.

2. Per-tenant cache fragmentation

Scoping the cache per tenant cuts hit rates from 60% to 30%. For deployments with hundreds of small tenants, the per-tenant cache may never warm up enough to be useful.

The mitigation: two-tier caching — a per-tenant tier (small, hot) and a global tier (large, warm) where only answers that are demonstrably tenant-agnostic land. The global tier handles questions like "what is your refund policy" that have the same answer regardless of tenant; the per-tenant tier handles questions like "what's my balance" that do not.

3. Prompt cache misses from token-count drift

The system prompt grows by 50 tokens over a deployment month as the team adds context. Each prompt-cache hit becomes a miss. Cost rises silently.

The mitigation: freeze the system prompt's cacheable prefix with a clearly marked boundary; mutable context goes after the cache boundary so the prefix never invalidates. The pattern is similar to immutable Docker image layers — the stable part is cached, the variable part is layered on top.

4. Streaming output that breaks downstream parsers

Streaming the response improves perceived latency but breaks downstream JSON-parsing logic that expects a complete response. Half the deployment now produces malformed output to downstream systems.

The mitigation: streaming for end-user-facing channels, complete responses for system-to-system integration. The same agent has two modes; the channel determines which.

5. The cost-tag fatigue problem

Every component, every tool, every Lambda gets a tag. The team's tagging discipline erodes over time; new resources land without tags; the cost dashboard becomes ambiguous.

The mitigation: enforced tagging via AWS Config rules that block resource creation without the required tags. The discipline becomes a CI gate, not a request to remember.

What this taught us about enterprise scaling

Five things hold up across the multimodal RAG deployments we ship:

1. The semantic cache is the single highest-leverage move on both cost and latency. Every deployment we run pays for the cache infrastructure within the first day. Teams that skip it are paying full per-query rates on workloads that have 40-65% repetition.

2. Cold-start matters more for buying conversations than for production usage. A deployment with a 4-second cold-start and a 0.8-second warm-state will lose every demo conversation. The cold-start budget is part of the architecture, not a Day 2 optimisation.

3. Multimodal cost surfaces are bigger and more variable than text-only. A workload that adds image inputs to a previously text-only pipeline typically sees 3-5× cost increase if the multimodal layer is not separately optimised. Per-modality tagging surfaces this; without it, the cost surprise is the procurement conversation.

4. The cost-and-latency dials are not independent. Optimising for one usually optimises for the other in this architecture. Teams that frame these as separate workstreams duplicate effort; teams that frame them as the same workstream ship faster.

5. The CFO conversation gets shorter every quarter once the dashboards are right. When per-feature, per-tier, per-modality cost is attributable in CloudWatch, the cost conversation becomes engineering, not negotiation. The team that shows up with the dashboard ships the budget approval; the team that shows up with an estimate does not.

Engaging with cmdev

CreativeMinds Development (cmdev) ships the cost and latency architecture as a standing part of every production AI engagement. We work with banking under CBN CSAT, energy operators under NMDPRA and NIS2, fintechs under NDPA, and healthcare networks under HIPAA-equivalent regional regimes — the cost discipline applies identically across them.

• Email: [email protected]
• Cloud security services: /services/cloud-security
• Companion architecture series: Amazon Bedrock for Production AI, Air-Gapped LLM Deployments, Custom Evaluation Frameworks, Day 2 — Mitigating Non-Deterministic Failures, Compliance Automator case study

Mayowa A. is CTO of CreativeMinds Development. He leads cmdev's AI engineering practice for regulated enterprises across Africa and the EU.