Cold-Start Latency and Cost for Multimodal RAG Pipelines

An operator-grade pattern from the CreativeMinds Development (cmdev) AI engineering practice. Companion to Cost Optimization on Amazon Bedrock and RAG with Bedrock Knowledge Bases.

Two Questions, One Architecture

By the second meeting of every engagement, the CFO joins. The conversation always splits down the same two seams. "How long will the first user wait?" comes first, almost without exception. "What does it cost per query at our volume?" comes second. Both questions feel independent. Neither one actually is.

The architectural decisions that drag cold-start latency below two seconds — connection pooling, model warming, prompt caching, semantic caching — also pull per-query cost down by 60 to 80 per cent. The decisions that lower the bill — model-tier routing, batch inference, smaller fine-tuned models for narrow tasks — also keep the latency envelope predictable. Optimising for one is optimising for both, if the architecture is right. It is closer to the way a well-insulated house simultaneously saves heating bills and improves comfort than to two separate engineering projects that happen to live in the same code base.

What follows is the architecture we ship for multimodal RAG workloads at enterprise scale. The same patterns apply to text-only RAG; the multimodal extensions — image embeddings, OCR pre-processing, mixed-modality retrieval — add specific cost surfaces that the text-only version does not encounter.

The First Query Pays for All the Others

The first query of the day in a typical Bedrock-backed RAG pipeline hits an unwarmed cascade of dependencies. Picture a cold engine on a winter morning — every component takes a moment longer than it will at temperature. The latency budget we measure across deployments:

Cold-start dominates the user's first experience. Warm-state dominates the rest of their session. The architectural moves that cap cold-start under two seconds are the ones that matter most for buyer trust — the demo where the answer takes seven seconds to start streaming is the demo that does not advance to procurement. The room sees the latency, not the elegance of the underlying pipeline.

Six Layers, Six Dials

Every layer has its own dial for cost and latency. The discipline is to tune each one independently against the eval harness from the eval frameworks article, not all at once — turn one knob, measure, turn the next.

The Highest-Leverage Move Is the Cheapest One

Most user queries are not unique. In the customer-support workloads we have profiled, 40 to 65 per cent of incoming queries are semantically equivalent to one asked in the previous 30 days. The wording differs; the answer is the same. Caching at the query level is the move that drops both the average cost and the average latency materially in one stroke. It is the librarian who learns by week three which book each regular wants and has it ready at the desk.

The architecture is straightforward. The cache key is the query embedding from Cohere Embed v3 or Titan v2, stored in Redis (Cluster on ElastiCache for most workloads, Enterprise Cloud for higher scale), Pinecone, or pgvector on Aurora. On lookup, the incoming query is embedded and the cache searched for the nearest neighbour with similarity above a threshold of typically 0.92 to 0.95; a hit returns the cached answer. Cache entries expire after a configured window — 24 to 72 hours for support workloads, 5 to 15 minutes for time-sensitive workloads like incident triage where answers go stale fast. Invalidation is explicit: a KB ingestion event flushes the cache, because the corpus has changed underneath.

The economics are unambiguous. A cache hit costs roughly $0.0001 in Redis lookup plus embedding generation. A cache miss costs the full pipeline, $0.02 to $0.10 per query depending on model tier. The break-even sits at around a 5 per cent hit rate, and typical workloads see 40 to 65 per cent. The cache infrastructure pays for itself within hours of going live.

There is one trap, and it is the one most teams fall into: scoping the cache globally. A query from User A and the same query from User B may have different correct answers because of row-level security or per-tenant filters, and a global cache will happily serve A's answer to B. The cache key has to be (query_embedding, tenant_id, user_role), not just (query_embedding). The hit rate drops to 25 to 45 per cent because of the additional dimensions. That is still material, and the alternative is a privacy breach dressed as a performance optimisation.

A Prompt Prefix That Pays Itself Back

Bedrock prompt caching (from Part 7 of the Bedrock series) caches the prefix of the prompt across invocations, billing the cached portion at roughly 10 per cent of the standard input rate. For a multimodal RAG pipeline, the cached prefix is typically the system prompt at 3,000 to 8,000 tokens, the tool descriptions for the agent at 1,500 to 4,000 tokens, and the static context — company facts, current date, persona definition — at 500 to 2,000 tokens. The total comes to 5,000 to 14,000 tokens of recurring prefix per invocation, billed at full input rate without caching, billed at a tenth with it. The savings compound on every call inside the TTL window.

Three operational rules hold across deployments. Keep the cached prefix stable: any change invalidates the cache, so version-control the prefix and treat changes as deliberate events rather than incremental edits. Default the TTL to 5 minutes; high-volume workloads keep the cache warm naturally, while low-volume workloads see frequent misses and lower realised savings. Measure the cache hit rate via CloudWatch — Bedrock surfaces it cleanly — and treat anything below 60 per cent as a prefix-stability problem worth investigating. For a workload running 100,000 queries a day with a 7,000-token cached prefix, the recurring input-token savings run $300 to $400 a day at Sonnet pricing. Real money, paid by a discipline that takes a week to set up.

Triage by Tier

The cascade pattern from Part 7 of the Bedrock series applies cleanly to multimodal RAG. Think of the router as an emergency-room triage nurse: the patient with a paper cut does not see the surgical team, and the patient with chest pain does not wait behind paper cuts. The routing table for a typical workload:

The router itself runs on Haiku at roughly $0.001 per routing decision and classifies in under 200 milliseconds. The savings on the queries it routes to cheap models pay for the router thousands of times over on any workload with the typical query distribution.

For multimodal extensions, the routing decision tree picks up a modality classifier: is the input text-only, image-only, or mixed? Image-only and mixed queries route through Amazon Textract for documents or Titan Multimodal for natural images before reaching the reasoning model. The pre-processing step is cheaper than sending the image directly to Claude for full multimodal reasoning when the question is well-defined and what is actually needed is OCR plus a downstream text query, not a vision-language model thinking about the image as a whole.

When a Distilled Model Earns Its Keep

The customisation patterns from the Model Customization article apply directly. A workload that runs 50,000 daily queries through a narrow extraction task — invoice line items, contract clause classification, compliance-flag identification — should not pay Sonnet rates for each one. A distilled Llama 4 Scout or fine-tuned Titan Text Lite reaches 85 to 95 per cent of Sonnet's quality on the narrow task at 15 to 25 per cent of the per-call cost. The combined topology is straightforward: Claude Sonnet handles the open-ended reasoning step, the custom-tuned smaller model handles the recurring extraction step, and the router decides which call goes to which. Below roughly 10,000 daily queries the customisation overhead is not yet justified. Above it, the savings compound monthly.

Beyond the Cache — Catching the First Query

The semantic cache handles warm-state cost. Cold-start latency is a different problem, and several moves combine to bring it under the two-second budget.

Provisioned concurrency on the agent Lambda eliminates the Lambda cold init at a cost of $10 to $30 a month per provisioned concurrency unit and a saving of 1.2 to 3.0 seconds per cold start. Connection pool reuse on the Bedrock client — botocore.config.Config with tcp_keepalive=True and pooled connections — shaves 100 to 200 milliseconds per call after the first. OpenSearch Serverless on provisioned OCUs rather than serverless-only avoids the cold-start scale-out latency on the vector store, at a baseline of $700 to $2,000 a month for the elimination of KB-side variance. A synthetic warm-up query issued every four minutes keeps the Bedrock prompt cache warm even during low-traffic windows. Streaming the response to the user from the first token cuts perceived latency dramatically even when end-to-end latency has not changed. Combined, these moves reliably hit 1.4 to 1.8 seconds cold-start and 600 to 900 milliseconds warm.

What Images Cost That Text Does Not

A pure-text RAG pipeline has predictable cost structure. Adding image inputs surfaces new cost vectors that demand explicit instrumentation, because the bill arrives without breakdown if the team has not designed it in.

Amazon Textract for document images — PDFs, scanned forms — runs $0.0015 per page for standard text extraction and more for forms-and-tables. Cache the Textract output per document hash to avoid re-running on the same content. Titan Multimodal Embeddings for natural-image retrieval costs $0.0006 per 1,000 tokens for image-plus-text inputs; bulk image-ingestion runs should use Bedrock Batch Inference for roughly 50 per cent savings. Claude vision input — Claude Sonnet accepts image input directly, with per-image pricing that scales by token consumption based on image size — earns its place when the question requires reasoning over the image content rather than just OCR. The trick is per-modality cost tagging from day one. Without it, the bill arrives as "Bedrock — $X" and the team has no idea which feature drove the spend. With it, the dashboard attributes spend by modality, by feature, by tier.

Five Frictions, Five Fixes

Production deployments surface the same five frictions almost regardless of the workload, and we have engineered past each.

The first is cache poisoning from a single bad query. A query that returns a wrong answer gets cached, and subsequent semantically-similar queries return the same wrong cached answer. The cache becomes a poison vector. The fix is cache invalidation on negative feedback — when a user flags an answer as wrong, the cache entry for that query embedding is invalidated immediately. For high-stakes workloads, a more aggressive variant ties quality-score-aware caching to faithfulness scores, so low-faithfulness answers are never cached at all.

The second is per-tenant cache fragmentation. Scoping the cache per tenant cuts hit rates from 60 to 30 per cent, and for deployments with hundreds of small tenants the per-tenant cache may never warm up enough to be useful. The fix is two-tier caching — a per-tenant tier that is small and hot, and a global tier that is large and warm but accepts only answers that are demonstrably tenant-agnostic. The global tier handles questions like "what is your refund policy" that have the same answer regardless of tenant. The per-tenant tier handles questions like "what is my balance" that do not.

The third is prompt-cache misses from token-count drift. The system prompt grows by 50 tokens over a deployment month as the team adds context, and each prompt-cache hit becomes a miss. The cost rises silently. The fix is to freeze the system prompt's cacheable prefix with a clearly marked boundary, and let mutable context land after the boundary so the prefix never invalidates. The pattern is similar to immutable Docker image layers — the stable part is cached, the variable part is layered on top.

The fourth is streaming output that breaks downstream parsers. Streaming improves perceived latency but breaks downstream JSON-parsing logic that expects a complete response. Half the deployment now produces malformed output to systems that integrate with it. The fix is to stream for end-user-facing channels and return complete responses for system-to-system integration. The same agent has two modes, and the channel determines which one.

The fifth is the cost-tag fatigue problem. Every component, every tool, every Lambda gets a tag, and the team's tagging discipline erodes over time as new resources land without tags. The fix is enforced tagging via AWS Config rules that block resource creation without the required labels. The discipline becomes a CI gate, not a request to remember.

What Holds Up Across Deployments

Five observations hold up across the multimodal RAG deployments we ship.

The semantic cache is the single highest-leverage move on both cost and latency. Every deployment pays for the cache infrastructure within the first day. Teams that skip it are paying full per-query rates on workloads with 40 to 65 per cent repetition, which is the operational equivalent of leaving the tap running.

Cold-start matters more for buying conversations than for production usage. A deployment with a 4-second cold-start and a 0.8-second warm state will lose every demo, and the team will spend the next quarter wondering why the proof-of-concept did not advance. The cold-start budget belongs in the architecture from day one, not in a Day 2 optimisation list.

Multimodal cost surfaces are bigger and more variable than text-only ones. A workload that adds image inputs to a previously text-only pipeline typically sees a 3 to 5x cost increase if the multimodal layer is not separately optimised. Per-modality tagging surfaces this before procurement does. Without it, the cost surprise becomes the procurement conversation.

The cost and latency dials are not independent. Optimising one usually optimises the other in this architecture. Teams that frame them as separate workstreams duplicate effort. Teams that frame them as the same workstream ship faster.

The CFO conversation gets shorter every quarter once the dashboards are right. When per-feature, per-tier, per-modality cost is attributable in CloudWatch, the cost conversation becomes engineering, not negotiation. The team that arrives with the dashboard gets the budget. The team that arrives with an estimate does not. The first user is still waiting; what does the second meeting say to the CFO when they ask why?

Engaging with cmdev

CreativeMinds Development (cmdev) ships the cost and latency architecture as a standing part of every production AI engagement. We work with banking under CBN CSAT, energy operators under NMDPRA and NIS2, fintechs under NDPA, and healthcare networks under HIPAA-equivalent regional regimes — the cost discipline applies identically across them.

• Email: [email protected]
• Cloud security services: /services/cloud-security
• Companion architecture series: Amazon Bedrock for Production AI, Air-Gapped LLM Deployments, Custom Evaluation Frameworks, Day 2 — Mitigating Non-Deterministic Failures, Compliance Automator case study

Mayowa A. is CTO of CreativeMinds Development. He leads cmdev's AI engineering practice for regulated enterprises across Africa and the EU.