Building an impressive Bedrock demo takes an afternoon. Getting the same thing into production, where it handles real customer questions without inventing a refund policy, is where the actual work sits. Here is what that gap looks like.
What Bedrock gives you
Bedrock is a managed API over foundation models from several providers, plus the surrounding pieces: retrieval, guardrails, agents, and evaluation. You do not manage GPUs, and requests stay within your AWS account boundary rather than going to a third-party endpoint — which is usually the reason a regulated customer chooses it.
Two things to check before you design around it. First, model availability differs by region, so confirm what is actually enabled in ap-south-1 rather than assuming. Second, you have to request access to each model family in the console before you can call it — a step that trips up most first attempts.
Choosing a model, cheaply
The reflex is to pick the largest model available. Do not start there. Build your evaluation set first — thirty to fifty real questions with the answers you would accept — then run the cheapest reasonable model against it.
A large fraction of production workloads are classification, extraction, summarisation, or routing, and smaller models handle those at a fraction of the token cost and latency. Save the big model for the step that genuinely needs reasoning, and route to it rather than defaulting to it.
Your data is the hard part, not the model
Any useful internal assistant needs your documents, and that means retrieval. Bedrock Knowledge Bases handles the pipeline — chunking, embeddings, a vector store, and retrieval at query time — with OpenSearch Serverless or Aurora PostgreSQL with pgvector behind it.
Retrieval quality is where these projects live or die, and it comes down to unglamorous decisions:
- Chunking — too small and context is lost, too large and irrelevant text crowds the prompt. Test against your evaluation set rather than accepting the default.
- Metadata — tag chunks with source, date, and department so you can filter. "According to a policy document from 2019" is a wrong answer even when the retrieval worked.
- Freshness — decide who re-syncs the knowledge base when documents change, and automate it. Stale retrieval is worse than no retrieval, because it looks confident.
- Citations — return the source with the answer. It is the only practical way for a user to check the model, and it changes how much people trust the system.
Guardrails, and what they are actually for
Bedrock Guardrails filter both input and output — denied topics, content filters, and detection of personally identifiable information you would rather not see in a log. Contextual grounding checks can flag responses that are not supported by the retrieved passages.
They are a control, not a guarantee. Anything with a financial, legal, or medical consequence needs a human in the path, and the interface should make it obvious to the user that they are reading a generated answer.
Cost, before it surprises you
Billing is per input and output token, and the input side is what catches teams out. Each retrieval-augmented request carries your system prompt, the retrieved chunks, and the conversation history — so a short user question can become a very long prompt.
- Cap conversation history rather than resending the entire thread on every turn.
- Set CloudWatch alarms on invocation volume in week one, not after the first bill.
- Cache answers to repeated questions. Support queries cluster hard around the same twenty topics.
- On-demand pricing is right until traffic is steady and predictable; provisioned throughput is a commitment, so measure before you buy one.
How to start
Pick one narrow use case with a bored human doing it today — first-line support triage, extracting fields from invoices, summarising call notes. Build the evaluation set before the prompt. Ship it to five internal users, watch what they ask, and fix the retrieval.
The projects that fail are the ones that start with "an AI assistant for the whole company" and no definition of a correct answer.