Paul: Today, the spotlight is on Marcelo Acosta.
He’s an AWS-certified Solutions Architect with more than 30 years in technology, focused on data, machine learning, and practical AI systems. He writes Build with AWS, where he helps software and cloud engineers build the skills, judgment, and real-world experience needed to move into AI engineering.
Enough chitchat. Let’s get into it 👀 ↓
A customer wants to return one item from a gift set. Your chatbot promises a refund. It found the refund rules, but missed the gift-return window.
The missing rule is already in your documents. So why did the chatbot miss it? And what would help it get the answer right? A broader first search might be enough. Or perhaps it needs an agent that can recognize what’s missing and look again.
In this article, I’ll show how I built and tested both versions of a RAG support chatbot on Amazon Bedrock for a fictional outdoor store. We’ll follow the path from policy documents to customer answers, then compare quality, response time, and model cost to see whether the extra searches pay off.
The two options
A question comes in, the application searches your documents, and the model uses the returned text to write an answer. That’s retrieval-augmented generation, or RAG. Each piece of returned text is a passage.
The choice is how much control to give the model over that search:
The single search gives the model more material immediately. The agent can choose a follow-up query based on what it has read. For the gift question, it could ask specifically about the return window for gift recipients.
Both are reasonable ways to find a missing rule. I wanted to see which produced better customer answers.
What I tested
The store has 24 policy articles covering returns, gifts, shipping, warranties, and repairs. They add up to about 29 KB.
The documents live in S3. Bedrock Knowledge Bases splits them into passages of around 200 tokens. Titan Text Embeddings V2 converts those passages into numerical representations, stored in S3 Vectors, so Bedrock can find text related to a question. GLM 5 on Bedrock writes the answers.
A Python script calls Bedrock’s Retrieve API for passages and Converse for answers. It also records the searches, responses, token counts, and API times.
I chose nine passages using eight practice questions: nine was the smallest setting tested that retrieved all the required policy text. One practice question resembled the gift-return case.
In a retrieval request, the setting is numberOfResults:
response = client.retrieve(
knowledgeBaseId=knowledge_base_id,
retrievalQuery={"text": question},
retrievalConfiguration={
"vectorSearchConfiguration": {
"numberOfResults": 9,
"overrideSearchType": "SEMANTIC",
}
},
)I added three checks to the main comparison: answer from only three passages, ask the model to check that draft using the same passages, and give it every document. The entire collection fits in a prompt, so sending everything was worth trying.
All versions used the same model and answer instructions. When the agent exhausted its search allowance, the script put the collected passages into a fresh prompt for the final answer. This happened on nine questions, including the pickup example below.
How the answers were scored
I tested 24 customer questions: eight direct lookups, eight combining policies, and eight asking for information the documents didn’t contain. Five versions produced 120 answers.
An AI evaluator checked the answers against rules written before the run. To pass, an answer needed correct policies and calculations, evidence for its claims, and an acknowledgment of missing information. Code checked citation IDs and recorded costs and timings.
This was a custom evaluation workflow. I wanted to inspect each answer alongside the exact passages it received. Bedrock handled retrieval and generation; the scoring used the saved outputs.
The single search gave the better result
API time covers search and generation. Model cost is estimated from input and output tokens; storage, embeddings, and vector operations are separate. The three-passage answer also supplies the draft for “Check the first answer,” whose cost includes both calls.
The agent cost 23% more than the nine-passage search and took 72% longer, while producing two fewer passing answers. Sending every document matched the nine-passage score at almost three times the model cost.
The agent did find the missing gift-return policy from the opening. The wider single search found it too. Other answers showed why successfully finding a rule was only part of the job.
Finding the rule didn’t guarantee a correct answer
The store offers a 60-day trial for hiking boots. For an order collected in a store, that clock starts when the customer collects it.
The three-passage search missed the collection rule. The nine-passage search found it and correctly used collection as the start date.
The agent found the rule too. Its final answer named both delivery and collection as the start date, then said it couldn’t establish which one applied.
The customer now has three answers to one question. None of the extra searching helps them choose.
This gives us a specific next step: test whether the answer consistently applies the collection rule when that rule is present. The search had already supplied what the model needed.
More text also broke a correct refund
A customer paid USD 120 for boots, plus USD 6 for delivery. The boots qualified for a return, with a USD 7 return-label fee. The refund should be:
120 − 7 = USD 113.
The three-passage answer got this right. Asking the model to check that draft kept it right. The nine-passage, agent, and all-document versions each returned USD 107. They deducted the separately paid delivery charge from the boot price again.
The extra text contained a second statement that delivery charges were nonrefundable. I suspect the repeated rule encouraged the extra subtraction; removing that passage and repeating the test would help check that explanation.
The arithmetic is correct for the expression the model chose: 120 − 6 − 7 = 107. The error is the extra shipping deduction. A calculator given that expression would produce the same wrong refund.
I’d test whether asking for a separate breakdown of the item’s paid amount, refundable delivery, and deductions prevents the mix-up. Each amount needs support from the order and the policy. If the store already has a returns service, the chatbot could request a refund quote using the order, returned items, and return method. That service would apply the store’s maintained rules across orders; nobody would be coding “this answer is 113.”
When an agent is worth trying
An agent helps when the next useful search depends on what the previous result reveals. Imagine investigating a product fault: one document identifies a component, which leads to a service bulletin, which identifies affected product versions. The agent can follow that trail.
For a direct lookup such as “How much is a return label?”, one search may be enough. AWS’s guidance recommends standard retrieval for direct lookups and agentic retrieval for more involved questions, including comparisons and searches across knowledge bases.
The agent also needs to recognize when searching cannot help. In this experiment, ten of its 25 extra searches went to questions whose answers weren’t in the documents. They produced no additional passing answers.
For example, a policy can explain how refunds are processed. A customer’s current refund status requires an order-status service. Giving the chatbot access to that service would address the missing information.
I’d test an agent on investigations like the product-fault example. For requests with a fixed sequence of lookups, application code can run the sequence directly. A chatbot can also route only the harder questions to an agent; the evaluation needs to show whether that routing improves the result.
From this experiment to a production chatbot
This experiment gives me a version to develop further. Before release, I’d test new customer questions, run them repeatedly, and have someone who knows the policies check a sample of the judgments. I’d compare errors, total customer waiting time, and operating cost against targets agreed before launch.
For recurring evaluations, I’d add Bedrock RAG evaluations. Its bring-your-own-responses option accepts saved answers and retrieved passages from a custom application. That would let me keep this search setup and use managed evaluation to track two things separately: whether retrieval found the necessary information, and whether the answer used it correctly.
The application would verify the customer’s identity, order, payment, and return history before approving a refund. A statement in a test question becomes a lookup against a real system in production. For calculated answers, check both the selected amounts and the arithmetic. Code can also verify that each citation points to a passage the model received.
For an agent, AgentCore Evaluations can assess the sequence of tool calls. Tracing with OpenTelemetry and CloudWatch helps locate slow calls and failed searches. I’d set limits on searches and retries, protect customer data in logs, and provide a support handoff when the chatbot gets stuck.
Document access and freshness also need testing: the right customer must receive the right version of the right policy.
My choice: keep the single search
For this chatbot, I’d keep the single search with nine passages. It produced more passing answers than the agent, cost less, and responded faster. My next test would check whether an explicit refund breakdown prevents the extra deduction, then compare the updated version on new customer questions.
I’d add an agent when its follow-up searches improve answers enough to meet the service’s quality, cost, and response-time targets. This agent gave me no reason to add that complexity yet.
The practical check is simple: when an answer fails, look at the passages it received. If the policy was missing, work on retrieval. If the policy was present, work on how the answer applies it. That tells you which part of the chatbot to fix before paying for another search.
See you next time!
Marcelo
Whenever you’re ready, here is how I can help you
Go from agent user to agent builder. Master the foundations of AI agents and turn fragile demo code into reliable, production-ready systems with my course, Agent Engineering: Building Multi-Agent Systems (made with Towards AI).
35 lessons. Pure foundations from scratch. 4 mini-projects. 2 production systems. A certificate and direct access to me & industry experts in our Discord.
Built for software and data professionals transitioning into AI engineering. Rated 5/5 with 300+ students. The first 7 lessons are free:
Not ready to commit? Start with our free Agent AI Engineering Guide, a 6-day email course on the mistakes that silently break AI agents in production.
Resources
GitHub repository · Compare the answers
Images & videos
If not otherwise stated, all images and videos are created by the author.
