I connected an AI agent to a Shopify store for pre-sales questions. It almost promised a refund the store doesn't offer.

I built a RAG-based agent trained on a store's product catalog, FAQs, and return policy to answer pre-sales questions on the storefront. It worked well — until it didn't, on exactly the wrong order.

The Build

The agent pulled from three sources: product descriptions, a sizing/shipping FAQ doc, and the return policy page. It handled the bulk of pre-sales traffic — sizing questions, shipping estimates, stock availability — without a human touching it.

Where It Worked

For routine questions, it was fast and accurate, because the answers were clearly stated in the source documents it was retrieving from. This is exactly the kind of narrow, well-documented Q&A that RAG agents are good at.

Where It Broke

A high-value customer asked: "Can I return if it doesn't fit?" The agent answered: "Yes, we offer a full refund on all orders." That's not the store's actual policy — returns are store credit only. The agent hadn't lied intentionally; it had generalized from a "returns accepted" line in the FAQ into a broader, incorrect claim about refunds specifically.

The Fix

The root issue was the system prompt gave the model too much room to be "helpful" instead of strictly accurate. The fix was narrowing its permission to answer at all:

Before (too permissive):

You are a helpful sales assistant. Answer questions truthfully.

After (constrained to source material):

You are a sales assistant. Only answer using the provided product catalog and policy documents. If the exact answer is not explicitly stated there, say:

"I'm not sure — let me connect you with our support team at support@store.com."

The key change: it's no longer allowed to synthesize an answer that sounds plausible. If the specific fact isn't in the source doc, it has to defer.

What I Learned

"Truthfully" is not a constraint an LLM can reliably self-enforce — it still has to generate something, and without a hard boundary it'll generate the most plausible-sounding completion rather than admit it doesn't know. The fix isn't a smarter model. It's a stricter prompt that makes "I don't know" an explicitly acceptable answer.