I built an AI agent to answer "does this fit my [product]?" DMs. It confidently said yes to the wrong match.

Compatibility questions — "will this part work with my [model]" — are one of the most common DMs an online parts or accessories store gets. I built an agent to answer them automatically. It handled the clear cases well. On anything slightly ambiguous, it guessed, confidently, and got it wrong often enough that I had to pull it back and rebuild the grounding from scratch.

The Build

The agent took an incoming DM, pulled out the product being asked about and the item it needed to be compatible with, and answered whether they matched — sourced from the model's general knowledge, not from an actual compatibility catalog.

Where It Worked

For very common, unambiguous product/model combinations — a clearly stated, well-known pairing — it did fine. The failure mode wasn't "always wrong," it was inconsistent in exactly the cases that mattered most.

Where It Broke

Real customers don't describe products the way a spec sheet does. A model name alone often doesn't specify the variant, and compatibility genuinely differs across sub-models or configurations that share the same general name. Customers also frequently get details slightly wrong themselves, and the agent, working from general knowledge instead of a real compatibility table, would fill those gaps with a plausible-sounding guess rather than flagging the ambiguity. Checking a sample of its answers against the real compatibility data, it was wrong on roughly one in five.

The Fix

The fix was grounding every compatibility answer in real compatibility data instead of the model's general knowledge, and making ambiguity an explicit stop condition instead of something to guess past:

System: You answer product compatibility questions using ONLY the
compatibility table provided via lookup_compatibility(product_id, target_model).
Never state a compatibility match from general knowledge.

If the customer's details are incomplete (missing variant or
configuration) and the compatibility table shows different answers
for different variants, do not guess — ask which variant they have
before answering.

The second half mattered as much as the first. Grounding in real data fixes wrong answers on clear questions; explicitly requiring the agent to ask a follow-up on genuinely ambiguous ones is what stops it from confidently picking one variant when the honest answer is "it depends, which one do you have."

What I Learned

"Ground it in real data" isn't a complete fix by itself if the underlying question is genuinely ambiguous — a lookup tool answers the question you ask it, and if the question itself is missing information, the tool will happily return an answer for whichever variant it defaults to. The agent needed permission to say "I need one more detail before I can answer that" as an acceptable, expected response — not a fallback for when everything else fails, but a normal part of handling a real compatibility question correctly.