Why does your agent sound so confident when it's wrong?

By Simon Butler

You're building an agent and you want it grounded in real knowledge, not guessing. The standard advice is to embed or retrieve your own documents so it can't make things up. But Gartner has put a number on what happens when the documents themselves are wrong: organisations report an average cost of $12.9 million a year from poor data quality alone, before an agent has read a word of it.

Key takeaways

• Grounding an agent in your own documents doesn't stop it being wrong if the documents are wrong. It just gives the wrong answer a citation, which is harder to catch on review than a guess.
  
  
• A wiki stores whatever gets written down. A canon only holds what has been reviewed and released, or has met stated readiness criteria, and that gate is the actual difference.
  
  
• Keeping a canon current doesn't require a dedicated reviewer on every entry. Route validation to whatever signal already exists, calls, chat decisions, outcome data, and fall back to a last-validated date where nothing else applies.
  
  
• How much review an entry needs should scale with what it costs to be wrong, not the same ceremony applied everywhere.

Why doesn't grounding an agent in your own documents stop it from being wrong?

Because grounding your agent only answers where the agent gets its information, not whether that information is still true. The industry answer to "don't let it hallucinate" is to give the agent something to read: embed your documents so retrieval is fast, or fetch them on demand so it always sees the live version. Most companies moved their operating knowledge into Notion, Confluence, or a wiki of some kind a year or two ago, and nobody has gone back to check what's still accurate.

What makes a wrong answer with a citation more dangerous than a hallucination?

Because it looks exactly like a correct answer, not like a mistake. A hallucination is at least legible as one: it reads as invented, ungrounded, easy to flag on review. An agent that retrieves a real document and states what it says is doing exactly what it was built to do, and the answer still arrives with a source attached. It passes the review a hallucination would have failed.

The actual risk in grounding an agent on ungoverned knowledge is that it faithfully repeats something that used to be true. That's a harder failure to catch than something invented from nothing.

What actually turns a pile of documents into something worth grounding an agent in?

Governance: nothing enters, or stays, just because someone wrote it down. Call it a canon. What makes an entry part of one is that it earned its place, a proposal was reviewed and released, or it met stated readiness criteria that vouch for it.

That single rule changes what retrieval is doing. An agent reading from a wiki is reading whatever survived. An agent reading from a canon is reading what was checked. The retrieval code, the prompt, the model: none of that has to change. The knowledge base underneath it does.

Once an entry is trustworthy, how do you keep it that way without a full-time reviewer on every doc?

By routing validation to whatever signal already exists for that kind of knowledge, rather than assigning someone to re-check everything on a schedule.

Where the signal comes from:

• Calls and meetings, through transcripts from any call the team's bot already sits in
  
  
• Decisions raised in chat, wherever the team already discusses changes, Slack or Teams
  
  
• Individual emails, in principle, though privacy makes this the hardest channel to wire up
  
  

What actually triggers a proposal:

• An outcome signal, like an OKR or usage data tied to the claim
  
  
• A bet's stated readiness criteria being met, for instance "at least one paying engagement adopts the feature"

Where neither fires and nothing is being actively said about an entry at all, the fallback is a timer on its last-validated date, checked on a cadence such as a quarterly review. A timer is a watchdog, not a sensor: it catches silence, not change.

Once entries carry that kind of evidence, maintaining a canon stops being a standing project. Agents listening to the channels where things actually get decided can raise a proposal to promote or revise an entry the moment the readiness bar is met, and a person still has to approve it before it counts.

How the loop works

What the agents watch: calls and meetings, decisions in chat, outcome signals, a bet's readiness criteria, and a last-validated timer.

Those signals feed a drafting agent. New content enters the canon as a bet, labelled unvouched, with its readiness criteria stated. A change to already-vouched content becomes a proposal. When a human or an agent notices a bet's criteria are met, that also becomes a proposal.

Every proposal hits a human decision gate: approve promotes it to a validated entry with an owner, a date, and a status; reject sends it back to being a bet. Retrieval then serves vouched entries as validated and bets flagged as bets to the consuming agent, which writes back what it learned as a fresh proposal.

What are the limits of this, and where doesn't it apply?

It isn't free, and it isn't the right shape for everything. Review has real fatigue attached to it, someone has to be the person who decides, and bootstrapping a first canon out of an existing pile of half-trusted documents takes real time up front. None of that goes away because the model got better.

It also doesn't fit fast-moving operational state. Transactional data changes too quickly to need a review gate; it needs a database, not a canon. A canon is for the knowledge an agent needs to act with judgement, not the numbers it needs to act at all.

Does every entry need the same amount of review?

No. Rigour scales with what it costs to be wrong, not with a fixed process applied to everything. What makes an entry checkable is that it carries an owner, a date, and a status. What makes retrieval clean is that there's exactly one answer per question, not three versions with no way to tell which one is current. What's actually tunable is how much ceremony sits in front of a write. A pricing claim that lands in every proposal earns a reviewer. A note in a runbook needs a date and a name of who suggested it, and nothing more.

We built this because we needed it for our own drafting agents, not because it made a good pitch. Our own sales and marketing material, this post included, is rendered from a canon of the same shape: entries with an owner, a status, and a review history, and a proposal queue we can show on a call. Some of it is still labelled experimental rather than standard, on purpose, so anyone reading knows which parts are settled and which are still a bet.

The knowledge base quality problem doesn't disappear once you have a canon. It gets a mechanism instead of a hope.

None of this asks you to trust agents more. It asks you to stop trusting documents by default, whether it's an agent reading them or a person. The discipline that made source control ordinary for code, propose a change, review it, release it, applies just as well to the knowledge a company runs on. What's different now is that agents can do the drafting and the noticing, so the discipline stops costing a full-time job to maintain.

FAQ

Does adding more documents to an agent's retrieval corpus fix hallucination?

Not by itself. If the corpus contains stale or incorrect information, retrieval gives the agent a specific wrong answer to state instead of a vague one, and that answer arrives with a citation attached. That makes it harder to catch on review than an ungrounded guess would have been.

What's the actual difference between a wiki and a canon?

A wiki holds whatever anyone writes down, with nothing stopping it from drifting once it's there. A canon requires membership to be earned: an entry enters, or stays, because a proposal was reviewed and released, or because it met stated readiness criteria. The gate is the difference, not the storage format.

How do you keep a knowledge store current without assigning someone to review every entry?

Route validation to whatever signal already exists for that kind of knowledge, calls and outcome data for claims that get tested, decisions in chat for things people actively discuss changing. Where nothing is being said about an entry, fall back to a timer on its last-validated date as a watchdog rather than a sensor.

Talk to us

If your agents are reading from a knowledge base nobody has fully vouched for lately, we're happy to talk through what a canon would look like for yours. Get in touch, or subscribe to the newsletter for more from the team on how we build this ourselves.

Sources

• Gartner, "Data Quality" ($12.9M average annual cost of poor data quality, from Gartner's 2020 Magic Quadrant for Data Quality Solutions survey of 154 reference customers)