How to Check an AI Agent's Work Without Redoing It

To check an AI agent's work without redoing it, match your effort to the risk: spot-check a sample of low-stakes work, read how it reached important answers, compare reports with a source you trust, and keep a person on anything that moves money. Check everything at first, then check less as a job proves reliable.

If you recheck every line, the agent saves you nothing. If you never check, you find its mistakes when a customer does. There's a practical middle ground, and it's similar to how you'd review a new hire: closely at first, then lightly and on purpose.

Reading the work behind an answer is much quicker than doing it again. You're looking for the right data, the right period and sensible steps. In Concierge, click Show work under an agent's reply. "Work behind this answer" lists the data read and the commands run on its computer, and progress lines show browser steps as they happen.

• Did it use the right source?: Orders from your store, not a stale export.
• Did it use the right period?: Check the start and end dates, and the time zone.
• Did it use your definitions?: For example, whether net sales excludes shipping.
• Did it leave anything out?: Test orders may need excluding, but real orders do not.

Build a short test set before you rely on an agent: 10 to 20 real requests where you know the right answer. Include a few awkward ones. Run them when you start, and again after you change its instructions, playbooks or model.

• A number you can confirm in Shopify, such as last month's orders.
• A policy question with a clear answer, and one with an exception.
• A request it should refuse or hand to a person.
• A question it can't answer from what it has. It should say so rather than guess.
• A customer who hasn't verified who they are, asking for private details.

For customer agents, use Playground in Concierge to try these without touching live customers.

You can ask an agent to do some of the checking itself. A good instruction or playbook includes lines like these:

• "Confirm the product rows add up to the total before you reply."
• "State which dates and orders you counted."
• "If data is missing, say what is missing. Don't estimate."
• "Tell me how sure you are, and what would change your answer."
• "Compare this with last month and point out anything that changed by more than 20%."

This doesn't replace your review, since an agent can be wrong about its own work too. It does make mistakes easier to see, and a total that doesn't add up becomes an obvious flag.

An AI hallucination is a false statement delivered with confidence. The best defense for customer answers is to prevent it rather than catch it. Concierge's approved answers keep agents to your policies, product facts and brand rules, and not the open web or old notes. So check your sources as well as your answers: an out-of-date return policy will produce a confident, out-of-date reply.

• Sample at random: Read a set number each week, so you see ordinary conversations and not just complaints.
• Read every handoff: These show where the agent ran out of knowledge or confidence.
• Check repeat contacts: A customer who writes back about the same thing may not have been helped the first time.
• Look at the outliers: Read the longest conversations and those where a customer sounds unhappy.
• Score a few things consistently: Was it correct, did it follow policy, was the tone right? Our QA checklist gives you a starting point.

In Concierge, every conversation is in Operate, then Inbox, and past work is in Operate, then History. Teammates can step in on any conversation, and the agent stops replying until it's handed back.

• Fix the result: Correct the report, or follow up with the customer.
• Find the cause: Was it a missing definition, an old source, weak instructions or too much access?
• Fix the cause in the right place: Use a memory for one rule, a playbook for a method, a capability change for access and a source update for facts.
• Add it to your test set: That mistake now gets checked every time.
• Consider the wider effect: If it happened once, search for other cases of it.

In Concierge, you can correct an agent in plain words, for example "From now on, put refunds first in the weekly report." It saves that as a memory, with Undo and Edit right there. See memory vs playbooks.

Step down in stages, one job at a time. The right timing depends on the job and how much of it you see, so set your own bar in advance.

• Stage 1: check everything: Compare every result with the manual version.
• Stage 2: check key numbers: After a clean stretch, check only totals and anything that looks unusual.
• Stage 3: sample: Review a few each week, and read anything the agent flags.
• Back to stage 1: Return here after you change the model, the instructions or the underlying process.

• Read five random pieces of work, and open the work behind two of them.
• Read the week's handoffs, or a sample if there are many.
• Compare one key report total with your trusted source.
• Check that last week's corrections held.
• Add any new mistakes to your test set.

• Round or oddly neat numbers: Real store data is rarely tidy. Check where they came from.
• No dates or sources: An answer that doesn't say what it counted is hard to trust or check.
• Totals that don't add up: Ask it to reconcile the parts with the whole.
• A very different answer to the same question: This often points to a missing definition.
• Sudden changes after an update: Rerun your test set after any change to instructions or models.
• Confident answers to things it can't know: Ask it what it would need to be sure.

Concierge records every request, approval and action in the conversation, shows the work behind each answer, keeps customer replies to approved sources and hands off to your team when it should. The data analysis and weekly report pages show how reports are checked before they are sent.

Keep reading: how to keep a human in the loop, the first 30 days and rolling out a support agent in stages. Glossary: audit trail, AI hallucination and human in the loop.

Compare it with a source you trust, read how it reached the answer and test it on questions where you already know the result. Put your effort where mistakes would cost the most, and sample the rest.

Review closely at the start and after any change to its instructions or model. Once a job is reliable, a short weekly review of a random sample and all handoffs is a sensible routine for most small teams.

Anything that moves money or is hard to undo. In Concierge, agents never move money, so a teammate approves refunds. Sensitive customer situations and legal or safety issues should also go to a person.

You can't remove the risk entirely, but you can reduce it. Limit customer answers to approved sources, tell the agent to say what is missing instead of guessing, and test it with questions it can't answer.