--- title: "A QA Checklist for AI Customer Service | Chargezen" description: "How to check an AI agent before it talks to customers and keep checking after launch: accuracy, tone, policy, actions, handoffs and a weekly review routine." url: "https://chargezen.com/blog/ai-customer-service-qa-checklist" lang: "en" updated: "2026-10-18" --- Canonical URL: https://chargezen.com/blog/ai-customer-service-qa-checklist # A QA Checklist for AI Customer Service: Review Your AI Agent Before and After Launch DTC Strategy · 8 min read ![Jerry Abrahamson](https://chargezen.com/assets/blogpost1.6581a30d.png) Jerry Abrahamson Founder & CEO • Oct 18, 2026 ![A QA Checklist for AI Customer Service: Review Your AI Agent Before and After Launch](https://chargezen.com/og-default.png) Quality assurance in customer service used to mean a team lead reading a handful of tickets each week and scoring them against a rubric. That still works for people. An AI agent needs the same care, with two differences: it answers far more conversations than any one person, and when it gets something wrong, it tends to get it wrong the same way every time. That is good news, in a way. A mistake you find once can be fixed once. This checklist covers how to review an AI agent before it talks to a single customer, what to look at once it is live, and how to turn what you find into fixes that stick. ## What you are checking for Most QA rubrics for human agents score empathy, accuracy, process and resolution. For an AI agent, split it into six questions. Use them before launch and in every review after. * **1\. Is the answer true?**: Does it match your actual policies, product facts and the customer's real order? An answer that sounds right but invents a return window is the most expensive kind of mistake. * **2\. Is it grounded?**: Can you point to where the answer came from: a help article, a policy, the order record? If you can't, neither could the agent. * **3\. Did it follow the rules?**: Refund limits, discount rules, what it may change on its own and what needs a person. * **4\. Did it do what it said?**: If it says the address is changed or the subscription is paused, is that actually true in your store? * **5\. Did it hand off at the right moment?**: Not too early, which wastes your team's time, and not too late, which frustrates the customer. * **6\. Does it sound like you?**: Tone, length, and the words your brand does and doesn't use. ## Before launch: the pre-flight checklist Don't switch an AI agent on for every customer and hope. Spend a few hours testing it first. The goal is to see its worst answers while only your team can see them. ## Step 1: Write down your top 30 questions Pull them from your inbox. Most stores find the same handful at the top: where is my order, change my address, skip or pause my subscription, return or exchange, a discount code that didn't work, product fit or ingredients. Add five awkward ones: an angry customer, a request outside your policy, a question about a product you don't sell, someone asking for a person, and someone trying to get another customer's details. ## Step 2: Check the knowledge before you check the answers * Is every policy the agent needs written down somewhere it can read: returns, shipping times, subscription changes, warranties? * Do any two documents disagree? An old shipping page and a new one will produce answers that flip between them. * Are exceptions written as rules ("we refund damaged items without a return if the customer sends a photo") rather than left in someone's head? * Is anything in there that customers should never hear, like internal margins or supplier names? Most wrong AI answers trace back to missing or conflicting source material, not to the AI itself. Fixing the source fixes every future answer at once. The glossary entry on [grounded answers](https://chargezen.com/glossary/grounded-answers) explains why this matters. ## Step 3: Run the test conversations Ask each of your 30 questions through the channels customers actually use: the chat on your site, a text to your number, an email. Use a real test customer with real orders so the agent has something to look up. For each answer, score the six questions above as pass or fail and note why. ## Step 4: Test every action, not just every answer * Ask it to change an address, then check the order in Shopify. * Ask it to skip or pause a subscription, then check the subscription. * Ask for a refund above your limit and confirm it waits for a person instead of going ahead. * Ask for something it shouldn't do at all and confirm it says so plainly. An agent that can act needs [approval rules](https://chargezen.com/glossary/approval-workflow) before launch: which actions it may take alone, which need a yes from a teammate, and who that teammate is. ## Step 5: Test the handoff Ask for a person. Get angry. Ask something it can't answer. Then check what your team receives: does the teammate see the conversation, who the customer is and what has been tried, or do they have to start over? A [handoff](https://chargezen.com/glossary/human-handoff) that loses context undoes most of the good work before it. ## Step 6: Decide your launch bar Write down what "ready" means before you look at the scores, so you don't talk yourself into launching. A reasonable bar: no wrong facts about policy or orders, every action does what it says, every handoff arrives with context, and any tone problems are minor. Fix and retest until you hit it. Many teams start with one channel, or with after-hours conversations only, and widen from there. ## After launch: what to review, and how often Testing before launch catches the questions you thought of. Customers will ask the ones you didn't. Plan a review routine before you go live. | When | What to review | Why | | --- | --- | --- | | Daily, first two weeks | Every conversation the agent handed to a person, plus any it handled that involved a change to an order or subscription | Early mistakes repeat. Catch them while volume is small. | | Weekly | A sample of resolved conversations across your top topics, every conversation with a complaint or a repeat contact, and every approval request | Shows whether "resolved" really means solved | | Monthly | Trends: what topics grew, where handoffs cluster, which answers changed after a policy update | Finds gaps in your knowledge and rules | | After any policy or product change | Ask the agent the affected questions yourself | Old answers outlive old policies | A starting cadence. Adjust it to your volume. ## How to sample without reading everything * **Read every risky conversation**: Refunds, cancellations, complaints, anything that ended in a handoff, and anything where the customer wrote back about the same issue within a few days. * **Sample the routine ones**: A small, fixed number per topic each week, such as five order-status and five subscription conversations. Fixed numbers per topic stop one busy topic from crowding out the others. * **Rotate reviewers**: Different people notice different problems. The person who wrote the return policy will spot a wrong return answer that a new hire won't. * **Keep a log**: For each failure: the conversation, which of the six checks it failed, the cause (missing knowledge, wrong rule, bad tone, bad action) and the fix. The log is how you see patterns. ## A scoring sheet you can copy Keep it short enough that reviewers actually use it. Score each conversation pass or fail on these lines, with a one-line note on every fail. * Facts: every fact about policy, product and order was correct. * Source: the answer came from your approved information or the customer's records. * Rules: it stayed inside refund, discount and change limits, and asked for approval where required. * Actions: every change it said it made is visible in the store. * Handoff: it handed off when it should have, with context, and not before. * Tone: it sounded like your brand and kept answers to a sensible length. * Outcome: the customer's problem was solved, or the customer knows exactly what happens next. ## Turning findings into fixes A review only pays off if it changes what the agent does next. Match each kind of failure to the fix that prevents it. | What went wrong | Usual cause | Fix | | --- | --- | --- | | Wrong fact | Missing, old or conflicting knowledge | Update or remove the source document | | Right facts, wrong process | The method lives in someone's head | Write it as step-by-step instructions the agent follows | | Did something it shouldn't | Approval rules too loose | Require approval for that action | | Handed off too often | Rules too tight, or knowledge missing | Add the knowledge, or loosen one rule at a time | | Off-brand tone | No written voice guidance | Add brand rules with examples of good and bad replies | | Said it was done when it wasn't | The action failed silently | Treat as urgent; check how the tool reports results | Fix the source, not the single conversation. Resist the urge to fix individual conversations by hand and move on. If the cause is still there, the same answer goes out tomorrow. ## Mistakes to avoid * Only reading conversations the AI handed off. The worst mistakes are often confident answers nobody questioned. * Judging by deflection alone. A customer who gave up looks the same as one who was helped. See [ticket deflection](https://chargezen.com/glossary/ticket-deflection) and [AI resolution rate](https://chargezen.com/glossary/ai-resolution-rate) for the difference. * Testing once and never again. Every new product, policy or promotion is a new test. * Loosening approval rules all at once. Widen what the agent may do alone one action at a time, after it has handled that action well with approval. * Keeping the review in a spreadsheet nobody reads. Put the fix in the agent's knowledge and rules, where it changes behavior. ## How Chargezen helps Concierge is Chargezen's team of digital employees, and it is built to be reviewed. Here is how each part of this checklist maps to what it does: * **Answers from your approved information**: Concierge answers from the policies, product facts and brand rules you give it. See [approved answers](https://chargezen.com/concierge/features/governed-answers). * **Approvals you control**: Choose what each agent may do alone and what waits for a yes. Refunds and other big moves send a request that says exactly what will happen. See [approvals and audit trail](https://chargezen.com/concierge/features/approvals). * **Proof before "done"**: Concierge checks real results before it reports success, says plainly what is unfinished, and keeps a record of every step. * **Show work**: On answers built from your data, Show work lists the data the agent read and the commands it ran, with their output, so you can check a number instead of trusting it. See [data analysis](https://chargezen.com/concierge/features/data-analysis). * **Every conversation in one inbox**: Your team sees what Concierge said and did on every channel and can take over with one click. See the [team inbox](https://chargezen.com/concierge/features/team-inbox) and [human handoff](https://chargezen.com/concierge/features/human-handoff). * **Fixes that stick**: Write your methods as [playbooks](https://chargezen.com/concierge/features/playbooks). When a teammate takes over, Concierge notes how they handled it, and you can review, edit or forget anything it has learned. See [learning](https://chargezen.com/concierge/features/learning). Concierge Starter is $20 a month with 20,000 credits. Pro is $199 a month with 199,000 credits, or $1,990 a year. You can also connect your own OpenAI or Anthropic account. See [pricing](https://chargezen.com/pricing). Related terms: [AI hallucination](https://chargezen.com/glossary/ai-hallucination), [human in the loop](https://chargezen.com/glossary/human-in-the-loop) and [audit trail](https://chargezen.com/glossary/audit-trail). Related guides: [customer service metrics for online stores](https://chargezen.com/blog/ecommerce-customer-service-metrics), [conversational AI for ecommerce](https://chargezen.com/blog/conversational-ai-ecommerce) and [when an AI agent should bring in a person](https://chargezen.com/blog/ai-escalation-human-handoff). ## Frequently asked questions ## How many AI conversations should I review each week? Read every risky conversation (refunds, cancellations, complaints, handoffs, repeat contacts) and a small fixed sample of routine ones per topic. In the first two weeks after launch, read more. Cut back once a week's review finds nothing new. ## Should AI conversations use the same QA rubric as human agents? Mostly. Keep the parts about accuracy, policy and outcome. Add two checks that matter more for AI: whether every answer is grounded in your own information, and whether every action it claimed actually happened. ## What is the most common cause of wrong AI answers? Missing or conflicting source material: an old policy page, two documents that disagree, or an exception that was never written down. Fix the source and every later answer improves. ## When can I let an AI agent act without approval? When it has handled that specific action correctly with approval for a while, the action is easy to undo, and the stakes are low. Order-status answers can go first. Refunds and cancellations usually keep an approval step longer. ![Jerry Abrahamson](https://chargezen.com/assets/blogpost2.32f70dbd.png) Written by ### Jerry Abrahamson Jerry is the Founder and CEO of Chargezen, helping brands build sustainable omni-channel businesses. With over 10 years of experience in omni-channel commerce and SaaS, he's passionate about helping brands create recurring revenue streams. [](https://twitter.com/chargezenjerry)[](https://linkedin.com/in/jerry-a) ## Related Articles [ ![AI Receptionist for Plumbers: Answer Every Leak, Clog and Quote Call](https://chargezen.com/og-default.png) DTC Strategy · 10 min read ### AI Receptionist for Plumbers: Answer Every Leak, Clog and Quote Call What plumbing calls an AI receptionist should answer, what it should hand to a person, the safety lines it must follow, a sample call and a setup checklist. ](https://chargezen.com/blog/ai-receptionist-for-plumbers)[ ![AI Receptionist for HVAC Companies: No-Heat Nights and Heat-Wave Mornings](https://chargezen.com/og-default.png) DTC Strategy · 9 min read ### AI Receptionist for HVAC Companies: No-Heat Nights and Heat-Wave Mornings How an HVAC company's AI receptionist should handle no-heat and no-cool calls, tune-ups and replacement quotes with safety lines and a sample call. ](https://chargezen.com/blog/ai-receptionist-for-hvac)[ ![AI Receptionist for Roofers: Storm Surges, Leaks and Insurance Questions](https://chargezen.com/og-default.png) DTC Strategy · 8 min read ### AI Receptionist for Roofers: Storm Surges, Leaks and Insurance Questions How an AI receptionist should answer roofing calls after a storm: leaks, inspections, insurance questions and quotes, with safety lines and a sample call. ](https://chargezen.com/blog/ai-receptionist-for-roofers) ## Ready to Grow Your Subscription Revenue? Join leading brands using Chargezen to reduce churn, increase revenue, and automate operations. [Get Started](https://admin.chargezen.com/signup)