AI Customer Service Quality Assurance: How to Measure AI Performance

A customer service AI can have a 90% automated resolution rate and still be delivering poor service. It might resolve nine in ten conversations without a human, score well on customer satisfaction and rarely escalate, and still be quietly giving customers incorrect information in a small but important category of cases, perhaps around refund eligibility or account security, for weeks.

As AI takes on more customer service work, QA needs to answer a harder question: is the system actually doing the job well, or simply handling more interactions?  In February 2026, Gartner found that 91% of surveyed customer service and support leaders were under pressure from executives to implement AI, with customer satisfaction, operational efficiency and self-service success among their top priorities. Quality assurance is what tells an organisation whether that pressure is translating into genuinely better service.

AI customer service QA is broader than traditional contact centre QA because it has to assess the system, not just the conversation.

Traditional QA evaluates a sample of human interactions against a scorecard: greeting, verification, empathy, accuracy, process adherence, resolution, closing. It asks whether the agent performed well.

AI requires a broader question: was the system behaving correctly? That means examining the customer’s input, the context retrieved, the model’s response, the action taken, the escalation decision and the final outcome. An AI system can produce a perfectly polite, well-structured response that is completely wrong. Equally, it can give a technically correct answer while failing to recognise that the customer needed human help.

Rather than tracking dozens of indicators, it is more useful to focus on seven.

1. Accuracy and relevance: Was the information correct and relevant to what the customer was trying to achieve?

2. Groundedness: Accuracy asks whether the answer is right. Groundedness asks whether there is appropriate evidence for it. The two aren’t the same: a response can be grounded in a retrieved source and still be wrong, if that source is outdated or was misread by the model. This distinction matters most for retrieval-augmented generation (RAG) systems, where a confident-sounding answer isn’t automatically a supported one.

3. Resolution rate: Did the customer’s issue actually get resolved, rather than simply receiving a response?

4. Escalation quality: Did the AI escalate the right cases? Too little escalation creates risk; too much makes automation pointless. Escalation should be treated as part of quality, not as an AI failure. Last month, Gartner uncovered that 87% of customers said companies using GenAI for customer service should still provide access to a human agent. A good evaluation programme therefore needs to test not only whether AI can resolve an interaction, but whether it knows when to hand it over.

5. Policy compliance: Did the AI stay within organisational rules on refunds, discounts, cancellations, regulated advice and authentication? This is closely tied to how well the system’s guardrails are designed and enforced.

6. Customer effort: Did the AI make the customer’s problem easier or harder to solve?

7. Business outcome: Did the interaction produce the desired result, such as retention, conversion, a completed transaction or reduced repeat contact?

Building a credible evaluation process starts with a test set of realistic customer scenarios: routine questions, ambiguous questions, difficult questions, edge cases, adversarial questions, outdated-information scenarios, emotionally charged interactions and cases that should trigger escalation. For each scenario, organisations need to define what a good response looks like: the expected answer, acceptable variations, prohibited behaviour and the conditions under which the AI should escalate rather than attempt a resolution itself.

NIST takes a similarly broad approach to AI evaluation, focusing on measuring the capabilities and limitations of AI systems through structured testing. Its Generative AI Evaluation Program covers generative AI specifically. In August 2026, NIST released an initial public draft of its TEVV-Athlon framework, which sets out a structured approach to evaluating AI systems, including large language models and agentic systems.

It’s tempting to reduce evaluation to asking another large language model whether a response was good. That can be a useful component of an evaluation stack, but it shouldn’t be the whole of it.

Automated evaluation is useful for large-scale monitoring: factual checks, policy checks, semantic similarity, groundedness scoring and response classification. Human evaluation is better suited to difficult or subjective cases, such as empathy, appropriateness, complex reasoning and interactions involving vulnerable customers. Real-world monitoring looks at what happens after deployment, tracking repeat contacts, complaints, escalations, customer corrections and failed transactions. Adversarial testing deliberately tries to make the system fail. NIST’s own ARIA Evaluation Planning Manual sets out a similar structure for holistic AI evaluation, combining model testing, red teaming and user testing rather than relying on any single method.

A chatbot mostly produces a response. An AI agent can retrieve information, call APIs, update CRM records, issue refunds, change subscriptions, send messages and trigger workflows. That means quality assurance needs to evaluate actions as well as answers.

For every action an AI agent takes, ask: was it authorised? Was it appropriate? Was it based on correct information? Was it completed correctly? Could it be reversed? Should a human have approved it first? Did it create any unintended downstream consequences elsewhere in the customer’s account or journey?

As AI becomes more autonomous, QA has to become behavioural assurance.

A workable AI QA programme can be built around six steps.

1.       Define quality: Agree what “good” means for each specific AI use case.

2.       Create evaluation scenarios: Build them from real customer interactions rather than hypothetical ones.

3.       Establish a baseline: Measure the current system before making any changes.

4.       Test before deployment: Run the AI against the agreed scenarios.

5.       Monitor continuously: Sample live interactions and track failures.

6.       Feed failures back into the system: Use them to improve prompts, knowledge bases, retrieval, guardrails, workflows, models and escalation rules.

That creates a continuous loop: test, deploy, monitor, identify failure, improve, retest.

Gartner revealed in July 2026 that customers were roughly three times more likely to turn to third-party generative AI tools than a company’s own chatbot when resolving customer service issues, and that use of third-party GenAI in service interactions had nearly doubled in a year while use of company-provided chatbots stayed flat. That raises the bar for what customers expect from AI-powered support: they are already using general-purpose AI tools for service-related tasks, so company-provided AI is no longer operating in isolation.

The goal isn’t to maximise the number of customer interactions handled by AI. It is to know, with evidence, when the AI is handling them well, and when it should not be. That means measuring accuracy, groundedness, resolution, compliance, customer effort and business outcomes, and testing AI against realistic and difficult scenarios rather than easy ones.

As AI moves from answering questions to taking actions, the consequences of getting that judgement wrong become greater. QA therefore needs to measure not just what AI says, but what it does, when it acts and when it knows to hand the customer to a human.