A support agent can sound polished and still fail the customer. It may pull an expired policy, skip a required account check, or promise a refund it cannot authorize.
That is why I build AI agent test cases from real support tickets, not clean demo prompts. Historical tickets expose the language, missing details, product states, and policy boundaries that an agent will face after launch.
The goal is not to prove that the model can write. The goal is to prove that it can complete support work safely and consistently.
Why AI agent test cases start with closed tickets
A closed ticket contains more than a customer question. It often records the issue, the account context, the policy used, the actions taken, the final outcome, and the point where a human had to step in.
That makes tickets more useful than generic benchmark questions. A benchmark can show whether an agent handles a broad class of tasks. A ticket set shows whether it handles your billing rules, feature names, account states, integrations, and escalation process.
I still look at public benchmarks. tau-bench is a good example because it tests customer-service-like workflows with policies, APIs, multiple turns, and resulting system state. That last point matters. A friendly answer is not a pass if the agent changed the wrong subscription or failed to create a required escalation.
In practice, I treat support tickets as the raw material for a controlled evaluation set. Each case should answer four questions:
- What did the customer need?
- What facts did the agent need to retrieve or verify?
- What action was permitted or required?
- What outcome would count as safe and complete?
Closed tickets are a starting point, not a source of automatic truth. Some were resolved poorly. Others were closed because the customer stopped responding. A few contain workarounds that conflict with current policy.
A ticket becomes useful test data only after someone verifies the intended outcome against the current policy, product behavior, and account rules.
I prefer tickets that show a complete path. A short exchange that ends with “fixed” may have little value. A case that shows the initial issue, relevant system state, a policy reference, and a confirmed resolution can test an agent properly.
Strip risk before you label data
Support data is full of material an evaluation system should not retain. Names, email addresses, phone numbers, order numbers, payment details, IP addresses, account IDs, and internal notes can all appear in a single conversation.
I remove or replace personal information before exporting tickets into a test repository. Use consistent placeholders where identity matters. For example, replacing a customer name with Customer A preserves conversation flow without retaining the person.
I also remove information that gives the model an unfair advantage. An internal note saying “refund already approved by finance” should not remain visible if the production agent would never receive it.
Each retained case needs a source record. I keep the ticket’s creation date, product area, issue type, channel, relevant policy version, and a link or ID for the approved source material. If a policy has changed, I update the expected behavior or retire the case.
This matters most for refunds, security incidents, account ownership, legal requests, and plan changes. A response can be factually plausible and still violate a current rule.
I don’t use model confidence as proof of correctness. A high confidence score may describe the model’s preference, not whether it retrieved a valid source or took the right action. Evidence and system traces are more useful than a confident sentence.
Choose coverage before volume
A test set built from the hundred most common tickets will look impressive and miss the failures that cost the most. Password resets may dominate volume, while account takeovers, duplicate charges, and cancellation disputes create more risk.
I start with 50 to 100 high-signal cases for an early baseline. Once the workflow is stable, I expand toward a few hundred cases across ticket types, policy boundaries, and known failures.
The mix should reflect both volume and consequence.
| Ticket group | What the agent must prove | Common failure |
|---|---|---|
| Routine how-to requests | It retrieves current product documentation | It invents a feature or outdated step |
| Billing and cancellation | It checks account facts and follows policy | It promises a refund without authority |
| Technical troubleshooting | It gathers required details before advising | It guesses at a fix from incomplete context |
| Account access and security | It protects identity and escalates correctly | It exposes account data or skips verification |
| Multi-issue tickets | It separates requests and sequences actions | It solves one issue while ignoring another |
| Escalated cases | It recognizes the boundary of automation | It keeps chatting when a human must intervene |
The important cases are often buried in old escalations. I search for tickets reopened after a seemingly correct answer, tickets marked as poor CSAT, tickets transferred between teams, and tickets where a human corrected an earlier response.
I also group duplicate issues before sampling. If 300 customers reported the same outage, I don’t need 300 copies of nearly identical wording. I keep a few that vary by channel, tone, account status, or product plan.
A ticket from last year can still be useful if the behavior remains valid. If the case depends on a retired plan, a discontinued integration, or an old privacy rule, it belongs in an archive, not the active suite.
Turn one ticket into a repeatable case
A raw transcript is hard to grade. A repeatable case has a controlled input, a documented context, an expected outcome, and clear pass conditions.
I don’t ask, “Did the answer sound good?” That question produces inconsistent reviews. I ask whether the agent found the needed evidence, followed policy, used the right tool, and stopped when human review was required.
A cancellation dispute is a common example. The customer’s wording may be simple: “I canceled last week. Why was I charged again?” The actual work is not simple.
The agent may need to check the billing provider, cancellation timestamp, renewal date, plan status, and refund policy. It may need to explain the charge without promising a refund. If the data conflicts, it should escalate rather than improvise.
Keep the expected outcome operational
I structure each case around observable behavior:
- Record the customer message exactly as the agent will receive it, after redaction.
- Add only the account state, tool outputs, and documents available to the production workflow.
- State the approved sources the agent should use, including policy version and knowledge-base section.
- Define required actions, such as checking a subscription record or creating a handoff.
- Define prohibited actions, such as issuing a credit, changing an address, or revealing account details.
- Write the expected final state, not only the preferred wording.
This format handles valid variations in language. Two answers can phrase a policy differently and both pass. An answer that uses the wrong billing provider or skips account verification should fail, even if it sounds empathetic.
For retrieval-based agents, I record the source passages needed for a correct answer. That lets me measure retrieval separately from generation. If the agent never retrieved the current cancellation policy, prompt edits are unlikely to solve the real problem.
I also require an evidence trail for factual claims. The agent does not need to expose private reasoning. It does need a trace that shows which documents, account fields, and tools informed the result.
Score response quality and tool execution separately
A support agent has at least four jobs: understand the request, retrieve evidence, take action, and communicate the result. One overall score hides too much.
I score each of those jobs separately. That helps me identify whether a failure came from retrieval, tool permissions, policy logic, or response generation.
| Evaluation area | Pass condition | Failure worth flagging |
|---|---|---|
| Answer correctness | The response matches current policy and ticket facts | It gives an unsupported or outdated answer |
| Evidence grounding | Claims map to approved documentation or account data | It cites unrelated material or no source |
| Tool execution | Calls use correct parameters and create the intended state | It queries the wrong account or takes an unauthorized action |
| Handoff behavior | The agent escalates at the right boundary | It keeps attempting a high-risk task |
| Conversation control | The agent asks for missing details without looping | It repeats questions or ignores a second issue |
Customer-service agents need outcome-based tests. Toloka’s overview of AI agent evaluation makes the same distinction between a fluent response and an agent that completes a multi-step task correctly.
A correct tool call is not enough either. The final account state matters. If an agent successfully calls an API but cancels the wrong subscription, that is a hard failure.
I use automated grading for repeatable checks. Did it call the required tool? Did it avoid forbidden actions? Did it cite an approved source? Did the final state match the expected result?
Then I use calibrated human review for ambiguous cases. An LLM judge can score structure and policy adherence at scale, but I regularly compare its verdicts with expert reviewers. If the judge rewards polished but unsafe answers, it needs a better rubric.
A response can be well-written, correctly cited, and still fail because the agent took the wrong action.
Add ugly inputs and known failure modes
Clean tickets test happy paths. Production support is rarely clean.
I create variations from real ticket patterns: misspellings, vague requests, angry language, multiple issues in one message, copied error logs, incomplete account details, and requests that conflict with policy.
A customer might ask for a refund while also reporting unauthorized access. The agent should not treat that as one billing question. It should recognize the security concern, follow the appropriate verification path, and avoid revealing sensitive information.
I also build tests around known failures. If customers repeatedly receive an outdated answer after a documentation change, that failure should stay in the suite until it is fixed. If a model once followed an instruction hidden in pasted text, keep a safe version of that prompt as a regression test.
The strongest adversarial cases test the agent’s refusal behavior. Give it a request where no authorized evidence exists. A safe agent should say it cannot verify the claim, explain the next step, or transfer the case. It should not fill the gap with a plausible answer.
Broad benchmark lists can help teams compare evaluation methods and datasets. This AI agent benchmark reference is useful for that research. Still, no outside benchmark can replace tickets that contain your real operational constraints.
Make evaluation part of each release
I run the active suite before changing the model, prompt, retrieval index, policy documents, tools, or routing logic. A small wording change can alter tool selection. A documentation update can lower retrieval quality. A new plan can make old answers unsafe.
Each release should produce a comparison against the prior version. I track pass rate by ticket cohort, tool-call errors, unsupported claims, failed handoffs, retrieval misses, and regressions in high-risk cases.
Production monitoring comes next. I sample live conversations for human QA and compare results with offline tests. A staged rollout by ticket type is safer than sending all traffic to a new agent at once.
Watch operational metrics alongside evaluation scores:
- Resolution rate shows whether customers reached a usable outcome.
- Reopen rate catches answers that looked complete but created follow-up work.
- Escalation rate shows whether the agent knows its limits.
- Time to resolution shows whether automation reduced work instead of adding another loop.
- Policy and security failures need their own count, regardless of overall resolution.
I treat new failures as new test candidates. The support queue changes as the product, policies, and customer behavior change. A fixed dataset becomes less reliable over time.
The ticket queue is your test lab
The most valuable AI agent test cases come from situations where a real customer needed a correct result, not a convincing sentence. Support tickets give you the context, ambiguity, and consequences that clean prompts leave out.
I start small, verify every expected outcome, and separate response quality from actions and evidence. That produces a test suite the team can use during development and after release.
A support agent earns trust when it knows what to do, what not to do, and when to hand the work to a person.
Frequently asked questions
How many support tickets are needed for AI agent testing?
I begin with 50 to 100 reviewed cases across core workflows and high-risk exceptions. That is enough to catch obvious regressions before launch. Expand the set as recurring failures, new policies, and product changes create new test scenarios.
What makes a support ticket unusable as a test case?
Discard or rework tickets with unresolved outcomes, missing policy context, outdated product behavior, or sensitive data that cannot be safely redacted. A ticket is not a valid test because it was closed. The required action and expected result must be verifiable.
