Classifying Contact-Center Requests
Engineering

Classifying Contact-Center Requests

A request classifier with 86 percent accuracy can still send one in four cancellations to the wrong team. A worked confusion matrix shows where to look and what to fix before routing goes automatic.

Short answer

Classifying contact-center requests means assigning each incoming call, chat or email to a class that decides its queue or next step, so it reaches the right team the first time. Define classes by queue, train only on what is known at routing time, measure per-class recall and precision in a confusion matrix, and send uncertain or sensitive requests to a person.

What does classifying contact-center requests mean?

Contact-center request classification means assigning every incoming request (a call transcript, a chat message, an email or a form) to a class that determines a queue or a next step. The goal is not to tag the topic but to send the request to the right team the first time. That is why a good classifier is defined not by its accuracy figure but by how many requests reach the wrong queue and what that costs.

The first rule follows directly: every class must correspond to a distinct action. If two classes go to the same queue and trigger the same next step, separating them only makes the model's job harder, so merge them. Conversely, if requests that go to the same queue carry different urgency (a cancellation versus a general enquiry, for example), keep them apart.

How do you choose the classes?

Settle the class list before the model, together with agents and tested against real requests. Four checks cover most of it:

  • Does it map to an action? Every class should have a target queue or an automatic next step.
  • Do people give the same label? Have two agents label the same sample independently. Cohen's kappa, listed among the classification metrics in the scikit-learn documentation, is meant for measuring agreement between human annotators; according to that page, scores above 0.8 are generally considered good agreement and zero or below means no agreement. Do not expect a model to separate two classes that people cannot.
  • Is there room for uncertain requests? Without a "human triage" class, the model forces every request somewhere.
  • How are multiple intents handled? For requests such as "my bill is wrong and I also want to cancel my subscription", either write an approved priority rule or send them to human triage.

Microsoft's custom text classification documentation warns in the same direction: classes whose training data is very similar are frequently confused, and it suggests merging them or labelling more examples that tell the two apart. According to the same page, a class with fewer than 15 labelled instances in the training data can lead to lower accuracy. For urgent or sensitive requests (safety reports, formal complaints, risk to life), add keyword and channel rules on top of the model's decision; missing these requests costs more than sending them to the wrong queue.

Rules, a trained model or a language model?

Choose the method by the number of classes, the labelled data you have and how quickly things change. With a few classes that are worded distinctively (for example, billing disputes arriving through their own form), keyword and channel rules may be enough, and they are the easiest to explain. If you have labelled historical requests and the classes stay stable for months, a trained text classifier is usually fast and cheap to run. If classes change often or labelled data is scarce, a language-model step that receives the class definitions and a few examples in the prompt is more flexible, but its latency and cost per request must be measured and it must be evaluated with the same confusion matrix. In every case, routing is usually not agent work but a fixed workflow with a single text-understanding step. Whichever you choose, the evaluation steps below stay the same.

How do you prepare training and test data?

Use only information that is available at routing time. A closed ticket contains the resolution note, the transfer history and the final category; the first message contains none of them. Train on the full ticket and you will see good test results and poor live results. Label each example by asking "which queue should this have gone to, given the first message?", not "what did this request turn out to be?".

Split the test set by time: train on earlier months and test on later ones. Product launches, price changes and campaigns shift the mix of requests, and a randomly split test set will not show that drift. Report results separately by channel and language. Before using call recordings for training or evaluation, remove unnecessary personal data and confirm the notice and legal basis with your legal team.

On the phone channel, the classifier does not hear the call; it reads the speech-to-text output. Microsoft's speech recognition accuracy documentation describes contact-center audio as low quality (8 kHz, compressed, sometimes two people on one channel) and puts recognition quality for this scenario at medium, with a word error rate under 30 percent. According to the same page, product and people's names cause substitution errors, and noise and overlapping speakers cause deletion errors. So test the classifier on the real output of the recognition system you use, not on clean human-written transcripts.

How do you read a routing confusion matrix?

In a confusion matrix, rows show the true class and columns show the queue the model sent the request to; scikit-learn uses the same layout. Write counts, not percentages: a cell tells you how many times one specific misroute happened. Below is a hypothetical review set of 600 requests:

True class ↓ / Queue sent to →BillingDeliveryCancellationHuman triageTotal
Billing27010515300
Delivery8196412220
Cancellation14652880

Compute recall from each row and precision from each column. Recall for billing is 270/300, or 90 percent; for delivery it is 196/220, about 89 percent. For cancellation it is 52/80, only 65 percent. Of the 61 requests sent to the cancellation queue, 52 really are cancellations, so precision is about 85 percent. In total, 518 requests reached the right queue (about 86 percent) and 35 went to human triage.

The overall picture looks fine, but the real finding sits in one cell: 14 genuine cancellation requests went to the billing queue. That means a customer who wants to cancel waits with the wrong team and most likely calls again. Google's course on classification metrics makes the same point: on imbalanced data accuracy misleads, and a model that never predicts a class occurring 1 percent of the time still scores 99 percent accuracy. According to scikit-learn, macro averaging gives every class equal weight and so highlights the performance of infrequent but important classes. The next step is to read those 14 requests one by one; if phrases such as "cancel my bill" are genuinely ambiguous between the two classes, write a rule for them or send them to human triage.

Which metric matters for which error?

Write down, for each class, the cost of missing it and the cost of a false alarm; the metric to prioritise follows from those costs. Google's course recommends prioritising recall when false negatives are more expensive, while the example in Microsoft's documentation finds precision more relevant when a support ticket with the wrong class is forwarded to the wrong team.

SituationCost of missing itCost of a false alarmMetric to prioritise
Urgent or safety-related requestHighLow (a person reviews it)Recall
Cancellation requestHigh (repeat calls, lost customers)MediumRecall
Class that triggers an automatic actionMediumHigh (wrong action)Precision
General enquiryLowLowF1

What should happen when the model is unsure?

Low-confidence, multi-intent and sensitive requests should go to a trained agent. Set a confidence threshold per class and measure what moving it does: as Google's course explains, raising the threshold tends to reduce false positives and increase false negatives, and lowering it does the opposite. Count the human-triage queue as its own column in the matrix, because its capacity is limited. If that queue overflows, reducing the number of automatically routed classes is safer than forcing uncertain requests into one.

On the agent's screen, show the predicted class and a short reason without hiding the original message. Make it possible to correct the route with one click, and record every correction as a label for the next evaluation.

How do you run the pilot?

Bring the classifier in over three stages. First run it in shadow mode: the model predicts, people still route, and the two are compared. Then show the prediction to agents as a suggestion. Start automatic routing only with classes that meet their error limit, and in a single queue. At every stage compare against the current manual routing: transfer rate, time to reach the right queue, first-contact resolution and agent corrections, all broken down by class and channel. Repeat the measurement after product or policy changes, and when a new request type appears, review the labelling rules before widening automatic routing. We cover a similar flow for written channels in our article on the email triage agent with human review.

What should you monitor after go-live?

A classifier starts to age the day it goes live, because request wording, products and campaigns change. Every week, build a new confusion matrix from agent corrections and compare it with the previous week. Track sudden shifts in the class mix, the share of requests sent to human triage and the transfer rate, all per class. If the correction rate climbs for a particular class, take a random sample of its recent requests, have it relabelled and check whether the model recognises the new wording. Update thresholds and the class list from these figures, and confirm the labelling rules still hold before retraining the model.

Common mistakes

  • Creating many topic classes instead of defining classes by queue.
  • Training on closed tickets and running on first messages.
  • Testing the phone channel on clean transcripts.
  • Reporting success as overall accuracy and missing rare classes.
  • Ignoring the capacity of the human-triage queue.
  • Not recording agent corrections.

Systems that classify incoming requests, route them to the right team and keep sending behind human approval fall under the request-triage use case described on our AI agent development service page.

Key takeaways

  • Define classes by queue or next step, merge classes that people cannot tell apart, and keep a human-triage class.
  • Train and test only on what is known at routing time, split test data by time, and use real speech-to-text output for calls.
  • Read the confusion matrix in counts: recall from rows, precision from columns, and the costly cells first.
  • Pick recall or precision per class from the cost of a miss versus a false alarm, not from overall accuracy.
  • Set per-class confidence thresholds and count the human-triage queue's workload as its own destination.
  • Move from shadow mode to suggestions to automatic routing one queue at a time, measured against manual routing.

Frequently asked questions

How many classes should a contact-center request classifier have?

As many as there are distinct queues or next steps, and no more. If two classes go to the same queue with the same next step, merge them; if people labelling the same requests disagree on two classes, the model will not separate them reliably either. Always add a human-triage class for uncertain requests.

Why is accuracy a poor measure for request routing?

Because request classes are imbalanced. A model can be right on most billing and delivery requests and still miss a large share of a small but costly class such as cancellations. Read recall and precision per class from the confusion matrix, and use macro averaging to keep rare classes visible.

Should I prioritise precision or recall?

It depends on the class. Where missing a request is expensive, as with urgent or cancellation requests, prioritise recall. Where a wrong prediction triggers an automatic action or sends work to the wrong team, prioritise precision, and decide this per class rather than once for the whole model.

Can I train a routing model on closed tickets?

Only on the part of the ticket that existed at routing time. Closed tickets contain resolution notes, transfers and final categories that the first message does not have, so a model trained on them looks good in testing and performs worse live. Label each example by the queue it should have gone to from the first message.

How do speech recognition errors affect call classification?

On the phone channel the classifier reads the speech-to-text output, not the call. Microsoft describes contact-center audio as low quality, with product names and overlapping speakers causing recognition errors, so test the classifier on the real transcripts your recognition system produces.

All articles