What does classifying contact-center requests mean?
Contact-center request classification means assigning every incoming request (a call transcript, a chat message, an email or a form) to a class that determines a queue or a next step. The goal is not to tag the topic but to send the request to the right team the first time. That is why a good classifier is defined not by its accuracy figure but by how many requests reach the wrong queue and what that costs.
The first rule follows directly: every class must correspond to a distinct action. If two classes go to the same queue and trigger the same next step, separating them only makes the model's job harder, so merge them. Conversely, if requests that go to the same queue carry different urgency (a cancellation versus a general enquiry, for example), keep them apart.
How do you choose the classes?
Settle the class list before the model, together with agents and tested against real requests. Four checks cover most of it:
- Does it map to an action? Every class should have a target queue or an automatic next step.
- Do people give the same label? Have two agents label the same sample independently. Cohen's kappa, listed among the classification metrics in the scikit-learn documentation, is meant for measuring agreement between human annotators; according to that page, scores above 0.8 are generally considered good agreement and zero or below means no agreement. Do not expect a model to separate two classes that people cannot.
- Is there room for uncertain requests? Without a "human triage" class, the model forces every request somewhere.
- How are multiple intents handled? For requests such as "my bill is wrong and I also want to cancel my subscription", either write an approved priority rule or send them to human triage.
Microsoft's custom text classification documentation warns in the same direction: classes whose training data is very similar are frequently confused, and it suggests merging them or labelling more examples that tell the two apart. According to the same page, a class with fewer than 15 labelled instances in the training data can lead to lower accuracy. For urgent or sensitive requests (safety reports, formal complaints, risk to life), add keyword and channel rules on top of the model's decision; missing these requests costs more than sending them to the wrong queue.
Rules, a trained model or a language model?
Choose the method by the number of classes, the labelled data you have and how quickly things change. With a few classes that are worded distinctively (for example, billing disputes arriving through their own form), keyword and channel rules may be enough, and they are the easiest to explain. If you have labelled historical requests and the classes stay stable for months, a trained text classifier is usually fast and cheap to run. If classes change often or labelled data is scarce, a language-model step that receives the class definitions and a few examples in the prompt is more flexible, but its latency and cost per request must be measured and it must be evaluated with the same confusion matrix. In every case, routing is usually not agent work but a fixed workflow with a single text-understanding step. Whichever you choose, the evaluation steps below stay the same.
How do you prepare training and test data?
Use only information that is available at routing time. A closed ticket contains the resolution note, the transfer history and the final category; the first message contains none of them. Train on the full ticket and you will see good test results and poor live results. Label each example by asking "which queue should this have gone to, given the first message?", not "what did this request turn out to be?".
Split the test set by time: train on earlier months and test on later ones. Product launches, price changes and campaigns shift the mix of requests, and a randomly split test set will not show that drift. Report results separately by channel and language. Before using call recordings for training or evaluation, remove unnecessary personal data and confirm the notice and legal basis with your legal team.
On the phone channel, the classifier does not hear the call; it reads the speech-to-text output. Microsoft's speech recognition accuracy documentation describes contact-center audio as low quality (8 kHz, compressed, sometimes two people on one channel) and puts recognition quality for this scenario at medium, with a word error rate under 30 percent. According to the same page, product and people's names cause substitution errors, and noise and overlapping speakers cause deletion errors. So test the classifier on the real output of the recognition system you use, not on clean human-written transcripts.
How do you read a routing confusion matrix?
In a confusion matrix, rows show the true class and columns show the queue the model sent the request to; scikit-learn uses the same layout. Write counts, not percentages: a cell tells you how many times one specific misroute happened. Below is a hypothetical review set of 600 requests:
| True class ↓ / Queue sent to → | Billing | Delivery | Cancellation | Human triage | Total |
|---|---|---|---|---|---|
| Billing | 270 | 10 | 5 | 15 | 300 |
| Delivery | 8 | 196 | 4 | 12 | 220 |
| Cancellation | 14 | 6 | 52 | 8 | 80 |
Compute recall from each row and precision from each column. Recall for billing is 270/300, or 90 percent; for delivery it is 196/220, about 89 percent. For cancellation it is 52/80, only 65 percent. Of the 61 requests sent to the cancellation queue, 52 really are cancellations, so precision is about 85 percent. In total, 518 requests reached the right queue (about 86 percent) and 35 went to human triage.
The overall picture looks fine, but the real finding sits in one cell: 14 genuine cancellation requests went to the billing queue. That means a customer who wants to cancel waits with the wrong team and most likely calls again. Google's course on classification metrics makes the same point: on imbalanced data accuracy misleads, and a model that never predicts a class occurring 1 percent of the time still scores 99 percent accuracy. According to scikit-learn, macro averaging gives every class equal weight and so highlights the performance of infrequent but important classes. The next step is to read those 14 requests one by one; if phrases such as "cancel my bill" are genuinely ambiguous between the two classes, write a rule for them or send them to human triage.
Which metric matters for which error?
Write down, for each class, the cost of missing it and the cost of a false alarm; the metric to prioritise follows from those costs. Google's course recommends prioritising recall when false negatives are more expensive, while the example in Microsoft's documentation finds precision more relevant when a support ticket with the wrong class is forwarded to the wrong team.
| Situation | Cost of missing it | Cost of a false alarm | Metric to prioritise |
|---|---|---|---|
| Urgent or safety-related request | High | Low (a person reviews it) | Recall |
| Cancellation request | High (repeat calls, lost customers) | Medium | Recall |
| Class that triggers an automatic action | Medium | High (wrong action) | Precision |
| General enquiry | Low | Low | F1 |
What should happen when the model is unsure?
Low-confidence, multi-intent and sensitive requests should go to a trained agent. Set a confidence threshold per class and measure what moving it does: as Google's course explains, raising the threshold tends to reduce false positives and increase false negatives, and lowering it does the opposite. Count the human-triage queue as its own column in the matrix, because its capacity is limited. If that queue overflows, reducing the number of automatically routed classes is safer than forcing uncertain requests into one.
On the agent's screen, show the predicted class and a short reason without hiding the original message. Make it possible to correct the route with one click, and record every correction as a label for the next evaluation.
How do you run the pilot?
Bring the classifier in over three stages. First run it in shadow mode: the model predicts, people still route, and the two are compared. Then show the prediction to agents as a suggestion. Start automatic routing only with classes that meet their error limit, and in a single queue. At every stage compare against the current manual routing: transfer rate, time to reach the right queue, first-contact resolution and agent corrections, all broken down by class and channel. Repeat the measurement after product or policy changes, and when a new request type appears, review the labelling rules before widening automatic routing. We cover a similar flow for written channels in our article on the email triage agent with human review.
What should you monitor after go-live?
A classifier starts to age the day it goes live, because request wording, products and campaigns change. Every week, build a new confusion matrix from agent corrections and compare it with the previous week. Track sudden shifts in the class mix, the share of requests sent to human triage and the transfer rate, all per class. If the correction rate climbs for a particular class, take a random sample of its recent requests, have it relabelled and check whether the model recognises the new wording. Update thresholds and the class list from these figures, and confirm the labelling rules still hold before retraining the model.
Common mistakes
- Creating many topic classes instead of defining classes by queue.
- Training on closed tickets and running on first messages.
- Testing the phone channel on clean transcripts.
- Reporting success as overall accuracy and missing rare classes.
- Ignoring the capacity of the human-triage queue.
- Not recording agent corrections.
Systems that classify incoming requests, route them to the right team and keep sending behind human approval fall under the request-triage use case described on our AI agent development service page.