A triage model for a public support queue, and an honest answer to whether replying faster changes how conversations end.
Eileen Ip · portfolio project
A support queue receives thousands of messages. Some of them are about to go badly — the customer will be left talking to nobody, or will keep coming back angrier. A triage model has to guess which, at the moment the message arrives, with nothing but that message to go on. Underneath it sits a second question a CX lead usually assumes the answer to: does replying faster actually help?
Both are operational. The first decides what gets worked first; the second decides whether speed is worth buying.
The Customer Support on Twitter dataset (Kaggle) is 2,811,774 tweets — a flat table, not conversations. Threads have to be reconstructed from two inconsistent reply columns: a parent pointer and a child list, either of which can be missing, and either of which can name a tweet that is not in the file (172,500 of them do). Treating the links as one undirected graph and taking its connected components recovers 798,197 conversations.
Four brands were selected to span industries — AmazonHelp, Delta, TMobileHelp and Tesco — giving 146,506 conversations. AmazonHelp is 56% of them, so every pooled figure in this report is mostly an Amazon figure, and the per-brand breakdowns are the honest unit.
Nothing in this data says a conversation went badly. The label is constructed, and constructing it defensibly was the hardest thinking in the project.
Four candidate definitions were implemented from the spec and run across the corpus, then 100 conversations were sampled — stratified by brand and by candidate, including 20 that no candidate flagged — and read by hand, blind to which candidate had fired. The verdicts: 67 went badly, 15 did not, 18 could not be called.
Three things came out of that hour that no amount of modelling would have produced:
That third point forced a decision: conversations ghosted after a handoff are censored — excluded rather than counted as “did not go badly”, because calling them fine asserts something nobody observed. 55,057 conversations were set aside on that basis, leaving 91,449 to model, 21.5% of them positive.
The leakage boundary is the design: features come from the first inbound message only, labels from everything after it. It is enforced by perturbation tests rather than by inspection — rewrite the rest of the thread and the features must not move; rewrite the opening message and the labels must not move.
The split is by time, not at random: train on conversations opened before 2017-11-16, test on those after (62,899 and 28,550). A random split would let the model train on November and test on October.
| model | PR-AUC | notes |
|---|---|---|
| keyword rule (the baseline a support tool ships with) | 0.251 | base rate is 0.230 |
| TF-IDF + logistic regression | 0.368 | ROC-AUC 0.685 |
| the same, calibrated | 0.361 | isotonic, on a time-held-out slice |
The keyword rule is barely better than picking at random. The learned model is a real but modest improvement: it catches 19.0% of all bad outcomes in the worst 10% of the queue, roughly 1.9x better than chance. Per brand it ranges from 0.16 to 0.40.
Calibration mattered more than the headline metric. Before it, a predicted 0.74 came back at 0.40 actual — useless for sizing a shift. After it the scores track, and the range compresses so that nothing scores above 0.6, which is the model declining to call any single message more than coin-flip risky.
At the chosen threshold — fast-track the worst 10% of the queue, score above 0.40:
That last number belongs next to the first. Tightening the lane to 5% barely moves precision and gives up half the bad outcomes; widening it to 25% prioritises a quarter of everything, which is close to not prioritising.
| first reply within | conversations | left unanswered |
|---|---|---|
| 0-5m | 22,129 | 25.2% |
| 5-15m | 27,445 | 26.1% |
| 15-60m | 26,158 | 18.0% |
| 1-4h | 10,691 | 12.9% |
| 4-24h | 4,206 | 13.1% |
| 24h+ | 820 | 28.2% |
Every tenfold increase in first-reply time goes with -5.9 percentage points in the rate of conversations left unanswered. Negative: slower replies go with fewer bad endings. The shape is not monotonic either — the fastest bucket (0-5m) is among the worst, the 1-4h bucket is the best, and the longest delays turn back up.
It survives the controls. Within brand the trend is -4.8pp; within predicted-difficulty band, which is the closest this design gets to holding difficulty constant, -3.2pp.
It also survives the judgement call most likely to have produced it. Censoring removed 37.6% of conversations, and a thread ending in “DM us” is both fast-replied and a negative, so deleting fast negatives could have invented the effect. Put them back and the naive trend is -2.1pp, the within-brand trend -2.9pp. Half the magnitude, same direction.
This is association, not causation. Nobody randomised who got a fast reply. Two confounders are named in the design and neither is closed by stratifying: brands answer easy questions faster, and a queue having a bad day is slow and bad at once. The plausible mechanism — that fast replies are largely templated acknowledgements that resolve nothing — is a hypothesis this data cannot test.
The outcome is unobservable for most of this corpus, and that is the first thing to say. 83% of conversations end with the customer not replying, and 45% of those end with the brand pushing them to DM or a link — so the ending happened somewhere this dataset does not reach. Conversations handed off that way are excluded rather than scored as fine, which is the defensible half of the problem. The other half is not fixed: the remaining ghosted conversations stay in the negative class, and their outcomes are equally unknown. Excluding all of them would leave only the 17% that end in the open, where the label is close to tautological. This is a middle position, chosen deliberately, and it should be argued with rather than assumed.
The label is a strict subset of the problem, and a human reads the problem as much larger. Reweighting the hand audit back to the population puts the share of conversations that went badly near 56%, with a range of 0.34 to 0.78; the label used here fires on 13.4%. Nothing in this report describes “bad support conversations” in general. It describes conversations where the customer was visibly left talking to nobody.
The label rests on one person’s judgement, on one afternoon. 100 conversations, one reader, no second opinion — so there is no inter-rater agreement figure, and no way to separate the definition from the person who applied it. The 18 conversations that could not be called are evidence the task is genuinely hard, not evidence of carelessness. A second reader on the same 100 is the cheapest meaningful improvement available to this project.
The agreement figures are weaker evidence than they look. Of 82 decidable conversations, 67 were called bad — so chance agreement is high, kappa is punished, and only 15 negatives exist to be wrong about. The ranking between candidate definitions is directional, not statistically settled.
Public tweets are a biased sample of support. Angry customers tweet; the merely confused email. Anything here describes the public face of a support queue, never the queue itself — and the brands in it know they are being watched, which is not true of a ticket system.
Two months, four brands, one of them dominant. The corpus is effectively October to December 2017, so no seasonality can be claimed and there is no before-and-after to compare. AmazonHelp is 56% of the conversations, so every pooled figure is mostly an Amazon figure and the per-brand breakdowns are the honest unit — model quality alone ranges from 0.16 to 0.40 across them.
The reply-speed result is association and cannot be made into anything stronger. Nobody randomised who got a fast reply. Stratifying by brand and by predicted difficulty narrows the two confounders the design names — brands answer easy questions faster, and a queue having a bad day is slow and bad at once — but narrowing is not closing. The mechanism that would explain it, that fast replies are largely templated acknowledgements, is a hypothesis this data cannot test.
The model is modest and the calibration says so out loud. After isotonic calibration nothing scores above 0.6. That is not a display bug; it is the model declining to call any single message more than coin-flip risky, and it caps what triage can be expected to do here.
The four definitions of “went badly” written before looking at anything. All of them. Each was high-precision and low-recall against a human reading: they agree when they fire and miss most of what a person calls bad. The rule that replaced them — the customer was left talking to nobody and never said it got sorted — came out of an hour of reading, not out of the brief, and it agrees with 91% of what it flags. The lesson is not that the four were badly chosen; it is that they were chosen in the wrong order.
Half of the human rule could not be implemented, and pretending otherwise would have broken the evaluation. The reader distinguished a customer ghosting after asking for assistance (bad) from one ghosting after a question (undecidable). That distinction lives in the opening message, which is the model’s only input — using it in the label would have made the outcome a partial restatement of the feature and flattered every number downstream. The rule stays out of the ghosting case entirely rather than guessing at intent.
The keyword baseline support tools actually ship with. 0.251 PR-AUC against a 0.230 base rate — close to useless. Worth knowing before paying for one.
VADER, for detecting a resolved conversation. It scores “thanks for nothing, worst service ever” at +0.735, more positive than “thank you, sorted!” at +0.420. Sarcastic thanks is exactly how an angry customer signs off, so a sentiment threshold alone marks the angriest endings as resolutions. The check now requires an explicit negation list to clear before sentiment is consulted at all.
The first model’s strongest single feature was one customer’s account number. Raw @mentions went into the vectoriser, so it was memorising individuals rather than reading language — which cannot generalise, and quietly inflates any score where the same customer appears either side of the split. Stripping mentions and URLs cost 0.011 PR-AUC, and that number is the honest size of what the memorisation was buying.
My own summary statistic for the reply-speed analysis. The first version compared the slowest reply bucket to the fastest. The slowest bucket holds as few as 8 conversations inside some strata, so the headline was being set by noise and flipped sign between neighbouring strata for that reason alone. Replaced with a size-weighted slope over cells of at least 100, with a test that pins exactly that failure.
Calibration, on the first attempt, exactly where it mattered. The raw model predicted 0.74 where reality was 0.40, and 0.64 where it was 0.50 — overconfident precisely at the top of the queue, which is the only part a triage threshold reads. A score that overstates risk there cannot be used to size a shift, which is the entire point of calibrating.
The transformer variant is the interesting one, because it worked and was not shipped anyway. MiniLM sentence embeddings, replacing TF-IDF with everything else held constant, score 0.374 PR-AUC against 0.361 — a gain of +0.012 that is not noise (95% CI [+0.005, +0.019] over 400 paired bootstrap resamples, ahead in every one).
It still does not run, for two reasons that are not about the metric. At the operating point the two models catch 19.2% and 19.0% of bad outcomes — the same queue, the same rota, 0.3 percentage points apart. And the dashboard explains every score in plain words, which is exact for a linear model over TF-IDF terms and simply unavailable for embeddings; a triage score a support lead cannot interrogate is one they will not use. Shipping the weaker model is the judgement, and it reverses in one line if the priorities change.
Getting the comparison at all took more engineering than the comparison deserved. Encoding 91,449 messages on a box already at 93% memory was killed outright once; the encoder was rewritten to write into a memory-mapped file and record its position, so a kill costs one batch of 32 rather than hours. Raising the thread count from 2 to 8 moved the measured rate from 1.7 to 14.5 texts a second.
Route on the score, and staff for the miss rate. Fast-track the worst 10% of arrivals — 159 conversations a day, a lane one person can actually work. That lane sees 44% bad outcomes against 23% in the queue at large, which is worth having. But it catches 19% of them, so 295 bad conversations a day still arrive through the normal queue. Any staffing plan that treats the fast lane as coverage rather than prioritisation will fail, and the second number belongs on the same slide as the first.
Do not buy a faster SLA on this evidence — test it first, and the test is cheap. The relationship between first-reply time and outcomes runs the wrong way for the “answer faster” theory, and it survives both controls this design allows. That is not proof that speed is worthless; it is a strong reason to check what your fast replies contain before funding more of them. The specific test: classify brand first replies as substantive or holding, and see whether the relationship survives inside each class. If it disappears among substantive replies, the finding was never about speed — it was about templated acknowledgements, and the fix is what the reply says, not how quickly it arrives.
Treat the score as an argument, not a verdict. The reason the dashboard shows the drivers behind every score in plain words is that a triage list a team cannot interrogate is one they will quietly stop trusting. The same reasoning is why the marginally better sentence-encoder model is not the one running. If the score is going to change who waits, the people it affects need to be able to say “that one is wrong, and here is why” — and be right sometimes.
Before any of this runs for real, fix the outcome. The blocker is not the model, it is that a public thread hides what happened for 83% of conversations. A real support desk has a ticket system that records resolution, and that changes both the label and its reliability. Rebuild it there before trusting any of these figures operationally.
Then re-audit the label on a schedule, with more than one reader. The definition came from one person reading 100 conversations in an afternoon, which is both the strongest part of this project and its narrowest point of dependence. Two readers on the same sample would give an agreement figure that currently does not exist. And because “went badly” is a judgement rather than a fact, it drifts with the product and the queue — a definition set once and never revisited is a model slowly measuring the wrong thing.
What would change the conclusions. A ticket-system outcome could easily show the label was missing most bad conversations. A second reader could show the definition was idiosyncratic. And a brand whose fast replies are substantive rather than templated could reverse the reply-speed finding entirely — that is the result to go looking for, because it is the one that would make this analysis wrong in a useful way.
Authorship: the project spec reserves Limitations, “What didn’t work” and the Recommendation for Eileen, because they are the sections an interviewer probes hardest. She asked for them to be drafted on 2026-09-15. They are the agent’s words, written from this project’s own numbers rather than from impressions, and they should be read and made her own before she defends them live. Every other section is generated from the pipeline’s artifacts.