support-triage

Support Triage: Which Conversations Are About to Go Bad

Which incoming support messages need a human now — judged at the moment they arrive, from the customer’s first message alone — and does replying faster actually change how a conversation ends?

Built on the Customer Support on Twitter dataset: 2,811,774 real tweets, rebuilt into 798,197 conversations, narrowed to 146,506 across AmazonHelp, Delta, TMobileHelp and Tesco.

Live dashboard → · Report · Stakeholder deck

The headline

Replying faster goes with worse outcomes, not better. Every tenfold increase in first-reply time is associated with -5.9 percentage points in the rate of conversations left unanswered. Conversations answered within 0-5m end badly 25% of the time; those answered in 1-4h, 13%.

It holds within brand (-4.8pp) and within predicted-difficulty band (-3.2pp), and survives at -2.1pp when the study’s largest judgement call is reversed. It is association, not causation — nobody randomised who got a fast reply.

The triage model reaches PR-AUC 0.361 against 0.251 for the keyword rule support tools ship with, on a 23% base rate. Fast-tracking the worst 10% of the queue — 159 conversations a day — finds 44% that go badly and catches 19% of all bad outcomes, while leaving 295 a day unprioritised.

The variant that won, and was not shipped

The spec asked for one transformer variant. MiniLM sentence embeddings, in place of TF-IDF and with everything else held constant, score 0.374 PR-AUC against 0.361 — a real gain of +0.012, not noise: 95% CI [+0.005, +0.019] over 400 paired bootstrap resamples, ahead in 100% of them.

It is still not what runs, for two reasons that have nothing to do with the metric:

It buys nothing where the decision is made. At the operating point the two models catch 19.2% and 19.0% of bad outcomes respectively — the same queue, the same staffing, a difference of 0.3 percentage points.
It costs the explanation. The dashboard shows why each message scored what it did, in plain words, because a linear model over TF-IDF terms makes that exact rather than approximate. Embeddings have no such story, and a triage score a support lead cannot interrogate is one they will not use.

Shipping the weaker model here is the judgement, not an oversight. It is reversible in one line if the priorities change.

The part that took judgement

Nothing in this data says a conversation went badly. Four candidate definitions were written from the brief, then 100 conversations were read by hand, blind to which candidate had fired: 67 went badly, 15 did not, 18 could not be called.

Three things came out of that hour:

All four candidates were high-precision and low-recall — they agree with a human when they fire, and miss most of what a human calls bad.
The rule a human was actually using was not among them. It became the label: the customer was left talking to nobody, and never said it got sorted.
Every undecidable conversation had the same shape — the customer stopped replying after the brand answered. That is 83% of the corpus, and 45% of those end with the brand pushing the customer to a private channel. For most support conversations, the outcome happens where this dataset cannot see it.

Those conversations are censored — 55,057 of them excluded from training and evaluation rather than counted as “did not go badly”, because calling them fine asserts something nobody observed.

How it is built

stage	what it does
`src/threads.py`	reference implementation of conversation reconstruction, one dict per tweet
`src/reconstruct.py`	the same rules over flat arrays, which is what the real file runs through
`src/subset.py`	cuts the chosen brands out, with text
`src/outcomes.py`	the candidate labels, the chosen one, and the censoring rule
`src/features.py`	arrival-time features — first inbound message only
`src/audit.py` · `src/build_label_tool.py` · `src/score_audit.py`	the blind hand audit and its scoring
`src/dataset.py` · `src/model.py`	modelling table, baseline, TF-IDF model, calibration
`src/threshold.py`	the operating point, framed as staffing
`src/reply_speed.py` · `src/sensitivity.py`	the reply-speed analysis and its robustness check
`src/dashboard.py` · `src/deliverables.py` · `src/case_study.py`	the four deliverables

Two implementations of the reconstruction exist because the readable one cannot run the real file — 2.8M tweets as Python objects is several GB, and the machine had 0.6 GB free. The array version indexes by tweet id directly (ids run 1 to 2,987,950 across 2,811,774 rows). Both are pinned to the same hand-built 15-thread fixture, so the tested code is the code that touches the data.

The leakage boundary

Triage happens before anyone replies, so features may read only the first inbound message and labels only what comes after it. This is tested by perturbation, not inspection: rewrite every tweet after the opener and the features must not move; rewrite the opener and the labels must not move. Reading code and agreeing it looks clean is how leaks survive.

Running it

pip install -r requirements.txt
python -m src.reconstruct     # rebuild conversations from the reply graph
python -m src.brand_report    # the evidence for choosing brands
python -m src.subset          # cut the chosen brands out, with text
python -m src.audit           # label the subset, draw the blind audit sample
python -m src.build_label_tool  # writes label.html for the hand audit
python -m src.score_audit     # score the candidates against the verdicts
python -m src.dataset         # modelling table
python -m src.model           # baseline, model, calibration
python -m src.threshold       # the operating-point table
python -m src.reply_speed     # the reply-speed analysis
python -m src.sensitivity     # does the censoring decision make the result?
python -m src.dashboard && python -m src.deliverables
pytest -q                     # support-triage: 102 tests

The raw dataset and everything derived from it are gitignored; python -m src.reconstruct rebuilds them from the Kaggle download.

What this cannot tell you

The outcome is unobservable for most conversations. 83% end with the customer not replying. Those handed off to a private channel are excluded; the rest stay in the negative class and their outcomes are just as unknown. A middle position, not a clean one.
The label is narrower than the problem. Reweighting the hand audit to the population puts the share that “went badly” near 56%; this label fires on 13%.
Public tweets are a biased sample of support. Angry customers tweet; the merely confused email.
Two months, four brands, one of them 56% of the data. No seasonality, no before-and-after, and pooled figures are mostly AmazonHelp figures.

A note on authorship

Limitations, “What didn’t work” and the Recommendation in the report are reserved for the author and are currently marked [DRAFT — EILEEN TO REPLACE]. Everything else is generated from the pipeline’s own artifacts, so no figure in any document is typed by hand.