How to know whether your AI is actually working
Not whether it is impressive. Whether it is working. Those are different questions and the second one is harder, because the metrics that feel meaningful are usually the ones that cannot tell you anything.
Three months after a deployment goes live, somebody asks whether it was worth it. This is a reasonable question and most businesses cannot answer it, because nothing was measured before the system arrived and nothing meaningful has been measured since.
The metrics that tell you nothing
Accuracy
Accurate against what? Unless you froze a set of real examples with known-good answers before the system went live, an accuracy figure is a vendor's opinion with a decimal point attached.
Messages handled
Volume is not value. A system that handles a thousand messages badly has a bigger number than one that handles two hundred well and escalates the rest correctly.
Time saved
Almost always modelled rather than measured, and the model is usually built by whoever is selling the system. If nobody timed the old way, the saving is arithmetic performed on a guess.
Are people using it without being told to? Adoption is the only metric that cannot be gamed, because it is a choice your staff make every day. A tool people prefer is working. A tool people are reminded to use is not, whatever the dashboard says.
What to measure instead
1. Adoption, unforced
Track usage with no mandate behind it. One deployment went from zero to 1,536 captured conversations in its first month, without a training programme, because working inside the system became easier than working around it. That number meant more than any accuracy figure, because it was a thousand-plus individual decisions to use the thing.
2. The escalation rate, and its direction
How often does the system correctly hand something to a human? You want this number to be non-zero, which surprises people. A system that never escalates is not confident, it is unsupervised. Watch the trend rather than the level: it should fall slowly as the knowledge base gets better, and a sudden jump means something changed in your business that the system has not been told about.
3. Response time at the edges
The average is not interesting. The 11pm and Sunday numbers are, because that is the window the system exists to cover. If your out-of-hours response time has not moved, you have bought a very expensive way to do what you were already doing.
4. Cost per unit of work, measured not estimated
Ours is instrumented per request. It is how we found that Thai runs roughly three times English on tokens, which means sizing a non-English workload from an English baseline gives you a badly wrong number. You cannot discover things like that from a monthly invoice.
5. The thing that used to go wrong
The best measure is usually specific to you and known before you start. Enquiries that went unanswered overnight. Follow-ups nobody got to. Quotes that were never chased. Pick the one that hurt, count it now, and count it again in ninety days.
Build the eval set before you go live
Freeze a set of real examples with known-good answers before a model touches production. Twenty is enough to start. Without that baseline, every later claim about improvement is unfalsifiable.
And a corpus that is not linked to outcomes is an archive, not a dataset. We once found 1,536 captured conversations that could not be joined to the results they produced, because of a single field never being populated. The data existed and answered nothing.
Ninety days is the honest review point
Earlier than that and you are measuring novelty. Later and you have already absorbed the cost of something that was not working. Pick the numbers at the start, write them down, and look at them once.
Seeing this in your own operation?
We build AI workers that run inside small service businesses, then stay and operate them. If any of the above sounds like your week, the conversation is short and there is nothing to install.
Related: What it costs compared with hiring · Who is accountable for the decision
Want this running on your phones?
Tell us how many calls you miss in a week and what an average job is worth. If an answering service will not pay for itself on those numbers, we will say so.