You cannot improve what you do not measure — but most chatbot dashboards measure the wrong things. They proudly display total conversations, total messages, and an ‘engagement’ score, none of which tell you whether the chatbot is actually helping your customers or your business.
Volume is not value. A chatbot that handles ten thousand conversations badly is worse than one that handles a thousand well. Here are the six metrics that actually tell you whether your chatbot is working, and the vanity numbers you can safely ignore.
The vanity metrics to ignore
Start by tuning out the numbers that feel good but mean little. Total conversations, total messages sent, and generic ‘engagement’ tell you how busy your bot is, not how useful it is. A rising conversation count could mean growing traffic — or it could mean your bot is failing so often that people keep trying. On its own, volume is noise.
1. Containment (self-service) rate
This is the headline efficiency metric: the share of conversations the chatbot resolves without handing off to a human. It is the number that most directly drives your return on investment, because every contained conversation is one your team did not have to staff.
But containment alone can lie, which is why it needs the next metric right beside it.
2. Resolution rate
Containment tells you the bot ended the conversation without a human. Resolution tells you whether it actually solved the problem. A high containment rate paired with a low resolution rate is not success — it is deflection theatre, where customers give up rather than get helped.
Measure resolution by asking (a quick ‘did this solve your problem?’ at the end of a conversation) and by watching whether the same customer returns with the same question. Containment times resolution is the true picture of automation quality.
3. Escalation rate and handoff quality
Not every conversation should be contained, so escalation is not a failure — it is a feature. What matters is how often the bot escalates and how smoothly it does it. Track the escalation rate to spot where the bot is out of its depth, and check that every handoff carries the full conversation so customers never repeat themselves. A clean handoff is often what earns customers’ trust in the bot for everything else.
4. Lead capture and conversion rate
If your chatbot has a sales or lead-generation job, measure whether it does it: the share of relevant conversations that produce a qualified lead, a booked meeting, or a captured contact. This is the metric that connects your chatbot directly to revenue, and it is the one your sales team will care about most.
5. Customer satisfaction (CSAT)
A simple satisfaction rating on chatbot conversations keeps you honest. It is entirely possible to have great containment and resolution numbers while customers quietly hate the experience. CSAT catches the gap between ‘the bot ended the conversation’ and ‘the customer felt helped.’ Watch the trend, not just the absolute number.
6. Fallback (‘I don’t know’) rate
This is your diagnostic metric. Every time the bot cannot answer, it is telling you where your knowledge base is weak. A rising fallback rate is an early warning that your content has gaps or that customers are asking about something new. Treat the fallback log as a running to-do list for improving your knowledge base.
How the six work together
These metrics are most powerful in combination. Containment times resolution times CSAT tells you the real story of how well your bot serves customers. Escalation rate and fallback rate point you to exactly what to fix. And lead capture ties the whole thing back to business outcomes. Any single one of them, viewed alone, can mislead.
Turn metrics into a habit
Numbers only help if you act on them. Set a target for each, review them on a regular cadence — weekly is a good start — and map every metric to an action. Low resolution? Improve the answers. High fallback? Fill the content gaps. Rising escalations in one topic? That topic needs work. A chatbot you measure this way stops being a black box and becomes something you can steadily improve.
The bottom line
‘We handled 10,000 conversations’ is a number that means nothing. Containment, resolution, escalation, capture, CSAT, and fallback are the six that turn a chatbot from a vanity project into a measurable part of your operation. Track those, ignore the rest, and you will always know whether your automation is earning its place.
Monology surfaces these numbers in its built-in analytics, so you can see real containment, capture, and fallback rates — and act on them — rather than guessing.
