Chatbot resolution audit: check a six-conversation cohort

What exactly should count as a resolution?

A chatbot resolution audit should start with a question that a reader can independently answer from the same records. Our question is deliberately narrow: in a fixed practice group, how many conversations ended with an explicit customer confirmation, without a human reply, and without a later unresolved return before the observation cutoff? This is a local review rule, not a replacement for any platform metric.

Write a boundary card first: “Practice set B1–B6; observation ends at the final supplied event; unit is one conversation; confirmed bot-only resolution requires explicit confirmation, no human reply and no subsequent unresolved return within this record.” A cohort simply means a group selected using the same rule. Here all six conversations belong to it, and the bot participated in each.

The denominator is the number you divide by. It stays six for both calculations below. We are not dividing by messages, customers who responded, or conversations that happen to look successful. If a thread contains several messages, it still contributes one conversation to this exercise. Fixing that choice before reading the outcomes prevents an attractive percentage from quietly changing the question.

For a broader starting point, the artificial intelligence guide covers choosing a useful first task. This exercise does not install a bot, compare providers, calculate a bill, or establish how any real company performs. Its output is a small, inspectable acceptance record.

Copy this fictional six-conversation record

Imagine a small equipment rental desk practising an audit before it has any production report. Every event below is invented for teaching. These are not customer interviews, measured model responses or results from a business trial. There is no hidden transcript. Use the final supplied event as the observation cutoff and make no assumption about what happens afterwards.

Count the identifiers aloud or mark them on paper: B1 through B6, with no duplicate and no missing row. B6 contains a later event in the same conversation. It does not become a seventh conversation just because the customer returned. Keep the original confirmation and the subsequent objection together, rather than preserving whichever message supports your preferred claim.

B4 and B5 both lack a final confirmation, but their evidence is different. One follows a question; the other follows an answer. Preserving that difference helps you interpret a platform definition later. It does not justify guessing that either customer was satisfied, dissatisfied or unavailable. Those facts are not in the practice record.

Turn each record into an evidence-backed decision

Create a second table beneath the source record. Use five headings: conversation ID, final category, supporting event, included in confirmed bot-only resolution, and included in confirmed resolution by either bot or employee. Keep the source separate from your decisions. Otherwise a correction to your judgement can accidentally rewrite the material you were supposed to inspect.

A no in these columns is not automatically a failed bot response. It means the record does not meet this particular inclusion rule. B3 is a successful customer outcome involving an employee. B5 is unknown under our confirmation requirement. B6 has evidence that the earlier success no longer describes the final position. Different reasons deserve different labels.

If you later receive additional events, preserve this record as V1 and make a dated V2. Do not silently replace the original cutoff. The customer service writing templates address reply preparation and handovers; this ledger instead checks a completed reporting claim.

Calculate two rates without moving the denominator

Open a calculator on your phone or computer. Enter 2 divided by 6 multiplied by 100 for confirmed bot-only resolution. The result is 33.333…%, which you can display as 33.3% to one decimal place. For confirmed resolution by either party, enter 3 divided by 6 multiplied by 100. That result is 50%. Write the fractions beside the percentages so rounding cannot conceal the counts.

Your accepted statement is: “For these six fictional conversations, at the supplied cutoff and under our explicit-confirmation rule, confirmed bot-only resolution is 2/6 = 33.3%; confirmed resolution including employee help is 3/6 = 50%.” This does not estimate a population rate or forecast production savings.

Now challenge three tempting statements. “The bot resolved half the group” incorrectly includes B3 in the bot-only numerator. “Ignore the three unfinished records and the bot resolved two thirds” changes the agreed denominator. “B6 was resolved once, so add it to success” ignores the final supplied objection. Mark each claim revise and write the exact row causing the problem.

Reconcile the categories as another independent check: two bot-confirmed, one human-assisted, one unanswered clarification, one silent-after-answer and one reopened-unresolved equals six. The chart and percentage audit provides another worked example of keeping a claim aligned with its source numbers. No chart is needed here.

Keep local audit rules separate from vendor definitions

A provider may legitimately use a different definition. Intercom documents both confirmed and assumed Fin resolutions. Leaving after an answer without requesting more help can be treated differently from leaving after a clarification question. Our explicit-confirmation rule is intentionally narrower. A difference from our result is therefore not, by itself, proof that the provider dashboard is wrong.

Zendesk distinguishes solved tickets from closed tickets. A later reply to a solved ticket can reopen or update it, while a reply to a closed ticket can create a follow-up. Consequently a closure label, a follow-up record and an explicit customer confirmation are not interchangeable pieces of evidence. Check how your own reporting system links records before applying a conversation-based rule.

For an actual comparison, record the metric name, official definition, selected period, filters, unit and denominator together. Keep any billing outcome definition on a separate line. This walkthrough neither checks invoices nor says how a vendor should charge you. It also makes no claim that one product resolves more issues than another. Definitions explain scope; they do not supply comparative performance data.

If customer feedback is missing, leave that gap visible. Do not invent a positive survey response to complete the table. The survey question review can help prepare a later feedback question, but writing that question is not the same as receiving a customer answer.

What can you save and reopen in fifteen minutes?

You can complete the core exercise without a support subscription or an AI account. On Windows, open Start, type Notepad and open the application. Create a new document headed “Practice cohort V1: fictional data, not company performance.” Paste the boundary card, all six source records, your decision ledger and both checked calculations. Keep the unknown cases visible rather than deleting them to shorten the file.

Press Ctrl+S, choose Documents, enter chatbot-cohort-V1.txt and save. Close the editor. Open the saved file from that folder and check B6’s later objection, the denominator of six and both final percentages.

For an optional AI second opinion, provide only this fictional record to an assistant you already use. Copy this request: “Classify each ID using its final supplied event. Keep all six conversations in the denominator. Require explicit confirmation for confirmed bot-only resolution. Do not attribute human-assisted work to the bot. Distinguish an unanswered clarification from silence after an answer. Return the source evidence, decisions and two rates separately. Invent no events.” No particular assistant or account was tested for this article.

Compare the proposed output with the human answer key above. If the assistant includes B3 or B6 in bot-only success, correct the inclusion rule rather than accepting a fluent summary. If it cannot be accessed, write “optional AI comparison not performed.” Your manually checked ledger can still be complete. This exercise does not send messages, change ticket statuses or alter a live support system.

A suggested fifteen-minute allocation is three minutes for the boundary and records, four for classification, four for arithmetic and four for saving and reopening. This is a planning allowance, not a measured speed promise. In my earlier writing about perfection and effectiveness, I described work delayed by pursuing every detail. Applied here, that principle means completing one traceable six-row decision before redesigning a dashboard; it is not a claim of personal chatbot trial results. For installation work, use the separate website chatbot setup guide.