--- title: "How do you know if a chatbot resolved the issue? — MING Labs" description: "Check chatbot resolution beyond the closed chat. Use an illustrative ten-case review to distinguish same-issue returns, unrelated contacts and unknown outcomes." canonical: https://www.minglabs.com/insights/answers/how-do-you-know-if-a-chatbot-resolved-the-issue lang: en last-updated: 2026-10-23 --- # How do you know if a chatbot resolved the issue? [Home](/) [Insights](/insights) [Answers](/insights/answers) How do you know if a chatbot resolved the issue? [English](/insights/answers/how-do-you-know-if-a-chatbot-resolved-the-issue)[Deutsch](/de/insights/answers/hat-der-chatbot-das-kundenproblem-geloest) Answer Check what happened to the customer’s issue after the chat ended. Separate a system’s resolved label, the customer’s feedback and evidence that the requested outcome occurred. Then follow the same issue across reopened chats and new contacts for a defined period. Our proposed review keeps repeat contacts and missing evidence visible; a quiet inbox alone does not prove that the problem was solved.[\[S1\]](#source-s1) Invented example · 10 issues · 7 days each Two of the ten issues return to support. 1. Same issue returns 2. Same issue returns 3. Link unclear 4. Different issue 5. No return observed 6. No return observed 7. No return observed 8. No return observed 9. No return observed 10. No return observed All 10 issues were initially marked resolved. Within seven days of each mark, two return about the same issue, one follow-up cannot be linked reliably and seven have no observed same-issue return. One of those seven has a follow-up about a different issue. The observed repeat share is 2 ÷ 10 = 20%. This does not establish an 80% success rate. Last updated: October 2026 | Next review: November 2026 Proposed review method [Machine-readable record ](/insights/answers/how-do-you-know-if-a-chatbot-resolved-the-issue.json) By MING Labs · Editorial team The answer at a glance 1. [A resolved label needs a definition.](#labels) 2. [Follow the issue when the customer opens a new chat.](#cohort) 3. [Use the return reason to choose the repair.](#decision) ## A closed conversation is only one piece of evidence. Suppose a customer asks for a refund, receives an explanation and leaves the chat. Two days later, they email because the refund never arrived. The first conversation can look complete while the customer’s task is still unfinished. This is an invented situation, but the measurement question is real: a public support practitioner asked Intercom to track new contacts after assumed resolutions, including contacts that start a new conversation.[\[S4\]](#source-s4) Start by reading what your system calls resolved. Intercom distinguishes positive customer feedback from an assumed resolution when the customer leaves without asking for further help.[\[S2\]](#source-s2) Zendesk’s current verified-resolution tier includes an LLM assessment of the conversation text.[\[S3\]](#source-s3) These are product definitions. Neither label, by itself, is a record that a refund reached the customer. We would keep three fields beside each other: the system’s label, the customer’s response and the evidence of the requested outcome. For the refund, that last field needs an appropriate transaction status. For an explanation, a reviewer can check whether it was correct and answered the question. Define the needed evidence with the person responsible for the service before scoring cases. ## Does your service need a clearer definition of done? Discuss one customer request and the evidence needed after the chat. [Discuss my support workflow](/contact?type=project&topic=support-resolution) Tell us what you have in mind. We will respond personally. ## Follow ten original issues, even when the next contact has a new number. Our example begins with ten invented issues initially marked resolved by a chatbot. Each has a full seven-day observation period starting at that mark. We follow the initial issue, rather than counting every message as a separate success or failure. The chosen number and period explain the calculation; they are not a sample-size recommendation or a service target.[\[S1\]](#source-s1) Issue 01 returns in the same chat on day two. Issue 02 returns in a new email on day four; a reviewer confirms that it concerns the same unresolved problem. A later contact potentially related to issue 03 cannot be linked reliably. Issue 04 has a new, unrelated question. For issues 05–10, no further contact is observed in the available records. These are assumed observations, not records from a MING client. The two same-issue returns count once each. The unclear link stays visible, and the unrelated question does not become a failure of the original answer. | Initial issues | What the full seven-day window shows | Count | | --- | --- | --- | | 01–02 | Confirmed return about the same unresolved issue | 2 | | 03 | Follow-up cannot be linked reliably | 1 | | 04 | Return concerns a different issue | 1 | | 05–10 | No return observed in the available records | 6 | | All initial issues | Initially marked resolved, each with complete follow-up | 10 | The observed same-issue repeat share is 2 ÷ 10 = 20%. Seven issues have no observed same-issue return and one has an unclear link. Do not rename the remaining 80% “success”: an unobserved return or a customer who gives up can remain invisible. This is our proposed issue-cohort measure, not a reproduction of either vendor’s dashboard metric.[\[S1\]](#source-s1) ## Keep the evidence that connects a return to the original problem. A new ticket number can hide a repeat; the same customer can also have several unrelated problems. Use the identifiers and records already authorised for your service, and inspect the reason for contact. A matching topic label is a clue, not sufficient proof of the same unresolved issue. If the connection cannot be established, keep it unknown. A short review record makes the denominator and the judgement inspectable. The same fields work for both a reopened conversation and a new contact. | Record | Why it matters | | --- | --- | | Initial issue and requested outcome | Defines the unit counted once and what a solution would mean. | | Resolved mark, window end and included channels | Shows whether the full period has elapsed and what could be observed. | | Follow-up reference and same-issue judgement | Separates a repeat, another topic and an unclear connection. | | Outcome evidence and reviewer | Links the conclusion to a record and a responsible person. | Keep cases whose window is still open in a waiting group. They enter the calculation when their full period has elapsed. Also retain counts for all incoming issues, including those the chatbot never handled or passed to a person. A strong result on a narrow, easy subset does not describe the whole service. For the wider evidence chain, use our [audit-trail guide](/insights/answers/what-belongs-in-an-ai-agents-audit-trail) . ## Repair the reason customers have to come back. Return to the refund example. If the answer promised a payment that was never initiated, inspect the action and its confirmation. If the payment is still within an agreed processing time, the next improvement may be a clearer expectation and a visible status. If the chatbot cannot safely handle the request, the service needs a working handover. A lower repeat percentage is not a reason to make it harder to reach a person. Review the confirmed returns and unclear links with the service owner, then choose one correction. Compare equivalent issue types, channels and completed windows after that change. Keep correctness, waiting time and human effort beside the repeat share; our [quality-metrics answer](/insights/answers/which-metrics-show-whether-an-ai-agent-is-improving) explains those separate checks. Start with one recurring customer request and establish what would prove it finished. Evidence and limits - The ten issues, events and seven-day period are invented. The calculation illustrates a proposed MING review; it does not establish a customer result, reliable sample size or recommended threshold. - Missing channels, uncertain identity and customers who stop trying limit what repeat contact can establish. No observed return is not verified resolution. - The public practitioner question supports the usefulness of asking; vendor documentation explains its own labels as read on 23 October 2026. We have not tested those products or compared their accuracy. Citable short version MING proposes reviewing a fixed cohort of issues initially marked resolved by a chatbot, with a complete follow-up window for each. Count issues with a confirmed same-issue return once, retain unclear links separately and verify the actual customer outcome. Our invented example has two returns among ten issues over seven days each: a 20% observed repeat share, not proof of an 80% success rate. Ten invented issues and current vendor definitions. A public practitioner question establishes the problem; no customer outcome or industry benchmark is measured. ## Which customer issue needs a clearer outcome? Bring one recurring support request and the signals your team currently uses. We can discuss what completion means and how to review the follow-up. [Discuss my support workflow](/contact?type=project&topic=support-resolution) Tell us what you have in mind. We will respond personally. [Read about agent quality metrics ](/insights/answers/which-metrics-show-whether-an-ai-agent-is-improving) ## Sources \[S1\] Seven-day issue-cohort example MING Labs · 23 October 2026 Supports: 2 same-issue returns / 10 initially marked-resolved issues = 20%; 1 unclear link, 7 without observed same-issue return. Full seven-day follow-up for each. Original proposed method and ten invented issues, n=10. No customer sample, product test or benchmark. \[S2\] [Reporting metrics & attributes](https://www.intercom.com/help/en/articles/7022438-reporting-metrics-attributes) Intercom · Read 23 October 2026 Supports: Confirmed and assumed resolution labels differ; recontact metrics depend on windows, denominators and topic coverage. Current vendor definitions, not independently validated customer outcomes. \[S3\] [About automated resolution tiers](https://support.zendesk.com/hc/en-us/articles/9570369117338-About-automated-resolution-tiers) Zendesk · Read 23 October 2026 Supports: Verified resolution includes an LLM assessment of conversation text. Current vendor definition for its resolution tiers, not a test of the evaluator. \[S4\] [Calculate recontact rate after (assumed) AI resolution](https://community.intercom.com/ideas/calculate-recontact-rate-after-assumed-ai-resolution-14553) Christian Osmundsen, Intercom Community · 15 May 2026; read 23 October 2026 Supports: A support practitioner asks to connect assumed resolution with subsequent contact, including a new conversation. Public first-person feature request; self-identified company affiliation, no measured demand volume or effectiveness data. ## Frequently asked questions Is every repeat contact a chatbot failure? No. A customer may ask a different question, add information or follow an agreed next step. Read the reason and link the contact to the original issue before classifying it. Several messages about one unresolved issue count as one affected initial issue in our proposed method. How long should the follow-up window be? Long enough for the outcome to become observable in that workflow. A password reset and a refund have different completion times. Seven days is an assumption in our example, not a recommended universal window. Give every included issue the full chosen period. Does no repeat contact mean the issue was solved? No. The customer may stop trying or use an unobserved channel. Keep evidence of the requested outcome separate from the absence of another message. A confirmed refund in the transaction record answers a different question from a quiet conversation. Can we compare this percentage with the vendor dashboard? Only after aligning the unit, denominator, time window, channels and treatment of unknowns. Our example follows initial issues; a dashboard may count new conversations during a calendar period. The rates can legitimately differ. What if no issues have completed the window? Report the number still awaiting follow-up and leave the repeat share undefined. Zero eligible issues does not mean a zero percent repeat rate. ## Available actions - **Copy answer for my AI**: copies the [AI briefing](https://www.minglabs.com/insights/answers/how-do-you-know-if-a-chatbot-resolved-the-issue.md) to the clipboard in the browser. --- Canonical: https://www.minglabs.com/insights/answers/how-do-you-know-if-a-chatbot-resolved-the-issue Machine-readable index: https://www.minglabs.com/llms.txt