Why 80% reliability is not enough for AI in customer service

By Bernardo Gontijo, founder of NeftPublished on Updated on 9 min read

Summary

An 80% success rate leaves 20% of evaluated cases unsuccessful, but does not tell you which failures reach customers or how much they cost to fix. When buying customer service AI, demand evidence of outcomes, repeated tests and a path to a human team. A preprint found that only 22.6% of passes cleared four trust checks in an academic benchmark; this percentage does not measure real customer interactions.

What 80% reliability means for the customer

An 80% success rate means one failure in every five attempts, on average, within the sample and definition of success used. It does not mean every fifth conversation will fail, or that your operation will see the same result. Before accepting the figure, ask: success at answering, performing an action or resolving the entire request?

Consider a duplicate bill: the response may be well written but contain the wrong bill. Or the correct document may never reach the customer. If someone has to check every delivery, the automation benefit must be calculated after that work. A detected failure handed over to the team also has a different impact from a false completion that goes unnoticed.

Felipe Blanes, from Amazon AGI Lab, discusses this loss of trust in the “trust cliff” chapter at 9:35. The channel's published description summarizes the idea as follows:

“80% reliability feels like more work.”

AI Engineer, description of Felipe Blanes's talk, chapter starting at 9:35. The original wording is “80% reliability feels like more work”; quoted from the editorial description, not a verbatim transcript of the talk.
Video: Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab (AI Engineer (@aiDotEngineer), YouTube, ). Published October 8, 2026; playback starts at the chapter on trust, at 9:35. Talk in English.

Blanes works at the Amazon laboratory building AI tools, including Nova Act. His talk draws on experience with browser-agent customers, not a controlled customer service study. It dates from October; the research below received an expanded version in November. Neither establishes a universal rate that makes every customer interaction safe.

Why a good benchmark score can be misleading

In the “benchmark illusion” chapter at 2:22, the talk contrasts static evaluations with unexpected customer behavior. For buyers, the implication is practical: a demo using questions selected by the provider does not replace tests using your operation's actual requests.

Ask for cases involving incomplete information, changes in intent and unavailability of the queried system. Check that the set includes complaints and out-of-scope requests as well as frequently asked questions. An average can hide an AI that handles simple queries well but fails precisely where the consequences are greatest.

The preprint A Trust Layer for Agent Evaluation, by Sediqin, Dalmia, Kovvuri and Mukherji, evaluated five agent configurations on 108 Agents’ Last Exam tasks. In the subset of 50 tasks repeated five times per configuration, between 18% and 46% did not stay in the same score band. This measures score instability, not a customer service error rate.

During procurement, ask for the distribution of results by request type, not just the best run. Record which versions of the model, instructions and integrations were tested. If the offered configuration has changed, the previous evidence must be reassessed. A report without that identification is difficult to reproduce and compare.

The four criteria for a verifiable outcome

The Trust Layer checks four dimensions: support from the benchmark's grading logic, traceable computation, agreement between claimed completion and what actually happened, and stability across repeated runs. In this subset, only 19 of the 84 recorded passes cleared all four: 22.6%, with a 95% confidence interval of 15.0% to 32.6%.

The work is a preprint, with no confirmed peer review, by authors affiliated with Centific Research. The cited sample is academic, not customer service data. The application below is a proposed evaluation approach for buyers, not an experimental validation by Neft:

  1. Verifiable rule: define what counts as resolution before testing. For a duplicate bill, this could be the document for the requested billing period, made available to the correct customer. A friendly response does not replace this criterion.
  2. Traceable path: ask for evidence of the query and action that produced the result. The record must distinguish information obtained from the system from a response generated without confirmation.
  3. True completion: compare the conversation's “completed” claim with the system's actual state. If the ticket was not created, the message must acknowledge the failure and explain how to continue.
  4. Repetition: rerun the same scenarios and relevant variations. Separate persistent failures from unstable results; both need to be addressed before expanding the scope.

These questions make the demo auditable. The provider can show examples without exposing personal data: fictional identifiers, a test environment and restricted-access records. What matters is connecting request, execution and outcome while preserving privacy.

What Brazilian regulation requires of customer service

Brazilian Decree No. 13,135/2026, published on October 13, repealed Decree No. 11,034/2022 and took effect twenty days after publication (Articles 24–25). Article 1 defines the providers covered, including federally regulated products and services and digital financial services regulated or supervised by Brazil's Central Bank. It does not automatically apply to every business.

Article 3, paragraph 3, allows automation, including AI, provided that a real-time human service option is assured during the periods and under the conditions specified. Article 4 requires security, privacy and effective resolution, among other things; Article 7 requires continuity until actual completion. Also check the applicable sector-specific regulation.

The decree sets no minimum AI accuracy percentage. The 80% mentioned in Article 15 refers to the share of telephone requests subject to a particular waiting-time limit, not the quality of automated responses. During procurement, test the human option and request continuity. A benchmark score does not prove regulatory compliance.

What the sources allow us to conclude

What the source showsWhat changes in customer serviceLimitation
Trust Layer: 22.6% of 84 passes cleared all four checksAsk for evidence beyond the aggregate scoreAcademic ALE sample; a preprint, not an estimate of customer service quality
Talk: the need for oversight can destroy trustInclude checking and correction when assessing the benefitExperience reported by an Amazon professional, not a controlled study
The sources' findings and Neft's proposed application are distinct.

How to assess reliability before buying

Define acceptance criteria by service and failure severity, involving frontline staff and those responsible for operations. An informational query may allow review; an action that changes rights or monetary amounts requires stronger controls. Do not let a single rate conceal these differences.

  • What is the denominator? Ask for the number of cases, sample origin and exclusion criteria. Abandonments and handoffs must be reported separately from autonomous resolutions.
  • What happens when it fails? Observe the path to the human team, the context provided and the response to the customer when nobody is available. Test this in the demo.
  • What does oversight cost? Measure time spent reviewing and correcting as well as time saved. Include rework caused by incorrect completion claims.
  • Who controls changes? Agree on approval owners, version history and rollback to the previous configuration. Changing instructions can affect results even without changing the model.
  • How will testing continue? Define the sample reviewed after launch, the metrics monitored and the events that halt or restrict automation.

The talk organizes learning into an evaluation loop at 4:41, starting with defining customer success at 4:56. In your operation, turn discovered failures into new test cases. The guide on testing AI before going live explains this preparation.

Reliability in practice: what Neft offers

The public Corsan case study documents the adoption of video customer service with Neft in 2022 and first-year results. This is experience with human-led video service; it is not proof of autonomous AI reliability. This distinction prevents a real case from being used to support a different promise.

For its current AI solution, Neft's public page describes allowed and prohibited topics, actions that are disabled, simulated, subject to human approval or automatic within policy, and configuration publishing with history. When needed, the team takes over with the same reference number and a conversation summary.

These controls help define what will be automated and retain an exit for cases that require people. The page does not provide an accuracy percentage or numerical AI reliability SLA. Procurement should therefore evaluate the chosen service and its resolution criteria without treating features as a statistical guarantee.

Explore Neft's AI customer service and bring a real workflow, the errors your operation cannot accept and the records you need to receive to the discussion. This makes it possible to assess a concrete proposal, with clear responsibilities and limits, before expanding automation.

Frequently asked questions

Does 80% reliability mean one in five customers will have problems?
Not necessarily. It means a 20% failure rate in the sample and definition used by the test. To estimate customer impact, you need to know which requests were evaluated, how failures were handled and whether the sample represents the operation.
What percentage should I require of customer service AI?
There is no universal percentage. Define criteria by service, consequences of error and recovery capacity. Separate correct resolution, process compliance, handoffs and rework. A high average does not compensate for serious failures hidden in a particular request type.
Does the study's 22.6% measure the quality of customer service agents?
No. It is the proportion of academic benchmark passes that cleared all four Trust Layer checks. It is neither a customer service resolution rate nor a measurement of any customer service provider.
Is handing over to a person an AI failure?
It depends on the objective. If the case is out of scope, handing it over with context may be the correct response. Record the handoff separately from autonomous resolution and assess whether the person could continue the interaction and resolve the request.

Sources

Videos, studies and documents used in this article. Preprints are versions published by their authors before peer review. Article researched and written with AI assistance. The preprint's figures and limitations, decree text, Neft's public pages, and YouTube metadata, description and chapters were checked on November 9, 2026. A secondary library transcript was used to locate material; the published quote comes from the channel description. Procurement recommendations are an editorial application, not outcomes measured by the sources.

  1. Video

    Why 80% Reliability Isn't Good Enough (opens in a new tab)

    Felipe Blanes (Amazon AGI Lab), AI Engineer channel (@aiDotEngineer) · published on

    Date, description and chapters verified on YouTube: 2:22, 4:41, 4:56 and 9:35. The quote comes from the channel's editorial description; professional experience, not a controlled study.

  2. Research paper (preprint)

    A Trust Layer for Agent Evaluation (opens in a new tab)

    Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri and Abhishek Mukherji (Centific Research) · published on

    Version 2, expanded in November; first version dated October 5, 2026. ALE sample: 108 tasks in the study; 50 repeated tasks per configuration and 84 passes in that subset. Abstract and sections 4.5–4.6.

  3. Legislation

    Brazilian Decree No. 13,135/2026 (opens in a new tab)

    Office of the President of Brazil · published on

    Customer service, AI and human assistance: Articles 1, 3, 4, 7 and 15. Repeal and effective date: Articles 24–25; published in Brazil's Official Gazette on October 13, 2026.

  4. Reference page

    Neft AI customer service (opens in a new tab)

    Neft · accessed on

    Scope, action authorization, versions and handoff to the team; no accuracy-rate promise.

  5. Reference page

    How Corsan began serving customers by video call with Neft (opens in a new tab)

    Neft · accessed on

    Human-led video service case; not used as evidence of AI reliability.

Want to serve your customers by video?

Take the free assessment or try Neft for 15 days, with no credit card.