How To Test AI Voice Agents

The rapid deployment of AI voice agents across B2C industries spanning healthcare, financial services, telecommunications, and retail has created an urgent need for rigorous, standardized evaluation methodologies. Unlike traditional software, voice agents operate at the intersection of acoustic signal processing, natural language understanding, real-time inference, and human conversational dynamics. A failure in any one of these dimensions can silently erode customer satisfaction, reduce conversion rates, and expose organizations to compliance risk.

This guide synthesizes findings from an analysis of over 4 million production voice agent calls across more than 10,000 deployed agents to provide practitioners, product leaders, and AI engineers with a definitive evaluation blueprint. The guidance contained within these pages reflects the state of the art as of 2026, drawing on research from Hamming AI, Braintrust, Maxim AI, and Nurix AI.

Why Evaluation Is Non-Negotiable

Voice agents are uniquely punishing environments. A text chatbot that produces a slightly awkward response causes mild friction; a voice agent that mishears a user, fails to stop speaking when interrupted, or returns a hallucinated non-sequitur in a customer-facing call can cause immediate abandonment and lasting brand damage. The consequences of poor performance are measured in real time, users do not scroll back; they hang up.

Production data indicates that response latency beyond 1.5 seconds materially increases the risk of call abandonment and user frustration, so latency should be monitored as a primary experience metric. Word Error Rate above 10% in clean audio is a strong warning sign for downstream recognition quality and can contribute to misrouting and execution errors. Hallucinated non-sequitur responses, even at sub-1% rates, can undermine user trust and produce incorrect task actions in high-stakes domains such as appointment scheduling and financial transactions.

Scope and Structure

This guide is organised into five sections. Following this introduction, the Executive Summary distills the most critical benchmarks and strategic imperatives for leadership audiences. Pages 3 and 4 provide an exhaustive treatment of all key metrics organised by the 4-Layer Framework: Infrastructure, Execution, User Behavior, and Business Outcomes. The final page provides a curated reference list for further reading.

The 4-Layer Framework, developed and validated by Hamming AI, represents the most comprehensive publicly documented methodology for voice agent quality assessment. It moves beyond simplistic accuracy metrics to encompass the full lifecycle of a voice interaction: from raw audio signal quality and latency through intent classification accuracy and tool reliability, all the way to macro business outcomes such as call containment and first-call resolution rates.

Audience

This document is written for technical practitioners building and operating voice AI systems, product managers defining quality standards and KPI thresholds, and business leaders seeking to understand the quantitative relationship between voice agent performance and commercial outcomes. The benchmarks cited throughout are drawn from large-scale production environments and represent attainable targets for well-engineered systems, not theoretical ideals. 

Whether you are evaluating a first-generation deployment or optimising a mature system handling hundreds of thousands of calls per month, the frameworks and thresholds documented here provide the empirical foundation for data-driven quality governance.

Executive Summary

AI voice agents are transforming customer operations across every major B2C vertical. Their commercial value, measurable in reduced operational cost, higher booking conversion, and improved customer satisfaction, is directly contingent on the quality of their performance across four interdependent evaluation dimensions. This document presents the authoritative 4-Layer framework and the specific metric thresholds that distinguish world-class deployments from underperforming ones.

Source: Hamming AI, The 4-Layer Voice Agent Quality Framework

Strategic Imperatives

Tail Latency Governs User Experience. Median latency alone is an insufficient proxy for user experience. P99 events, those affecting the worst 1% of calls, create disproportionate brand damage and must be monitored and bound independently. Organizations should set automated alerts for any P95 latency exceeding 50% above baseline.

Hallucinations Are a Silent Revenue Risk. Hallucinated Unrelated Non-sequitur (HUN) rates must be kept below 1% in normal operating conditions. Even at this low rate, hallucinations in appointment scheduling or financial query contexts can trigger incorrect actions with direct commercial consequences. Downstream propagation of hallucinations must be reduced to zero, as multi-turn error compounding is significantly more damaging than isolated incidents.

Containment Without Resolution Is a Trap. Containment rates above 80% that are not matched by equivalent First Call Resolution rates indicate that users are being looped within automated systems without genuine resolution. This configuration produces the worst possible outcome: high automation cost alongside poor customer experience.

Prompt Changes Are the Leading Cause of Regression. Every prompt modification must trigger a full regression evaluation suite. Deployments should be automatically blocked if TSR drops more than 3%, latency increases more than 10%, WER rises more than 2%, or escalation rates increase by more than 5%.

Audio-Native Debugging Is Essential. Transcript-only analysis is systematically blind to the signals that most reliably predict user abandonment: prosody shifts, changes in speech rate, overlapping speech, and acoustic markers such as sighs. A comprehensive evaluation stack must integrate audio-native analysis alongside lexical methods.

Metrics in Depth: Infrastructure & Execution

Layer 1: Infrastructure

1.1 Latency Metrics

Latency in a voice agent is measured end-to-end: from the moment a user stops speaking to the moment agent audio begins playing. This measurement: Time to First Audio (TTFA), encompasses Voice Activity Detection, Speech-to-Text, NLU/intent classification, LLM reasoning, any tool invocations, and TTS synthesis. Each component contributes a compounding latency budget. The P50 target of 1.5 - 1.7 seconds represents the threshold at which conversational naturalness is preserved; the P99 ceiling of 8 seconds bounds the absolute worst-case user experience.

Source: Hamming AI, The 4-Layer Voice Agent Quality Framework

1.2 Word Error Rate (WER)

WER quantifies automatic speech recognition accuracy using the formula: WER = (Substitutions + Insertions + Deletions) / Total Reference Words × 100. It is the primary diagnostic for the STT component and directly predicts downstream NLU quality. Enterprise-grade production targets WER below 5% for clean audio. Performance degrades predictably with acoustic environment quality: office noise typically adds 3–5% WER and heavy outdoor noise adds 8–12%, though these figures reflect controlled test conditions and may vary across ASR providers and deployment environments.

Source: ai-coustics, Behind Quail Voice Focus 2.0: [5]; VoiceToNotes, State of AI Transcription Accuracy

Layer 2: Execution

2.1 Task Success Rate & Intent Accuracy

TSR measures the percentage of interactions in which the agent successfully completes the user’s intended task. It is the single most commercially relevant execution metric because it directly correlates with conversion, resolution, and customer satisfaction. Intent classification accuracy must exceed 95% to prevent the misrouting that drives up escalation rates and degrades TSR. Appointment scheduling deployments require TSR above 90% due to direct revenue impact; failure to meet this threshold has measurable consequences on booking conversion rates.

Source: Hamming AI, The 4-Layer Voice Agent Quality Framework

2.2 Hallucination: HUN Rate & Factual Accuracy

The Hallucinated Unrelated Non-sequitur (HUN) Rate measures how frequently a voice agent produces a response semantically disconnected from user input. In production voice environments, even a 1% HUN rate means that 1 in every 100 turns generates a potentially incorrect or task-breaking response. Downstream propagation, where an erroneous statement is built upon in subsequent turns, must be reduced to zero, as compounding errors are significantly more damaging than isolated hallucinations. The FActScore methodology provides one rigorous framework for measuring factual alignment at the atomic-claim level in generative outputs.

Nurix AI’s evaluation framework further structures hallucination detection across the reasoning and output layers, combining reference-based scoring, semantic alignment algorithms, and multi-stage turn verification to provide comprehensive coverage.

Source: Hamming AI, Voice Agent Evaluation Metrics [1b]; Nurix AI, How We Evaluate Voice AI Models [4]; Min et al., FActScore (EMNLP 2023)

Metrics in Depth: User Behavior & Business Outcomes

Layer 3: User Behavior

3.1 Barge-In Detection & Interruption Handling

Barge-in handling is one of the most technically demanding aspects of voice agent design. When a user speaks while the agent is outputting audio, the system must correctly identify the interruption, stop speaking within 200 milliseconds, and coherently recover context for the subsequent response. Failures, whether false positives that stop the agent on background noise, or false negatives that cause the agent to talk over users, are among the most cited sources of user frustration and call abandonment.

Source: Hamming AI, How to Evaluate Voice Agents

3.2 Sentiment Trajectory & Reprompt Rate

Sentiment trajectory tracks the emotional arc of a call using audio-native signals: tone, speech rate, prosody shifts, and acoustic markers such as sighs, that transcript analysis cannot capture. A downward sentiment trajectory is the strongest available leading indicator of imminent call abandonment or escalation request. Reprompt Rate measures how frequently the agent must re-ask for clarification, serving as a proxy for combined STT and NLU quality under real operating conditions. A Reprompt Rate above 10% is a reliable indicator that ASR or NLU performance is degraded in the production environment.

Source: Hamming AI Voice Agent Evaluation Metrics [1]; Braintrust, How to evaluate voice agents [2]; Maxim AI, Voice Simulation [3]

Layer 4: Business Outcomes

4.1 Containment Rate & Escalation Patterns

Containment Rate = (Agent-Handled Calls / Total Calls) × 100. Leading contact centers achieve containment above 80%, though the majority of mature deployments operate in the 60–75% range. The critical insight is that containment above 80% without a matching FCR rate indicates users are trapped in automated loops: a configuration that erodes satisfaction while appearing efficient on surface-level dashboards. Escalation Rate should remain below 30% for deployments targeting 70% containment, and below 20% where containment consistently exceeds 80%. It should be further analysed by pattern type to distinguish structural agent failure from legitimate complexity routing.

Source: Hamming AI, How to Evaluate Voice Agents

4.2 First Call Resolution & Regression Testing

FCR measures the proportion of calls resolved on first contact without repeat calls or escalations. In B2C environments, FCR above 75% is the baseline for acceptable automation performance, with leading systems achieving 85%+. FCR is the most direct link between voice agent quality and customer lifetime value.

Regression testing protects FCR and all other business outcome metrics during the continuous development cycle. Every prompt modification triggers a full evaluation suite; deployments are automatically blocked if any regression gate threshold is breached. Shadow mode testing: running new prompt versions alongside live traffic without affecting users, provides a production-equivalent validation environment for safe iteration.

Source: Hamming AI, Voice Agent Evaluation Metrics [1]; Braintrust, How to Evaluate Voice Agents [2]

Compliance Considerations

Voice agents deployed in regulated B2C environments must satisfy applicable data protection and security standards alongside performance quality requirements. In healthcare contexts, deployments are subject to HIPAA Security Rule safeguards governing PHI disclosure, role-based access controls, and audit logging, with proposed 2024 amendments pending final rule as of 2026. Voice agents that process payment information must comply with PCI DSS v4.0.1 encryption and access control requirements. Platforms handling sensitive customer data across industries should maintain SOC 2 Type II certification to provide ongoing assurance of security, availability, and processing integrity controls. These compliance requirements operate independently of; but in parallel with, the performance evaluation framework documented in this guide.

Why Choose telli

The evaluation framework outlined in this article reflects genuine engineering complexity: orchestrating latency budgets across ASR, NLU, LLM reasoning, and TTS; tracking word error rate across acoustic conditions; and building observability into hallucination, containment, and first-call resolution metrics. Most organizations do not have the bandwidth to build and maintain this infrastructure in-house.

telli abstracts this complexity so businesses can go live with production-grade voice agents in days rather than months. Instead of assembling a stitched stack of ASR, LLM, and TTS vendors and instrumenting each layer separately, teams deploy directly on telli’s platform and inherit its latency-optimised infrastructure and built-in monitoring out of the box.

Monitoring and improving performance doesn’t require a separate evaluation stack either. telli gives every customer real-time dashboards covering conversion rates, call duration, and outcome tracking, alongside custom call outcomes. Teams define their own structured fields, beyond the pre-built defaults, to capture exactly what matters for their use case, whether that’s booking confirmations, qualification status, or payment completion. Every call is analysed for sentiment and summarized automatically, and full recordings and transcripts remain available for deeper review.

Just as important, telli closes the loop between measurement and improvement. Teams can test and refine scripts directly in telli Prompt Editor using Charlie, our AI prompt assistant, compare agent versions side by side, and flag calls for quality review, letting the AI agent self-improve iteratively rather than requiring a full engineering cycle for every prompt change. Combined with warm transfer to a human agent whenever a call needs one, telli lets businesses focus on outcomes, not infrastructure.

Sources

1. hamming.ai - The 4-Layer Voice Agent Quality Framework
2. ai-coustics.com - ASR Performance and Word Error Rate (WER) Under Acoustic Variation
3. hamming.ai - Voice Agent Evaluation Metrics, Definitions, and Benchmarks
4. hamming.ai - Interruption Handling and Escalation Complete Framework
5. voicetonotes.ai - Production System Speech Recognition and Transcription Accuracy Benchmarks
6. arxiv.org - FActScore Factual Precision Evaluation Methodology
7. nurix.ai - Multi-Layer Voice AI Evaluation and Hallucination Detection
8. getmaxim.ai - Audio-Native Evaluation and Voice Simulation Methodologies
9. braintrust.dev - Voice Agent Evaluation Best Practices and Shadow Testing
10. hhs.gov - HIPAA Security Rule Technical Safeguard Standards
11. pcisecuritystandards.org - PCI DSS v4.0.1 Data Security Standard for Payment Processing
12. aicpa.org - SOC 2 Type II Trust Services Criteria for Security Assurance
13. telli.com - telli Studio Prompt Editor and Continuous Script Optimization