How AI Voice Agents Work: Architecture & Call Flow Guide

The majority of explainers about AI voice agents begin and end at “it's AI that talks on the phone.” Which, of course, is accurate. But the most interesting stuff is happening in the gap between someone opening their mouth and hearing a reply. Quite a lot is happening, actually, and all of it in under 2 seconds.

For a moment, let's imagine a customer calls their dentist to postpone a Thursday appointment to the following week. On the customer's side, it's a simplethirty-second interaction. But on the system's side, the picture is far more complex: the call is parsed, understood, verified against the scheduler, confirmed, and then spoken back to the caller before they have a chance to hang up or ask to be transferred. Throughout this article, we will use this example as we unpack each piece of the process.

As an aside, this is precisely the type of problem that telli's voice agent platform is designed to solve, Businesses commonly use AI voice agents for tasks such as lead qualification and appointment booking, where conversations need to happen naturally while integrating with business systems. So we'll use this example to show how a real system like telli would deal with each step of the process, rather than discuss it in purely theoretical terms.

The AI Voice Agent Pipeline

When we boil them down, all AI voice agents run more or less the same series of steps:

Caller speaks → Speech Recognition → Language Understanding → LLM Reasoning → Knowledge Retrieval → Tool Calls → Speech Generation → Caller hears a response

Each one is absolutely essential to the pipeline functioning, and each is critically interdependent on the others. Having a fantastic language model, for example, won't help all that much if your transcription engine is messy and doesn't correctly hear what the customer is saying. Great audio quality will make little difference if your speech engine is slow and makes the customer wait a second or two. Every link in this chain is just as important as any other.

Here's a general description of what each link in the pipeline is responsible for:

Let's follow the rescheduling example through the stages of the pipeline.

Step 1: Converting Speech to Text

The moment a caller utters “I need to move my Thursday appointment”, a Speech-to-Text (STT), or Automatic Speech Recognition (ASR) engine is already active behind the scenes. Providers like Deepgram, Speechmatics, Whisper, and Google's Speech models implement it in a myriad of ways, but all aim for the same result: to produce a punctuated transcript of the spoken word as close to real time as possible.

Every other downstream component inherits errors originating in the transcription; a mistake turning “move my Thursday appointment” to “remove my Thursday appointment” changes the customer's meaning entirely and can cause undue friction.

Accents, background noise, poor phone connections, and even simply muttering make this stage complex but critical. So, most real-time systems utilize noise reduction and multilingual voice recognition rather than relying on a single pass.

The factors that actually affect transcription accuracy, according to research on ASR performance are:

This is not a minor discrepancy. A 2020 study widely cited by researchers showed a significant difference in the error rate between ASR models listening to African-American speakers compared to white American speakers, mostly due to data limitations rather than technology issues. It's a good reminder that transcription quality varies across speakers, and that accuracy numbers based on only a small range of accents may not actually reflect real-world performance.

Step 2: Understanding What the Caller Actually Means

Getting the words right isn't the same as getting the meaning right. This is where Natural Language Understanding (NLU) and intent detection come into play. For instance, “Can you send me the invoice?” and “Can you resend me the invoice? sound nearly identical, grammatically speaking, but necessitate different outcomes.

For the purpose of this exercise, our caller says, “I need to move my Thursday appointment.” The system must determine that intent (reschedule, not cancel), identify the correct entity (this specific Thursday appointment), remember details mentioned earlier in the call (who the caller is, whether they’ve been authenticated), and discern their sentiment (whether they are irritated, excited, etc.).

Step 3: How the LLM Decides What to Say

People usually misunderstand this step the most. It's not the case that an LLM driving a voice agent is simply freestyling answers like it would in a chatbot demo. Instead, it's operating within defined boundaries, including a system prompt defining business policies, historical chat context, information retrieved from the business's knowledge base, and pre-authorised tool actions.

When the caller wants to reschedule Thursday’s appointment, the LLM is not inventing a plausible statement about rescheduling. Rather, it's considering such questions as: “Do we allow same-week reschedules?” “Is the calendar tool accessible?” “What additional information (a new date, for example) is required to act?”

Only once there is sufficient information does it respond by either requesting that the caller provider additional information or triggering an appropriate function. This is a fundamental difference between generating text and performing reasoning tasks with tools. It's also the difference between saying something that sounds plausible and actually doing what needs to be done.

Depending on the underlying LLM (GPT, Claude, Gemini, Llama, etc.), hallucination risk increases substantially when models can respond freely using their training data rather than using verified, retrieved information. Good systems actively try to avoid such situations.

Step 4: Retrieving Company-Specific Information

Just because an LLM has the ability to generate intelligent text doesn’t mean it “knows” your clinic's pricing, or the ins and outs of your cancellation policies, or when your store hours fall. LLM-driven agents must retrieve this information on demand.

That is the job of Retrieval-Augmented Generation (RAG), an approach outlined in a 2020 paper that enhances an LLM's existing (or "parametric") knowledge with external information retrieved when the LLM needs it. Instead of using what the LLM learned during its training (which is general and may be nonexistent for company-specific details), the system pulls relevant documents, like FAQs, internal documents, CRMs, or policies, from the company's knowledge base, transforms that data, and injects it into the language generation stage.

The difference comes down to where the knowledge lives. A trained model has it baked in, so updating it means retraining or fine-tuning, and its answers about your specific business are often generic or just wrong. Retrieval pulls the same information live from your own documents and databases, updates the moment you update your knowledge base, and grounds every answer in something real rather than a hallucinated guess.

In our example, the agent must confirm whether or not our caller will incur a fee for a same-week reschedule, as this is not a detail built into the generic LLM.

It needs to be retrieved from the clinic's own policy documentation. This is one area where practice diverges from theory.

telli's agents, for instance, can access an uploaded corpus of documents (FAQs, price lists, policies) and choose themselves during the call whether to use semantic search, keyword search, or both. If the policy is found in a PDF, for example, it’s read and used directly by the agent rather than guessed. If the policy isn't located, the agent reverts to the prompt rather than fabricating an answer.

Step 5: Calling External Systems

What’s the point of a talking AI if it can’t do anything else? Businesses want AI agents that can take action: consult with their CRMs, create customer profiles, verify information, or transfer calls. All these tasks rely on integrations, APIs, and webhooks.

In our scenario, once our caller is authenticated, the agent would query their calendar for open Thursday slots, reschedule the appointment, update the customer profile, and make sure everything’s logged in the CRM, all without human involvement.

telli implements this by enabling agents to trigger "tool calls" mid-conversation. These can include functions that end calls or initiate a custom HTTP endpoint to check calendar availability. The caller has no idea if tool calls are involved; they just hear, "Let me check that for you."

Step 6: Converting Text to Speech

Finally, when the LLM determines what it needs to say, Text-to-Speech (TTS) converts that response back into the audio the caller hears. Modern systems offer sophisticated neural voices as opposed to decade-old, syllable-spliced robotic voices, impacting the way the agents pace speech, enunciate, and convey tone.

The factors that distinguish human-like speech from artificial speech are:

• Prosody: natural rise and fall in intonation, not flat delivery
• Pronunciation: handling names, numbers, and abbreviations correctly
• Pacing: neither rushed nor unnaturally slow
• Latency: a sub-second response that keeps the exchange feeling live
• Streaming: audio starting before the full response has even finished generating

Latency is a bigger deal here than in almost any other place in the pipeline. A voice that's perfect, but two seconds late, feels broken. Humans expect an answer rhythm similar to how another person would answer. Across languages, people wait for an average of 200 ms between the end of one sentence and the beginning of another.

Voice AI, which stitches together several providers for speech recognition, language model, and speech synthesis, is usually between 600 ms and 1,700 ms end-to-end, and generally anything over 800 ms in industry benchmarks is already noticeably late.

telli’s own experience here is a good example of how important this step is to get right. telli builds conversational AI agents for B2C sales calls, where a stilted, inhuman voice not only sounds bad but actively hurts sales conversion, as callers hang up the moment it crosses the threshold of what feels "human". That’s why telli prioritizes a low-latency, voice-cloning-ready TTS provider, rather than leaving voice quality as an afterthought behind the LLM.

How AI Voice Agents Handle Interruptions

Real conversations aren't turn-based in the way text chat is. We interrupt each other, pause to think, or trail off. This feature of real conversations is known as barge-in and is amongst the hardest problems to solve in the AI voice industry.

If our caller suddenly says "actually, wait, let's cancel instead" halfway through the reschedule confirmation, the system needs to detect that interruption immediately, stop talking, and re-process the new input rather than finishing its sentence and only then noticing the caller changed their mind. To detect such shifts in intent and interruption correctly, the system requires continuous real-time voice activity detection rather than only detection at the beginning and end of a turn.

How Human Handoffs Work

The best AI conversations don't all end with the agent, either. If the AI isn't capable of handling the request, or the caller requests a person explicitly, the call must be warm-transferred, where the caller is directed to the right live agent, accompanied by a summary of what has already been discussed, to save the customer from repeating information.

It's a stage at which many good voice AI systems fail. When a hand-off occurs but the transcript or context is lost between the AI and the human agent, it's no better than starting a call with a completely cold caller. The intention, CRM data, and already-confirmed details must all follow along with the hand-off to the live agent.

telli's warm transfer approach puts the caller on hold, briefs the human agent on the conversation with an automated summary, and then connects the caller and the agent. If the transfer times out or lands on voicemail, the call reverts back to the AI agent.

Common Technical Challenges

Voice AI has real failure modes. Here's what they look like in practice, rather than the usual generic pros-and-cons list:

• Latency comes from sequential processing across STT → LLM → TTS. Fixed with streaming architecture and parallelised inference.
• Hallucinations happen when the model answers from general knowledge instead of retrieved facts. Fixed with strict grounding and retrieval-first prompting.
• Background noise degrades input quality from the caller's environment. Fixed with noise suppression and confidence thresholds.
• Poor documentation means a thin or outdated knowledge base. Fixed with regular knowledge base audits.
• API failures come from third-party integration timeouts or errors. Fixed with retry logic and graceful fallback messaging.
• Unexpected questions happen when the caller goes off-script. Fixed with fallback responses and defined escalation paths.

What Makes Enterprise AI Voice Agents Different

The requirements of a small business testing its first AI receptionist are somewhat different from an enterprise company scaling its AI-powered phone interactions. In short, here's what generally separates enterprise deployments from smaller operations:

• Compliance: call recording, data residency, specific industry regulations (financial services, healthcare, etc.)
• Observability: transparency into how each call decision was made, beyond just the transcript.
• Analytics: tracking conversions, analysing drop-offs, monitoring trends across calls.
• Prompt versioning: structured rollouts with the ability to roll back changes.
• Testing: simulated call testing before any new deployment reaches production.
• Monitoring: real-time alerting if latency or error rates spike.
• Knowledge management: a framework for ensuring the retrieval layer stays up-to-date with changing policies

These are the differences between a demo that works and a system that holds up across ten thousand calls a month.

For example, telli’s observability feature uses the concept of ‘auto-structured fields.’ During a call, the system breaks down each interaction into key fields defined for each agent, such as boolean value, category, number, or a free text field. These are pulled directly from the transcript, strictly adhering to the schema. If no value can be identified for a given field in the transcript, it is left null. This makes what otherwise is thousands of messy unstructured calls searchable and actionable by transforming them into structured, comparable data. That’s truly the heart of what it means to build something “enterprise-ready”.

Common Misconceptions

A few things worth correcting directly:

• "LLMs memorize your company's data." They don't, they retrieve it using RAG from company sources like FAQs, PDFs, knowledge bases, etc.
• "Voice AI is just speech recognition with a text bot behind it." Speech recognition is only one of many moving parts that AI voice agents must manage — and typically not the most difficult one.
• "A more powerful model automatically means a better voice agent." While models continue to improve, a better model will not improve the overall performance of the AI voice agent if it lacks reliable data retrieval and tool access to execute actions correctly.

Putting It All Together

If we remove all of the layers between each step of the voice AI process, the only job of the whole system is to take a spoken query and generate an appropriate result in a normal conversational time frame. It starts with speech recognition, translating words from audio to text. Language understanding then interprets those words. A large language model decides what to do based on those words and intent. Retrieval supplies the facts specific to that business. Tool calls then execute the actions, and speech synthesis finally converts the text response back to speech. Any one of these components can cause the entire pipeline to break down, no matter how well the others are functioning.

Sources used

1. researchgate.net - Speech recognition challenges in noisy environments
2. pnas.org - Racial disparities in automated speech recognition
3. arxiv.org - Retrieval-Augmented Generation (RAG)
4. docs.telli.com - Knowledge Base overview
5. docs.telli.com - Tool Calls overview
6. researchgate.net - Turn-taking universals in conversation
7. telnyx.com - Voice AI latency benchmark
8. cartesia.ai - telli's low-latency voice integration
9. telli.com - AI voice agent barge-in / interruption handling
10. docs.telli.com - Call Transfer
11. docs.telli.com - Call Analysis