Voice AI — agents
that handle phone calls
A voice agent is an AI that answers the phone, speaks in natural language, understands the customer and performs real actions — books an appointment, updates the CRM, transfers to a rep. In this guide: the STT→LLM→TTS pipeline, the latency secret, the platforms (Vapi, LiveKit, Realtime API), and how to build a business voice agent that connects to your automation.
What a voice agent is and why it took off in 2026
A voice agent (Voice AI Agent) is a system that lets an AI hold a real-time, two-way voice conversation — to hear, understand, think and answer in a natural voice, while performing actions in other systems. Unlike the old IVR ("press 1 for service"), the customer simply talks normally and the agent understands.
Two things caused the jump: models fast enough for real-time responses, and TTS that sounds human (like ElevenLabs). The result — voice agents that already answer call centers, book appointments and make outbound calls, 24/7, at a cost of pennies per minute.
The pipeline — STT → LLM → TTS
Every voice agent, under the hood, runs the same chain of three stages that happen in a loop throughout every second of the call:
On top of these three is a telephony layer (Twilio, Telnyx) that connects to a real phone number, and an orchestration orchestration layer that times everything and handles interruptions. A newer approach — Speech-to-Speech Speech-to-Speech (like the OpenAI Realtime API) — skips the text stage and talks directly voice-to-voice, for exceptionally low response time.
Latency & Turn-taking — the secret to a natural conversation
The difference between a good voice agent and a bad one is response time. In a human conversation, silence of more than a second feels awkward. The professional target: under 800ms from the moment the customer stops talking to the moment the agent starts answering. To get there, every stage in the pipeline must run in streaming — not wait for completion.
- Endpointing — detecting when the customer really finished speaking (and did not just pause). A wrong detection makes the agent talk over the customer.
- Barge-in / interruption — when the customer starts talking while the agent is talking, the agent must go silent immediately and listen. This is essential for a natural conversation.
- Backchanneling — "uh-huh", "got it" — small sounds that signal the agent is listening.
- Filler words — the agent can say "one moment, checking..." while it fetches information, to fill the silence.
You can wire STT+LLM+TTS together manually, but the orchestration (endpointing, barge-in, telephony, latency) is the hard engineering. That is why most builders use a platform that handles it — and focus on the business logic.
The leading platforms in 2026
| Platform | Approach | Best for |
|---|---|---|
| Vapi | Open orchestration | Free choice of components |
| LiveKit Agents | Open source (WebRTC) | Full control, self-host |
| OpenAI Realtime API | Speech-to-Speech | Exceptionally low latency |
| ElevenLabs Agents | Managed, excellent voice | Fast setup |
- Vapi — an orchestration layer that connects best-of-breed components (choose your own STT, LLM and TTS) without locking you in. Very popular for fast building with flexibility.
- LiveKit Agents — open source based on WebRTC. The choice when you need full control, self-hosting and scale.
- OpenAI Realtime API — a direct Speech-to-Speech approach, skipping the text. The lowest latency, but less control over each stage.
- ElevenLabs Agents — a managed solution with the best voice on the market and fast setup in the interface.
Building a voice agent — the anatomy
Setting up a voice agent in Vapi (or similar) comes down to configuring five components:
- System Prompt — who the agent is, its goal, tone of voice, and what is allowed and forbidden. This is the heart of the personality and behavior.
- Voice — choosing the TTS and the voice, including language support and speaking speed.
- Tools / Functions — actions the agent can perform: checking calendar availability, creating a CRM lead, sending an SMS. This is whereMCP and webhooks come in.
- Phone number — connecting a number via Twilio/Telnyx for inbound and outbound calls.
- Handoff to a human — when and how the agent transfers to a human rep when it gets stuck.
The real power comes when the agent triggers n8n or Make.com via a webhook: "book an appointment" → a workflow that checks the calendar, creates an event, sends an SMS confirmation and updates the CRM — all during the call. The agent is the voice; the automation is the hands.
Business use cases
In many places you must disclose that it is an AI, and outbound calls have consent rules (opt-in) and time restrictions. Recording calls requires notice. Check the regulations in your country before running a voice agent with customers — especially for outbound calls.