How to build realtime voice AI that sounds like a real conversation

Key takeaways

• Realtime voice gives users a different way to interact with an AI product: instead of typing or navigating menus, they speak naturally and receive answers with live pacing, intonation, and emotional context.
• A strong realtime voice experience is not just a fast model response. It is the management of a live interaction where the conversation unfolds in real time and does not always fit the clean sequence of "user spoke, system answered."
• The model is only one part of the system. Production requires voice-native infrastructure, separation between the conversational layer and deep reasoning, and event-driven session-state control.
• Guardrails and evals remain the hardest areas. Safety checks must keep up with live audio, and qualities like tone, timing, and conversational flow are hard to evaluate with standard text tests.

Realtime voice as a product surface

Most voice-enabled AI applications are still built as a chain: speech becomes text, text goes to a model, the model thinks, and then a speech synthesizer reads the answer aloud. It works, but it feels like a pipeline, not a conversation.

Realtime voice changes the mechanics. The user speaks naturally, and the system responds while taking pace, tone, and emotional context into account. The experience becomes faster and closer to talking with a person than to operating an interface.

These agents are especially useful where voice works better than text: customer support, onboarding, coaching, accessibility scenarios, and replacing long IVR menus or departmental transfers. If the conversational form itself reduces friction, realtime voice should be treated as a separate product surface rather than as a "spoken chat."

Chained voice vs realtime voice

In the classic chained approach, separate models handle speech-to-text, language processing, and text-to-speech. Audio exists only at the input and output. Between them are discrete steps that add latency and make the interaction feel less natural.

Realtime voice works differently: one model natively listens, understands, and speaks while handling audio and transcript simultaneously. Input and output are continuous. That makes timing, intonation, and interruption handling part of the product rather than technical details.

System note

a comparison diagram of two architectures. On the left, a chained pipeline with separate speech-to-text, LLM, and text-to-speech blocks, with latency appearing between them. On the right, a realtime voice model that simultaneously receives audio, understands context, and generates spoken output in a live stream.

The realtime experience is attractive precisely because it feels immediate. But that also makes the system more complex: in a conversation, nothing politely "waits for its turn." The model operates inside a live session, and everything around it - state, safety, orchestration, and control - must update at the same speed.

In a chained system there is a clear order: the user speaks, the system replies, then the next step begins. Realtime voice has no such structure. Participants may speak at the same time, silence may appear, the user may interrupt the agent mid-answer, or ask a follow-up before the sentence is finished.

That is why realtime voice is not only a response-generation problem. It is a live-interaction coordination problem.

What production needs

To make realtime voice work at scale, the architecture has to be designed specifically for live interaction. The article highlights three required layers.

Voice-native infrastructure

A realtime voice session must handle audio streaming, turn ordering, interruptions, connection lifecycle, and agent actions. If the product runs in telephony, it also needs call support.

The first foundational layer is a voice-native session layer. RTC frameworks provide the place where you manage participants, stream audio, and run agents in telephony or WebRTC environments. The original article specifically mentions LiveKit as a practical choice because it provides a low-latency WebRTC stack, noise cancellation, and jitter reduction out of the box. Building this layer yourself is usually not worth it because of the complexity.

Separate "speaking" and "thinking"

For realtime voice, a multi-agent architecture with separated responsibilities is often useful.

Realtime voice models are strong at conversational audio streaming, but they are not always ideal for deep reasoning. Tool calling, retrieval, planning, and structured decision-making are often better handled by a separate model.

A practical pattern is responder-thinker architecture.

Responder is the realtime voice agent. It maintains the live interaction: it listens, speaks, handles interruptions, and preserves the flow of the conversation. Its priorities are speed, clarity, and emotional continuity.

System note

a responder-thinker diagram. The realtime responder is inside a live voice conversation with the user, while a separate thinker works outside the main audio path: it calls tools, performs retrieval, plans, and returns results to the responder for natural insertion into the conversation.

Thinker is a separate agent running on a reasoning-capable model. It works out of band and handles tools, search, planning, and complex decisions. The responder can call it when necessary and then bring the result back into the conversation.

Sometimes the thinker performs reasoning itself. Sometimes it orchestrates a set of specialized agents. The important point is not to force the realtime voice layer to do all the heavy work. That keeps the responder fast and conversational, while the thinker handles tasks that need time, structure, and broader context.

The authors note that future frontier models may reduce the need for this separation. But for now, in their experience, this pattern works more reliably than a single-agent design.

Event-driven control

Realtime voice systems continuously produce events: the user started speaking, paused, interrupted, the transcript updated, the response began streaming, an external tool returned, the session state changed.

All of these events need to be captured, streamed, and stored as the basis of the current conversation state. Without that, the system loses the ability to intervene accurately at the right moment.

An event-driven approach solves this problem. The system catches events as they happen, updates the session state, and triggers the required actions.

Lightweight handlers should stay on the realtime path so responsiveness is not harmed. Heavier tasks - updating the state machine, logging metrics, removing sensitive data, writing to databases, and ending the session - are better moved to asynchronous background flows.

As the product grows, the number of background tasks quickly increases. Even a small feature change can add new event flows and dependencies. That is why concurrency and event structure need to be designed in advance, otherwise the system becomes hard to understand and hard to evolve reliably.

The event-driven layer is not only about engineering reliability. It also helps manage the conversation itself: pacing, silence, interruptions, and session-ending timing. For example, if the user is approaching a time limit, the system can gently prompt the responder to help wrap up the conversation. If the interaction gets stuck, it can provide a clarifying instruction.

A strong realtime system always knows who is speaking, how the dialogue is evolving, and which conditions have already been met. This state is updated continuously through an event stream and lets the system give the agent the right prompt at the right time.

Guardrails must work at realtime speed

Guardrails are mandatory in any user-facing AI product: they handle safety, compliance, misuse prevention, and reliability. In turn-based systems there are comfortable checkpoints: after the user turn or before the response is delivered.

Realtime voice removes those comfortable checkpoints. User input arrives continuously. Audio output may already be streaming. The final transcript often appears later than the sound itself. If you wait for complete messages, the realtime feeling breaks.

That means safety checks have to run in parallel with the conversation. One approach is to stream audio into a buffer and asynchronously inspect transcript fragments as they appear. This allows guardrails to operate almost in real time without blocking the interaction.

System note

a realtime guardrails diagram. The audio stream passes through buffering and transcription, transcript fragments are checked in parallel by a safety layer, and when a guardrail triggers the system redirects the conversation, changes the agent behavior, or ends the session without a noticeable delay for the user.

If a guardrail triggers, the system has to respond in context: redirect the conversation, adjust behavior, or end the session. The goal is to preserve safety without noticeably degrading the user experience.

Realtime evals are harder than standard evals

The hardest part of evaluating realtime voice systems is that the key qualities cannot be seen from the transcript alone. Timing, interruptions, flow, and tone exist in time.

Standard eval pipelines work well for text: provide a realistic scenario, get a response, evaluate the result. In realtime systems, the input is live audio, and the important properties show up in the dynamics: how the agent handles overlapping speech, how quickly it responds, and how it recovers after interruption.

Manual voice testing captures these qualities well, but it does not scale. Transcript-based testing scales well, but it misses the main signal that separates a good realtime experience from a bad one.

A practical answer is to combine several layers:

• Agent-to-agent evaluations: a second realtime agent plays a defined user persona and talks to the system under test, while a third LLM-as-a-judge evaluates the interaction.
• Non-functional metrics: time-to-first-audio and transcript-level sentiment analysis provide quantitative proxy metrics for conversation quality.
• Manual qualitative review: manual review is still necessary to catch tone, naturalness, and subtle issues that automated metrics miss.

No single method covers everything. For production, you need all three layers, and even then the tooling for evaluating realtime audio is still noticeably less mature than tooling for text AI.

Conclusion

Realtime voice changes the shape of the product. The user perceives not only the words, but also pauses, interruptions, recovery speed, tone, and the overall flow.

That is why the model is only part of the system. Production-ready realtime voice requires a voice-native session layer, separation between conversational and reasoning layers, event-driven control around the live session, and guardrails that do not break the realtime feeling.

The weakest part of the stack is still quality evaluation. There is still no single reliable way to test the qualities that make realtime voice feel "alive": timing, tone, interruption handling, and conversational naturalness. Teams therefore have to combine automated tests, agent-to-agent runs, and manual review.