AI & Agents · Advanced
Real-Time Voice Agent With MCP Tools
Build a sub-second speech-to-speech agent that calls live tools through a Model Context Protocol server you write, with barge-in, streaming audio, and turn-taking handled cleanly.
You build a full-duplex voice agent that listens through the browser, sends audio over WebRTC to the OpenAI Realtime API, and invokes tools hosted on a FastMCP server you write in Python. The agent can look up weather, query a database, or call any API you wire up, and it speaks the result back in under a second. This project is worth building because it closes the gap between a toy voice demo and a production-grade assistant that actually does things, teaching you the plumbing that every real voice product needs.
What you build
- Captures microphone audio in the browser and opens a low-latency WebRTC connection to the OpenAI Realtime API
- Handles barge-in: the user can interrupt the agent mid-sentence and it stops speaking immediately
- Routes tool calls from the model to a FastMCP server over SSE, so the agent can query live data at runtime
- Streams synthesized speech back to the browser with sub-200ms time-to-first-audio on typical connections
- Implements clean turn-taking with voice activity detection events so the agent never talks over the user
- Exposes a simple tool-registration decorator on the MCP server so new capabilities can be added in a few lines
What it teaches
- How the OpenAI Realtime API session lifecycle works, including ephemeral token issuance, WebRTC signaling, and audio track management
- How to implement barge-in and voice activity detection using server-sent events from the Realtime API
- How to write an MCP server with the FastMCP Python SDK and expose tools over the SSE transport
- How to bridge model-generated tool calls from the Realtime API to an MCP server and return structured results mid-conversation
- How to keep API keys out of the browser by proxying session creation through a server-side route handler
- How turn-taking and audio buffer management work in a full-duplex streaming audio system
How it works
- 1
Browser mic
- getUserMedia
- WebRTC track
↓ audio stream
- 2
Next.js server
- ephemeral token
- signaling proxy
↓ WebRTC offer
- 3
OpenAI Realtime API
- VAD
- barge-in
- tool_call event
↓ tool call
- 4
FastMCP server
- tool handler
- live API call
↓ tool result
- 5
Realtime API (response)
- TTS synthesis
- audio delta events
↓ audio chunks
- 6
Browser speaker
- MediaStream playback
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add a second MCP server tool that reads from a Postgres database so the agent can answer questions about real application data.
- Implement session memory by storing conversation turns in a vector store and injecting relevant history as system context at the start of each turn.
- Replace the browser WebRTC client with a SIP bridge so the agent can handle inbound phone calls.
