Realtime

Realtime serves low-latency voice and text conversations over a WebSocket, or over a phone call through Twilio. A realtime model is a saved preset that bundles the pieces of a conversation: what generates the answers (a chat model or an agent), the speech-to-text and text-to-speech models, how the server decides the caller has finished speaking, and who speaks first.

Realtime is an enterprise module. Without an active enterprise licence the page is not available, and the client endpoints answer 402 Payment Required. See Licensing.

Operators open it as Realtime from the service launcher. The area has four pages:

Page	What it is for
Realtime	The list of realtime models, with summary tiles.
Playground	Talk to a model from the browser, let it ring you, or have it phone a number.
Sessions	Logs of every WebSocket and phone session, with turns, transcript and latency.
Telephony	Twilio connections used for inbound and outbound calls.

What you see on the Realtime page

The tiles at the top summarise the sessions of the active project: total sessions (with a count per transport), end-to-end latency, total tokens, and audio processed. When voice turns have been recorded the latency tile shows the voice end-to-end p50 (speech end to first audio byte) with the p95 underneath; otherwise it shows the average first-token latency.

Below the tiles is a table with one row per realtime model: name and key, the generator (an Agent badge appears when an agent is bound), the voice pipeline (STT and TTS badges, or Text only), status, and creation date. You can filter by name, key, description and status. Row actions are View details, Edit, Open in playground, Enable/Disable and Delete.

Disabling a model makes new connections with its key fail; sessions that are already running are not interrupted.

Create a realtime model

1. Open Realtime and choose Create realtime model.

2. Identity. Give the model a name and a key. Clients select the model by key, and the key cannot be changed after creation. Keys use lowercase letters, digits, dashes and underscores.

3. Response generator. Choose one of:

   • Chat model — pick a model whose category is llm, and optionally add instructions (the system prompt), a temperature and a maximum number of output tokens. A fast model keeps answer latency low.
   • Agent — pick an agent; its model, system prompt, tools and knowledge apply to every response. Choose which agent version to use: Published (the latest published version, falling back to the draft), Draft, or a pinned version number. You can also set a tool status message that is announced and spoken as soon as the agent starts calling tools, and turn on a soft waiting sound during tool calls.

4. Speech-to-text. Pick a model whose category is stt. Leave it empty for text-only sessions. You can set the spoken language (auto-detect is slower and can mis-detect short phrases, so pick a language where you know it), up to 1000 characters of vocabulary hints, and the transcription mode (Batch or Streaming; streaming needs a provider with streaming transcription).

5. Text-to-speech. Pick a model whose category is tts, optionally a voice name, and the output audio format: PCM 16-bit (streams frame by frame, recommended), or G.711 μ-law / A-law for telephony-style clients. Without a TTS model, answers are text only.

6. Turn detection. Choose the detector:

   • Semantic VAD — voice activity detection plus a turn-completion model. It waits while the sentence sounds unfinished and answers as soon as it is complete. You can set eagerness (auto, low, medium, high).
   • Server VAD — voice activity only; the turn ends after a fixed silence.

   Tuning fields cover the pause before the first end-of-turn check, the silence that ends the turn, the maximum turn silence, the speech threshold (raise it in noisy environments), and how long the caller must speak to interrupt the agent. A switch controls whether callers can interrupt the agent (barge-in) at all. An Advanced group exposes finer detector settings. Fields left empty use the engine defaults shown as placeholders.

7. Conversation start. Choose who speaks first (User or Agent), and the first message: None, Fixed text (spoken word for word) or Generated (the agent writes the opening from your instructions). You can protect a legal notice from barge-in by turning off Caller can interrupt the first message. When the user speaks first you can set a time after which the agent greets anyway.

8. Telephony. Optionally pick a Twilio connection, a caller ID, who speaks first on outbound calls, and whether inbound calls should be answered with this model.

9. Choose Create realtime model. A pre-flight summary on the form lists what is still missing, for example that no STT model is selected for voice input.

Variables

The instructions and the first message accept placeholders. Values come from the session (the session.update event, the playground, or the call request). Agents also receive them as runtime_context.metadata.variables. Variable names start with a letter and use letters, digits, _, . and -.

Try it in the playground

The Playground page of Realtime (or Open in playground on a row) connects the browser to a model. Pick the model, then a mode:

• I call — you speak and the model answers.
• Agent calls me — choose Ring me; accepting the call makes the agent speak first.
• Phone call — enter a number in international format and the model dials out through a Twilio connection.

You can override the turn detection for the test (semantic, server, or push-to-talk), set session variables, and apply them to a live session. Tabs show the live transcript, per-turn latency, and the raw events. The playground warns when the model has no STT model (voice input cannot work), no TTS model (answers appear in the transcript only), or when turn detection fell back to energy-based detection. Voice sessions need microphone access in the browser.

Review sessions

The Sessions page of Realtime lists every session in the active project with its start time, model, transport (WebSocket or Phone), caller and callee for phone sessions, status (pending, active, ended, error), number of turns, p50 end-to-end latency, tokens and duration. Filter by model, transport, status and date range.

Opening a session shows:

• Overview — session details, call details reported by the carrier for phone sessions (direction, call status, provider, answered by, duration), latency percentiles, and the variables the session used.
• Turns — each user turn and its answer with a latency breakdown across turn detection, STT, LLM, TTS and the total, how the turn ended, tool calls, and whether it was interrupted.
• Transcript — the conversation as recorded.
• Live — for a running session, the transcript and latency as they happen.

On a running phone session you can Hang up the call. You can also send a message into a running session; it arrives as a user turn marked as an external notification, and goes through the same guardrails as spoken input. You choose whether the assistant answers it, and whether it arrives after the current answer or right now.

Connect Twilio for phone calls

The Telephony page of Realtime holds the Twilio connections.

1. Choose New connection and enter a name, an optional key (it becomes part of the webhook URL, so it cannot change later), the Twilio Account SID and Auth Token, a default caller ID in E.164 format, and an optional inbound realtime model. The Auth Token is encrypted at rest. When you edit a connection, leave the token empty to keep the stored one.
2. Copy the voice webhook (and optionally the call status webhook) shown on the page into your Twilio number, under Voice configuration, as HTTP POST.
3. Use Test connection to check the credentials, see the phone numbers on the account, and see whether the webhook is set. The page also lists warnings, for example that no number sends its calls to this deployment yet, or that the default caller ID is not on the account.

Things to know:

• Twilio must reach the deployment over https. Set REALTIME_PUBLIC_URL to a public https address if Console runs locally behind a tunnel. The page shows the URL Twilio calls back and warns when it is not reachable.
• Incoming webhook requests are verified with the X-Twilio-Signature header, so the Auth Token stored here must be the account's current primary token.
• Realtime models used for calls need both an STT and a TTS model.
• If no inbound realtime model is set on the connection, incoming calls are declined.
• Twilio trial accounts have restrictions that can stop the agent from being connected to a call; the page explains them when it detects a trial account.

Connect from your own application

Clients open a WebSocket to /api/client/v1/realtime and select a model with ?model=<key> (or an agent with ?agent=<key>), authenticating with an API token. Header authentication is preferred. A token passed in the query string (?api_key=) is accepted only when it is scoped to the realtime service.

The same API token surface can place and hang up phone calls and manage realtime models. See the Realtime SDK reference for the connection helper and event types.

Permissions and limits

Topic	Behaviour
Plan	Enterprise licence required. Without it, the client API answers 402.
Permission	The realtime service (RBAC and API token service permissions). A response generator that is a chat model or an agent also requires the matching models or agents access for the caller.
Scope	Models, sessions and connections are project-scoped; a session outside the active project is not found.
REALTIME_MAX_SESSIONS	Optional cap on live sessions per node. At the cap, new connections receive a server_busy error and the socket is closed with code 1013. Unset means unlimited.
REALTIME_MAX_FRAME_BYTES	Largest WebSocket message accepted. Default 1 MiB.
REALTIME_MAX_PENDING_INPUT_BYTES	Buffered audio waiting for turn detection. Default 8 MiB.
REALTIME_MAX_SOCKET_BUFFER_BYTES	Unsent backlog on a client socket before frames are skipped. Default 8 MiB.
REALTIME_PUBLIC_URL	Public https URL that Twilio calls back.

Related

• Realtime SDK reference
• Audio API reference for speech and transcription endpoints
• Agent models
• Model Hub