OpenAI Releases Realtime API: Speech-to-Speech Voice Agents in One Call

The News: OpenAI launched the Realtime API on October 1st, 2024 -a WebSocket-based API that handles speech-to-speech conversations without chaining Whisper (speech-to-text) → GPT-4 (reasoning) → TTS (text-to-speech). Single endpoint, bidirectional audio streaming, sub-second latency (official announcement).

What Changes: Building voice agents goes from managing 3 separate APIs with complex state synchronization to opening a WebSocket, streaming audio in, getting audio back. Massive simplification.

Who This Matters For: Anyone building phone agents, voice assistants, AI receptionists, or conversational interfaces. The old pipeline typically had a second or more of latency. Realtime API aims for well under a second, which feels much closer to talking to a human.

The Old Way (Pain)

Before Realtime API, voice agent pipeline looked like this:

User speaks →
  [Microphone] →
    [Buffer audio until pause detected] →
      [Send to Whisper API] (200ms) →
        [Whisper returns text] →
          [Send text to GPT-4] (1,200ms) →
            [GPT-4 returns text response] →
              [Send to TTS API] (300ms) →
                [TTS returns audio] →
                  [Speaker plays audio]

Total latency: often well over a second

Problems:

1. Latency stacks: Each step adds delay. A pause of more than a second feels unnatural in conversation.
2. State management nightmare: Track which audio chunk maps to which text, which response.
3. No interruptions: User can't interrupt mid-response (agent has to finish speaking).
4. Complex error handling: If any step fails, you need to restart or handle partial state.

Managing state between Whisper, GPT and TTS can eat a large share of development time. The happy path is easy; it's handling interruptions, timeouts and partial failures that hurts.

The New Way (Realtime API)

User speaks →
  [Microphone] →
    [WebSocket stream to Realtime API] →
      [API processes audio, responds with audio stream] →
        [Speaker plays audio]

Total latency: sub-second

Key improvements:

1. Single API call: Open WebSocket, stream audio bidirectionally.
2. Native interruption handling: User can cut in mid-response, API handles it.
3. Lower latency: No intermediate text conversion -audio to audio directly (internally uses GPT-4, but optimized pathway).
4. Function calling works: Agent can call tools mid-conversation, return to natural speech.

Technical Details

WebSocket-Based Streaming

const ws = new WebSocket(
  'wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01',
  {
    headers: {
      'Authorization': `Bearer ${OPENAI_API_KEY}`,
      'OpenAI-Beta': 'realtime=v1'
    }
  }
);

// Stream audio to API
ws.on('open', () => {
  ws.send(JSON.stringify({
    type: 'response.create',
    response: {
      modalities: ['audio', 'text'],  // Can request text too for display
      instructions: 'You are a helpful assistant.',
    }
  }));

  // Stream microphone audio
  microphoneStream.on('data', (audioChunk) => {
    ws.send(JSON.stringify({
      type: 'input_audio_buffer.append',
      audio: audioChunk.toString('base64')  // PCM16, 24kHz, mono
    }));
  });
});

// Receive audio responses
ws.on('message', (data) => {
  const event = JSON.parse(data);

  if (event.type === 'response.audio.delta') {
    // Stream audio to speaker
    speakerStream.write(Buffer.from(event.delta, 'base64'));
  }
});

Supported Features

1. Modalities: Request audio only, text only, or both simultaneously (useful for transcription display while speaking).

2. Interruption handling: Built-in. User speaks → API detects speech → stops current response → processes new input.

3. Function calling: Agent can invoke tools mid-conversation.

// Define function
{
  type: 'response.create',
  response: {
    tools: [
      {
        type: 'function',
        name: 'get_weather',
        description: 'Get current weather',
        parameters: {
          type: 'object',
          properties: {
            location: { type: 'string' }
          }
        }
      }
    ]
  }
}

// API calls function, waits for result, continues conversation

4. Voice selection: 6 voices available (alloy, echo, fable, onyx, nova, shimmer).

Pricing

Realtime API pricing (as of October 2024):

Example cost (10-minute customer support call):

• Input: 10 min × $0.06 = $0.60
• Output: 8 min × $0.24 = $1.92
• Total: $2.52 per call

vs Old pipeline:

• Whisper: 10 min × $0.006 = $0.06
• GPT-4 Turbo: ~4K tokens in, 2K tokens out = $0.10
• TTS: 2K chars × $0.000015 = $0.03
• Total: about $0.19 per call

In this example, Realtime API is more than 10x more expensive. Trade-off: Pay premium for better UX (lower latency, natural interruptions).

When worth it: High-value interactions (sales calls, premium support). Not worth it for high-volume, price-sensitive use cases (call center tier-1 support).

Latency Comparison

What to expect when you build the same voice agent both ways:

Why Realtime is faster:

1. No intermediate text conversion (audio→audio path optimized internally)
2. Streaming starts immediately (no waiting for full response)
3. Single API call (no network hops between Whisper/GPT/TTS)

What This Means for Builders

1. Voice agent UX just became competitive with humans

Sub-second latency approaches the speed of human turn-taking, where gaps between speakers are typically a few hundred milliseconds. The old second-plus delays felt robotic.

Impact: Voice agents go from "tolerable" to "actually pleasant to use".

2. Complexity drops drastically

No more state machines tracking audio chunks, text chunks, TTS mappings. WebSocket connection + stream handling = done.

Impact: Smaller teams can build production voice agents (previously required complex orchestration expertise).

3. Function calling in voice becomes practical

Old way: User says "What's the weather?" → Noticeable wait → Agent says "It's sunny, 72 degrees" (agent called weather API during the wait, felt slow).

New way: User says "What's the weather?" → Short pause → Agent says "It's sunny, 72 degrees" (still calls API, but latency is low enough that it feels close to instant).

Impact: Voice agents can take actions (book meetings, query databases, trigger workflows) without breaking conversation flow.

4. Cost is a blocker for high-volume use cases

Using the example above, at about $2.50/call (10 min average), running 1,000 calls/day = $2,500/day, or roughly $75K/month.

Old pipeline: 1,000 calls × $0.19 = $190/day, or roughly $5,700/month.

Impact: Realtime API is premium tier. High-volume/low-margin use cases (call centers, automated phone menus) stick with old pipeline for cost reasons. High-value use cases (sales, premium support, concierge services) migrate to Realtime for UX.

Migration Path

If you have existing voice agent on Whisper→GPT→TTS:

Option A: Full migration (if cost acceptable)

• Replace pipeline with single Realtime API WebSocket
• Simplify codebase, improve latency
• Accept a much higher per-call cost

Option B: Hybrid (optimize for cost+UX)

• Use Realtime API for premium users/high-value calls
• Keep old pipeline for standard tier
• Route based on user segment or call type

Option C: Wait (if cost-sensitive)

• Stick with existing pipeline
• Monitor Realtime API pricing (likely to decrease over time as OpenAI optimizes)
• Migrate when cost drops or revenue justifies premium

Our recommendation: Build new voice projects on Realtime API (simpler dev experience, better UX). Migrate existing projects only if latency is causing measurable UX issues and cost is acceptable.

Limitations & Gotchas

1. Preview model only

gpt-4o-realtime-preview-2024-10-01 is currently the only model. No GPT-4 Turbo or GPT-3.5 options (can't reduce cost by using cheaper model).

Expected: OpenAI will add model tiers (realtime-mini for cost-sensitive use cases).

2. No streaming text fallback

If WebSocket drops, can't gracefully degrade to text-only mode. Need to handle reconnection.

3. Audio format constraints

Input: PCM16, 24kHz, mono, little-endian.

Output: PCM16, 24kHz, mono.

Most browsers/devices support this, but some edge cases (old phones) might need transcoding.

4. Rate limits

Currently: 100 concurrent WebSocket connections per API key.

If building multi-tenant voice app, need connection pooling or multiple API keys.

Competitive Landscape

Anthropic: No realtime voice API yet (Claude supports text-only).

Google: Gemini multimodal supports audio but not streaming realtime (upload audio file, get response).

ElevenLabs: Has realtime conversational AI (ElevenLabs Conversational AI) with low latency and competitive pricing, and lets you plug in the LLM of your choice.

Deepgram: Realtime speech-to-text + streaming (pairs well with LLM) but still requires chaining steps.

OpenAI is first to market with true end-to-end realtime speech-to-speech at GPT-4 intelligence level.

Expect: Anthropic and Google to launch competing offerings. Pricing pressure will drive costs down.

Frequently Asked Questions

Can I use Realtime API for phone calls (PSTN)?

Yes, but need telephony provider that supports WebSocket audio streaming. Providers adding support:

• Twilio: Beta support for Media Streams → WebSocket
• Vonage: Experimental WebSocket integration
• Bandwidth: Roadmap item

Does it work in browser (web apps)?

Yes. WebSocket API works in browsers. Use getUserMedia() to capture microphone, stream to Realtime API over WebSocket.

Sample: OpenAI Realtime Console demo (web-based voice chat).

What languages does it support?

Same as Whisper (100+ languages). However, voice output currently supports English-optimized voices. Non-English works but accent may be English-biased.

Expected: Localized voice options in future releases.

Bottom Line

Realtime API is a step-function improvement for voice agent UX. Latency drops sharply, complexity drops dramatically and interruptions work natively. But cost per call is many times higher.

Use it if: Building high-value voice interactions where UX matters more than cost (sales, premium support, consumer apps).

Skip it if: Building high-volume, cost-sensitive use cases (call center tier-1 support, automated phone menus).

Watch for: Pricing evolution (likely to drop), competitor launches (Anthropic, Google), model tiers (cheaper realtime-mini variant).

Voice agents just became viable for mainstream products. The question is whether your use case can afford the premium.

---

Further Reading:

• OpenAI Realtime API Docs
• Realtime API GitHub Examples
• Building Voice Agents with Realtime API (Guide)