audio-streaming-agent

audio-streaming-agent

Build a voice-in / voice-out agent on Gemini Live via ADK 2.0.

When to use

• Phone-style assistants
• In-car voice systems
• Accessibility (screen reader replacement)
• Hands-free workflows (medical, manufacturing)

Agent config

from google.adk.agents import LlmAgent
from google.adk.streaming import StreamingMode

root_agent = LlmAgent(
    name="voice_assistant",
    model="gemini-2.5-flash-live",
    instruction=(
        "You are speaking with the user via voice. "
        "Keep responses under 30 seconds of speech. "
        "Don't read URLs or code; describe them in plain language."
    ),
    streaming_mode=StreamingMode.BIDI,
    response_modalities=["AUDIO"],
    voice_name="Aoede",
    input_audio_config={
        "sample_rate_hz": 16000,
        "encoding": "LINEAR16",
    },
    output_audio_config={
        "sample_rate_hz": 24000,
        "encoding": "LINEAR16",
    },
)

Voice options

Common Gemini Live voices: Aoede, Puck, Charon, Kore, Fenrir. Test each — characteristics differ (warmth, pace, pitch).

Mic input on the client

const audioCtx = new AudioContext({ sampleRate: 16000 });
navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
  const source = audioCtx.createMediaStreamSource(stream);
  const processor = audioCtx.createScriptProcessor(4096, 1, 1);
  source.connect(processor);
  processor.connect(audioCtx.destination);
  processor.onaudioprocess = (e) => {
    const pcm = e.inputBuffer.getChannelData(0);
    const int16 = new Int16Array(pcm.map(s => Math.max(-1, Math.min(1, s)) * 0x7FFF));
    ws.send(int16.buffer);
  };
});

Turn detection

Gemini Live handles VAD (voice activity detection) server-side. Override:

turn_detection={
    "type": "server_vad",
    "threshold": 0.5,
    "silence_duration_ms": 700,
}

Interruptions

When the user starts speaking mid-response, ADK fires an interruption event:

from google.adk.callbacks import on_interruption

@on_interruption
async def handle_interrupt(ctx, event):
    print(f"User interrupted at {event.timestamp}")
    # Optional: log, reset agent state, etc.

Validation

• Audio plays back cleanly (no clicks/gaps) on the client
• VAD correctly detects end-of-utterance (test with pauses)
• Latency: first audio chunk < 1s after user finishes speaking
• Interruption event fires when user speaks over the agent

See also

• gemini-live-bootstrap for general streaming setup
• bidirectional-tool-streaming for tools that run during voice convo