vision-streaming-agent

vision-streaming-agent

Stream live camera or screen frames to a Gemini Live agent for real-time analysis.

When to use

• Live tutoring / form coaching (yoga, surgery, music)
• Accessibility (describe surroundings for visually impaired)
• Inspection / QA workflows
• Screen-share copilot

Agent config

from google.adk.agents import LlmAgent
from google.adk.streaming import StreamingMode

root_agent = LlmAgent(
    name="vision_coach",
    model="gemini-2.5-flash-live",
    instruction=(
        "You are watching the user via their camera. "
        "Provide concise feedback as they perform the task. "
        "Only speak when there's something useful to say — don't narrate."
    ),
    streaming_mode=StreamingMode.BIDI,
    response_modalities=["AUDIO"],
    input_video_config={
        "frame_rate": 1,           # 1 fps is plenty for most tasks
        "max_resolution": "720p",
    },
)

Client-side: capture frames

const video = document.querySelector("video");
const stream = await navigator.mediaDevices.getUserMedia({ video: true });
video.srcObject = stream;

const canvas = document.createElement("canvas");
const ctx = canvas.getContext("2d");

setInterval(() => {
  canvas.width = video.videoWidth;
  canvas.height = video.videoHeight;
  ctx.drawImage(video, 0, 0);
  canvas.toBlob(blob => {
    blob.arrayBuffer().then(buf => {
      ws.send(JSON.stringify({
        type: "video_frame",
        mime: "image/jpeg",
        data_b64: arrayBufferToBase64(buf),
      }));
    });
  }, "image/jpeg", 0.7);
}, 1000);

Frame rate vs cost

FPS	Use case	Cost
0.5	Static scene check	$
1	Coaching, inspection	$$
2-4	Sports, fast motion	$$$
5+	Real-time gameplay	$$$$

Gemini Live charges per frame; start low and bump up if needed.

Combining audio + video

LlmAgent(
    ...,
    response_modalities=["AUDIO"],
    input_audio_config={...},
    input_video_config={...},
)

User speaks AND camera streams; agent integrates both.

Validation

• Frames decode server-side (check adk web event log)
• Agent's commentary references visible objects (sanity test)
• Frame rate doesn't exceed configured cap
• Bandwidth budget under user's connection limits

See also

• audio-streaming-agent to add voice
• gemini-live-bootstrap for the underlying transport setup