watch-video

Watch Video

Purpose

Convert a video file into a sequence of JPEG frames so Claude can analyze its visual content through the native Read image tool. Claude cannot read video files directly, but it can read images — this skill bridges the gap.

When to invoke

Invoke when any of these match:

• The user shares a path to a .mp4, .mov, .webm, .mkv, .avi, or similar file and asks Claude to look at it, watch it, describe it, summarize it, verify a fix in it, or check what happens in it.
• The user shares a screen recording demonstrating a bug.
• The user asks "what is shown in this video" or "did the app behave correctly in this recording".
• A previous step recorded a Playwright/Maestro/Detox test run and the user wants Claude to inspect the result.

Do NOT invoke for: live streams, audio-only files, GIFs (use Read directly), or videos already provided as still frames.

Prerequisites

ffmpeg and ffprobe must be available on PATH:

brew install ffmpeg          # macOS
sudo apt-get install ffmpeg  # Debian/Ubuntu

The script verifies this and emits a clear error if missing.

Workflow

1. Run scripts/extract_frames.sh <video_path> — pass the path verbatim, including spaces and special characters. The script copies the file to a safe temp path internally.
2. Parse the JSON output for output_dir, frames[], frame_count, video_duration_seconds, and strategy.
3. If frame_count is large (>30), sanity-check that a smaller frame set isn't sufficient before reading them all — each Read of a JPG consumes vision tokens.
4. Read each path in frames[] with the Read tool. The frames are timestamp-ordered: frame_001.jpg is earliest, the highest-numbered frame is latest.
5. Analyze the frames to answer the user's question. When relevant, reference frames by their timestamp (compute as index * (1 / effective_fps) for fps mode, or use the visible content for scene-detection mode).
6. Optionally clean up: rm -rf "$OUTPUT_DIR" after the analysis is complete and reported.

Default behavior (auto strategy)

With no flags, the script picks an adaptive frame rate based on duration so the total stays in a useful range (~8–25 frames):

Video duration	Default fps	Approx frames
≤ 10s	1.0	up to 10
≤ 30s	0.5	up to 15
≤ 60s	0.333	up to 20
≤ 2 min	0.2	up to 24
≤ 5 min	0.125	up to ~37
> 5 min	0.0667	depends

Frames are downscaled to 590px wide by default — readable at mobile-screenshot scale without flooding the context window.

Useful flags

Flag	Purpose	Example
--max-frames N	Cap total frames to N (auto-calculates fps)	--max-frames 20
--fps F	Force a specific frame rate	--fps 1
--scene-threshold T	Extract one frame per detected scene change. T in 0.0–1.0; lower = more sensitive	--scene-threshold 0.3
--width W	Output width in pixels (height auto-scaled)	--width 800
--quality Q	JPEG quality 1 (best) – 31 (worst), default 4	--quality 2
--output-dir D	Custom output directory (default: mktemp -d)	--output-dir /tmp/myframes

Common usage patterns

Bug-reproduction screen recording (short, mobile)

scripts/extract_frames.sh "/path/to/ScreenRecording.mp4"

Auto-mode handles it. Read all frames; describe what changes between them.

Long tutorial — pick scene transitions only

scripts/extract_frames.sh tutorial.mp4 --scene-threshold 0.3

Returns one frame per visual scene change (e.g., slide transitions). Best for presentations, edited footage.

Read text on UI screens (OCR-style)

scripts/extract_frames.sh screencast.mp4 --width 1180 --quality 1 --max-frames 25

Higher resolution + best JPEG quality so small UI text remains legible.

Strict frame budget

scripts/extract_frames.sh long.mp4 --max-frames 12

The script computes an fps that yields ~12 evenly-spaced frames regardless of duration.

Output format

The script prints a single JSON object on stdout. Example:

{
  "output_dir": "/tmp/watch-video-frames.AbC123",
  "video_duration_seconds": 19.67,
  "strategy": "auto",
  "effective_fps": 0.5,
  "frame_width": 590,
  "frame_height": 1278,
  "frame_count": 10,
  "frames": [
    "/tmp/watch-video-frames.AbC123/frame_001.jpg",
    "/tmp/watch-video-frames.AbC123/frame_002.jpg"
  ]
}

On failure, a JSON object with an "error" key prints to stderr and the process exits with code 1.

Troubleshooting

• "ffmpeg not found" — install ffmpeg and re-run.
• "could not read video duration" — the file is corrupt, truncated, or uses a codec ffmpeg can't decode. Try ffprobe <file> directly to see ffmpeg's diagnostics.
• "no frames extracted" — the filter selected zero frames. With --scene-threshold, the threshold is too high; lower it (try 0.2). Otherwise the video may be very short — pass --fps 2 to force extraction.
• Frames look blurry / small UI text unreadable — increase --width (try 1180 for iPhone-portrait recordings) and lower --quality (try 1 or 2).
• Too many frames blow context — pass --max-frames 15 to cap.
• Filename has spaces / colons / quotes — pass it verbatim; the script normalizes the path internally. Do not pre-escape.
• "video file not found" but the file is clearly there — the path string may not be byte-identical to the filename. iOS screen recordings notoriously contain a Unicode narrow no-break space (U+202F, bytes e2 80 af) between the time and "AM"/"PM" — it looks like a regular space but isn't. Get the path with shell completion or find -print0 | xargs -0 rather than typing it; that preserves the exact bytes.