vision-link
Give MCP-compatible AI assistants the ability to watch and understand videos.
vision-link is an MCP server and optional Claude Code plugin that extracts frames via ffmpeg and processes audio via multiple backends (Gemini API, local Whisper, or OpenAI API). Your AI client receives frames as images and audio transcription with timestamps — the server is a perception layer, not an interpretation layer.
Features
- Multimodal perception — Your AI client sees video frames directly and reads audio transcriptions with timestamps
- YouTube URL support — Pass a YouTube URL directly; the MCP server downloads it with
yt-dlp, preserving source metadata and captions for context
- Flexible backends — Choose between cloud APIs or fully local processing
- Adaptive extraction — The server supports varying fps, time ranges, and resolution based on the request context
- Auto-installation — Whisper models download automatically on first use
- Quick Setup wizard — Three modes: Quick (one-click), Advanced (full control), Custom (pick what to configure)
- Auto-diagnostics —
/vision-link:doctor checks dependencies, config, and fixes common issues automatically
Platform Support
Verified MCP Clients:
- ✅ Claude Desktop - Full support via MCP config
- ✅ Claude Code - Full support with plugin system + slash commands
- ✅ Cursor IDE - MCP support available
- ✅ Generic MCP clients - Standard MCP protocol
See docs/setup/ for platform-specific guides.
Quick Start
1. Install the plugin
Inside Claude Code, run these commands one at a time:
/plugin marketplace add https://github.com/0labs-in/vision-link
Then:
/plugin install vision-link
The MCP server will auto-install via npx from npm on first use — no build step required.
Alternative: local development
git clone https://github.com/0labs-in/vision-link.git
claude --plugin-dir /path/to/vision-link
2. Quick Setup (Recommended)
Inside Claude Code, run the interactive setup wizard:
/vision-link:setup-video-vision
Choose Quick Setup for instant configuration with best defaults, or pick Advanced/Custom for full control.
3. Troubleshooting
If you encounter any setup issues, run the diagnostic tool:
/vision-link:doctor
Doctor will:
- ✅ Check all dependencies (ffmpeg, whisper, yt-dlp)
- ✅ Verify your configuration
- ✅ Test PATH detection
- ✅ Auto-fix common issues
- ✅ Provide clear fix suggestions
Usage
Slash commands
/vision-link:watch-video path/to/video.mp4
/vision-link:watch-video tutorial.mp4 "what language is used in this tutorial?"
/vision-link:watch-video https://www.youtube.com/watch?v=... "summarize this video"
/vision-link:doctor
Conversational
Just mention a video file or YouTube URL — Claude will detect it:
"analyze this video for me: ~/Downloads/demo.mp4"
"take a look at the first second of ~/videos/bug-report.mov"
"summarize this YouTube Short: https://www.youtube.com/shorts/..."
The MCP workflow can adapt parameters automatically:
- "the first second" → extracts at original fps from
00:00:00 to 00:00:01
- "summarize this 1h lecture" → low fps, full duration
- "what text is on screen at 1:30?" → high resolution, narrow time window
Backends
| Backend | Audio processing | Cost | Setup |
|---|
| Gemini API | Native (speech + non-speech events) | Free tier: 1500 req/day | GEMINI_API_KEY env var |
| Local (Whisper) | whisper.cpp or Python openai-whisper | Free, fully offline | brew install whisper-cpp + auto model download |
| OpenAI API | OpenAI Whisper API | Paid per usage | OPENAI_API_KEY env var |
All backends extract video frames via ffmpeg so the connected AI client receives direct visual context.
Architecture
