web-kit

web-kit

简体中文

A skill bundle for AI coding agents (Claude Code, Copilot CLI, etc.) that gives them three lightweight web primitives: search, page reading, and authenticated download. Plus a self-hosted single-container backend that powers them all.

┌──────────────────────────┐         ┌──────────────────────────────────┐
│   skill/  (the AI skill) │  ◄───►  │   backend/  (the server side)    │
│   ─────────────────────  │         │   ─────────────────────────────  │
│   ask-search             │         │   SearxNG + Chrome+Playwright    │
│   crwlr                  │         │   CDP, all in one container,     │
│   cdp-download           │         │   self-healing in 4 layers       │
└──────────────────────────┘         └──────────────────────────────────┘
        │                                            ▲
        └─ uses ──── SEARXNG_URL=http://...:8082 ────┤
                     CDP_URL=http://...:9223 ────────┘

You can run them together (this repo's main story) or split:

• skill only — point at any existing SearxNG / CDP-enabled Chrome you already have.
• backend only — drop-in replacement for the typical 4-container searxng + browser-proxy stack.

---

skill/ — the AI skill

Three commands an agent can call:

Tool	What it does	When agents use it
ask-search "<query>"	SearxNG-aggregated web search, multi-engine	"search for X", "look this up", "找一下…"
crwlr crawl -o md "<url>"	Real-browser-rendered page → clean markdown	"read this page", "what does this URL say", "转成 markdown"
cdp-download <url>	File download via Chrome DevTools Protocol (uses browser cookies)	When wget/curl fails on auth-protected URLs

Compared to a vanilla agent's built-in web tools, this skill gives:

• Real JS rendering (the page the user actually sees, not a server-side stripped version)
• Persistent login cookies (login once via noVNC, all subsequent crawls are authenticated)
• Aggregation across many search engines (Google, DDG, Bing, Brave, plus 13 academic sources)
• Lightweight CLI surface — no MCP server / no extra runtime

Install (Claude Code example)

One runtime dependency: uv — runs all Python scripts and auto-installs their inline dependencies (crawl4ai, websocket-client, etc.) on first use.

# 1. Clone the repo and copy the skill into your skills dir
git clone https://github.com/huangyrcn/web-kit.git
cp -r web-kit/skill ~/.claude/skills/web-kit
chmod +x ~/.claude/skills/web-kit/scripts/*

# 2. Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 3. Tell the skill where its backend lives
export SEARXNG_URL=http://your-host:8082
export CDP_URL=http://your-host:9223

# 4. Smoke test
~/.claude/skills/web-kit/scripts/ask-search "hello world" -n 3
~/.claude/skills/web-kit/scripts/crwlr crawl -o md "https://example.com"

For Copilot CLI, Gemini CLI, or other platforms, see skill/SKILL.md — the doc lists the per-platform skill loading mechanism.

Required environment variables

Var	Used by	Example
SEARXNG_URL	ask-search	http://localhost:8082
CDP_URL	crwlr, cdp-download	http://localhost:9223
WEB_KIT_API_KEY	all three skills + gateway	$(openssl rand -hex 32)

If you already have a SearxNG instance and a Chrome with CDP exposed, you can stop here.

---

backend/ — the single-container server

Self-hosted backend that serves both SEARXNG_URL (port 8082) and CDP_URL (port 9223), plus a noVNC interface (port 6080) for one-time manual cookie login.

Why bundle into one container?

Self-hosted SearxNG with a JS-rendering search-proxy typically requires 4+ separate containers (searxng, redis, browser-proxy, novnc) glued together with networking. That's fragile — one container's failure cascades into the rest, and there's no single place to monitor or restart things.

This backend collapses it into one container, with supervisord managing all processes and four layers of self-healing:

1. Process-level — supervisord autorestart=true restarts any crashed process.
2. Application-level — _ensure_browser() reconnects CDP on each request if the underlying Chrome crashed in between.
3. Container-level — Docker HEALTHCHECK probes all three exposed ports.
4. Business-level — watchdog.sh probes internal health every 300s and triggers supervisorctl restart chrome search-proxy after 3 consecutive failures (catches "process alive but browser hung").

Quick start

cd backend
cp searxng-settings/settings.yml.example searxng-settings/settings.yml
$EDITOR searxng-settings/settings.yml      # set secret_key + any API keys

docker compose up -d                        # first build ~5-10 min for Chrome + SearxNG
sleep 90                                    # warm-up