fastCRW

Reference skill for building production-ready crw integrations. Covers verb selection, call surfaces (CLI/MCP/REST), post-filtering strategies, context-window hygiene, Hybrid RAG patterns, common pitfalls, and crw-specific operational considerations (SearXNG limits, renderer pool, proxy rotation). Load this when writing application code that embeds crw, designing a multi-step agent workflow, or debugging an integration that isn't behaving as expected.

Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.

Programmatic web search and scrape with context isolation. Use for any research task where you need to search the web, filter results, and extract specific information — without flooding your context window with raw HTML and boilerplate. This is the single biggest token-saver in the crw skill set. Triggered by "search for", "look up", "find", "research", "what's the latest on", or any query that requires current web information. Also use when asked to "search and filter", "find the important parts", or any task where you suspect the raw output will be large (multi-page scrapes, news aggregation, competitive research).

Extract a typed JSON object from one or more web pages against a JSON Schema with fastCRW. Use when you need structured data — "get the price and stock status", "extract all job listings as JSON", "pull structured fields from this page". Step 6 of the crw workflow ladder.

Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.