crw-scrape

crw-scrape — single-page extraction

When to use

• You have one (or a handful of) known URLs and want their content.
• Step 2 in the crw ladder: if you don't have a URL yet, go to crw-search (step 1) first. For many pages under a site, use crw-crawl (step 4). For a local PDF, use crw-parse (step 5).
• JS-heavy page? You usually don't need anything special — crw auto-detects and renders. This is a crw advantage: no separate "interact"/browser step.

Quick start

CLI (binary on PATH):

crw scrape "https://example.com"                       # → markdown to stdout
crw scrape "https://example.com" --format json -o page.json
crw scrape "https://example.com" --js --css "article.main"
crw scrape "https://example.com" --format links -o .crw/links.txt

MCP (inside an agent harness):

crw_scrape(url="https://example.com", formats=["markdown"], onlyMainContent=true)

REST (drop-in for Firecrawl SDKs — just swap the base URL):

curl -X POST "$CRW_API_URL/v1/scrape" -H "Authorization: Bearer $CRW_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"],"onlyMainContent":true}'

Options

Need	CLI flag	MCP / REST field
Output format	--format markdown|html|rawhtml|text|links|json	formats: [...]
Strip nav/footer/sidebar	(on by default; --raw to disable)	onlyMainContent: true
Force JS rendering	--js	renderJs: true (null = auto)
Wait after load	—	waitFor: 2000 (ms)
Keep only selectors	--css "article" / --xpath …	includeTags: ["article"]
Drop selectors	—	excludeTags: ["nav","footer"]
Pick renderer	—	renderer: "auto|lightpanda|chrome|chrome_proxy|playwright" (auto is default)
Save to file	-o FILE	(write the response yourself)
Structured JSON	--extract '<schema>'	extract: {schema: {...}} — see crw-extract
Use a proxy	--proxy URL --stealth	proxy, proxyRotation, stealth

Tips

• Quote URLs — ? and & are shell-special. Always wrap in quotes.
• Multiple URLs = run them concurrently. Fire several crw scrape … & and wait, or issue parallel MCP calls.
• Blank page / loading skeleton? Add --js / renderJs: true, optionally a waitFor. crw's auto-detect covers most SPAs without it.
• Don't dump huge pages into context. Write to .crw/, then grep/head. MCP truncates to ~15 000 chars (maxLength: 0 to opt out).
• Want a typed object, not prose? --format json returns the raw full-page object (metadata + content), not schema-extracted data. For structured extraction against a schema use --extract '<schema>' — this calls an LLM and requires a configured LLM provider. See the dedicated crw-extract skill.
• Source is a file, not a URL? Use crw-parse instead.

See also

• crw-search — find the URL first
• crw-map — discover all URLs on a site
• crw-crawl — scrape many pages at once
• crw-dynamic-search — filter scrape output in a subprocess to save context