scrape-website

Scrape a website

Trigger

Extracting content from a single web page — an article, product page, documentation, or any URL.

Workflow

1. Verify anakin-cli is authenticated by running anakin status.
2. Create the output directory if it doesn't exist:

   mkdir -p .anakin
   

3. Determine the output filename from the URL (use domain and path):
   • https://docs.react.dev/learn -> .anakin/docs.react.dev-learn.md
   • https://example.com/pricing -> .anakin/example.com-pricing.md
4. Check if $ARGUMENTS contains --browser. If the URL is a known SPA/JS-heavy site (React apps, Next.js, Angular, etc.), add --browser automatically.
5. Run the scrape:

   anakin scrape "$ARGUMENTS" -o .anakin/<filename>.md
   

6. Check the output file size and read incrementally:

   wc -l .anakin/<filename>.md && head -80 .anakin/<filename>.md
   

7. Summarize the content for the user. If the page is long, highlight the key sections and offer to read specific parts.

Commands

# Clean readable text (default)
anakin scrape "<url>" -o .anakin/output.md

# Structured data
anakin scrape "<url>" --format json -o .anakin/output.json

# Full API response with HTML and metadata
anakin scrape "<url>" --format raw -o .anakin/output.json

# JavaScript-heavy or single-page app sites
anakin scrape "<url>" --browser -o .anakin/output.md

# Geo-targeted scraping
anakin scrape "<url>" --country gb -o .anakin/output.md

# Custom timeout for slow pages
anakin scrape "<url>" --timeout 300 -o .anakin/output.md

Options

• --browser — Use headless browser (for JS-rendered/SPA pages)
• --country <code> — Country code for geo-targeting (default: us)
• --format <fmt> — Output format: markdown (default), json, or raw
• --timeout <seconds> — Max wait time (default: 120)
• -o, --output <path> — Save to file

Async behavior

Starts job, polls every 3s until complete.

Guardrails

• Always quote URLs to prevent shell interpretation of ?, &, # characters.
• Default to markdown format unless the user asks for structured data or raw output.
• Use --browser only when a standard scrape returns empty or incomplete content.
• On 429 errors, wait before retrying rather than looping immediately.
• Always use -o to save output to a file rather than flooding the terminal.

Output

• Scraped content in the requested format
• File path where results were saved
• If scrape fails or returns empty, retry with --browser. If still fails, check anakin status.