gobbler-webpage

Gobbler Webpage

Convert web pages to markdown using the Crawl4AI service.

Requires: Crawl4AI Docker container running (docker compose up -d crawl4ai or docker-compose up -d crawl4ai). Crawl4AI is configured for up to 2 GB RAM and 1 CPU; allocate enough Docker resources before starting the service.

CLI Options

Option	Short	Description	Default
--output	-o	Output file path (stdout if not specified)	-
--selector	-s	CSS selector to extract specific content	-
--clean	-c	Auto-strip nav/footer/sidebar boilerplate	--no-clean
--timeout	-t	Request timeout in seconds	30
--images/--no-images	-	Include images in output	--images
--format	-f	Output format: markdown, json, table	markdown
--provider	-p	Webpage conversion provider	crawl4ai
--skip-if-exists	-	Skip if output file already exists (batch-friendly)	-

Tips

• Use --clean for AI consumption — auto-detects main content (article, main, etc.)
• Use --selector for precise extraction — e.g., --selector "article" or --selector ".post-content"
• Use --skip-if-exists for batch processing — won't re-fetch existing pages

Fetch Single Page

# Basic fetch
gobbler webpage "https://example.com"

# Save to file
gobbler webpage "https://example.com" -o page.md

# Custom timeout
gobbler webpage "https://example.com" --timeout 60

# Exclude images from output
gobbler webpage "https://example.com" --no-images -o page.md

Extract with CSS Selector

# Extract specific content
gobbler webpage "https://example.com" --selector "article.main-content" -o article.md

Output Formats

# JSON format (includes metadata)
gobbler webpage "https://example.com" --format json

# Table format
gobbler webpage "https://example.com" --format table

Saving Output

When saving pages to a file, follow these steps:

Step 1: Check for default output directory

gobbler config get output.default_directory

Step 2: Save to the default directory

If a default directory is configured, use it with a descriptive filename:

gobbler webpage "https://example.com/article" -o "<default_directory>/Article Title.md"

Step 3: If no default directory is configured

If the config returns empty/null, save to the current directory or ask the user where to save:

gobbler webpage "https://example.com/article" -o "Article Title.md"

Alternative: Using the Convert Subcommand

gobbler convert webpage "https://example.com" -o page.md

Prerequisites

Start services before using:

cd /path/to/gobbler
docker compose up -d crawl4ai || docker-compose up -d crawl4ai

# Check health
curl http://localhost:11235/health