web-forager

Web Forager

The thing about information on the web is that it doesn't want to be found. It wants to hide behind cookie banners, keep itself to itself, and generally behave like a cat that knows it's time for the vet. Web Forager is the sort of dogged, slightly grubby assistant who goes out there anyway — accompanied by a duck of questionable temperament — rummages through DuckDuckGo, grabs pages directly when it can, and calls in Jina Reader when things get complicated. The results come back neatly converted for LLM consumption, which is to say, in a format that would make a librarian weep with either joy or despair, depending on the librarian.

A search-and-fetch toolkit for AI agents, available as an MCP server and as standalone Agent Skills:

1. Search the web via DuckDuckGo
2. Search news via DuckDuckGo News
3. Fetch and convert web pages (direct HTTP + trafilatura, Jina Reader fallback)

Also ships five Agent Skills that work independently — no MCP required — for research, fact-checking, news monitoring, competitive analysis, and technology evaluation.

Features

• DuckDuckGo web search with safe search controls
• DuckDuckGo news search with date-sorted results and source attribution
• Fetch and convert URLs to markdown or JSON (direct HTTP + trafilatura, Jina Reader fallback)
• LLM-friendly output format option for search results
• CLI for search, news, fetch, serve, and version commands
• MCP tools for LLM integration
• Five standalone Agent Skills for specialized research workflows
• Docker support for containerized deployment

Installation

Prerequisites

• Python 3.10 or higher
• uv (recommended) or pip

Install from PyPI (recommended)

# Using uv (recommended)
uv pip install web-forager

# Or using pip
pip install web-forager

Install with UVX (for Claude Desktop)

# Install UVX if you haven't already
pip install uvx

# Install the Web Forager package
uvx install web-forager

Install from source

For development or to get the latest changes:

# Clone the repository
git clone https://github.com/CyranoB/web-forager.git
cd web-forager

# Install with uv (recommended)
uv pip install -e .

# Or with pip
pip install -e .

Docker

Build and run with Docker:

# Build the image (uses version from latest git tag)
docker build --build-arg VERSION=$(git describe --tags --abbrev=0 | sed 's/^v//') -t web-forager .

# Or specify a version manually
docker build --build-arg VERSION=2.0.2 -t web-forager .

# Run the server (MCP servers use STDIO, so typically run within an MCP client)
docker run -i web-forager

Usage

Starting the Server (STDIO Mode)

# Start the server in STDIO mode (for use with MCP clients like Claude)
web-forager serve

# Enable debug logging
web-forager serve --debug

Testing the Search Tool

# Search DuckDuckGo (JSON output, default)
web-forager search "your search query" --max-results 5 --safesearch moderate

# Search with LLM-friendly text output
web-forager search "your search query" --output-format text

Testing the News Search Tool

# Search DuckDuckGo news (JSON output, default)
web-forager news "your search query" --max-results 10 --safesearch moderate

# Search with LLM-friendly text output
web-forager news "your search query" --output-format text

Testing the Fetch Tool

# Fetch a URL and return markdown
web-forager fetch "https://example.com" --format markdown

# Fetch a URL and return JSON
web-forager fetch "https://example.com" --format json

# Limit output length
web-forager fetch "https://example.com" --max-length 2000

# Include generated image alt text
web-forager fetch "https://example.com" --with-images

Version Information

# Show version
web-forager version

# Show detailed version info
web-forager version --debug

MCP Client Setup

This MCP server works with any MCP-compatible client. Use one of the setups below.

Python 3.10-3.13 is supported (3.14 not yet). Use --python ">=3.10,<3.14" with uvx to enforce. Verified with Python 3.12 and 3.13.

Claude Desktop