From ScraperAPI
Onboarding entry point for ScraperAPI: get API key, choose integration path (MCP, SDK, REST), and route to specific scraping skills.
How this skill is triggered — by the user, by Claude, or both
Slash command
/scraperapi:scraperapi-agent-onboardingThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
ScraperAPI gives agents and apps reliable access to web data: any URL returned as clean HTML or
ScraperAPI gives agents and apps reliable access to web data: any URL returned as clean HTML or markdown (with proxy rotation, CAPTCHA bypass, and JS rendering handled automatically), structured JSON from 20+ supported platforms (Amazon, Walmart, eBay, Redfin, Google SERP), and a crawler for multi-page extraction.
This skill is the entry point. Read it once, pick a path, then hand off to the narrower skill that owns that path.
All paths require a ScraperAPI key. If you don't have one:
# macOS / Linux
export SCRAPERAPI_API_KEY=<your-key>
# Windows PowerShell
$env:SCRAPERAPI_API_KEY = "<your-key>"
# .env file
SCRAPERAPI_API_KEY=...
Verify the key works before doing real work:
curl "https://api.scraperapi.com/?api_key=$SCRAPERAPI_API_KEY&url=https://httpbin.org/ip"
A JSON response with an origin IP confirms the key is valid. A 401 means the key is wrong or
inactive — do not retry with the same key, surface the error.
| Situation | Path |
|---|---|
| LLM agent needs web data during this session | Path A — MCP Server |
| Adding ScraperAPI to app code | Path B — SDK / REST integration |
| Quick request, or environment without Node/Python | Path C — REST API directly |
| Need an API key first | Path D — Auth only (above) |
If your task spans paths, do them in order: get key → MCP or SDK → test one real request before building.
Use this when the consumer is an LLM agent (Claude Code, custom agent loop) that should call ScraperAPI as tools. The MCP server exposes up to 22 tools — scrape, Google, Amazon, Walmart, eBay, Redfin, and crawler — over a single connection.
No installation required. All 22 tools available.
Add to your claude_desktop_config.json, Claude Code project settings, or MCP client config:
{
"mcpServers": {
"ScraperAPI": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.scraperapi.com/mcp",
"--header",
"Authorization: Bearer ${SCRAPERAPI_API_KEY}"
]
}
}
}
Requires SCRAPERAPI_API_KEY set in the environment. npx mcp-remote handles the remote
connection — no separate install step needed.
If you prefer a local server or the remote endpoint is unavailable. Requires Python 3.11+.
pip install scraperapi-mcp-server
{
"mcpServers": {
"ScraperAPI": {
"command": "python",
"args": ["-m", "scraperapi_mcp_server"],
"env": { "API_KEY": "<YOUR_SCRAPERAPI_API_KEY>" }
}
}
}
The local server exposes only the scrape tool. For structured Google search data on the local
variant, use scrape with autoparse: true and outputFormat: "json" on a Google search URL.
Hand off to the scraperapi-mcp skill. It owns tool selection, parameter optimization, credit
cost guidance, and error recovery for all 22 MCP tools. Also check the
MCP setup reference for variant detection and
troubleshooting.
Use this when you're building an application, script, or workflow that calls ScraperAPI from code.
Always run one real request before writing integration logic — catches auth, credit, and parameter issues before they hide inside your app's error paths:
import os, requests
r = requests.get(
"https://api.scraperapi.com/",
params={"api_key": os.environ["SCRAPERAPI_API_KEY"], "url": "https://httpbin.org/ip"}
)
print(r.status_code, r.text[:200])
A 200 with valid JSON confirms you're wired up.
| Job in the app | API | Key docs |
|---|---|---|
| Fetch any URL as HTML or markdown | Standard API | docs |
| Render JS-heavy pages (React, Vue, SPAs) | Standard API + render=true | docs |
| Structured JSON: Amazon, Google, Walmart, eBay, Redfin | Structured Data API | docs |
| Batch scraping / 100+ URLs | Async Jobs API | docs |
| Recurring, scheduled scraping without managing infra | DataPipeline (dashboard) | docs |
pip install scraperapi-sdk
import os
from scraperapi_sdk import ScraperAPIClient
client = ScraperAPIClient(os.environ["SCRAPERAPI_API_KEY"])
result = client.get("https://example.com", render=True, country_code="us")
print(result)
npm install scraperapi-sdk
const scraperapi = require('scraperapi-sdk')(process.env.SCRAPERAPI_API_KEY);
const result = await scraperapi.get('https://example.com', { render: true, country_code: 'us' });
console.log(result);
PHP, Ruby, and Java SDKs are also available — see docs.scraperapi.com for setup guides.
Use this when the environment can't run pip or npm, or when you only need a few requests.
Base URL: https://api.scraperapi.com/
Auth: api_key query parameter (or apiKey in the JSON body for async)
# Basic scrape
curl "https://api.scraperapi.com/?api_key=$SCRAPERAPI_API_KEY&url=https://example.com"
# With JS rendering
curl "https://api.scraperapi.com/?api_key=$SCRAPERAPI_API_KEY&url=https://example.com&render=true"
# Structured data — Google SERP
curl "https://api.scraperapi.com/structured/google/search?api_key=$SCRAPERAPI_API_KEY&query=web+scraping"
# Structured data — Amazon product
curl "https://api.scraperapi.com/structured/amazon/product?api_key=$SCRAPERAPI_API_KEY&asin=B09V3KXJPB"
# Submit an async job
curl -X POST "https://async.scraperapi.com/jobs" \
-H "Content-Type: application/json" \
-d "{\"apiKey\": \"$SCRAPERAPI_API_KEY\", \"url\": \"https://example.com\"}"
# Poll async job status
curl "https://async.scraperapi.com/jobs/<jobId>?apiKey=$SCRAPERAPI_API_KEY"
For the full parameter surface (premium proxies, country targeting, session stickiness, binary targets), see the API reference.
| Code | Meaning | Action |
|---|---|---|
200 | Success | Return / parse the body |
401 | Invalid API key | Surface error — do not retry |
403 | Blocked by target site | Retry with premium=true, ultra_premium=true, or a different country_code |
429 | Concurrent limit hit | Back off; switch to async for batch work |
500/503 | Transient failure | Retry with exponential backoff (3–5 attempts) |
Route to the narrowest skill or reference that fits the task:
| User says… | Go to |
|---|---|
| "scrape this URL" / "get this page" | scraperapi-mcp → scrape tool |
| "search Google for…" / "find URLs about…" | scraperapi-mcp → google_search |
| "get Amazon / Walmart / eBay data" | scraperapi-mcp → SDE tools |
| "real estate / Redfin listings" | scraperapi-mcp → Redfin tools |
| "crawl a website / extract all links" | scraperapi-mcp → crawler_job_start |
| "scrape 100+ URLs / batch job" | Async Jobs API — Path C above |
| "schedule recurring scraping" | DataPipeline — https://docs.scraperapi.com/ |
| "debug a failed scrape or blocked request" | scraperapi-mcp error recovery section |
When in doubt, prefer structured data endpoints over raw HTML scraping for supported platforms —
they return clean JSON with no parsing logic, are more reliable, and often cost fewer credits than
raw scraping with render=true.
npx claudepluginhub scraperapi/scraperapi-skills --plugin scraperapiProvides UI/UX resources: 50+ styles, color palettes, font pairings, guidelines, charts for web/mobile across React, Next.js, Vue, Svelte, Tailwind, React Native, Flutter. Aids planning, building, reviewing interfaces.
Searches MemPalace before answering questions about past work, people, projects, or prior decisions. Returns verbatim stored content instead of guessing from model memory.