web_analysis_marketplace

Web Analysis Marketplace

A plugin marketplace for automated website traffic data extraction. Works with both Claude Code and OpenClaw.

What It Does

Automatically fetches website traffic data from SimilarWeb Pro — including visits, rankings, engagement metrics, geographic distribution, traffic sources, and top referral domains.

Example: run /fetch_website_flow_analysis_v2 github.com and get a structured overview + referral CSV in seconds.

Quick Start

Claude Code

# 1. Install the marketplace and plugin
/plugin marketplace add rolanyan/web_analysis_marketplace
/plugin install similarweb_analysis@rolanyan/web_analysis_marketplace

# 2. Restart Claude Code, then login to SimilarWeb
/similarweb_analysis:sw_login

# 3. Configure all environment variables (see "Environment Variables" section below)

# 4. Fetch data
/similarweb_analysis:fetch_website_flow_analysis_v2 github.com

OpenClaw

# 1. Install the plugin
cd plugins/similarweb_analysis && npm install

# 2. Add plugin to OpenClaw config
openclaw config set plugins.load.paths '["path/to/web_analysis_marketplace/plugins/similarweb_analysis"]'

# 3. Configure all environment variables (see "Environment Variables" section below)
#    Additionally set OpenClaw plugin config:
openclaw config set plugins.entries.similarweb-analysis.config.proxyBizId "$PROXY_BIZ_ID"
openclaw config set plugins.entries.similarweb-analysis.config.proxyAuthKey "$PROXY_AUTH_KEY"
openclaw config set plugins.entries.similarweb-analysis.config.proxyAuthPwd "$PROXY_AUTH_PWD"

# 4. Login to SimilarWeb (requires dev-browser, run manually)
python3 plugins/similarweb_analysis/scripts/sw_login.py

# 5. Fetch data (in OpenClaw conversation)
> 帮我获取 github.com 的流量数据
# Agent automatically calls similarweb_fetch(domain: "github.com")

Prerequisites

• Python 3.10+ with requests package
• SimilarWeb Pro account (cookie-based authentication)
• Proxy IP pool (Qingyun residential proxy recommended for v2)
• dev-browser plugin (Claude Code only, required for /sw_login)

Login Flow

The /sw_login command (or python3 scripts/sw_login.py) handles SimilarWeb authentication:

1. Checks if existing cookie is still valid
2. Launches dev-browser and navigates to SimilarWeb
3. Clicks "Sign in with Google" automatically
4. Prompts you for Google email and password (password input is hidden)
5. Fills Google credentials automatically (two-step: email → password)
6. Handles Google 2FA if needed (falls back to manual browser interaction)
7. Handles SimilarWeb 6-digit device verification code if prompted
8. Extracts and saves the session cookie

Run this whenever API requests start returning 401/403 errors.

Available Commands

Claude Code

Command	Description
/similarweb_analysis:fetch_website_flow_analysis_v2	Fetch data via API + proxy (recommended)
/similarweb_analysis:fetch_website_flow_analysis	Fetch data via browser automation (fallback)
/similarweb_analysis:sw_login	Login to SimilarWeb and refresh cookie
/similarweb_analysis:sw_check_cookie	Check cookie validity

OpenClaw

Tool	Description
similarweb_fetch	Fetch domain traffic data (calls sw_fetch.py)
similarweb_check_cookie	Check cookie validity (calls sw_check_cookie.py)

Note: sw_login requires interactive terminal input (Google credentials, verification code) and cannot run as a non-interactive OpenClaw tool. Run python3 scripts/sw_login.py directly in your terminal when cookie expires.

Output

Data is saved to web_data/{domain}/:

File	Content
overview.md	Traffic overview — visits, ranks, engagement, geography, traffic sources
referrals_incoming.csv	Top 100 referral domains with industry, rank, traffic share
raw_api_data.json	Raw JSON from 11 SimilarWeb APIs (v2 only)

Environment Variables

All 5 variables must be configured before first use. Set them in ~/.claude/.env or your shell profile (~/.zshrc / ~/.bashrc):

# Qingyun proxy IP pool
export PROXY_BIZ_ID="your-biz-id"       # 青云「业务标识」, e.g. qccjgh6k
export PROXY_AUTH_KEY="your-auth-key"    # 青云「AuthKey」, also the key= param in extraction URL
export PROXY_AUTH_PWD="your-auth-pwd"    # 青云「AuthPwd」
export PROXY_API_URL="https://overseas.proxy.qg.net/get"

# SimilarWeb cookie file path
export SW_COOKIE_FILE="/path/to/sw_cookies.txt"

Variable	Qingyun Dashboard	Description
PROXY_BIZ_ID	业务标识	Business identifier, e.g. qccjgh6k
PROXY_AUTH_KEY	AuthKey	Used for API extraction (key= param) and proxy auth
PROXY_AUTH_PWD	AuthPwd	Proxy authentication password
PROXY_API_URL	—	Qingyun proxy API endpoint
SW_COOKIE_FILE	—	SimilarWeb cookie file path