hasdata

HasData CLI

Real-time web data via the HasData CLI. Returns structured JSON — or HTML/markdown for raw scraping — for any of 40+ APIs covering search engines, e-commerce, real estate, jobs, maps, social, and travel.

Run hasdata --help to list all APIs, or hasdata <api-slug> --help for per-API flags.

Prerequisites

Must be installed and authenticated. Verify with:

hasdata version

If the binary isn't found or isn't authenticated, see rules/install.md. For output handling and security guidelines, see rules/security.md.

Run one small request to verify install + auth:

mkdir -p .hasdata
hasdata google-serp --q "hello world" --pretty -o .hasdata/install-check.json

Workflow

Pick the right API for the task. Most APIs return structured JSON — no HTML parsing needed.

1. Search the web — google-serp, google-news, google-ai-mode, bing-serp for SERPs and news
2. Scrape any URL — web-scraping for HTML/markdown/AI-extracted JSON from arbitrary pages
3. Find structured data on a known platform — use the platform-specific API (Amazon, Zillow, etc.) instead of generic scraping
4. Discover places or businesses — google-maps, yelp-search, yellowpages-search
5. Track prices, jobs, listings — combine search APIs with detail APIs (e.g. amazon-search → amazon-product, zillow-listing → zillow-property)

Need	Command	Cost (credits)
Google search results	google-serp / google-serp-light	10 / 5
Google news	google-news	10
Google AI Overviews	google-ai-mode	5
Bing search results	bing-serp	10
Scrape any URL	web-scraping	10
Google Maps places	google-maps, google-maps-place	5
Google Maps reviews / photos	google-maps-reviews, google-maps-photos	5
Amazon product / search	amazon-product, amazon-search	5
Amazon seller / their products	amazon-seller, amazon-seller-products	5
Shopify catalog	shopify-products, shopify-collections	5
Google Shopping	google-shopping, google-immersive-product	10 / 5
Zillow / Redfin	zillow-listing, zillow-property, redfin-listing	5
Airbnb	airbnb-listing, airbnb-property	5
Yelp / YellowPages	yelp-search, yelp-place, yellowpages-*	5
Jobs	indeed-listing, indeed-job, glassdoor-*	5
Instagram profile	instagram-profile	5
Google Flights	google-flights	15
Google Trends / Events / Images	google-trends, google-events, google-images	5

For detailed per-API reference, run hasdata <command> --help.

Generic scraping vs platform APIs:

• Use the platform-specific API when the user wants data from Amazon, Zillow, Yelp, Indeed, etc. — these return structured JSON ready to use.
• Use web-scraping only for arbitrary URLs without a dedicated endpoint, or when you need HTML/markdown/screenshots.
• Don't pipe web-scraping output through extra parsing if a platform API exists.

When to Load References

• SERPs, news, AI Overviews, Trends, Events, Images -> hasdata-search
• Scraping arbitrary URLs (HTML/markdown/JSON/AI extraction) -> hasdata-scrape
• Google Maps places, reviews, photos -> hasdata-maps
• Amazon, Shopify, Google Shopping -> hasdata-ecommerce
• Zillow, Redfin, Airbnb listings/properties -> hasdata-realestate
• Indeed, Glassdoor jobs -> hasdata-jobs
• Yelp, YellowPages business search -> hasdata-business
• Instagram profile -> hasdata-social
• Google Flights -> hasdata-flights
• Install, auth, or setup problems -> rules/install.md
• Output handling and safe file-reading patterns -> rules/security.md

Output & Organization

Always write results to .hasdata/ with -o to keep the context window clean. Add .hasdata/ to .gitignore. Use --pretty for readable JSON.

hasdata google-serp --q "react hooks" --pretty -o .hasdata/serp-react-hooks.json
hasdata web-scraping --url "https://example.com" --output-format markdown -o .hasdata/example.md
hasdata amazon-search --q "wireless mouse" --pretty -o .hasdata/amazon-mouse.json

Naming conventions:

.hasdata/{api}-{query-or-id}.json
.hasdata/serp-{query}.json
.hasdata/amazon-{query}.json
.hasdata/zillow-{location}-{type}.json
.hasdata/scrape-{site}-{path}.md

Always quote URLs and queries — shell interprets ?, &, and spaces specially.

Never read entire output files at once. Use jq, grep, or head to inspect only what's needed:

wc -l .hasdata/file.json && head -50 .hasdata/file.json
jq '.organicResults[] | {title, link}' .hasdata/serp-react-hooks.json
jq '.products[] | {title, price, asin}' .hasdata/amazon-mouse.json

Working with Results

Most HasData responses are JSON. Extract what you need with jq:

# SERP: titles + links
jq -r '.organicResults[] | "\(.title)\t\(.link)"' .hasdata/serp.json

# Amazon: products under $50
jq '.products[] | select(.price.value < 50) | {title, price: .price.value, url}' .hasdata/amazon.json

# Zillow: addresses + prices
jq -r '.properties[] | "\(.address)\t$\(.price)"' .hasdata/zillow.json

# Yelp: name + phone + rating
jq -r '.places[] | "\(.title)\t\(.phone // "no-phone")\t\(.rating)"' .hasdata/yelp.json

For web-scraping --output-format markdown, the result is raw markdown — read with head / grep directly.

Parallelization

Independent calls run in parallel — useful for fanning out from a search to many detail pages:

# Search Amazon, then fetch each product in parallel
hasdata amazon-search --q "yoga mat" --pretty -o .hasdata/yoga-search.json
for asin in $(jq -r '.products[].asin' .hasdata/yoga-search.json | head -10); do
  hasdata amazon-product --asin "$asin" --pretty -o ".hasdata/yoga-$asin.json" &
done
wait

Use & + wait for fan-out, but keep concurrency reasonable — every call consumes credits.

Credits & Rate Limits

Each API call consumes credits (5–15 per call, listed in the help output). View remaining credits and account status at https://app.hasdata.com.

Common flags that affect cost:

• web-scraping --js-rendering (default true) — required for SPAs, no extra cost
• web-scraping --proxy-type residential — needed for hard-to-scrape sites; same cost as datacenter
• google-flights --deep-search — slower, more thorough; same cost

If a request fails with a 401, the API key is missing or invalid — see rules/install.md. 429s auto-retry up to --retries 2 (default).