crw-watch

crw-watch — change tracking and diffing

When to use

• You want to know what changed between two snapshots of a page.
• Step 7 in the crw ladder. Assumes you can already scrape the page — see crw-scrape (step 2).
• You want a self-hosted, stateless diff primitive you control. Firecrawl offers change tracking only as a managed cloud feature; crw exposes the same primitive as a REST endpoint that runs on your own infra — you own the snapshots, the cadence, and the data.

Architecture: crw is stateless

crw stores nothing between calls. The caller owns the snapshots:

1. Scrape now        → store snapshot (markdown / json)
2. Scrape later      → call /v1/change-tracking/diff with current + previous
3. On status=changed → alert / act
4. Repeat on a cron

Diff modes

Two modes, composable:

Mode	Wire string	What it produces
Git-style text diff	"gitDiff" (alias: "git-diff")	Unified-diff text + parse-diff AST in diff.text / diff.json
Per-field JSON diff	"json"	Path-keyed map {"$.field": {"previous":…,"current":…}} in diff.json; requires schema

Default (omit modes): ["gitDiff"]. Combine both: "modes": ["gitDiff", "json"].

Quick start

Single page diff (REST):

curl -X POST "$CRW_API_URL/v1/change-tracking/diff" \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "current": {
      "markdown": "# Pricing\nPro plan: $49/mo"
    },
    "previous": {
      "markdown": "# Pricing\nPro plan: $39/mo",
      "contentHash": "<hash from prior result>"
    },
    "modes": ["gitDiff"]
  }'

Response shape:

{
  "success": true,
  "data": {
    "status": "changed",
    "firstObservation": false,
    "contentHash": "<new hash>",
    "snapshot": { "markdown": "...", "contentHash": "..." },
    "diff": {
      "text": "@@ -1,2 +1,2 @@\n # Pricing\n-Pro plan: $39/mo\n+Pro plan: $49/mo",
      "json": { "files": [...] }
    }
  }
}

Batch diff (discriminated by presence of batch key):

curl -X POST "$CRW_API_URL/v1/change-tracking/diff" \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "batch": [
      { "url": "https://example.com/pricing", "current": {"markdown": "..."}, "previous": {"markdown": "..."} },
      { "url": "https://example.com/about",   "current": {"markdown": "..."} }
    ],
    "modes": ["gitDiff"]
  }'

Shared modes/schema/prompt/contentType at the top level are defaults; each batch item can override them individually.

Inline during a scrape — pass changeTracking as a format on /v1/scrape:

curl -X POST "$CRW_API_URL/v1/scrape" \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown", "changeTracking"],
    "changeTracking": {
      "modes": ["gitDiff"],
      "previous": { "markdown": "...", "contentHash": "..." }
    }
  }'

Request fields

Single mode: { current, previous?, modes, schema?, prompt?, contentType?, tag?, goal?, judgeEnabled? }

Batch mode: { batch: [...items], modes, schema?, ... } where each item is { url?, current, previous?, modes?, schema?, ... }.

Field	Type	Notes
current.markdown	string	Current page content (gitDiff / mixed)
current.json	object	Current extracted JSON (json / mixed)
previous.markdown	string	Prior snapshot for gitDiff
previous.contentHash	string	Persist from prior result's snapshot.contentHash
modes	string[]	["gitDiff"] (default), ["json"], or both
schema	JSON Schema	Required for json mode; defines tracked fields
prompt	string	Natural-language extraction prompt (alternative to schema)
contentType	string	If binary/non-text, triggers byte-hash comparison only
tag	string	Opaque caller ID echoed back on the result
goal	string	Natural-language filter for meaningful changes (AI judge, M2)
judgeEnabled	bool	Enable AI judgment (M2 feature; accepted but not yet applied)

The goal field (AI judge)

goal is a natural-language filter for what counts as a meaningful change, fed to an LLM judge. It is accepted by the server now but applied in a future milestone (M2). Guidance for when it lands:

• Be specific: "Alert when the listed price changes; ignore copy rewrites and nav updates" beats "detect important changes".
• Narrow the scope: "Only flag changes to the Features table, not the hero section".
• The judge returns {meaningful, confidence, reason, meaningfulChanges[]} in the result's judgment field.

Cron pattern (self-hosted)

#!/usr/bin/env bash
# cron-check.sh — run every hour via cron or a scheduler
SNAPSHOT_FILE=".crw/snapshot.json"
CURRENT=$(crw scrape "https://example.com/pricing" --format markdown)

if [ -f "$SNAPSHOT_FILE" ]; then
  PREV_MARKDOWN=$(jq -r '.markdown' "$SNAPSHOT_FILE")
  PREV_HASH=$(jq -r '.contentHash' "$SNAPSHOT_FILE")
  RESULT=$(curl -s -X POST "$CRW_API_URL/v1/change-tracking/diff" \
    -H "Authorization: Bearer $CRW_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"current\":{\"markdown\":$(jq -Rsc . <<<"$CURRENT")},\"previous\":{\"markdown\":$(jq -Rsc . <<<"$PREV_MARKDOWN"),\"contentHash\":\"$PREV_HASH\"},\"modes\":[\"gitDiff\"]}")
  STATUS=$(echo "$RESULT" | jq -r '.data.status')
  if [ "$STATUS" = "changed" ]; then
    echo "CHANGED: $(echo "$RESULT" | jq -r '.data.diff.text')"
    # → send alert, write to DB, trigger webhook, etc.
  fi
  echo "$RESULT" | jq '.data.snapshot' > "$SNAPSHOT_FILE"
else
  # First observation — store the snapshot
  curl -s -X POST "$CRW_API_URL/v1/change-tracking/diff" \
    -H "Authorization: Bearer $CRW_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"current\":{\"markdown\":$(jq -Rsc . <<<"$CURRENT")},\"modes\":[\"gitDiff\"]}" \
    | jq '.data.snapshot' > "$SNAPSHOT_FILE"
fi

Tips

• Persist snapshot from each result as the next call's previous. The snapshot field in the response contains the normalized content and contentHash — store it, don't recompute it.
• firstObservation: true means no previous was supplied. The server sets status: "changed" and returns snapshot but produces no diff. Store it as your baseline.
• json mode needs current.json (+ optionally a schema). Without structured input it produces no diff — use gitDiff mode for plain markdown.
• Batch is more efficient at scale. One HTTP round-trip for N pages instead of N calls. Top-level modes/schema as defaults keeps the body compact.
• Data sovereignty. You supply previous; crw computes and returns. Nothing is stored server-side. Your snapshots, your infra, your retention policy.

See also

• crw-scrape — get the current page content to feed into the diff
• crw — ladder overview and routing rules