bot-repair

Bot Repair Playbooks

Practical repair procedures for broken ClawNet bots. Follow the decision tree to pick the right playbook, then always run post-repair verification.

Repair Decision Tree

Bot not responding
├── Check heartbeat
│   ├── Timeout → Playbook 1 (restart)
│   │   ├── Fixed? → Done
│   │   └── Still down → Check logs
│   │       ├── Missing module → Playbook 3 (dependency fix)
│   │       ├── Invalid key → Playbook 2 (env var fix)
│   │       ├── Out of memory → Playbook 5 (resource fix)
│   │       ├── Network error → Playbook 6 (connectivity check)
│   │       └── Unknown error → Playbook 4 (full redeploy)
│   └── Returns error code
│       ├── 500 → Check logs for stack trace
│       └── 503 → Sandbox starting up, wait and retry
└── Bot responds but degraded
    ├── Slow responses → Playbook 5
    ├── Intermittent errors → Check logs, may need restart
    └── Partial functionality → Check env vars and dependencies

Playbook 1: Simple Restart

When: Bot stopped responding, no obvious error cause

# Restart the sandbox
clawnet bot restart <bot-name>

# Wait for it to come back up (give it 30 seconds)
sleep 30

# Verify heartbeat
curl -s --max-time 10 https://<sandbox-url>/api/heartbeat

Success criteria: Heartbeat returns 200 within 60 seconds of restart.

Playbook 2: Environment Variable Fix

When: "undefined" errors, "Invalid API key", missing config

# Check current env vars
clawnet bot exec <bot-name> 'env | sort'

# Check specific vars (show only first 8 chars for security)
clawnet bot exec <bot-name> 'echo $OPENAI_API_KEY | head -c 8'
clawnet bot exec <bot-name> 'echo $ANTHROPIC_API_KEY | head -c 8'

# Set missing or update env var
clawnet bot env set <bot-name> VAR_NAME "value"

# Remove a bad env var
clawnet bot env remove <bot-name> VAR_NAME

# Restart after env change
clawnet bot restart <bot-name>

IMPORTANT: Never print full API keys. Only show first 8 characters to confirm they're set.

Playbook 3: Dependency Fix

When: "Module not found", "ERR_MODULE_NOT_FOUND", package errors

# Check package.json
clawnet bot exec <bot-name> 'cat /app/package.json'

# Check if node_modules exists and looks healthy
clawnet bot exec <bot-name> 'ls -la /app/node_modules/.package-lock.json'

# Reinstall dependencies
clawnet bot exec <bot-name> 'cd /app && bun install'

# If bun install fails, try cleaning first
clawnet bot exec <bot-name> 'cd /app && rm -rf node_modules && bun install'

# Restart after dependency fix
clawnet bot restart <bot-name>

Playbook 4: Snapshot Recovery (Full Redeploy)

When: Bot is totally wedged, previous fixes didn't work, or code is corrupted

# Stop the broken bot
clawnet bot stop <bot-name>

# Redeploy from template
clawnet bot deploy --name <bot-name> --yes

# Verify the new deployment
curl -s --max-time 30 https://<sandbox-url>/api/heartbeat

# Check logs for clean startup
clawnet bot logs <bot-name>

Warning: Redeploying resets the sandbox. Any in-sandbox state not stored in Vercel Blob will be lost.

Playbook 5: Memory/Resource Issues

When: "heap out of memory", high memory warnings, slow responses

# Check memory usage
clawnet bot exec <bot-name> 'ps aux --sort=-%mem | head -5'

# Check disk usage
clawnet bot exec <bot-name> 'df -h /app'

# Kill runaway processes if any
clawnet bot exec <bot-name> 'kill -9 <pid>'

# Restart with clean state
clawnet bot restart <bot-name>

Playbook 6: Network/Connectivity Issues

When: "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND"

# Test DNS resolution from inside sandbox
clawnet bot exec <bot-name> 'nslookup api.openai.com'

# Test external connectivity
clawnet bot exec <bot-name> 'curl -s --max-time 5 https://api.openai.com'

# Check if the issue is specific to one external service
clawnet bot exec <bot-name> 'curl -s --max-time 5 https://api.anthropic.com'

If external services are down, this is not a bot issue — document and wait.

Post-Repair Verification (ALWAYS DO THIS)

After ANY repair, run this verification sequence:

# 1. Heartbeat check
curl -s --max-time 10 https://<sandbox-url>/api/heartbeat

# 2. Check logs for new errors (last 50 lines)
clawnet bot logs <bot-name> | tail -50

# 3. Fleet status to confirm it shows healthy
clawnet bot list

# 4. Wait 2 minutes and check again (catch delayed failures)
sleep 120
curl -s --max-time 10 https://<sandbox-url>/api/heartbeat

A repair is not complete until verification passes. If verification fails, escalate to the next playbook or alert the admin.