From opengov-garden
Manages the complete Extract-Transform-Load pipeline for OpenGov procurement data extraction.
How this skill is triggered — by the user, by Claude, or both
Slash command
/opengov-garden:etl-pipelineThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Manages the complete Extract-Transform-Load pipeline for OpenGov procurement data extraction.
Manages the complete Extract-Transform-Load pipeline for OpenGov procurement data extraction.
Authentication & Session Management
Project Inventory Collection
opengov_projects tableOpportunity Extraction
Incremental Updates
# Run complete ETL pipeline
python main.py run
# Or via unified engine
python unified_engine.py --all
# Step 1: Authentication
python scripts/etl/step_1_auth_session.py
# Step 2: Project inventory
python scripts/etl/fetch_project_data.py
# Step 3: Sequential extraction
python scripts/etl/step_3_sequential_extraction.py
# Step 4: Gateway API extraction (if available)
python scripts/etl/gateway_api_extractor.py
# Long-running extraction with timeout
timeout 600 python scripts/etl/step_3_sequential_extraction.py
# Quick test run
timeout 30 python scripts/etl/step_1_auth_session.py
etl-atomic-writes.md: Ensure atomic database updatesetl-retry-backoff.md: Handle transient failuresetl-idempotency-patterns.md: Safe re-runsetl-incremental-loads.md: Efficient delta processingdata-storage-formats.md: Canonical output pathssupabase startconfig/env/.envplaywright installdata/db/opengov_state.dbopengov schemadata/exports/project-artifacts/<project_id>/logs/opengov_<timestamp>.log.agent/skills/opengov-harvester/anti-detection/SKILL.mdnpx claudepluginhub heaney-investments/opengov-garden --plugin opengov-gardenDesigns data pipelines and ETL processes covering extraction, transformation, loading, data quality checks, orchestration, and patterns for batch, streaming, CDC, ELT. Useful for building pipelines, data flows, syncing, or moving data between systems.
Builds ETL/ELT data pipelines with extraction, transformation, loading, error handling, scheduling, and monitoring. Activates for build ETL, data pipeline, move data from X to Y, or sync data requests.
Extracts structured knowledge from Airtable bases into linked entity pages in a vault. Uses a TypeScript/Bun pipeline with Sonnet workers and Opus reviewers for quality gating.