From example-skills
Designs ETL/ELT data pipelines with extraction, transformation, loading patterns, orchestration via Airflow/dbt/Kafka, error handling, and data quality validation.
How this skill is triggered — by the user, by Claude, or both
Slash command
/example-skills:data-pipeline-architectThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
This skill provides guidance for designing robust, scalable data pipelines that move data reliably from sources to destinations.
This skill provides guidance for designing robust, scalable data pipelines that move data reliably from sources to destinations.
To begin pipeline design, gather:
Batch Pipelines - For periodic bulk processing:
Streaming Pipelines - For real-time requirements:
Hybrid Approaches - Lambda or Kappa architecture:
ETL (Transform before Load):
ELT (Transform after Load):
Extraction Layer:
Transformation Layer:
Loading Layer:
┌─────────────────────────────────────────────────────────┐
│ Pipeline Execution │
├─────────────────────────────────────────────────────────┤
│ ┌─────────┐ ┌───────────┐ ┌──────────┐ │
│ │ Extract │───▶│ Transform │───▶│ Load │ │
│ └────┬────┘ └─────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────┐ ┌───────────┐ ┌──────────┐ │
│ │ Retry │ │ Dead Letter│ │ Rollback │ │
│ │ w/Backoff│ │ Queue │ │ Checkpoint│ │
│ └─────────┘ └───────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
Implement checks at each stage:
| Stage | Check Type | Example |
|---|---|---|
| Extract | Completeness | Row count matches source |
| Extract | Freshness | Data timestamp within SLA |
| Transform | Validity | Values in expected ranges |
| Transform | Uniqueness | Primary keys unique |
| Load | Reconciliation | Target matches source totals |
| Load | Integrity | Foreign keys valid |
Essential metrics to track:
Alert on:
-- Timestamp-based incremental
SELECT * FROM source
WHERE updated_at > {{ last_run_timestamp }}
-- CDC-based (Change Data Capture)
-- Captures inserts, updates, deletes from transaction log
-- Delete + Insert pattern
DELETE FROM target WHERE date_partition = '2024-01-15';
INSERT INTO target SELECT * FROM staging WHERE date_partition = '2024-01-15';
-- Merge/Upsert pattern
MERGE INTO target t
USING staging s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT ...
references/orchestration-patterns.md - Airflow, Dagster, Prefect patternsreferences/data-quality-checks.md - Validation frameworks and rulesreferences/pipeline-templates.md - Common pipeline architecturesnpx claudepluginhub a-organvm/a-i--skills --plugin document-skillsProvides a checklist for code reviews covering functionality, security, performance, maintainability, tests, and quality. Use for pull requests, audits, team standards, and developer training.