data-pipeline-builder

Status: v0.1 — minimum viable. Full skill (project-specific templates) lands in Phase 2.

When to invoke

Trigger files: any build_*.py / build_*.ts that reads xlsx, or new *.xlsx → *.json dataflow being designed. Trigger words: "convert the xlsx", "build features", "regenerate the data", "the xlsx changed".

The Rule

Any xlsx → engine data pipeline must have these three stages, in this order:

1. Reader — pure function xlsx_path → list[dict]. No engine imports. Pure data extraction.
2. Schema validator — typed schema (zod, pydantic, dataclass). Reject malformed rows with row number + reason; do not silently coerce.
3. Emitter — writes versioned JSON to data/v<version>/<name>.json. Never overwrites a prior version (see versioned-design-doc-keeper).

A diff job (scripts/diff-pipeline-output.sh) runs after the pipeline and shows:

• New rows added
• Rows removed
• Cell-level diffs on retained rows

If the diff is non-empty, surface it to the user before committing the new JSON.

Minimum viable checklist (v0.1)

1. Locate the existing pipeline in the project (suixi has build_features_from_sources.py + build_indexes.py).
2. Confirm the three stages are separated. If reader/validator/emitter are tangled, refactor before adding new functionality.
3. Confirm versioned output paths — emitter writes to data/v<version>/, never to a flat data/.
4. Diff before commit — run the pipeline, save output to a temp dir, diff against the current data/v<version>/, surface changes.
5. Wire to rules-version-auditor — pipeline output is one of the three sources audited.

Reference patterns (lifted from suixi_battle)

• Reader stage: build_features_from_sources.py reads maps/sources/*.xlsx, returns list of hex feature dicts.
• Validator: implicit (Python dicts) — TODO upgrade to dataclass.
• Emitter: writes to maps/data/<version>/features.json.
• Diff: not yet automated — currently manual git diff.

Anti-patterns

• Reader that also imports engine code — couples data extraction to runtime, can't run in CI without full env.
• Validator that fills in defaults silently — turns malformed rows into wrong rows.
• Emitter that overwrites — see versioned-design-doc-keeper.
• "I'll just regenerate everything" — no. Regenerate only what changed, diff, commit incrementally.

Cross-references

• bg-superpowers:rules-version-auditor checks pipeline output is consistent with xlsx and code.
• bg-superpowers:versioned-design-doc-keeper ensures output paths are versioned.
• superpowers:test-driven-development for pipeline test design — write a failing test against expected output before adding a new column.

TODO (Phase 2 full implementation)

• Generate the three-stage skeleton from a project-local config.
• Auto-wire the diff job into pre-commit.
• Schema validator templates for TypeScript (zod) and Python (pydantic).