fabric-dataflow-migration-toolkit

Build bronze layer PySpark **or** Python notebooks that ingest raw data into Fabric lakehouses with Delta Lake format. Handle OData, Excel-via-SharePoint, CSV, Parquet, JSON, and API sources. Add ingestion metadata columns, enable schema evolution, and implement append-only audit trails. The `engine` input (`pyspark` default | `python`) selects the codegen idiom: PySpark/Spark-session or single-node polars + delta-rs. Output is always `.ipynb` (Jupyter JSON) for Fabric deployment, never `.py`. MUST BE USED when creating the first ingestion layer in a medallion architecture.

End-to-end Power BI Dataflow Gen1 → Microsoft Fabric medallion notebook migration orchestrator. Drives the full workflow: extract Gen1 dataflows, analyze M code (m-query-analyst), gather refactor decisions (migration-analyst), scaffold the Fabric medallion project, generate bronze + silver PySpark notebooks via builders, deploy via the Fabric CLI, validate (fabric-pipeline-validator), and maintain a single migration-design.md document that every stage reads and updates. MUST BE USED as the top-level agent for end-to-end Gen1 migration. Requires THREE user touch points: config Q&A, refactor Q&A, and a plan-mode approval. Run via `claude --agent fabric-dataflow-migration-toolkit:fabric-migration-orchestrator:fabric-migration-orchestrator`.

End-to-end Fabric migration validator. Invoked by the orchestrator at Stage 12. Runs static checks on every generated .ipynb notebook (valid JSON, lakehouse binding, read_bronze() contract for silver) and runtime checks against deployed lakehouses (row counts, schema match) when not in dry-run mode. Writes Section 10 (Validation Results) of `1 - Documentation/migration-design.md`. Does NOT write any other file.

Build silver layer notebooks (PySpark or Python engine) that clean, conform, and transform bronze Delta tables into analysis-ready datasets. Silver notebooks read exclusively from bronze tables via read_bronze() — never from external storage. Handle type casting, renaming, null handling, deduplication, and unpivot/pivot transforms. Use when creating the second transformation layer in a medallion architecture.

Mechanical Power Query M analysis. Parses .pq files exported from Power BI Dataflow Gen1 dataflows, classifies each query (output_entity / staging / transformation / parameter / helper), detects source type (CSV, Excel, AzureStorage, SQL Server, etc.), builds a dependency map, and scans for known M-conversion risk patterns. Produces JSON envelopes for the orchestrator to merge into the migration design document. NO USER INTERACTION — pure mechanical analysis. Spawned in two passes: inventory (Stage 3) and risk scan (Stage 4).

Extract Dataflow Gen1 (Power BI Dataflows) definitions from a Fabric/Power BI workspace and parse them into individual M query (.pq) files. Use when the user wants to: (1) export dataflow definitions from a Power BI workspace, (2) extract M code / Power Query code from Dataflow Gen1, (3) migrate Dataflow Gen1 to notebooks or Dataflow Gen2, (4) inventory all queries in a workspace's dataflows, (5) parse exported dataflow JSON files into .pq files. Triggers on mentions of "dataflow gen1", "export dataflow", "extract M code from dataflow", "dataflow migration", "Power BI dataflow export".

Execute Fabric CLI (fab) commands for notebook deployment, execution, and workspace management. Use when deploying notebooks to Fabric, running notebooks, listing workspace items, or managing Fabric resources. Supports authentication, import/export, synchronous job execution, and JSON output.

Query Fabric lakehouse SQL analytics endpoints with read-only access. Validate data after notebook execution, inspect table schemas, run row count checks, and export results to CSV. Use when validating pipeline output, debugging data issues, or exporting query results from Fabric lakehouses. Requires ODBC Driver 18 and Azure authentication.

Deploy multiple .ipynb notebook files to a Microsoft Fabric workspace in batch. Wraps the Fabric REST API (via `fab api`) for createOrUpdate notebook operations with optional folder placement. Supports dry-run, glob patterns, retry on rate-limit, and structured JSON output. Use when migrating multiple notebooks at once or as part of an orchestrated deployment pipeline.

Pre-flight validation for the fabric-dataflow-migration-toolkit. Verifies that the Fabric CLI (`fab`) is installed and authenticated, that Azure auth is current, and (optionally) that a target workspace + lakehouses exist and are accessible. Use BEFORE running long migrations to fail fast on auth/config issues. Outputs human-readable status or a JSON envelope for orchestrator integration.

3 hooks across 2 events