dataproduct-builder-databricks

dataproduct-builder-databricks

Skills for your favorite coding agent that build data products on Databricks with Declarative Automation Bundles and Lakeflow Spark Declarative Pipelines, integrated with Entropy Data.

Sibling plugin to dataproduct-builder-dbt. Pick the one that matches your stack.

It also supports a contract-driven approach: specify your requirements as a data contract, and the builder implements the data product in minutes.

Skills

The plugin ships nine skills:

• dataproduct-init scaffolds a new Databricks bundle from scratch: databricks.yml, a serverless Lakeflow pipeline resource, a Lakeflow Job that schedules it, the src/{input_ports,transformations,output_ports/v1}/ layout, pyproject.toml, README. Shells out to databricks bundle init lakeflow-pipelines and overlays the Entropy Data pieces on top.
• dataproduct-implement analyzes the input and output data contracts, generates @dp.table Python files under src/output_ports/v1/, resolves access agreements into @dp.view input ports under src/input_ports/, and runs databricks bundle validate to verify the result.
• dataproduct-deploy wraps databricks bundle validate → deploy → run with target selection and polls the pipeline run for completion.
• dataproduct-share reads entropy-data access list --provider-dataproduct and applies Unity Catalog GRANT SELECT for internal consumers or creates a Delta Share for external ones, per active access agreement.
• entropy-data-publish audits an existing bundle against the Entropy Data reference layout, adds missing ODPS/ODCS files, generates the GitHub Actions publish workflow, and registers git connections.
• datacontract-edit edits an output-port src/output_ports/v<N>/*.odcs.yaml using natural language and classifies the change as breaking or additive.
• datacontract-test runs datacontract test to verify the live data still matches the schema and quality rules.
• dataproduct-exampledata extracts sample rows, drops PII columns flagged in the contract, and uploads the scrubbed sample to Entropy Data.
• entropy-data-teams lists the teams configured in Entropy Data so the user can pick an owner.

Install

The skills are plain markdown, any coding agent that can read instruction files can run them.

For major coding agents, those can be installed as a plugin:

Claude Code

In your terminal:

claude plugin marketplace add https://github.com/entropy-data/dataproduct-builder-databricks
claude plugin install dataproduct-builder-databricks@dataproduct-builder-databricks -s project

OpenAI Codex

In your terminal:

codex plugin marketplace add https://github.com/entropy-data/dataproduct-builder-databricks
codex plugin add dataproduct-builder-databricks@dataproduct-builder-databricks

GitHub Copilot CLI

In your terminal:

copilot plugin marketplace add https://github.com/entropy-data/dataproduct-builder-databricks
copilot plugin install dataproduct-builder-databricks@dataproduct-builder-databricks

Other agents (Cursor, Aider, etc.)

Any agent that reads AGENTS.md picks up the routing manifest. Alternatively, copy the skills to the directory that your coding agent expects.

Connect

The skills authenticate against three systems. Configure each once.

Databricks — workspace auth via the Databricks CLI:

brew install databricks/tap/databricks
databricks auth login --host https://<your-workspace>.cloud.databricks.com

Entropy Data — API key registered with the entropy-data CLI (requires uv).

The skills use a per-project venv for both entropy-data and datacontract. After dataproduct-init scaffolds a project, run uv sync from the project root to install both CLIs at the pinned versions, then invoke them as uv run entropy-data … / uv run datacontract ….

The one exception is the first call to dataproduct-init itself, which runs against an empty directory (no pyproject.toml, no venv yet) and needs entropy-data available globally for its lookup step. Install once per machine:

uv tool install --upgrade entropy-data
entropy-data connection add default --api-key <your-api-key> --host <your-entropy-data-host>

Create a user-scoped key in the Entropy Data web UI (Organization Settings → API Keys → Create new API key, scope User (personal token)). For CI workflows, add a connection with a team-scoped or organization-scoped key.