gcp-dataform-rest-api-deploy

GCP Dataform REST API Deploy

Deploying Dataform .sqlx files via REST API requires base64-encoding, undocumented field names, and a polling loop — six places to get it wrong. This skill handles the full lifecycle without the Dataform CLI.

Quick Start

You: Deploy this new .sqlx model to Dataform and run it
Claude: [writes file to workspace → commits → pushes → compiles → invokes → polls until SUCCEEDED]

Or use the documented bash patterns directly in CI/CD pipelines, Cloud Workflows, or deployment scripts.

Installation

Claude Code

# Plugin install (recommended)
/plugin marketplace add wan-huiyan/gcp-dataform-rest-api-deploy
/plugin install gcp-dataform-rest-api-deploy@wan-huiyan-gcp-dataform-rest-api-deploy

# Git clone (always works)
git clone https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy.git ~/.claude/skills/gcp-dataform-rest-api-deploy

Cursor (2.4+)

# Per-project rule (most reliable)
mkdir -p .cursor/rules
# Copy SKILL.md content into .cursor/rules/gcp-dataform-rest-api-deploy.mdc with alwaysApply: true

# npx skills CLI
npx skills add wan-huiyan/gcp-dataform-rest-api-deploy --global

# Manual global install
git clone https://github.com/wan-huiyan/gcp-dataform-rest-api-deploy.git ~/.cursor/skills/gcp-dataform-rest-api-deploy

What You Get

• Complete 6-step lifecycle: writeFile → commit → push → compile → invoke → poll
• Failure diagnostics: Action-level error extraction showing exact SQL errors
• 6 documented gotchas that cause silent failures or cryptic errors
• Cloud Workflows integration notes for YAML-based orchestration

The Problem

The Dataform REST API (v1beta1) has non-obvious patterns that cause frustrating failures:

What Goes Wrong	Symptom	Root Cause
404 on API calls	Not Found on valid workspace	Colon-prefixed actions (:writeFile) get URL-encoded or appended wrong
Commit fails	Unknown name "fileOperations"	Wrong field name — API uses paths, not fileOperations
File write seems to fail	Empty {} response	That's actually success — empty response = OK
Invocation runs wrong SQL	Old version executes	Forgot to push workspace before compiling
Region mismatch	Various auth/not-found errors	Dataform region often differs from Cloud Run region
Date parsing errors	No matching signature for TIMESTAMP_SECONDS	Column types in BQ don't match what you'd expect from field names

How It Works

Step	API Endpoint	What It Does
1. Write	{workspace}:writeFile	Upload base64-encoded .sqlx to workspace
2. Commit	{workspace}:commit	Commit with author, message, and file paths
3. Push	{workspace}:push	Push workspace changes to default branch
4. Compile	{repo}/compilationResults	Compile from release config
5. Invoke	{repo}/workflowInvocations	Run specific targets
6. Poll	GET {invocation}	Poll state until SUCCEEDED or FAILED

Key Gotchas

1. Colon-Prefixed Action Endpoints

# WRONG — colon gets eaten or URL-encoded
${BASE}/${WS_PATH}writeFile

# RIGHT — colon is part of the path
${BASE}/${WS_PATH}:writeFile

2. Base64 Encoding Required

writeFile expects contents as base64, not raw text:

ENCODED=$(base64 -i path/to/file.sqlx)

3. Commit Uses paths, Not fileOperations

// WRONG
{"fileOperations": {"definitions/model.sqlx": {"writeFile": {}}}}

// RIGHT
{"paths": ["definitions/model.sqlx"], "author": {...}, "commitMessage": "..."}

4. Empty {} = Success

writeFile, commit, and push all return empty {} on success. Don't treat it as an error.

5. Dataform Region != Cloud Run Region

Dataform repos are often in europe-north1 while Cloud Run services are in us-central1. Always check your Dataform repo settings.

6. Release Config Full Path Required

projects/{project}/locations/{region}/repositories/{repo}/releaseConfigs/{config}

Checking Failure Details