Create and repair Redpanda Connect pipeline configurations and Bloblang transformation scripts using natural language. Automates component search, config generation, and script debugging with optional sample data validation.
Create and test Bloblang transformation scripts from natural language descriptions
Create or repair Redpanda Connect configurations with interactive guidance and validation
Search for Redpanda Connect components (inputs, outputs, processors, caches, rate-limits, buffers, metrics, tracers)
This skill should be used when users need to create or debug Bloblang transformation scripts. Trigger when users ask about transforming data, mapping fields, parsing JSON/CSV/XML, converting timestamps, filtering arrays, or mention "bloblang", "blobl", "mapping processor", or describe any data transformation need like "convert this to that" or "transform my JSON".
This skill should be used when users need to discover Redpanda Connect components for their streaming pipelines. Trigger when users ask about finding inputs, outputs, processors, or other components, or when they mention specific technologies like "kafka consumer", "postgres output", "http server", or ask "which component should I use for X".
This skill should be used when users need to create or fix Redpanda Connect pipeline configurations. Trigger when users mention "config", "pipeline", "YAML", "create a config", "fix my config", "validate my pipeline", or describe a streaming pipeline need like "read from Kafka and write to S3".
Own this plugin?
Verify ownership to unlock analytics, metadata editing, and a verified badge. GitHub access is read-only (username + org membership).
Sign in to claimOwn this plugin?
Verify ownership to unlock analytics, metadata editing, and a verified badge. GitHub access is read-only (username + org membership).
Sign in to claimBased on adoption, maintenance, documentation, and repository signals. Not a security audit or endorsement.
[![Build Status][actions-badge]][actions-url] [![Apache V2 API][godoc-badge]][godoc-url-apache] [![Enterprise API][godoc-badge]][godoc-url-enterprise]
Redpanda Connect is a stream processor that moves data between a wide range of [sources][inputs] and [sinks][outputs], with support for [hydration, enrichment, transformation, and filtering][processors] along the way.
That includes a rich set of change-data-capture (CDC) connectors — for Postgres, MySQL, MongoDB, Oracle, MSSQL, and more — so database changes can flow through your pipelines as first-class events.
It uses [Bloblang][bloblang-about] for mapping, runs as a single static binary or container image, and is easy to operate and monitor.
Stream Postgres changes into Apache Iceberg tables on S3, one Iceberg table per source table:
input:
postgres_cdc:
dsn: [REDACTED:DB Connection String]
schema: public
tables: [ orders, customers ]
stream_snapshot: true
output:
iceberg:
catalog:
url: https://glue.us-east-1.amazonaws.com/iceberg
warehouse: "123456789012"
auth:
aws_sigv4:
region: us-east-1
service: glue
namespace: cdc
table: ${! meta("table") }
storage:
aws_s3:
bucket: my-iceberg-warehouse
region: us-east-1
schema_evolution:
enabled: true
table_location: s3://my-iceberg-warehouse/cdc/
Linux:
curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/
macOS (Homebrew):
brew install redpanda-data/tap/redpanda
Docker:
docker pull docker.redpanda.com/redpandadata/connect
See the [getting started guide][getting-started] for more options.
rpk connect run ./config.yaml
With Docker:
# From a config file
docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run
# With inline overrides
docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
-s "input.type=http_server" \
-s "output.type=kafka" \
-s "output.kafka.addresses=kafka-server:9092" \
-s "output.kafka.topic=redpanda_topic"
The catalog includes AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob, Queue, Table), GCP (Pub/Sub, Cloud Storage, BigQuery), Kafka, NATS (JetStream, Streaming), NSQ, MQTT, AMQP 0.91 (RabbitMQ), AMQP 1, Redis, Cassandra, Elasticsearch, HDFS, HTTP (server, client, websockets), MongoDB, and SQL (MySQL, PostgreSQL, ClickHouse, MSSQL) — and a lot more in the [components documentation][about-categories].
Delivery guarantees can be a tricky subject. Redpanda Connect processes and acknowledges messages using an in-process transaction model with no disk-persisted state, so when it's connecting at-least-once sources and sinks it can guarantee at-least-once delivery — even through crashes, disk corruption, or other server faults.
That's the default, with no caveats, which keeps deployment and scaling straightforward.
Two HTTP endpoints are exposed for orchestration probes:
/ping — liveness probe; always returns 200./ready — readiness probe; returns 200 once both input and output are connected, otherwise 503.Redpanda Connect [exposes metrics][metrics] to Statsd, Prometheus, a JSON HTTP endpoint, and [other backends][metrics].
OpenTelemetry traces are [emitted natively][tracers], so you can visualize what's happening inside a pipeline end-to-end.
Redpanda Connect ships with tooling for configuration discovery, debugging, and organization — see the [configuration guide][config-doc].
Requires a currently supported Go version:
git clone [email protected]:redpanda-data/connect
cd connect
task build:all
React/TypeScript frontend + Go backend enforcement harness. 92 wired hooks across 101 scripts enforce patterns on every edit. Green != done: `test-warning-check` surfaces warnings on passing test/lint/type runs (DeprecationWarning, React `act()`, unhandled rejection, `@ts-ignore`); `ci-warning-audit` Stop-hook scans `gh run view --log` on green CI for deprecations / console warnings / skipped tests. 66 skills from TDD through CI-green PR, including /steelman anti-sycophancy guard + /snyk-ux-security per-path vuln sweep (JS + Go ecosystems; exploitability-triage first gate via `bun why`/`go mod why`/`govulncheck`; top-level direct dep bump first, parent dep bump second, resolutions/overrides/replace as last resort only with follow-up TODO to remove; React 18 pin + changelog-walked major bumps + no-deferral escalation; Go `snyk test --file=go.mod` + `go get -u` + `go mod tidy` + `govulncheck` verify). Session exit blocked while PR review threads remain unresolved. TypeScript escape hatches blocked at Edit. tsconfig strictness weakening blocked. Worktree isolation + branch safety hook-enforced. bun.lock + yarn.lock parity enforced (Snyk IO doesn't parse bun.lock). 21 LLM failure modes enforced: 7 Karpathy single-agent + 14 MAST multi-agent (Cemri et al. NeurIPS 2025). OWASP + STRIDE subset + snyk/bun audit. Core Web Vitals perf gate. MCP ban with CLI redirect (~20x token savings). Agent-browser wrap (~91% token reduction for AI browsing). 3-hat plan review (product/engineering/design). 9 agents (adds plan hats + karpathy reference), 5 routines. Opus 4.7 tuned, POSIX-friendly.
Agent Skills for Redpanda's four products — Streaming (Kafka-compatible engine), SQL (Oxla), Connect (incl. CDC connectors), and Cloud (Serverless, BYOC, Dedicated) — plus the rpk CLI. Grounded in Redpanda source, docs, and APIs.
npx claudepluginhub redpanda-data/connect --plugin redpanda-connectLenses Kafka agent skills (topic audit, consumer lag, perf review, schema, security, connectors, DLQ, python client scaffold) powered by the Lenses MCP server
Skills for streaming application developers, covering Kafka and Flink client libraries and Schema Registry
Transform raw dlt pipeline data into a Canonical Data Model. Build an ontology, design a CDM with Kimball dimensional modeling, write @dlt.hub.transformation functions, and validate the output.
Skills for drt — Reverse ETL for the code-first data stack
Implement event-driven APIs with message queues and event streaming
Official Claude plugin for Google Cloud Bigtable. Manage instances/tables, design schemas, query data using SQL or client libraries, and more