Redpanda Connect
[![Build Status][actions-badge]][actions-url]
[![Apache V2 API][godoc-badge]][godoc-url-apache]
[![Enterprise API][godoc-badge]][godoc-url-enterprise]
Redpanda Connect is a stream processor that moves data between a wide range of [sources][inputs] and [sinks][outputs], with support for [hydration, enrichment, transformation, and filtering][processors] along the way.
That includes a rich set of change-data-capture (CDC) connectors — for Postgres, MySQL, MongoDB, Oracle, MSSQL, and more — so database changes can flow through your pipelines as first-class events.
It uses [Bloblang][bloblang-about] for mapping, runs as a single static binary or container image, and is easy to operate and monitor.
Highlights
- Declarative pipelines — a stream topology fits in a single YAML file.
- At-least-once delivery by default — in-process transactions, no disk state required.
- A large connector catalog — cloud services, message brokers, databases, HTTP, and more.
- First-class CDC — change-data-capture connectors for Postgres, MySQL, MongoDB, Oracle, and MSSQL.
- Bloblang — a mapping language designed for stream data.
- Cloud-friendly — stateless and horizontally scalable, with metrics and tracing built in.
Example
Stream Postgres changes into Apache Iceberg tables on S3, one Iceberg table per source table:
input:
postgres_cdc:
dsn: [REDACTED:DB Connection String]
schema: public
tables: [ orders, customers ]
stream_snapshot: true
output:
iceberg:
catalog:
url: https://glue.us-east-1.amazonaws.com/iceberg
warehouse: "123456789012"
auth:
aws_sigv4:
region: us-east-1
service: glue
namespace: cdc
table: ${! meta("table") }
storage:
aws_s3:
bucket: my-iceberg-warehouse
region: us-east-1
schema_evolution:
enabled: true
table_location: s3://my-iceberg-warehouse/cdc/
Quickstart
Install
Linux:
curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/
macOS (Homebrew):
brew install redpanda-data/tap/redpanda
Docker:
docker pull docker.redpanda.com/redpandadata/connect
See the [getting started guide][getting-started] for more options.
Run
rpk connect run ./config.yaml
With Docker:
# From a config file
docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run
# With inline overrides
docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
-s "input.type=http_server" \
-s "output.type=kafka" \
-s "output.kafka.addresses=kafka-server:9092" \
-s "output.kafka.topic=redpanda_topic"
Connectors
The catalog includes AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob, Queue, Table), GCP (Pub/Sub, Cloud Storage, BigQuery), Kafka, NATS (JetStream, Streaming), NSQ, MQTT, AMQP 0.91 (RabbitMQ), AMQP 1, Redis, Cassandra, Elasticsearch, HDFS, HTTP (server, client, websockets), MongoDB, and SQL (MySQL, PostgreSQL, ClickHouse, MSSQL) — and a lot more in the [components documentation][about-categories].
Delivery guarantees
Delivery guarantees can be a tricky subject. Redpanda Connect processes and acknowledges messages using an in-process transaction model with no disk-persisted state, so when it's connecting at-least-once sources and sinks it can guarantee at-least-once delivery — even through crashes, disk corruption, or other server faults.
That's the default, with no caveats, which keeps deployment and scaling straightforward.
Observability
Health checks
Two HTTP endpoints are exposed for orchestration probes:
/ping — liveness probe; always returns 200./ready — readiness probe; returns 200 once both input and output are connected, otherwise 503.
Metrics
Redpanda Connect [exposes metrics][metrics] to Statsd, Prometheus, a JSON HTTP endpoint, and [other backends][metrics].
Tracing
OpenTelemetry traces are [emitted natively][tracers], so you can visualize what's happening inside a pipeline end-to-end.
Configuration
Redpanda Connect ships with tooling for configuration discovery, debugging, and organization — see the [configuration guide][config-doc].
Documentation
- [General documentation][general-docs]
- [Bloblang language guide][bloblang-about]
- Public Go APIs for building custom plugins
Build from source
Requires a currently supported Go version:
git clone [email protected]:redpanda-data/connect
cd connect
task build:all
Plugins with external dependencies
