Back up the Kafka behind your MSK Connect pipelines
Amazon MSK Connect is AWS's managed Kafka Connect: it runs your connectors so you don't operate a Connect cluster. But the topics those connectors read, and the offsets that track their progress, are still yours to protect. OSO Kafka Backup captures both — and restores them to a precise moment in time.
What MSK Connect handles — and what stays yours
Amazon MSK Connect is AWS's managed Kafka Connect service for Amazon MSK and compatible self-managed Kafka clusters. It deploys connector plugins, runs the workers, scales them between the minimum and maximum you configure, and bills per MCU-hour. For running an S3 sink, a Debezium source, or any Connect pipeline without operating Connect yourself, it does the job well.
What it does not do is back anything up. The shared-responsibility split looks like this:
| MSK Connect handles | Still yours to protect | |
|---|---|---|
| Connect workers | Provisioning, scaling, patching | — |
| Connector code | Plugin deployment and restarts | Connector configuration you author |
| Topic data | — | Every topic your pipelines read and write |
| Pipeline progress | Commits offsets for you | The offsets themselves, if the cluster is damaged |
If a bad deploy poisons a topic, or someone deletes one that a connector depends on, MSK Connect faithfully keeps processing the broken state. Recovery needs an independent, point-in-time copy — the disaster recovery use cases walk through those failure modes.
Backing up the topics behind your connectors
OSO Kafka Backup connects to your MSK cluster as a standard Kafka client — no Connect
cluster, no plugin packaging, a single CLI. Scope the backup to your pipeline topics
with include and exclude patterns, and offsets for every consumer group are captured
alongside the data. That includes the connect-* groups MSK Connect sink connectors
use to track their progress.
- Pipeline backup
- Run it
mode: backup
backup_id: msk-connect-pipelines
source:
bootstrap_servers:
- b-1.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
- b-2.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
security:
security_protocol: SASL_SSL
sasl_mechanism: SCRAM-SHA512
sasl_username: ${MSK_SCRAM_USER}
sasl_password: ${MSK_SCRAM_PASSWORD}
topics:
include:
- "orders.*"
- "payments.*"
exclude:
- "__consumer_offsets"
storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: msk-connect/pipelines
backup:
compression: zstd
include_offset_headers: true
kafka-backup backup --config msk-connect-backup.yaml
After a restore, sink pipelines resume from their preserved consumer group offsets instead of reprocessing from earliest or silently skipping to latest. Source connector offsets live in an internal Connect offsets topic on the cluster, which you can back up like any other topic. The configuration reference covers every topic-selection and offset option.
About using the S3 sink connector as a backup
AWS documents a do-it-yourself pattern that chains an S3 sink connector with a source connector to copy topic data out and back. It works, and if you already run MSK Connect it is tempting. But the sink writes query-oriented files with no point-in-time restore, no consumer offset capture, and no topic configuration — you own the restore tooling. The S3 sink connector comparison and our S3 sink setup guide cover where that approach stops being a backup.
Frequently asked questions
Does Amazon MSK Connect back up my Kafka topics?
No. MSK Connect runs and scales Kafka Connect workers for you, but it provides no backup or restore capability for topic data. Protecting the topics and offsets behind your pipelines remains your responsibility under the AWS shared-responsibility model.
Can I use an MSK Connect S3 sink connector as a Kafka backup?
You can copy topic data to S3 with it, and AWS documents a sink-plus-source pattern for restore. But the output is query-oriented files with no point-in-time recovery, no consumer group offset capture, and no topic configuration — the restore tooling becomes your project. A purpose-built backup captures and restores all three as one unit.
Does OSO Kafka Backup preserve MSK Connect connector offsets?
Sink connectors track progress as regular Kafka consumer group offsets, and consumer group offset preservation is built in — after a restore those pipelines resume where they left off. Source connector offsets live in an internal Connect offsets topic on the cluster, which can be included in backups like any other topic.
Does OSO Kafka Backup run as a Kafka Connect plugin on MSK Connect?
No, and it does not need to. It is a standalone CLI (with a Kubernetes operator) that connects to your MSK cluster as an ordinary Kafka client over SASL/SCRAM or mutual TLS. There is no Connect cluster to size, no plugin to package, and no MCU-hour cost for the backup path.
Does this work with self-managed Kafka Connect too?
Yes. The same backup captures topics and consumer group offsets whether your connectors run on MSK Connect, a self-managed Connect cluster, or no Connect at all — it operates at the Kafka protocol level, independent of how your pipelines are deployed.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.