Back up Amazon MSK to S3
Amazon MSK runs and patches your brokers, but under the shared-responsibility model your topic data is still yours to protect. Point OSO Kafka Backup at your MSK bootstrap servers and stream compressed topic records, consumer group offsets, and configuration into S3 — then restore any of it to a precise moment in time.
Who backs up your MSK data
Amazon MSK is a managed service: AWS handles broker provisioning, patching, and availability. It does not take logical backups of your topics. If a producer bug corrupts a topic or someone deletes the wrong one, MSK's replication faithfully keeps the broken state. That is the gap a backup fills — see disaster recovery use cases for the failure modes.
OSO Kafka Backup connects to MSK as an ordinary Kafka client over the broker endpoints you already use. It writes compressed backups to S3 in the same region, so the data never leaves AWS and lands in a store isolated from your cluster's failure domain.
Connecting to MSK
MSK exposes SASL/SCRAM on port 9096 and mutual TLS on port 9094. Both are supported natively. Use SASL/SCRAM with credentials from AWS Secrets Manager, or mutual TLS with a client certificate — whichever your cluster is configured for. The AWS S3 setup guide covers the bucket and IAM side.
- SASL/SCRAM
- Mutual TLS
- Run it
mode: backup
backup_id: msk-prod-daily
source:
bootstrap_servers:
- b-1.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
- b-2.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
security:
security_protocol: SASL_SSL
sasl_mechanism: SCRAM-SHA512
sasl_username: ${MSK_SCRAM_USER}
sasl_password: ${MSK_SCRAM_PASSWORD}
storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: msk/production
backup:
compression: zstd
include_offset_headers: true
mode: backup
backup_id: msk-prod-daily
source:
bootstrap_servers:
- b-1.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9094
- b-2.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9094
security:
security_protocol: SSL
ssl_ca_location: /etc/kafka/msk-ca.pem
ssl_certificate_location: /etc/kafka/client.crt
ssl_key_location: /etc/kafka/client.key
storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: msk/production
kafka-backup backup --config msk-backup.yaml
What gets backed up
Every backup captures topic records with their timestamps and headers, consumer group offsets, and topic configuration, compressed with Zstd or LZ4. Restores can target the original MSK cluster or a new one, and can stop at a precise millisecond — which is what makes recovery from a bad deploy possible. Running the backup on EC2 or EKS in the same VPC keeps the read path fast and the transfer cost low.
Migrating an MSK cluster from ZooKeeper to KRaft is a separate job. For that controlled cutover with offset continuity, see the Enterprise MSK KRaft migration pipeline.
Frequently asked questions
Does Amazon MSK back up my Kafka topics automatically?
No. MSK manages broker availability and patching, but under the AWS shared-responsibility model your topic data is yours to protect. MSK provides no logical, point-in-time topic backup, so an accidental delete or bad deploy is not recoverable without a separate backup.
How does OSO Kafka Backup authenticate to MSK?
As a standard Kafka client. Use SASL_SSL with the SCRAM-SHA512 mechanism and credentials from AWS Secrets Manager on port 9096, or mutual TLS with a client certificate on port 9094 — whichever your MSK cluster is configured for.
Where are MSK backups stored?
In an S3 bucket you choose, written under a configurable key prefix. Keeping the bucket in the same region as the MSK cluster keeps transfer fast and cheap, and S3 lifecycle policies can tier or expire older backups to match your retention policy.
Can I restore an MSK backup to a different cluster?
Yes. Restores can target the original MSK cluster or a brand new one — including a cluster in another region or account — with topic and partition remapping, so backup and restore doubles as an MSK migration path.
Is this the same as MSK Replicator?
No. MSK Replicator copies data between clusters in real time, so it also replicates corrupted or deleted data. Point-in-time backup restores the state from before an incident. Many teams run both — see the MSK Replicator comparison.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.