Backup Kafka to S3 in one config file
Point OSO Kafka Backup at your cluster and an S3 bucket. It streams compressed topic data, consumer group offsets, and topic configuration into the bucket — restorable to a precise millisecond.
How to backup Kafka to S3
To backup Kafka to S3 you need three things: a config file naming your brokers and bucket, IAM access to that bucket, and one CLI command. The backup captures topic records with timestamps and headers, consumer group offsets, and topic configuration — compressed with Zstd before upload.
Credentials resolve through the standard AWS credential chain (environment variables, instance profile, IRSA), so nothing sensitive lives in the file. The AWS S3 setup guide covers the IAM policy and bucket configuration.
- backup.yaml
- Run it
- restore.yaml
mode: backup
backup_id: "orders-to-s3"
source:
bootstrap_servers:
- broker-1.kafka.svc:9092
topics:
include:
- orders
- payments
storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/production
backup:
compression: zstd
kafka-backup backup --config backup.yaml
kafka-backup list --path s3://my-kafka-backups/backups/production
mode: restore
backup_id: "orders-to-s3"
target:
bootstrap_servers:
- broker-1.kafka.svc:9092
storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/production
restore:
# Stop before the bad deploy landed (Unix milliseconds)
time_window_end: 1701439200000
consumer_group_strategy: header-based
Pick a backup mode
One config key decides whether the backup runs once, snapshots to the current offsets, or streams continuously. Start with a snapshot for scheduled jobs, and move to continuous mode when you need a tighter recovery point.
| Mode | Config | Behavior |
|---|---|---|
| One-shot | defaults | Reads from start_offset until stopped |
| Snapshot | stop_at_current_offsets: true | Captures high watermarks at start, exits when every partition reaches them — right for cron and Kubernetes Jobs |
| Continuous | continuous: true | Streams new records to S3 as they arrive, checkpointing progress |
Adding an offset_storage section makes any mode incremental: re-running the same
backup_id picks up where the last run finished instead of starting over.
Verify and restore
A backup you have not restored is a guess. Run kafka-backup list against the bucket
to confirm segments and offsets landed, then rehearse a restore with dry_run: true
in the restore config. Point-in-time restore replays records up to an exact
millisecond, and consumer_group_strategy: header-based places consumer groups at
the matching offsets so applications resume without reprocessing.
For the full production runbook — scoped IAM, integrity checks, retention, and troubleshooting — read the Kafka-to-S3 production runbook on the blog.
Frequently asked questions
What does a Kafka backup to S3 contain?
Topic records with their timestamps and headers, consumer group offsets, and topic configuration. Data is compressed with Zstd or LZ4 before upload and written in an efficient segment layout under your key prefix.
Do I need static AWS credentials in the config file?
No. By default the standard AWS credential chain is used — environment variables, shared credentials file, EC2 instance profile, or IRSA on EKS. Explicit access_key and secret_key fields exist for S3-compatible endpoints that need them.
Can I restore a Kafka S3 backup to a different cluster?
Yes. Restore mode takes its own target bootstrap servers, so you can replay a backup into the original cluster, a DR cluster, or a fresh test cluster, and remap topic names with topic_mapping if needed.
How do I schedule regular Kafka backups to S3?
Run the CLI from cron or a Kubernetes Job with stop_at_current_offsets: true, so each run snapshots to the current high watermarks and exits. Add an offset_storage section to make each run incremental rather than a full backup.
Is replication to another cluster the same as an S3 backup?
No. Replication copies every write in real time — including deletes and corrupt records — to another live cluster. An S3 backup is an independent, point-in-time copy outside the cluster failure domain, which is what lets you rewind a mistake.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.