Back up the Kafka topics that matter — not the whole cluster
Select topics by name or pattern, exclude the noise, and back them up with their consumer group offsets. Restore any of them later, to any cluster, under any topic name.
Backup Kafka topics by name or pattern
You rarely need to backup every Kafka topic. Compliance usually cares about a handful
of business-critical streams, and internal or scratch topics just inflate storage.
Topic selection takes exact names, wildcard patterns, and exclusions — applied in that
order, so an exclude always wins.
- backup.yaml
- Run it
- restore.yaml
mode: backup
backup_id: "critical-topics"
source:
bootstrap_servers:
- broker-1.kafka.svc:9092
topics:
include:
- orders # Exact topic name
- payments # Another exact name
- "events-*" # Wildcard pattern
exclude:
- "__consumer_offsets" # Internal Kafka topic
- "*-internal" # Pattern exclusion
storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/topics
backup:
compression: zstd
kafka-backup backup --config backup.yaml
kafka-backup list --path s3://my-kafka-backups/backups/topics
mode: restore
backup_id: "critical-topics"
target:
bootstrap_servers:
- broker-1.kafka.svc:9092
storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/topics
restore:
create_topics: true
topic_mapping:
orders: orders_restored # Replay next to the live topic
consumer_group_strategy: header-based
Restore topics on your terms
Restores are as selective as backups. Each restore targets its own cluster, so a topic backed up in production can be replayed into staging for debugging or into a DR cluster during recovery.
| Restore control | Config key | What it does |
|---|---|---|
| Rename on restore | topic_mapping | Replay orders as orders_restored next to the live topic |
| Create missing topics | create_topics | Auto-create targets with a chosen replication factor |
| Partition subset | source_partitions | Restore only the partitions you need |
| Point in time | time_window_start / time_window_end | Replay records between two millisecond timestamps |
| Rehearsal | dry_run | Validate the whole restore without producing a record |
Consumer offsets travel with the topics
A topic restore that loses consumer group positions forces every application to choose
between reprocessing from the start and skipping to the end. Backups store the original
offsets in record headers, and consumer_group_strategy: header-based uses that
mapping to place each consumer group at the equivalent position after restore.
For a comparison of topic backup methods — Connect S3 sink, this CLI, and DIY consumer scripts — read the topic backup and restore guide on the blog.
Frequently asked questions
Can I backup a single Kafka topic?
Yes. Put one exact topic name in topics.include and only that topic is backed up. You can add more names or wildcard patterns later without touching the rest of the config.
Can I backup Kafka topics matching a pattern?
Yes. Include entries accept wildcard patterns such as "events-*" or "logs-2024-*", and exclude patterns are applied afterwards, so you can select a family of topics while dropping internal ones.
Are consumer group offsets included in a topic backup?
Yes. Original offsets are stored in record headers by default, and consumer group offsets are captured so restores can reposition each group with the header-based strategy instead of forcing a replay from earliest.
Can I restore a Kafka topic without overwriting the live one?
Yes. Set topic_mapping in the restore config to replay the backup under a different name, such as orders to orders_restored, in the same cluster or a different one.
Does topic retention make backups unnecessary?
No. Retention only controls how long the live cluster keeps records in place — it disappears with the cluster and cannot rewind a bad write. A backup is an independent copy you can restore selectively, to a point in time.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.