Back up Kafka consumer offsets, not just the records
A topic restore without group positions forces every application to reprocess from earliest or skip to latest. Capture offsets with the data and reposition each consumer group exactly where it left off.
Why Kafka consumer offsets backup needs its own plan
Kafka consumer offsets backup is the part most DIY approaches miss. Copying the
__consumer_offsets topic does not work: offsets are positions in a specific
cluster's log, and the restored records land at different offsets on the target.
The tool solves this two ways at once. Every backed-up record carries its original offset in a header, and a consumer group snapshot records each group's committed position per partition.
- backup.yaml
- Run it
- restore.yaml
mode: backup
backup_id: "orders-with-offsets"
source:
bootstrap_servers:
- broker-1.kafka.svc:9092
topics:
include:
- orders
- payments
storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/orders
backup:
compression: zstd
# Store each record's original offset in headers
# (required for consumer offset reset; on by default)
include_offset_headers: true
# Write consumer group offsets to storage after each cycle (v0.12.0+)
consumer_group_snapshot: true
kafka-backup backup --config backup.yaml
# Snapshot consumer group offsets without a full backup cycle
kafka-backup snapshot-groups --config backup.yaml
mode: restore
backup_id: "orders-with-offsets"
target:
bootstrap_servers:
- dr-broker-1.kafka.svc:9092
storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/orders
restore:
create_topics: true
# Map each group's old position to the new log via offset headers
consumer_group_strategy: header-based
# Reset these groups after the data restore completes
reset_consumer_offsets: true
consumer_groups:
- order-processor
- analytics-consumer
# Save the source-to-target offset mapping for audit
offset_report: /tmp/offset-mapping.json
Choose an offset reset strategy
The restore engine supports four strategies for consumer_group_strategy. Header-based
is the recommended default because it maps positions exactly, record by record.
| Strategy | How it finds the new position | When to use it |
|---|---|---|
header-based | Scans the target topic for the original offset stored in record headers | Default choice — exact repositioning |
timestamp-based | Matches record timestamps between source and target | Headers unavailable or stripped |
manual | You supply the target offsets | Special cases with custom bookkeeping |
skip | Leaves consumer groups untouched | Data-only restores, new consumers |
After the restore, each listed consumer group is repositioned before applications
reconnect. No group is forced to choose between reprocessing everything and losing
in-flight work. The offset_report file records every source-to-target mapping, so
you can verify positions before restarting consumers.
For the full recovery procedure — ordering, verification queries, and the failure modes to check — read the consumer offsets recovery runbook on the blog.
Frequently asked questions
Does backing up the __consumer_offsets topic preserve consumer positions?
No. Offsets in __consumer_offsets point at positions in the source cluster’s log. After a restore, records sit at different offsets on the target, so the copied positions are meaningless. Offset preservation needs translation, which is what offset headers and the header-based strategy provide.
How are consumer offsets captured during a Kafka backup?
Two mechanisms work together: include_offset_headers stores each record’s original offset in its headers, and consumer_group_snapshot writes every group’s committed offsets to storage after each backup cycle. You can also snapshot groups on demand with kafka-backup snapshot-groups.
How do consumer groups get repositioned after a restore?
Set consumer_group_strategy: header-based and list the groups under reset_consumer_offsets. The restore engine scans the target topic for each group’s original committed offset in the record headers, finds the equivalent target offset, and resets the group there.
Does offset preservation work when restoring to a different cluster?
Yes. That is the primary use case. Because positions are mapped through record headers rather than copied as raw numbers, groups land at the equivalent position on any target cluster — DR, staging, or a rebuilt production cluster.
Is consumer offset preservation an Enterprise feature?
No. Offset headers, consumer group snapshots, and all four restore strategies are part of the open source (MIT) tool. Enterprise adds Schema Registry sync, encryption, RBAC, and audit logging on top.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.