Kafka Consumer Offsets Backup: A Recovery Runbook
A Kafka consumer offsets backup must capture committed group positions with
the topic records those positions reference. In OSO Kafka Backup, enable
include_offset_headers and consumer_group_snapshot, then store both under
the same backup ID.
During recovery, restore records before resetting consumer groups. This order keeps a saved position tied to data that actually exists on the target cluster.
Consumer offsets are not useful by themselves. A recovery point needs topic data, original record offsets, and committed group positions captured together.
Understand the two offset records a backup needs
Kafka uses the word "offset" for related but different records. A sound backup keeps both.
The first is each record's position within a source topic partition. OSO Kafka
Backup stores that original position in record headers when
include_offset_headers: true.
The second is a consumer group's committed position for a topic partition. It
marks where that group should continue processing. OSO Kafka Backup writes
these positions to consumer-groups-snapshot.json.
Restored records may receive different offsets on the target cluster. The original headers provide a source-to-target mapping. The group snapshot says which source positions each group had committed.
Together, those records let the restore process map a saved group position to the matching target position. Neither record can replace the topic data.
For example, a group may have committed source offset 1500. If source records through 1500 expired before backup, saving that number does not recover them.
Choose a Kafka consumer offsets backup method
The right method depends on the recovery event. Disaster recovery needs topic data and group positions in the same process. A planned reset may only need a short-lived rollback snapshot.
| Method | What it preserves | Best use | Main boundary |
|---|---|---|---|
| Topic backup with group snapshot | Records, original offsets, and committed group positions | Disaster recovery and isolated restore | Must restore records before groups |
On-demand snapshot-groups | Committed positions for groups on backed-up topics | Add a group snapshot after a one-shot backup | Still depends on the matching topic backup |
| Offset rollback snapshot | Selected group positions | Safety point before migration or manual reset | Does not back up topic records |
| Replicator offset sync | Positions on another live cluster | Planned or automated failover | Bad writes and deletes can also replicate |
Copying __consumer_offsets | Kafka's internal compacted records | Not recommended as an application recovery method | Internal state is not a portable group restore plan |
Replication can reduce failover time. It is still not backup. A replicated position can move after an unwanted write or deletion reaches both clusters.
The Kafka backup strategies guide explains how replication and independent backups fit one recovery design. For consumer continuity, use a recovery point stored outside the Kafka cluster.
Define a consistent recovery point
Start with the topics and groups that belong to one application boundary. Include every input topic that the application must replay after recovery.
Map each group to its owned topics and partitions. Remove retired group IDs from the runbook, but keep inactive groups that still have recovery value.
For a tightly coordinated recovery point, drain or pause consumers before the final topic snapshot. Wait for in-flight work to finish, then capture committed positions as close as possible to the completed backup.
This pause is especially useful when processing creates external side effects. The offset commit may not prove that a database write or API request completed. Your application must define that business-level boundary.
Record the backup ID, capture time, topic set, and application release. Use the same backup ID when operators inspect, validate, or restore the recovery point.
Kafka can remove inactive group positions after its configured retention period. Topic retention can also remove old records. A backup starts from what the source cluster still holds, so schedule it before either window closes.
Configure topic data and consumer offset backup
The following snapshot config backs up two application topics to filesystem storage. The same offset fields work with S3, Azure Blob, or GCS.
mode: backup
backup_id: "checkout-recovery-point"
source:
bootstrap_servers:
- kafka-0.kafka.svc:9092
- kafka-1.kafka.svc:9092
- kafka-2.kafka.svc:9092
topics:
include:
- orders
- payments
exclude:
- "*-internal"
storage:
backend: filesystem
path: /var/lib/kafka-backup/data
prefix: production-checkout
backup:
compression: zstd
start_offset: earliest
stop_at_current_offsets: true
include_internal_topics: false
include_offset_headers: true
consumer_group_snapshot: true
stop_at_current_offsets: true captures high watermarks when the run starts.
The backup exits after each selected partition reaches its captured watermark.
include_offset_headers: true keeps original source offsets with backed-up
records. Those headers support group-position mapping after restore.
consumer_group_snapshot: true queries every known broker after the backup
cycle. It keeps groups with committed positions on the backed-up topics and
writes the result to the configured storage backend.
Leave include_internal_topics disabled. Do not add __consumer_offsets to
the topic list just to save group positions. Kafka manages that compacted
internal topic, and its records are not a portable restore interface.
The configuration reference documents supported storage, topic selection, security, snapshot, and offset settings. Add TLS or SASL fields there when the source cluster requires them.
Run and inspect the offset-aware backup
Run the reviewed config:
kafka-backup backup --config offset-aware-backup.yaml
The automatic group snapshot runs after the backup cycle. For a one-shot backup without that setting, capture the groups on demand:
kafka-backup snapshot-groups --config offset-aware-backup.yaml
The command uses the backup manifest to determine the topic scope. It queries all known brokers, which includes groups coordinated away from the bootstrap broker in a KRaft cluster.
Inspect the backup before relying on it:
kafka-backup describe \
--path /var/lib/kafka-backup/data/production-checkout \
--backup-id checkout-recovery-point \
--format json
jq '{snapshot_time, groups: [.groups[].group_id]}' \
/var/lib/kafka-backup/data/production-checkout/checkout-recovery-point/consumer-groups-snapshot.json
Check the expected topics, partition counts, and group IDs. For every critical group, confirm that the snapshot contains each assigned topic partition.
Treat group IDs and committed positions as operational metadata. Restrict access to the snapshot and avoid sending its full contents into general logs.
Restore records before resetting consumer groups
Stop the affected consumers before recovery. An active member can commit a new position while operators are calculating or applying the restored position.
First, run backup validation and validate the restore plan:
kafka-backup validate \
--path /var/lib/kafka-backup/data/production-checkout \
--backup-id checkout-recovery-point \
--deep
kafka-backup validate-restore \
--config offset-aware-restore.yaml \
--format json
The restore config should use header-based mapping and load the saved group list. A minimal offset section looks like this:
mode: restore
backup_id: "checkout-recovery-point"
target:
bootstrap_servers:
- scratch-kafka-0.kafka.svc:9092
topics:
include:
- orders
- payments
storage:
backend: filesystem
path: /var/lib/kafka-backup/data
prefix: production-checkout
restore:
create_topics: true
default_replication_factor: 3
consumer_group_strategy: header-based
include_original_offset_header: true
auto_consumer_groups: true
Use the three-phase command for the full ordered operation:
kafka-backup three-phase-restore --config offset-aware-restore.yaml
Phase one collects original offset headers. Phase two restores records and builds the mapping. Phase three resets the saved consumer groups.
Do not reset groups before the records exist. A committed target position without its referenced record range can create skips or out-of-range errors.
The offset management guide covers mapping, preview, reset, and rollback controls. The broader backup and restore guide covers topic recreation and storage choices.
Verify consumer continuity before release
A successful reset proves that Kafka accepted positions. It does not prove that the application will process the restored stream correctly.
Use an isolated cluster or mapped topics for the first drill. Keep outbound email, payment, and other side effects disabled.
Verify the recovery in this order:
- Compare the restored topic and partition set with the backup manifest.
- Confirm each group-position mapping points within the restored record range.
- Verify the expected group positions on the target cluster.
- Start one test consumer and observe its first record per partition.
- Check for missing work, repeated work, deserialization failures, and lag changes.
Some repeated processing can be correct after recovery. A consumer may have completed work but failed before committing its next position. Applications should make important side effects safe to retry.
Also test topic mappings. If orders restores as orders-recovered, the group
reset plan must reference the target topic produced by that mapping.
Add this drill to the wider disaster recovery plan. Record expected results and assign one owner to resolve every mismatch.
Troubleshoot offset recovery failures
Most failures come from mismatched scope or timing. Inspect the manifest, snapshot, mapping report, and target positions before changing offsets again.
| Symptom | Likely cause | Safe response |
|---|---|---|
| Expected group is absent | It had no committed position on a backed-up topic | Confirm its topic scope and source retention, then capture again if the source remains available |
| Saved position has no record mapping | Offset headers were disabled or required records expired before backup | Use timestamp-based or manual recovery only after reviewing the acceptable replay boundary |
| Restore loads the wrong groups | Backup ID or storage prefix points to another recovery point | Stop consumers and correct both identifiers before another reset |
| Group position changes during recovery | A consumer is still active | Stop all members, wait for the group to become inactive, then rerun the reviewed reset |
| Consumer starts with out-of-range errors | Target data does not cover the committed position | Restore the missing record range or choose a documented earlier position |
| Remapped topic gets no group position | Reset plan still names the source topic | Review the topic mapping and regenerated offset report before execution |
Never guess a replacement offset during an incident. Preview the reset, record the chosen replay boundary, and keep a rollback snapshot before any manual change.
Keep offset protection in the operating routine
Enable group snapshots on every recovery-grade topic backup. Alert when the snapshot is missing, empty, late, or stored under an unexpected backup ID.
Run deep backup validation on a schedule. Then rehearse an isolated restore, because integrity checks cannot test group coordination or application logic.
Before migrations and manual resets, create a separate offset rollback snapshot. This gives operators a short-term safety point for the position change. It does not replace the topic backup.
Review group inventory as applications change. Old group IDs create noise, while missing new groups leave a hidden recovery gap.
The reliable sequence is simple: capture records, save original offsets, snapshot committed group positions, validate storage, restore data, reset groups, and test consumers. Release application traffic only after every step matches the recovery plan.
Kafka consumer offsets backup FAQ
Frequently asked questions
How to backup Kafka?
Back up selected topic records and recovery metadata to storage outside the Kafka cluster. Include original record offsets and a consumer group snapshot, validate the stored segments, and test a restore before relying on the recovery point.
How to take backup of Kafka topic?
List the topic in a backup configuration, choose an independent storage backend, enable original offset headers and consumer group snapshots, then run kafka-backup backup with that configuration. Inspect and validate the resulting backup ID.
How to take Kafka backup?
Define the topics, storage backend, and backup ID in YAML, then run kafka-backup backup --config followed by integrity validation. For consumer continuity, store committed group positions with the same backup and restore records before resetting groups.
Consumer offsets become recoverable only when they match stored records. Keep topic data, original offset headers, and group snapshots under one backup ID, then prove the relationship with a controlled restore drill.