Offset Management

Managing consumer group offsets is critical when restoring Kafka data. This guide covers offset concepts and management strategies.

Understanding Offsets

What Are Offsets?

Kafka offsets are sequential IDs assigned to each message in a partition:

Partition 0:
┌────┬────┬────┬────┬────┬────┬────┐
│ 0  │ 1  │ 2  │ 3  │ 4  │ 5  │ 6  │  ← Offsets
└────┴────┴────┴────┴────┴────┴────┘
  ↑                        ↑
  Earliest                 Latest (High Watermark)

Consumer Group Offsets

Consumer groups track their position (committed offset) in each partition:

Consumer Group: order-processor
  orders-0: committed offset 5 (next to read: 6)
  orders-1: committed offset 3 (next to read: 4)
  orders-2: committed offset 7 (next to read: 8)

The Restore Challenge

When data is restored, original offsets don't match the new offsets:

Original Cluster:       Restored Cluster:
┌────┬────┬────┬────┐   ┌────┬────┬────┬────┐
│ 0  │ 1  │ 2  │ 3  │   │ 0  │ 1  │ 2  │ 3  │
└────┴────┴────┴────┘   └────┴────┴────┴────┘
Consumer at offset 2     Consumer still thinks offset 2
                         but data changed!

Offset Strategies

Strategy 1: Skip (Manual Later)

Don't modify offsets during restore. Handle manually afterward.

restore:
  consumer_group_strategy: skip

Best for:

Development/testing environments
When consumers will reprocess all data anyway
Manual offset management workflows

Strategy 2: Header-Based

Store original offset in message headers. Consumers read headers to track position. The backup already archives x-original-offset / x-original-timestamp by default (backup.include_offset_headers: true); the header-based strategy adds the restore-side set too (include_original_offset_header defaults to false but is implied by the strategy):

restore:
  consumer_group_strategy: header-based
  include_original_offset_header: true

Each restored message includes (binary little-endian values, not strings):

Headers:
  x-original-offset:    12345          (i64 LE, 8 bytes)
  x-original-timestamp: 1701234567890  (i64 LE, 8 bytes)
  x-source-partition:   0              (i32 LE, 4 bytes)

To restore without any of these headers, set restore.strip_offset_headers: true (v0.19.0+) — see Offset-tracking headers.

Strategy 3: Automatic Reset

Reset consumer offsets as part of the restore process.

restore:
  consumer_group_strategy: header-based
  reset_consumer_offsets: true
  consumer_groups:
    - order-processor
    - payment-service
    - analytics-consumer

Viewing Offset Mapping

After backup, view the offset mapping:

kafka-backup show-offset-mapping \
  --path /data/backups \
  --backup-id production-backup-001

Output:

Offset Mapping: production-backup-001
══════════════════════════════════════════════════════════════

Topic: orders
─────────────────────────────────────────────────────────────
Partition  First Offset  Last Offset  Record Count
─────────────────────────────────────────────────────────────
0          0             150233       150234
1          0             148891       148892
2          0             152456       152457

Topic: payments
─────────────────────────────────────────────────────────────
Partition  First Offset  Last Offset  Record Count
─────────────────────────────────────────────────────────────
0          0             78234        78235
1          0             76543        76544

Export as CSV for analysis:

kafka-backup show-offset-mapping \
  --path /data/backups \
  --backup-id production-backup-001 \
  --format csv > offset-mapping.csv

Manual Offset Reset

Using Kafka Tools

# Reset to earliest
kafka-consumer-groups \
  --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders \
  --reset-offsets \
  --to-earliest \
  --execute

# Reset to specific offset
kafka-consumer-groups \
  --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders:0 \
  --reset-offsets \
  --to-offset 150233 \
  --execute

# Reset to timestamp
kafka-consumer-groups \
  --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders \
  --reset-offsets \
  --to-datetime 2024-12-01T10:00:00.000 \
  --execute

Using kafka-backup CLI

Generate Reset Plan

kafka-backup offset-reset plan \
  --path /data/backups \
  --backup-id production-backup-001 \
  --groups order-processor,payment-service \
  --bootstrap-servers kafka:9092

Output:

Offset Reset Plan
══════════════════════════════════════════════════════════════

Consumer Group: order-processor
─────────────────────────────────────────────────────────────
Topic      Partition  Current  Target   Action
─────────────────────────────────────────────────────────────
orders     0          -        150233   SET
orders     1          -        148891   SET
orders     2          -        152456   SET

Consumer Group: payment-service
─────────────────────────────────────────────────────────────
payments   0          -        78234    SET
payments   1          -        76543    SET

Total partitions to reset: 5

Execute Reset

kafka-backup offset-reset execute \
  --path /data/backups \
  --backup-id production-backup-001 \
  --groups order-processor \
  --bootstrap-servers kafka:9092

Generate Shell Script

For review before execution:

kafka-backup offset-reset script \
  --path /data/backups \
  --backup-id production-backup-001 \
  --groups order-processor \
  --bootstrap-servers kafka:9092 \
  --output reset-offsets.sh

Generated script:

#!/bin/bash
# Offset reset script for backup: production-backup-001
# Generated: 2024-12-03T10:00:00Z

# Stop consumers before running this script!

kafka-consumer-groups --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders:0 \
  --reset-offsets --to-offset 150233 --execute

kafka-consumer-groups --bootstrap-server kafka:9092 \
  --group order-processor \
  --topic orders:1 \
  --reset-offsets --to-offset 148891 --execute

# ... more commands

Bulk Offset Reset

For large numbers of partitions, use bulk reset:

kafka-backup offset-reset-bulk \
  --path /data/backups \
  --backup-id production-backup-001 \
  --groups order-processor,payment-service,analytics \
  --bootstrap-servers kafka:9092 \
  --max-concurrent 100

Benefits:

~50x faster than sequential reset
Per-partition retry with backoff
Detailed progress reporting
Performance metrics (p50/p99 latency)

Offset Snapshots and Rollback

Create Snapshot Before Changes

kafka-backup offset-rollback snapshot \
  --path /data/offset-snapshots \
  --groups order-processor,payment-service \
  --bootstrap-servers kafka:9092 \
  --description "Before restore operation"

List Snapshots

kafka-backup offset-rollback list \
  --path /data/offset-snapshots

Output:

Offset Snapshots
══════════════════════════════════════════════════════════════
ID                          Created               Groups  Description
──────────────────────────────────────────────────────────────
snapshot-20241203-100000    2024-12-03T10:00:00Z  2       Before restore
snapshot-20241203-090000    2024-12-03T09:00:00Z  2       Before migration

View Snapshot Details

kafka-backup offset-rollback show \
  --path /data/offset-snapshots \
  --snapshot-id snapshot-20241203-100000

Rollback to Snapshot

If something goes wrong:

kafka-backup offset-rollback rollback \
  --path /data/offset-snapshots \
  --snapshot-id snapshot-20241203-100000 \
  --bootstrap-servers kafka:9092

Verify Offsets Match Snapshot

kafka-backup offset-rollback verify \
  --path /data/offset-snapshots \
  --snapshot-id snapshot-20241203-100000 \
  --bootstrap-servers kafka:9092

Three-Phase Restore

For complete automated recovery:

three-phase-restore.yaml
mode: restore
backup_id: "production-backup-001"

target:
  bootstrap_servers:
    - kafka-dr:9092

storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-west-2

restore:
  consumer_group_strategy: header-based
  reset_consumer_offsets: true
  consumer_groups:
    - order-processor
    - payment-service
    - analytics-consumer

kafka-backup three-phase-restore --config three-phase-restore.yaml

Phases:

Phase 1: Collect offset headers from backup
Phase 2: Restore data to target cluster
Phase 3: Reset consumer group offsets

Best Practices

Before Restore

Stop consumers - Prevent offset commits during restore
Snapshot current offsets - Enable rollback if needed
Document consumer groups - Know which groups need reset

During Restore

Use header-based strategy - Preserve original offset information
Enable dry-run first - Validate configuration
Monitor progress - Watch for errors

After Restore

Verify offset mapping - Check offsets are set correctly
Start consumers gradually - Monitor for reprocessing issues
Keep snapshots - Retain for potential rollback

Troubleshooting

Consumer Group Not Found

Error: Consumer group 'my-group' not found

The group may not exist if:

Consumers never started
Group was deleted
Using incorrect group name

Solution: Create the group by starting a consumer, or use --reset-offsets --to-earliest --dry-run first.

Offsets Out of Range

Error: Offset 150000 is out of range for partition 0 (current: 0-100)

Target offset doesn't exist in restored topic. This can happen with PITR restores.

Solution: Reset to --to-earliest or --to-latest instead of specific offset.

Consumer Still Running

Error: Consumer group has active members

Stop all consumers before resetting offsets:

# Check active members
kafka-consumer-groups \
  --bootstrap-server kafka:9092 \
  --group order-processor \
  --describe --members

Next Steps

Three-Phase Restore - Automated restore
Performance Tuning - Optimize operations
Disaster Recovery - DR planning

Understanding Offsets​

What Are Offsets?​

Consumer Group Offsets​

The Restore Challenge​

Offset Strategies​

Strategy 1: Skip (Manual Later)​

Strategy 2: Header-Based​

Strategy 3: Automatic Reset​

Viewing Offset Mapping​

Manual Offset Reset​

Using Kafka Tools​

Using kafka-backup CLI​

Generate Reset Plan​

Execute Reset​

Generate Shell Script​

Bulk Offset Reset​

Offset Snapshots and Rollback​

Create Snapshot Before Changes​

List Snapshots​

View Snapshot Details​

Rollback to Snapshot​

Verify Offsets Match Snapshot​

Three-Phase Restore​

Best Practices​

Before Restore​

During Restore​

After Restore​

Troubleshooting​

Consumer Group Not Found​

Offsets Out of Range​

Consumer Still Running​

Next Steps​

Understanding Offsets

What Are Offsets?

Consumer Group Offsets

The Restore Challenge

Offset Strategies

Strategy 1: Skip (Manual Later)

Strategy 2: Header-Based

Strategy 3: Automatic Reset

Viewing Offset Mapping

Manual Offset Reset

Using Kafka Tools

Using kafka-backup CLI

Generate Reset Plan

Execute Reset

Generate Shell Script

Bulk Offset Reset

Offset Snapshots and Rollback

Create Snapshot Before Changes

List Snapshots

View Snapshot Details

Rollback to Snapshot

Verify Offsets Match Snapshot

Three-Phase Restore

Best Practices

Before Restore

During Restore

After Restore

Troubleshooting

Consumer Group Not Found

Offsets Out of Range

Consumer Still Running

Next Steps