Replay Kafka Messages from a Backup

Quick Answer​

Replay Kafka messages from an immutable backup into a new topic or another cluster, starting at a chosen point in time. Production consumer offsets stay where they are. Resetting a consumer group offset re-reads the live log instead, including bad records that are still on the brokers.

Why do teams replay Kafka messages?​

A replay is a planned copy of older records. Analytics may need a past window again. A migration may need the same records on a new cluster. A new consumer may need history the live group has already passed.

This is not incident recovery. Point-in-time recovery restores a topic to the moment before corruption or a bad deploy. A replay leaves the live topic in place and writes the window somewhere else.

The source of that copy is a backup. A mirror such as MirrorMaker 2 keeps another cluster current, and it copies bad records along with the good ones. It cannot return an older immutable window.

How do I replay Kafka messages from a backup?​

Start from a backup you already have. The point-in-time restore guide uses the same restore keys. You set a time window, map the topic to a new name, and skip consumer groups.

Validate before you produce. kafka-backup validate-restore checks the backup, the target, and the config without writing records. dry_run: true validates the plan when kafka-backup restore runs, and it produces nothing. Set dry_run to false only after that plan looks right.

mode: restore
backup_id: "orders-backup-20241201"

target:
bootstrap_servers:
  - broker-1.kafka.svc:9092

storage:
backend: s3
bucket: company-kafka-backups
region: us-west-2
prefix: production/prod-cluster

restore:
create_topics: true
default_replication_factor: 3

# 2024-12-01 10:00:00Z through 14:00:00Z
time_window_start: 1733047200000
time_window_end: 1733061600000

topics:
  - orders

topic_mapping:
  orders: orders_replay

# Production groups stay on their current offsets
consumer_group_strategy: skip
dry_run: true

mode: restore
backup_id: "orders-backup-20241201"

target:
bootstrap_servers:
  - kafka-analytics-1.internal:9092
  - kafka-analytics-2.internal:9092

storage:
backend: s3
bucket: company-kafka-backups
region: us-west-2
prefix: production/prod-cluster

restore:
create_topics: true
default_replication_factor: 3

time_window_start: 1733047200000
time_window_end: 1733061600000

topics:
  - orders

topic_mapping:
  orders: orders_replay

consumer_group_strategy: skip
dry_run: true

kafka-backup validate-restore --config replay.yaml

# dry_run: true validates the plan and produces nothing
kafka-backup restore --config replay.yaml

# Set dry_run: false, then run the replay
kafka-backup restore --config replay.yaml

Can I reprocess Kafka messages into a side topic?​

Yes. topic_mapping writes the backed-up topic under a new name. orders: orders_replay restores orders into orders_replay on the target cluster. The live orders topic is not overwritten.

Set create_topics: true so the new topic is created. Set default_replication_factor for that new topic. The configuration reference documents both keys, and production clusters usually use 3.

A new group does not read that history by itself. auto.offset.reset defaults to latest, so a group with no committed offset starts at the log end. Set it to earliest when that consumer should read the replay from the start.

How do I replay a Kafka topic onto another cluster?​

Point target.bootstrap_servers at the destination brokers. The backup in object storage is the source, so the two clusters do not need a replication link. Keep topic_mapping when the destination topic should use a new name.

This is a restore, not a mirror. The destination gets the window you selected. It does not follow records written on the source after the backup.

For a restore whose job is "the topic is gone," use the topic restore runbook. This page is the planned replay: a chosen window, a side destination, and production offsets left alone.

How do I bound a replay with a time window?​

time_window_start and time_window_end are Unix time in milliseconds. The restore keeps records whose stored timestamp falls inside that window. Omit time_window_end to run through the end of the backup. Omit time_window_start to include every earlier record in the backup.

Kafka stores that timestamp on the record. message.timestamp.type is CreateTime by default, or LogAppendTime. With CreateTime, the broker keeps the producer timestamp. With LogAppendTime, the broker overwrites it with the broker time at append.

The filter uses that stored timestamp and does not tell the two types apart. It does not filter on a clock inside the payload. The point-in-time restore guide documents both limits.

How does replay differ from an offset reset?​

kafka-consumer-groups moves one inactive group with --reset-offsets. --to-datetime, --to-earliest, and --to-offset select the new position in the log. --dry-run is the default. --execute applies the change.

The command only changes the committed offset. The next read is the live log, so bad records still on the brokers are read again. Records that retention has already removed are not restored. An out-of-range offset is adjusted to the available end.

A backup replay does not move that offset. It produces a selected window into a topic you name. The live log, and the groups reading it, stay as they are.

Incident recovery is a different job. Point-in-time recovery ends the window before a bad write, and it can reposition groups inside that restored window.

How do I leave production consumer offsets alone?​

Set consumer_group_strategy: skip. Skip is the default, and it does not modify consumer offsets. Leave reset_consumer_offsets at false, which is also the default. Do not list production groups under consumer_groups.

kafka-backup three-phase-restore is the path that does move groups. It collects offset headers, restores the data, and then resets consumer group offsets. Skip that command when the replay must leave production groups alone. The offset management guide lists skip for a data-only restore.

When should I reposition a group after the replay?​

Move a group only when that group should read the restored topic. A new analytics consumer on orders_replay is the usual case. Production groups on orders stay on their current offsets. When the goal is to carry positions through a restore, use consumer offset backup.

KafkaOffsetReset resets the groups you list. to-earliest moves to the earliest offset. to-timestamp reads resetTimestamp in epoch milliseconds and selects the first offset at or after that time. from-mapping uses offsetMappingRef, and restoreName points at the restore that produced the mapping.

dryRun: true validates that reset without committing offsets. snapshotBeforeReset defaults to true and writes a rollback snapshot first.

KafkaOffsetRollback returns groups to a saved snapshot. snapshotRef.name is required. snapshotRef.offsetResetRef can name the KafkaOffsetReset that created the snapshot. dryRun validates without committing, and verifyAfterRollback defaults to true.

An empty consumerGroups list rolls back every group in the snapshot. On the CLI, kafka-backup offset-reset plan previews a reset from the backup offset mapping, and --dry-run defaults to true. kafka-backup offset-reset execute commits that plan. kafka-consumer-groups --reset-offsets can target the side topic while its group is inactive.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaOffsetReset
metadata:
name: reset-replay-consumer
spec:
kafkaCluster:
  bootstrapServers:
    - broker-1.kafka.svc:9092
consumerGroups:
  - analytics-replay
topics:
  - orders_replay
resetStrategy: to-timestamp
resetTimestamp: 1733047200000
dryRun: true
snapshotBeforeReset: true

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaOffsetReset
metadata:
name: reset-from-replay
spec:
kafkaCluster:
  bootstrapServers:
    - broker-1.kafka.svc:9092
consumerGroups:
  - analytics-replay
resetStrategy: from-mapping
dryRun: true
snapshotBeforeReset: true
offsetMappingRef:
  restoreName: orders-replay

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaOffsetRollback
metadata:
name: rollback-replay-reset
spec:
snapshotRef:
  name: pre-reset-snapshot
  offsetResetRef: reset-replay-consumer
kafkaCluster:
  bootstrapServers:
    - broker-1.kafka.svc:9092
consumerGroups:
  - analytics-replay
dryRun: true
verifyAfterRollback: true