Skip to main content

How to Backup and Restore Kafka Topics: A Step-by-Step Guide

· 9 min read
OSO Engineering
The team behind OSO Kafka Backup

This is the backup restore Kafka topic sequence you can run. A backup copies records and offsets to storage outside the cluster. A restore writes that copy back. Replication does not do this. It only copies new writes, including a delete.

The restore a Kafka topic page starts after the backup exists. This guide is the full path: back up, verify, then restore. The selective backup page covers include and exclude patterns.

Quick Answer

Back up the Kafka topic with kafka-backup backup and a YAML file that names the topic and the storage path. Verify that copy with kafka-backup validate. Restore it with kafka-backup validate-restore, then kafka-backup restore. Replication is not a backup. A replica copies a delete forward and cannot rewind the topic.

How do you backup and restore a Kafka topic?​

Install the CLI from the getting started guide. The backup and restore commands below come from the first backup tutorial and CLI basics. One topic is enough. create_topics and default_replication_factor are in the configuration reference. The tutorial leaves both unset.

How do you write the backup config?​

Create backup.yaml. Put topics under source.

backup.yaml
mode: backup
backup_id: "production-backup-001"

source:
bootstrap_servers:
- broker-1.kafka.svc:9092
- broker-2.kafka.svc:9092
- broker-3.kafka.svc:9092
topics:
include:
- orders

storage:
backend: filesystem
path: "/var/lib/kafka-backup/data"

backup:
compression: zstd
compression_level: 3
start_offset: earliest

Swap orders for your topic name. The same include list accepts a pattern such as "events-*".

How do you backup Kafka topics to S3?​

Replace the storage block with the S3 fields from the first backup tutorial. The rest of the file stays the same.

storage
storage:
backend: s3
bucket: my-backup-bucket
region: us-west-2
prefix: kafka-backups/production

Filesystem, Azure Blob, and GCS use that same file shape. The S3 integration and the backup Kafka to S3 post cover the bucket path. If the cluster requires SASL, the tutorial shows security under source.

How do you run the backup?​

kafka-backup backup --config backup.yaml

The 5-minute quickstart runs this command in Docker against a local broker. The backup reads the topic as a consumer. Producers and consumers stay up.

How do you verify the backup?​

List the backup, read the manifest, then validate the segments. --deep reads every segment.

kafka-backup list --path /var/lib/kafka-backup/data

kafka-backup describe --path /var/lib/kafka-backup/data \
--backup-id production-backup-001

kafka-backup validate --path /var/lib/kafka-backup/data \
--backup-id production-backup-001 --deep

validate can take the same facts from the backup file:

kafka-backup validate --config backup.yaml --deep

With the S3 block, --path is the bucket plus the prefix:

kafka-backup list --path s3://my-backup-bucket/kafka-backups/production

describe and validate take that same --path and --backup-id. A valid report means the segments can be read. It does not prove a restore. The next command checks the restore plan before any produce.

How do you restore the topic?​

Create restore.yaml from the tutorial, and set create_topics when the topic is gone. That key defaults to false. default_replication_factor defaults to 1, so set the factor you want on a new topic.

restore.yaml
mode: restore
backup_id: "production-backup-001"

target:
bootstrap_servers:
- dr-broker-1.kafka.svc:9092
- dr-broker-2.kafka.svc:9092

storage:
backend: filesystem
path: "/var/lib/kafka-backup/data"

restore:
create_topics: true
default_replication_factor: 3
dry_run: false

To stop before a known bad write, set time_window_end to Unix milliseconds. The tutorial example is 1701504000000.

kafka-backup validate-restore --config restore.yaml
kafka-backup restore --config restore.yaml

validate-restore checks the backup, the target cluster, and the config. It does not produce records. Keep dry_run: true until that report looks right. Then set dry_run: false and run the restore.

The restore runbook is the short page once a backup already exists. Use it for topic remap, purge, and the other failure cases.

How do you restore consumer group offsets?​

A data restore leaves consumer groups where they are until you reset them. The tutorial's three-phase command restores the records and resets the groups from offset headers.

dr-restore.yaml
mode: restore
backup_id: "production-backup-001"

target:
bootstrap_servers:
- dr-broker-1.kafka.svc:9092

storage:
backend: filesystem
path: "/var/lib/kafka-backup/data"

restore:
consumer_group_strategy: header-based
reset_consumer_offsets: true
consumer_groups:
- my-consumer-group
kafka-backup three-phase-restore --config dr-restore.yaml

Offset headers are written by default (include_offset_headers: true under backup). The consumer offset guide shows how those headers map after the restore.

What should a Kafka topic backup include?​

The copy has to live outside the cluster. Keep the records, the partition layout, and the consumer positions you will need later.

PieceWhere it is storedWhy it matters on restore
Records, keys, headers, timestampsBackup segmentsThis is the data you replay
Partition countManifest field original_partition_countcreate_topics uses it when the topic is missing
Consumer offsetsRecord headers, plus an optional group snapshotGroups can resume instead of starting over
A second live clusterReplication or MirrorMakerA live copy. The delete is copied too

Retention is a delete timer on the live log. When retention.ms expires, those segments are gone. When the topic itself is deleted, the log goes with it. See how to delete a Kafka topic.

See Kafka log compaction for how a backup of a compacted topic copies the latest value per key, not the full history.

Is replication a Kafka backup?​

No. In-cluster replicas and MirrorMaker 2 copy each write to another log. They copy a bad deploy and a topic delete too. A backup is a separate copy you can restore to an earlier time. The mirroring and replication post keeps those live copies separate from a restore.

When should you use another method?​

Use the CLI above when you may need the topic back. The other tools answer different jobs.

JobToolOffsets on restorePoint in time
Recover a topickafka-backupYes, with header-basedtime_window_end
Archive records to object storageConnect S3 sinkNoNo
Print a few recordskafka-console-consumerNoNo
Keep a second live clusterReplication or MirrorMakerLive positions onlyNo

When does a Connect S3 sink fit?​

Use a sink when Connect is already running and you want files in a bucket. The connector class is io.confluent.connect.s3.S3SinkConnector. A full config is in the S3 sink connector post.

That archive does not store consumer group positions. Replaying the files produces new offsets. Treat it as an archive. Use the CLI when the job is a topic restore.

When does a console consumer fit?​

Use it to read a small topic. This is the inspect command from the docs, with key and timestamp printing turned on:

kafka-console-consumer \
--bootstrap-server broker-1.kafka.svc:9092 \
--topic orders \
--from-beginning \
--max-messages 5 \
--property print.key=true \
--property print.timestamp=true

That text is not a restore format. A later produce does not keep the original offsets or timestamps. For a restore, use the YAML above.

How do you check the topic after restore?​

Read a few records, then print the end offsets.

kafka-console-consumer \
--bootstrap-server dr-broker-1.kafka.svc:9092 \
--topic orders \
--from-beginning \
--max-messages 5

kafka-run-class kafka.tools.GetOffsetShell \
--broker-list dr-broker-1.kafka.svc:9092 \
--topic orders

The backup verification guide turns that check into a scheduled test. The point-in-time recovery post shows how time_window_end limits which records come back.

Frequently asked questions

How do you backup a Kafka topic?

Run kafka-backup backup --config backup.yaml with the topic under source.topics.include and a storage backend. The copy keeps records, partition layout, and consumer offsets outside the cluster.

How do you restore a Kafka topic from a backup?

Run kafka-backup validate-restore --config restore.yaml, then kafka-backup restore --config restore.yaml. The restore file names the backup_id, the storage location, and the target brokers.

Can you restore a deleted Kafka topic?

Yes, when the backup was taken before the delete. Set create_topics to true, set default_replication_factor, and run kafka-backup restore. A replica still has the delete, so it cannot bring the topic back.

How do you backup a Kafka topic without downtime?

The backup reads the topic as a consumer, so producers and consumers keep running. For a topic that is still receiving writes, set backup.continuous to true.

How do you verify a Kafka backup?

Run kafka-backup validate --config backup.yaml --deep so every segment is read. Then run kafka-backup validate-restore before you produce any records.

How do you backup Kafka topics to S3?

Set storage.backend to s3 and set bucket, region, and prefix in the backup file. Run kafka-backup backup --config backup.yaml. List that backup with --path set to the s3 URI of the bucket and prefix.

Is Kafka replication a backup?

No. Replication copies each new write, including a bad write and a topic delete. A backup is a separate copy you can restore to an earlier time.