Migrate a Kafka cluster through object storage
The clusters never need to talk to each other. Back up the source, validate the copy, restore to the target — with topic remapping and consumer group positions intact.
When backup-based Kafka cluster migration wins
Kafka cluster migration usually starts with MirrorMaker, and for continuous live sync that is the right tool. Backup-based migration wins when the clusters cannot communicate directly, when you want to move a specific time range, or when cross-region bandwidth costs matter.
The transfer medium is object storage. The source cluster writes a compressed backup to a bucket, and the target cluster restores from it — one-time transfer, no persistent link, and a validated copy sitting between the two.
- migration-backup.yaml
- Validate the copy
- migration-restore.yaml
mode: backup
backup_id: "migration-20260730"
source:
bootstrap_servers:
- source-kafka-1:9092
- source-kafka-2:9092
topics:
include:
- "*"
exclude:
- "__consumer_offsets"
- "_schemas"
storage:
backend: s3
bucket: kafka-migration
region: us-west-2
prefix: migration/source-cluster
backup:
compression: zstd
# Required to carry consumer group positions to the target
include_offset_headers: true
source_cluster_id: "source-cluster"
kafka-backup backup --config migration-backup.yaml
# What landed in the bucket?
kafka-backup list --path s3://kafka-migration/migration/source-cluster
kafka-backup describe \
--path s3://kafka-migration/migration/source-cluster \
--backup-id "migration-20260730"
# Deep validation before the cutover window
kafka-backup validate \
--path s3://kafka-migration/migration/source-cluster \
--backup-id "migration-20260730" \
--deep
mode: restore
backup_id: "migration-20260730"
target:
bootstrap_servers:
- target-kafka-1:9092
- target-kafka-2:9092
storage:
backend: s3
bucket: kafka-migration
region: us-west-2
prefix: migration/source-cluster
restore:
create_topics: true
default_replication_factor: 3
# Rename topics during the move if the target uses new names
topic_mapping:
orders: orders_v2
# Map each group's source position to the target log
consumer_group_strategy: header-based
reset_consumer_offsets: true
consumer_groups:
- order-processor
- analytics-service
# Keep the source-to-target offset mapping for the cutover record
offset_report: /tmp/migration-offset-mapping.json
Migration scenarios this pattern covers
The same backup-and-restore pair handles every cluster-to-cluster move. Only the endpoints and mappings change.
| Scenario | Example |
|---|---|
| Version upgrade | Kafka 2.x → Kafka 3.x |
| Cloud migration | On-premises → AWS MSK |
| Provider switch | AWS MSK → Confluent Cloud |
| Region migration | us-east-1 → eu-west-1 |
| Environment cloning | Production → Staging |
| Cluster consolidation | Multiple clusters → One |
| ZooKeeper to KRaft | MSK ZK-mode → KRaft-mode (Enterprise) |
Backup-based migration vs MirrorMaker
MirrorMaker is excellent at continuous replication, and if you need the two clusters in sync for weeks, use it. For a one-time move the trade-offs change.
| Question | MirrorMaker | Backup and restore |
|---|---|---|
| Live sync during migration | Yes | No — one-time transfer |
| Network path between clusters | Required, continuous | None — object storage in between |
| Move a specific time range | No | Yes, restore time windows |
| Consumer offset preservation | Complex (checkpoint topics) | Built in via offset headers |
| Validated copy before cutover | No | Yes — validate --deep on the backup |
Run the restore with dry_run: true first to check topics, mappings, and the
window without producing records. After cutover, the offset_report file
documents where every consumer group landed — worth keeping with your
migration record. Rehearse the whole sequence against staging first; the
DR testing playbook on the blog covers how
to structure that rehearsal.
Frequently asked questions
Can I migrate a Kafka cluster without network connectivity between the clusters?
Yes. That is the core of the pattern: the source writes a backup to object storage such as S3, and the target restores from the same bucket. The two clusters never connect to each other, which suits migrations across accounts, VPCs, or providers.
Do consumer groups keep their positions after a cluster migration?
Yes, when the backup was taken with include_offset_headers: true. The restore maps each group’s committed source offset to the equivalent target offset with consumer_group_strategy: header-based, then repositions the groups you list. The offset_report file records every mapping.
Can I rename topics while migrating between Kafka clusters?
Yes. Add topic_mapping entries to the restore config, such as orders: orders_v2, and records from the source topic land in the renamed target topic. Partition remapping is available the same way with partition_mapping.
Should I use MirrorMaker or backup and restore for a Kafka migration?
Use MirrorMaker when you need continuous live sync between clusters that can reach each other. Use backup and restore for a one-time move, when clusters are network-isolated, when you want a validated copy before cutover, or when you need offset preservation without checkpoint-topic translation.
Does this handle the MSK ZooKeeper to KRaft migration?
The backup-and-restore pattern moves the data for any cluster pair. For MSK specifically, kafka-backup Enterprise adds a managed ZooKeeper-to-KRaft migration: topic replication, ACL migration, offset translation, and cryptographic evidence, with free plan and precheck commands.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.