Cluster to cluster migration: route first, then move
Every Kafka migration is a source-target pair, and the pair decides the procedure. Find your route below, then run the same six phases every safe move shares.
Which Kafka cluster to cluster migration is yours
A Kafka cluster to cluster migration is never generic. The right procedure depends on what the source and target are — a data center, MSK, another provider, or the same cluster with a different metadata mode. Route yourself first; the phases that follow are shared.
| Source → target | The move | Where it is covered |
|---|---|---|
| Self-managed → Amazon MSK | Backup to S3, restore into MSK — no network link | On-prem to MSK |
| MSK ZooKeeper-mode → MSK KRaft-mode | New cluster plus a managed pipeline with ACLs and offsets | MSK ZooKeeper to KRaft |
| Same cluster, ZooKeeper → KRaft | In-place metadata migration, not a data move | ZooKeeper to KRaft router |
| Provider switch or region move | Backup and restore with topic remapping | Cluster migration configs |
| Two live clusters, kept in sync | Replication, not migration — a different job | Kafka-to-Kafka replication |
The last row matters. MirrorMaker 2 and its relatives are built for continuous sync between clusters that both stay alive. A migration ends with one cluster retired, which is why it needs a validated copy and a clean cutover instead of an open-ended mirror.
The six phases every migration shares
Whatever the pair, the safe sequence is the same. Only the endpoints and auth blocks change — the full YAML for each phase is on the cluster migration configs page.
- Assess — inventory topics, partitions, consumer groups, and data volume on the source.
- Seed — back up the source to object storage while producers keep running.
- Validate — prove the copy is complete before the target sees a byte.
- Restore — create topics on the target and load the seed.
- Cut over — freeze producers, move the delta, reposition consumer groups.
- Verify — compare counts, check offsets, then repoint applications.
The commands below are the checkpoints for phases 3 through 6. Nothing here modifies the source cluster, so rollback stays open until you decommission it.
- Validate the seed
- Dry-run the restore
- Reposition consumers
- Verify
kafka-backup validate \
--path s3://kafka-migration/transfer/source \
--backup-id "cluster-move-seed" \
--deep
kafka-backup validate-restore --config migration-restore.yaml
kafka-backup restore --config migration-restore.yaml
kafka-backup offset-reset execute \
--path s3://kafka-migration/transfer/source \
--backup-id "cluster-move-seed" \
--groups order-processor \
--bootstrap-servers target-kafka-1:9092
# Or run restore + offset translation as one operation:
kafka-backup three-phase-restore --config migration-restore.yaml
kafka-backup describe \
--path s3://kafka-migration/transfer/source \
--backup-id "cluster-move-seed" \
--format json | jq '.statistics.records'
What decides success: offsets, not bytes
Copying records between clusters is the easy half. Partition offsets on the target start from zero, so every offset committed on the source points at the wrong record after the move. Consumers restart from earliest or latest and either reprocess history or skip data.
Backups taken with include_offset_headers: true carry the source position
inside each record. During restore, consumer_group_strategy: header-based
maps each group's committed offset to its exact target equivalent — the
mechanism is documented in the
offset translation architecture. That is
the difference between a migration your consumers notice and one they do not.
Frequently asked questions
How do I migrate data from one Kafka cluster to another?
Back up the source cluster to object storage, validate the copy, and restore it into the target — then move the final delta during a short producer freeze. The clusters never need a network path to each other, and consumer group offsets carry across via offset headers.
Can I migrate between different Kafka versions or providers?
Yes. The backup format depends on the Kafka protocol, not the broker version, distribution, or metadata mode. A backup from a Kafka 2.x or ZooKeeper-mode cluster restores to a Kafka 3.x KRaft cluster, MSK, or any other Kafka-compatible target.
Is MirrorMaker 2 the right tool for a cluster to cluster migration?
MirrorMaker 2 is built for continuous replication between two live clusters, and it is the right tool for that. A one-time migration is better served by backup and restore: you get a validated copy before cutover and exact offset translation instead of checkpoint approximation.
How much downtime does a Kafka cluster migration require?
Only the delta window. The seed backup moves the bulk of the data while the source keeps serving traffic. The producer freeze at cutover covers just the records produced since the seed — minutes of data, independent of total cluster size.
What should I check before decommissioning the source cluster?
Three things: record counts on the target match the backup manifest, every consumer group is repositioned (the offset_report file lists each mapping), and applications run against the target through a full business cycle. Keep the final backup until your retention policy clears it.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.