Kafka-to-Kafka Replication: How to Move Data Between Clusters
Kafka to Kafka replication continuously copies topics from one cluster to another using a replication tool. MirrorMaker 2 ships with Apache Kafka and is the default choice; Confluent, AWS, and others sell managed or commercial alternatives that trade money for operations.
The tool matters less than matching the mechanism to the job. Disaster recovery, cluster migration, geo distribution, aggregation, and environment seeding each pull toward different choices. And one boundary holds for all of them: replication keeps a second cluster near the present — it cannot give you yesterday back.
Start with MirrorMaker 2 unless a platform already holds you: Confluent estates should assess Cluster Linking, and MSK-to-MSK flows can use MSK Replicator. Whatever copies your topics, pair it with an independent backup, because every replicator forwards mistakes as faithfully as records.
How Kafka-to-Kafka replication works
One mechanical model covers almost every tool: a consumer reads selected topics from the source cluster, and a producer writes those records to the target. The copy is asynchronous. The target trails the source by some lag, which is a normal operating fact rather than a defect.
The notable exception is Confluent Cluster Linking, which mirrors topics at the broker level, byte-for-byte, with offsets preserved exactly and no Connect cluster to run. Every other mainstream option — MirrorMaker 2, Confluent Replicator, MSK Replicator — is a consume-and-produce design.
That design has three consequences worth knowing before you pick a tool:
- Consumer offsets do not carry over raw. An offset is a coordinate inside one cluster's log. The same record almost never sits at the same offset on the target, so committed positions must be translated.
- Delivery is at-least-once by default. A task restart can duplicate records, so consumers on either side of a cutover need idempotent handling.
- Only topic data flows. Schema Registry contents and ACLs need their own plan; they are not part of the record stream.
This is a different mechanism from the replication factor inside a single cluster, where followers copy partitions between brokers for availability — Kafka replication explained covers that layer. This article is about moving data between two separate clusters.
Five jobs that call for Kafka to Kafka replication
Cross-cluster replication earns its operational cost when the second cluster serves a purpose the first one cannot. Five jobs come up repeatedly:
Disaster recovery. A warm standby in a second region consumes the live stream so a regional failure does not take Kafka down with it. Pattern choices — active-passive, active-active, hub-and-spoke — are mapped in the Kafka geo replication guide.
Cluster migration. Replication lets you move to a new cluster — new version, new vendor, new datacenter — while producers keep writing, then cut over when the target is caught up. When downtime is acceptable, a staged backup-based migration is often simpler than operating a live link you will use once.
Geo distribution. Consumers read from a cluster near them instead of crossing an ocean per fetch. Topology and network design for this job live in the cross-datacenter replication guide.
Aggregation. Many edge clusters replicate into one central cluster for
analytics. Naming policy matters most here: MirrorMaker 2's alias prefixes
keep edge1.orders and edge2.orders from colliding in the middle.
Environment seeding. Production topics replicated into staging give load tests real shape. Filter deliberately — replication copies sensitive records as readily as harmless ones.
| Job | Best-fit mechanism | Watch for |
|---|---|---|
| Disaster recovery | MirrorMaker 2 or a managed replicator, warm standby | Offset translation, cutover topic names |
| Cluster migration | Live link for zero-downtime; backup and restore for staged moves | Whether a one-time move needs a live link at all |
| Geo distribution | MirrorMaker 2 across WAN links | Lag monitoring, network tuning |
| Aggregation | MirrorMaker 2 with alias prefixes | Topic name collisions at the center |
| Environment seeding | One-shot replication or restore into staging | Sensitive data crossing environments |
The tools that move data between Kafka clusters
Four tools cover nearly every real deployment, and each has a detailed guide on this site.
MirrorMaker 2 ships with Apache Kafka under the Apache 2.0 license. It is
built on Kafka Connect as three cooperating connectors covering records,
consumer positions, and heartbeats. It runs in a dedicated mode or on an
existing Connect cluster, against any Kafka it can reach — self-managed, MSK,
or anything that speaks the protocol. By default it prefixes remote topics
with the source alias, so orders from cluster primary arrives as
primary.orders. Start with the
MirrorMaker overview for architecture.
Confluent Replicator is Confluent's commercial connector for Confluent Platform estates. It keeps source topic names by default and can migrate schemas between registries, which MirrorMaker 2 cannot. Confluent's own documentation points most new designs at Cluster Linking instead. The Kafka Replicator guide covers it in depth, and the Replicator vs MirrorMaker comparison settles the head-to-head.
Confluent Cluster Linking is the one that is not consume-and-produce: it mirrors topics at the broker level with offsets preserved exactly, so consumer positions remain valid on the destination. The tradeoff boundary against backup is mapped in OSO Kafka Backup vs Cluster Linking.
MSK Replicator is AWS's managed replication service for MSK-to-MSK flows. It keeps topic names identical by default and removes the Connect infrastructure work, at a service price and inside AWS only. The MSK Replicator guide covers setup and failover.
Two names from older estates: Uber's uReplicator is archived and read-only, and LinkedIn's Brooklin is a multitenant data-movement system that includes Kafka-to-Kafka mirroring among broader connector duties.
| Tool | License / cost | Runs | Destination name for orders | Offset handling |
|---|---|---|---|---|
| MirrorMaker 2 | Apache 2.0, ships with Kafka | Dedicated mode or Connect cluster, any Kafka | primary.orders (default) | Checkpoint translation, any committing client |
| Confluent Replicator | Commercial, Confluent Platform | Connect in Confluent Platform | orders (default ${topic}) | Timestamp interceptor, Java consumers |
| Cluster Linking | Confluent Platform / Cloud | Broker-level, no Connect cluster | orders, offsets preserved | None needed — offsets identical |
| MSK Replicator | AWS-managed service | AWS, MSK-to-MSK only | orders (identical by default) | Managed translation |
A minimal MirrorMaker 2 flow
The shortest honest path to records flowing looks like this. Define both clusters, enable one direction, and set replication factors that match your broker count:
clusters = primary, secondary
primary.bootstrap.servers = primary-1:9092,primary-2:9092,primary-3:9092
secondary.bootstrap.servers = secondary-1:9092,secondary-2:9092,secondary-3:9092
primary->secondary.enabled = true
primary->secondary.topics = orders
replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3
Launch the dedicated MirrorMaker 2 process:
bin/connect-mirror-maker.sh connect-mirror-maker.properties
Then confirm the remote topic exists on the target and records arrive:
bin/kafka-topics.sh --bootstrap-server secondary-1:9092 --list
bin/kafka-console-consumer.sh --bootstrap-server secondary-1:9092 \
--topic primary.orders --from-beginning --max-messages 5
This article stops at first records flowing. Authentication, allowlist planning, offset verification, and a timed failover drill are covered step-by-step in the MirrorMaker 2 setup tutorial.
The two things that do not copy themselves
Consumer offsets. MirrorMaker 2 records source-to-target offset pairs and
emits per-group checkpoints on the target; emit.checkpoints.enabled
defaults to true. Writing translated positions into the target's
__consumer_offsets requires sync.group.offsets.enabled=true — the default
is false, and it applies only to groups with no active members there. The
MirrorMaker 2 offset sync guide walks
through the internals and failure modes.
Schemas. Replicating a _schemas topic copies bytes, not meaning: nothing
reconciles registry IDs or subjects on the other side. Confluent Replicator
can migrate schemas between registries; MirrorMaker 2 cannot. Protecting
registry state is its own problem either way, covered in the
Schema Registry backup guide — and in
OSO Kafka Backup, Schema Registry sync is an Enterprise feature.
Kafka-to-Kafka replication is not backup
Every mechanism above copies the live stream forward. A bad write, an application bug, or a topic deletion replicates to the target as faithfully as good data — usually within seconds. Both copies then sit inside running Kafka clusters, subject to broker retention and the same administrative mistakes. No replicator can reconstruct a topic as it existed before an incident.
Recovery to an earlier state needs a restore point outside the replication path. OSO Kafka Backup stores topic data and consumer group offsets together in S3, S3-compatible storage, Azure Blob, GCS, or a filesystem, with millisecond-precision point-in-time recovery and offset preservation built in, under an MIT license. Replication and backup are complements in a disaster recovery design, not substitutes.
Replication keeps cluster B as wrong as cluster A within seconds. Keep point-in-time restore points — topic data and consumer offsets — in your own object storage. Start with OSO Kafka Backup.
Choosing your path
MirrorMaker 2 is the default: it ships with Kafka, runs anywhere, and its checkpoint translation works for any client that commits offsets. Reach for a platform tool when the platform already holds you — Cluster Linking for new Confluent designs, Replicator where a licensed estate runs it today, MSK Replicator for AWS-native MSK pairs. Then add the piece no replicator provides: an independent backup, because the second cluster inherits your failures at replication speed.
Frequently asked questions
What is Kafka-to-Kafka replication?
Kafka-to-Kafka replication continuously copies topics from one Kafka cluster to another using a tool such as MirrorMaker 2, Confluent Replicator, Cluster Linking, or MSK Replicator. Most tools consume from the source and produce to the target asynchronously, so the target trails the source by some lag.
Does Kafka replicate messages to another cluster automatically?
No. Kafka's built-in replication factor copies partitions between brokers inside one cluster for availability. Moving data to a second cluster requires a separate cross-cluster replication tool, such as MirrorMaker 2, which ships with Apache Kafka but must be configured and operated deliberately.
What is the best tool to replicate data between Kafka clusters?
MirrorMaker 2 is the default choice: it ships with Apache Kafka, costs no license, and works between any reachable clusters. Confluent estates should assess Cluster Linking for new designs, and AWS teams replicating MSK to MSK can use the managed MSK Replicator service.
Do consumer offsets transfer when replicating between Kafka clusters?
Not as raw numbers. An offset is a position inside one cluster's log, so the same record usually sits at a different offset on the target. MirrorMaker 2 translates positions through checkpoints and can write them to the target for inactive groups; Cluster Linking preserves offsets exactly at the broker level.
Can Kafka-to-Kafka replication replace backups?
No. Replication copies deletions, corrupt writes, and application bugs to the target as faithfully as valid records, and both copies remain subject to broker retention. Restoring a topic to an earlier point in time requires an independent backup stored outside the replication path.