Kafka MirrorMaker 2 Architecture: How the Connectors, Topics, and Clusters Fit Together
The Kafka MirrorMaker architecture is three Kafka Connect connectors run by one or more worker processes, coordinating cross-cluster replication through two distinct layers of internal Kafka topics. MirrorMaker 2 ships no bespoke server of its own — it inherits everything about how it runs from the Connect framework underneath it.
That inheritance explains most of what confuses people operating MirrorMaker 2 in production: why it needs a Connect cluster at all, what its internal topics actually are, how one config file scales past two clusters, and what changed when exactly-once support arrived.
MirrorMaker 2 is three connector classes running inside standard Kafka Connect. Two separate layers of internal topics exist — Connect's own worker bookkeeping topics, and MM2's own offset-translation and heartbeat topics — and confusing the two is the most common source of "where did that topic come from" incidents.
MirrorMaker 2 is a Kafka Connect application
MirrorMaker 2 has no standalone daemon. It is three connector classes — MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector — running inside the standard Kafka Connect framework. MirrorSourceConnector reads selected source topics and writes their records to renamed remote topics. MirrorCheckpointConnector maps each consumer group's committed position on the source cluster to the equivalent position on the target. MirrorHeartbeatConnector produces periodic records that prove a replication path is reachable.
Running as Connect connectors means MirrorMaker 2 inherits Connect's entire operational model wholesale: workers (the processes that actually run connector code), tasks (the unit of parallel work a connector splits itself into), worker bookkeeping topics (where Connect stores connector configs, task offsets, and status), and group-coordination rebalancing (how Connect redistributes tasks when a worker joins or leaves). Those four terms are the spine of everything below.
If you haven't decided whether MirrorMaker 2 is the right tool yet, What Is Kafka MirrorMaker? covers the introductory architecture table and a minimal working config. This post assumes that decision is made and goes one level under it.
Two deployment shapes, one set of connectors
The same three connector classes can run in either of two places, and the choice is an operational one rather than a technical requirement.
A dedicated MirrorMaker cluster runs bin/connect-mirror-maker.sh, which starts worker processes whose only job is MirrorMaker 2's three connectors. A shared Connect cluster runs the same three connector classes as jobs submitted to a Kafka Connect cluster you already operate for other purposes — a pattern Apache Kafka's KIP-382 documents alongside dedicated mode.
The trade-off comes down to blast radius. A dedicated cluster isolates MirrorMaker 2's resource use, upgrade cycle, and failure domain from anything else running on Connect. A shared cluster reuses infrastructure you already monitor and secure, but a misbehaving unrelated connector — or a MirrorMaker connector that starts consuming excess memory — now shares fate with everything else on that cluster.
Running on Kubernetes adds operator-specific options to this same decision. MirrorMaker 2 on Kubernetes walks through the Strimzi KafkaMirrorMaker2 custom resource, a plain Deployment running the dedicated-mode script, and adding connectors to an existing Connect cluster, with manifests for each.
Two layers of internal topics — don't confuse them
MirrorMaker 2's internal topics exist in two layers, created by two different parts of the stack, and mixing them up is a common source of confusion when debugging a stuck replication flow.
Layer 1 — Kafka Connect's own worker bookkeeping topics. Every Connect deployment, MirrorMaker 2 or otherwise, needs somewhere to store connector configurations, task offsets, and connector status. These are configured per cluster alias — for example, us-west.offset.storage.topic names the offset-storage topic for workers connecting to the us-west cluster. This layer belongs to the Connect framework itself; it would exist even if MirrorMaker 2's connectors did nothing cross-cluster at all.
Layer 2 — MirrorMaker 2's own connector data topics. These are the topics the three connectors create to do the actual replication job: mm2-offset-syncs.<target>.internal (source-to-target offset pairs, written by MirrorSourceConnector), <source>.checkpoints.internal (per-group translated positions, written by MirrorCheckpointConnector), and heartbeats (connectivity proof, written by MirrorHeartbeatConnector).
| Layer | Topic examples | Created by | Purpose |
|---|---|---|---|
| 1 — Connect framework | <alias>.offset.storage.topic, <alias>.config.storage.topic, <alias>.status.storage.topic | Kafka Connect worker | Tracks connector configs, task offsets, and status for the framework itself |
| 2 — MirrorMaker 2 connectors | mm2-offset-syncs.<target>.internal, <source>.checkpoints.internal, heartbeats | MirrorSourceConnector, MirrorCheckpointConnector, MirrorHeartbeatConnector | Translates consumer offsets across clusters and proves replication connectivity |
The mechanics of layer 2 — the checkpoint flow, the tuning knobs, and where offset translation breaks down — are covered in full in MirrorMaker 2 offset sync. This post stops at naming the two layers; that one explains layer 2's machinery end to end.
How work is distributed — tasks, workers, and rebalancing
tasks.max sets the upper bound on how many parallel units of work a connector splits itself into. Kafka's own operations guidance recommends setting it to at least 2, higher depending on available hardware and the total number of topic-partitions being replicated. Kafka Connect then assigns those tasks across whichever worker processes are currently running.
When a worker joins the cluster — scaling out, or restarting after a crash — Connect's group-coordination protocol rebalances tasks across the now-larger set of workers. When a worker leaves, the same protocol redistributes its tasks to the survivors. None of this is MirrorMaker-specific machinery; it's the standard behavior any Kafka Connect deployment gets for free.
The practical implication: adding MirrorMaker worker processes increases the number of available task slots, not parallelism by itself. Real throughput is bounded by both task count and the number of partitions being replicated — a tasks.max of 8 against three partitions still leaves five tasks with nothing to do. MirrorMaker best practices covers concrete sizing guidance and the monitoring to confirm tasks are actually keeping up.
Scaling one config file to more than two clusters
The clusters property and the {source}->{target} flow-key syntax are not limited to a single pair. One configuration file can name any number of cluster aliases and enable independent replication flows between any subset of them.
A verified multi-region example from Apache Kafka's own geo-replication documentation names six cluster aliases — west-1, west-2, east-1, east-2, north-1, north-2 — and enables an active/active flow pair scoped to one data center at a time, for example west-1->west-2.enabled = true and west-2->west-1.enabled = true for the West DC pair, with separate flow definitions for the other regions in the same file.
Each flow carries its own topics, groups, and tasks.max settings, configured independently of every other flow in the file. That independence is what makes hub-and-spoke, mesh, and per-region active/active topologies all expressible from the same properties file — the architecture doesn't change between two clusters and twenty; only the number of flow entries does. Choosing which pattern fits your failure domains is a separate question: Kafka geo-replication compares hub-and-spoke against mesh, and Kafka active-active replication covers the two-way case in depth.
Exactly-once semantics changed the architecture in 3.5.0
Exactly-once semantics for dedicated MirrorMaker clusters shipped in Kafka 3.5.0. Enabling it requires exactly.once.source.support = enabled on the target cluster's brokers, plus dedicated.mode.enable.internal.rest = true on the MirrorMaker nodes themselves.
That second setting is the architectural addition: dedicated-mode MirrorMaker nodes now run an internal REST server so they can coordinate directly with each other, a mechanism defined in KIP-710. That communication path didn't exist in MirrorMaker 2's original design — before 3.5.0, coordination between MirrorMaker nodes happened entirely through Kafka Connect's standard group-membership protocol, with no direct node-to-node channel.
Without exactly-once enabled, MirrorMaker 2's default is at-least-once delivery — the offset-translation model covered in full here already assumes duplicates are possible after failover. Exactly-once removes that cross-cluster duplicate window for workloads that genuinely cannot tolerate reprocessing, at the cost of the added REST coordination path and the requirement that it stay on dedicated mode only — shared Connect clusters do not currently support it.
What the architecture does not give you
Every layer described above exists to keep a live second cluster in sync with the first. None of it is a restore point. A corrupted record, a bad delete, or a mistaken producer write moves through the connectors, both layers of internal topics, and the rebalanced tasks exactly as reliably as good data does.
OSO Kafka Backup writes topic data and consumer group offsets to S3, S3-compatible storage, Azure Blob, GCS, or a filesystem, independently of any live cluster's replication path. Point-in-time recovery works at millisecond precision, so a restore can return to a moment before the bad write reached any of MirrorMaker 2's connectors — something the live architecture in this post cannot do by design.
MirrorMaker 2's connectors, topics, and workers keep a live cluster in sync — OSO Kafka Backup adds the independent, point-in-time copy that survives when the whole architecture goes down with it. See the disaster recovery use case for where each layer fits.
The architecture in one paragraph
Three connectors, running as ordinary Kafka Connect workloads, split into tasks that Connect distributes and rebalances across worker processes. Two separate layers of internal topics back that work — one belonging to Connect itself, one belonging to MirrorMaker 2's own offset-translation and heartbeat logic. The same model scales from a two-cluster pair to a six-cluster mesh by adding flow entries, not by changing the architecture, and as of Kafka 3.5.0 it can add a direct node-to-node REST path for exactly-once delivery. Understanding this layer is what turns "MirrorMaker is stuck" from a mystery into a diagnosable problem.
Frequently asked questions
What are the three MirrorMaker 2 connectors?
`MirrorSourceConnector` replicates records from a source cluster to renamed remote topics on the target. `MirrorCheckpointConnector` maps each consumer group’s committed offset on the source to the equivalent position on the target. `MirrorHeartbeatConnector` produces periodic records that prove the replication path between clusters is reachable. All three run as ordinary Kafka Connect connectors.
Does MirrorMaker 2 need its own Kafka Connect cluster?
No. MirrorMaker 2’s three connectors can run in a dedicated cluster started with `bin/connect-mirror-maker.sh`, or as connectors added to a Kafka Connect cluster you already operate for other jobs. The choice is a resource-isolation and blast-radius trade-off, not a technical requirement — both deployment shapes run the identical connector code.
What internal topics does MirrorMaker 2 create?
Two layers. Kafka Connect itself creates worker bookkeeping topics per cluster alias, such as offset, config, and status storage topics, which exist for any Connect deployment. MirrorMaker 2’s own connectors separately create data topics for their job: an offset-syncs topic and a checkpoints topic for offset translation, and a heartbeats topic for connectivity proof.
Does MirrorMaker 2 support exactly-once replication?
Yes, for dedicated MirrorMaker clusters as of Kafka 3.5.0. It requires `exactly.once.source.support = enabled` on the target cluster’s brokers and `dedicated.mode.enable.internal.rest = true` on the MirrorMaker nodes, which brings up an internal REST API for node-to-node coordination defined in KIP-710. Shared Connect clusters do not currently support this mode.
Can MirrorMaker 2 replicate between more than two clusters?
Yes. A single MirrorMaker 2 configuration file can name any number of cluster aliases in its `clusters` property and enable independent replication flows between any subset of them, each with its own topics, groups, and task settings. That flexibility is how hub-and-spoke and mesh topologies get built from the same architecture as a simple two-cluster pair.