MirrorMaker 2 Offset Sync: How Offset Translation Really Works

Kafka MirrorMaker 2 offset sync is the mechanism that maps a consumer group's committed position on the source cluster to the equivalent position on the target cluster. The same record almost never sits at the same offset in both logs, so MirrorMaker 2 records source-to-target offset pairs in an internal offset-syncs topic and emits per-group checkpoints that failover consumers use to resume without reprocessing or skipping data.

Topic data that fails over without consumer positions is an incomplete disaster recovery story. Applications either replay hours of duplicates or silently skip records. This guide explains the machinery behind offset translation: the three internal topics, the checkpoint flow, automated group offset sync, the tuning knobs with their real defaults, and the places where the mechanism breaks down.

Key takeaway

MirrorMaker 2 never copies committed offsets verbatim — it translates them. The MirrorSourceConnector records offset pairs in mm2-offset-syncs.<target>.internal, the MirrorCheckpointConnector turns them into per-group checkpoints in <source>.checkpoints.internal, and sync.group.offsets.enabled = true applies them to the target's __consumer_offsets — but only for groups with no active members on the target.

Why offsets differ between Kafka clusters

A replicated record lands at a different offset on the target cluster in almost every real deployment. The target topic starts life empty at offset 0, while the source partition may begin far beyond it after retention or compaction has removed early segments. Aborted-transaction markers and records filtered out by the replication flow shift positions further as the stream progresses.

The consequence is fundamental: copying __consumer_offsets values verbatim across clusters is wrong by design. A group that committed offset 5,000,000 on the source might need offset 1,200,431 on the target for the same logical position. Translation, not copying, is what makes consumer failover correct.

MirrorMaker 2 also renames topics as it replicates them. The DefaultReplicationPolicy prefixes each remote topic with its source cluster alias, so orders from the primary cluster becomes primary.orders on the target. The MirrorMaker 2 setup tutorial covers the full configuration and verification workflow; this post stays inside the offset machinery.

How MirrorMaker 2 offset sync works: three internal topics

MirrorMaker 2 runs as a set of Kafka Connect connectors, and each one contributes a piece of the offset translation pipeline.

The MirrorSourceConnector copies records from source to target. Each time the gap between a source offset and its target offset changes by more than offset.lag.max, it emits an offset sync record — a per-partition pair of (source offset, target offset) — to mm2-offset-syncs.<target-alias>.internal. This topic lives on the source cluster by default; offset-syncs.topic.location can move it to the target, which helps when the source cluster is the one you expect to lose.

The MirrorCheckpointConnector does the translation. It reads the offset-syncs topic and the source cluster's committed group offsets, then emits checkpoints to <source-alias>.checkpoints.internal on the target cluster. One checkpoint carries a consumer group ID, a topic-partition, the group's source offset, and the translated target offset.

The MirrorHeartbeatConnector completes the picture but plays no part in translation: it produces beats to a heartbeats topic so downstream clusters can prove connectivity and measure replication lag.

Internal topic	Lives on	Written by	One record contains
mm2-offset-syncs.<target>.internal	Source cluster (default)	MirrorSourceConnector	Topic-partition, source offset, target offset
<source>.checkpoints.internal	Target cluster	MirrorCheckpointConnector	Group ID, topic-partition, source offset, translated target offset
heartbeats	Source, replicated to target	MirrorHeartbeatConnector	Timestamped beat proving the replication path works

Reading translated offsets with RemoteClusterUtils

The checkpoints topic is a queryable record of every group's translated position. The Kafka client library ships RemoteClusterUtils, which reads it and returns positions a failover consumer can seek to or commit:

Map<String, Object> props = new HashMap<>();
props.put("bootstrap.servers", "secondary-1:9092");

Map<TopicPartition, OffsetAndMetadata> newOffsets =
    RemoteClusterUtils.translateOffsets(
        props, "primary", "order-service", Duration.ofSeconds(30));

The properties point at the cluster you are failing over to. The method reads primary.checkpoints.internal there and returns the latest translated offset for each partition the order-service group had committed on the source. Topic names come back already renamed by the replication policy, so the map keys reference primary.orders, not orders.

Without custom code, the same result is reachable with standard tooling. Consume the checkpoints topic to find the translated offset, then apply it:

kafka-consumer-groups.sh --bootstrap-server secondary-1:9092 \
  --group order-service --reset-offsets \
  --topic primary.orders --to-offset 1200431 --execute

Wrapping this sequence in an automated runbook is exactly what the next section's feature does for you.

Automated group offset sync: sync.group.offsets.enabled

Since Kafka 2.7, MirrorMaker 2 can skip the manual step entirely and write translated offsets straight into the target cluster's __consumer_offsets. Enable it per flow:

primary->secondary.emit.checkpoints.enabled = true
primary->secondary.sync.group.offsets.enabled = true
primary->secondary.sync.group.offsets.interval.seconds = 30
primary->secondary.groups = order-service,payment-service

sync.group.offsets.enabled defaults to false and depends on checkpoints, so emit.checkpoints.enabled (default true) must stay on. With both active, a failover consumer group starts on the target cluster and finds its position already committed — no seek logic, no reset commands.

Two safety rules govern when MirrorMaker 2 actually writes those offsets, and both matter operationally:

1. It only writes offsets for groups with no active members on the target cluster. The moment a consumer in that group connects to the target, sync for the group stops silently.
2. It never rewinds a target group that is already ahead. If the group has committed beyond the translated position on the target, the write is skipped to avoid re-consuming records.

The practical rule: keep standby consumers stopped until cutover. A "warm" consumer connected to the target for readiness checks is enough to block offset sync for its whole group.

Group selection uses the groups and groups.exclude filters. The default exclude list is console-consumer-.*, connect-.*, __.*, and the group list refreshes every refresh.groups.interval.seconds (600 seconds). A group missing from the filter gets no checkpoints and no synced offsets — worth checking before assuming the mechanism failed.

Configuration reference and tuning

Every knob below is a per-flow MirrorMaker 2 property with its actual default. All of them accept the source->target. prefix used in connect-mirror-maker.properties.

Property	Default	What it controls
emit.offset-syncs.enabled	true	Whether the source connector records offset pairs at all
offset.lag.max	100	Max offset drift before a new sync record is emitted — the translation granularity
offset-syncs.topic.location	source	Which cluster stores the offset-syncs topic (source or target)
emit.checkpoints.enabled	true	Whether per-group checkpoints are produced
emit.checkpoints.interval.seconds	60	How often checkpoints are emitted
sync.group.offsets.enabled	false	Whether translated offsets are written to the target's __consumer_offsets
sync.group.offsets.interval.seconds	60	How often synced offsets are written
refresh.groups.interval.seconds	600	How often the consumer group list is re-scanned
offset-syncs.topic.replication.factor	3	Durability of the offset-syncs topic
checkpoints.topic.replication.factor	3	Durability of the checkpoints topic

offset.lag.max deserves the most thought. It bounds how stale a translation can be in record terms: with the default of 100, a translated position can trail the group's true position by up to 100 records per partition, which arrive as duplicates after failover. Lowering it tightens translation at the cost of more offset-sync traffic; offset.lag.max = 0 emits a sync for every batch.

Time-based staleness stacks on top. With defaults, a checkpoint can be 60 seconds old and the group offset sync another 60 seconds behind that — up to two minutes of drift before replication lag is even counted. Failover runbooks should assume the resumed position is minutes, not milliseconds, behind the source.

Where offset sync breaks down

Offset translation is at-least-once by construction. Failover consumers should expect duplicates — up to offset.lag.max records per partition plus interval staleness — and must be idempotent or deduplicate downstream.

Several failure modes recur in real incidents:

• Groups that commit rarely have little to translate. A checkpoint requires a committed source offset; a group committing every 10 minutes can lose up to 10 minutes of position on failover regardless of tuning.
• Filtered groups fail silently. A group caught by groups.exclude, or missing from groups, produces no checkpoints. The first anyone notices is often during a disaster recovery drill.
• Active target consumers block automated sync. The classic "offsets never appeared" incident traces back to a standby consumer someone left connected to the target cluster.
• The mapping itself can be lost. Offset syncs live on the source cluster by default. If the source dies and the offset-syncs topic goes with it, existing checkpoints on the target still work, but no new translation can happen.

Monitor the pipeline like any other replication path. The MirrorCheckpointConnector exposes checkpoint-latency-ms under the kafka.connect.mirror MBean namespace — rising latency there means translated positions are drifting further behind the source, exactly when you least want it. A tested failover drill that measures resumed consumer positions is the only proof the whole chain works.

Offset sync is failover plumbing, not a backup

Everything above keeps a live second cluster consistent for consumers. That is its whole scope. MirrorMaker 2 replicates deletions, corrupt records, and bad writes to the target just as faithfully as good data, and the offset mapping exists only while both clusters and the internal topics stay healthy. See the MirrorMaker comparison for a fuller account of where replication ends and backup begins.

An independent backup covers the failures translation cannot. OSO Kafka Backup stores topic data and consumer group offsets together in S3, S3-compatible storage, Azure Blob, GCS, or a filesystem, with point-in-time recovery at millisecond precision. Offset preservation is built in, so a restore returns applications to a consistent position even when the source cluster — and its offset-syncs topic — no longer exists. The consumer offsets backup guide covers that workflow, and the disaster recovery use case shows where each layer fits.

Consumer offsets that survive losing both clusters

MirrorMaker 2 translation only works while the clusters and internal topics are alive. Keep an independent, restorable copy of topics and consumer positions in object storage. Start with OSO Kafka Backup.

Translation is the difference between moving data and moving applications

MirrorMaker 2 offset sync turns cross-cluster replication into something applications can actually fail over to. The offset-syncs topic records the raw mapping, checkpoints turn it into per-group positions, and automated group offset sync applies them — governed by two safety rules that reward keeping standby consumers stopped. Size offset.lag.max against your duplicate tolerance, watch checkpoint latency, verify with drills, and pair the pipeline with an independent offset backup for the failures replication faithfully copies.

Frequently asked questions

Does Kafka replicate messages on each cluster?

Native Kafka replication copies partition data among brokers inside one cluster and keeps offsets identical because replicas share one log. MirrorMaker 2 copies records between separate clusters, where the same record lands at a different offset. That difference is why cross-cluster consumer failover needs offset translation rather than a plain copy of committed offsets.

How does Kafka replication work?

Inside a cluster, follower replicas fetch each partition from its leader, so offsets match everywhere. Across clusters, MirrorMaker 2 consumes source records, produces them to renamed remote topics, and records source-to-target offset pairs in an offset-syncs topic. A checkpoint connector uses those pairs to translate each consumer group committed position into the equivalent target position.

Why is replication required in Kafka?

Replication keeps data available across a defined failure boundary. In-cluster replicas absorb broker failures; MirrorMaker 2 extends coverage to a cluster or site outage, and its offset sync lets consumers resume on the target near where they stopped. Neither layer restores an earlier state — deletions and corruption replicate too, which is what independent backups exist for.

How to backup Kafka?

Run a dedicated backup tool that copies topic data and consumer group offsets to storage outside the cluster, such as S3, Azure Blob, or GCS. MirrorMaker 2 checkpoints are not a backup: they exist only while the clusters and internal topics are healthy. OSO Kafka Backup stores both records and offsets independently with point-in-time restore.