27 posts tagged with "Replication"

The Kafka Replicator vs MirrorMaker decision comes down to ownership. MirrorMaker 2 is the replication framework included with Apache Kafka: free, vendor-neutral, and built on Kafka Connect. Kafka Replicator — almost always Confluent Replicator — is Confluent's commercial connector with deeper Confluent Platform integration, including Schema Registry migration and Control Center monitoring.

The feature lists overlap heavily. The real differences sit in licensing, destination topic naming, consumer offset translation, and schema handling. Neither tool can restore yesterday's data, so neither replaces backup.

Kafka MirrorMaker 2 offset sync is the mechanism that maps a consumer group's committed position on the source cluster to the equivalent position on the target cluster. The same record almost never sits at the same offset in both logs, so MirrorMaker 2 records source-to-target offset pairs in an internal offset-syncs topic and emits per-group checkpoints that failover consumers use to resume without reprocessing or skipping data.

Topic data that fails over without consumer positions is an incomplete disaster recovery story. Applications either replay hours of duplicates or silently skip records. This guide explains the machinery behind offset translation: the three internal topics, the checkpoint flow, automated group offset sync, the tuning knobs with their real defaults, and the places where the mechanism breaks down.

In-sync replicas (ISR) are the subset of a Kafka partition's replicas that are currently caught up with the leader — the leader itself, plus every follower that has fetched to the leader's latest offset within replica.lag.time.max.ms (30 seconds by default). The ISR is the mechanism behind every guarantee Kafka makes: only ISR members confirm acks=all writes, and only ISR members can become leader without losing data.

Replication factor is a static promise; the ISR is the live measurement of how much of that promise currently holds. This guide explains how replicas join and leave the ISR, what the high watermark commits, how min.insync.replicas and acks form a durability contract, and what happens during leader election when the ISR has shrunk.

Kafka under-replicated partitions (URPs) are partitions whose in-sync replica (ISR) count has dropped below their replication factor. One or more follower replicas has either died or fallen behind the leader for longer than replica.lag.time.max.ms — 30 seconds by default. Every broker reports the count through the UnderReplicatedPartitions JMX gauge, and any value above zero means the cluster is running with less redundancy than you configured.

URP is the most-watched health metric in Kafka operations for a simple reason: it measures how much failure margin remains before an outage becomes data loss. This guide covers what the metric means, the commands that find the failing broker, a fix for each root cause, and the alert thresholds that separate routine maintenance from a real incident.

Kafka active-active replication runs two or more clusters that all serve producers and consumers, with bidirectional replication keeping them converged. Every region writes locally, reads locally, and receives the other regions' data within seconds.

Kafka has no native multi-master mode, so active-active is an architecture you assemble — from a cross-cluster replication tool plus application-level discipline. Three problems define the assembly: replication loops, consumer offset divergence, and the complete absence of conflict resolution.

Kafka Replicator is Confluent's Kafka Connect connector for asynchronously copying selected topics from one Kafka cluster to another. It can preserve topic settings, translate consumer positions, and fit into an existing Confluent Platform operating model.

Replicator keeps a second Kafka cluster close to the source's current state. It does not create historical restore points, so it cannot recover the state before a bad write or deletion.

A Kafka MirrorMaker 2 setup needs two cluster aliases, a directional flow, topic and group filters, and a running dedicated Connect process. You must then verify remote topics, copied records, translated consumer offsets, and measured replication delay.

MirrorMaker 2 provides asynchronous cross-cluster replication. It does not keep a historical restore point, and a running connector does not prove that an application can fail over safely.

Kafka replication factor is the number of brokers that store each topic partition. A factor of three gives each partition one leader replica and two follower replicas. Kafka can then keep serving the partition when one broker fails, provided another eligible replica is current.

More replicas improve tolerance for broker and disk failure. They also consume more storage, network bandwidth, and recovery capacity. The right value depends on your failure domains and durability target, not a universal rule.

Kafka MirrorMaker 2 is Apache Kafka's framework for copying records and selected metadata between independent Kafka clusters. It runs on Kafka Connect, reads from a source cluster, and writes to a target cluster through a directional replication flow.

MirrorMaker helps with regional availability, cluster migration, and centralized data access. It does not create an independent historical backup, and it cannot restore a topic to a time before a bad record or deletion.

Kafka replication stores copies of each partition on multiple brokers. One replica leads reads and writes while followers copy its ordered log and stand ready to take over. This design keeps partitions available through many broker failures. It does not create a historical backup or protect every copy from a bad delete.