27 posts tagged with "Replication"

Kafka cross-region replication comes down to a small set of named topologies: active-passive, active-active, hub-and-spoke, and mesh. Most teams need one of the first two, and both already have a full implementation guide on this site. This post covers the other two — the ones that show up as a single sentence in every comparison article and never get built out: hub-and-spoke fan-in with real MirrorMaker 2 configuration, and an honest account of what mesh actually costs before you reach for it.

Kafka database replication almost never means what the words suggest. Searchers using that phrase are usually not asking about Kafka's own broker-to-broker replication — they're asking how to get changes out of a database and into Kafka, so downstream systems can react to them. That's change data capture (CDC), and Kafka Connect with a connector like Debezium is the standard way to do it.

This post untangles the terms, walks through the CDC architecture that "Kafka database replication" actually describes, and covers something that architecture's own documentation rarely does: what happens to those topics when you need to recover from a failure.

The original Kafka MirrorMaker (kafka.tools.MirrorMaker, run through the kafka-mirror-maker.sh script) was deprecated in Kafka 3.0.0 under KIP-720 and fully removed in Kafka 4.0.0. If a cluster is still running it, moving that job onto MirrorMaker 2 isn't a config patch — it's a new config file format, a topic-naming decision, and, for the first time, real consumer-offset translation between clusters.

This post is the flag-by-flag migration guide: what changes, what the old options map to, and the order of operations that keeps a rollback path open until the cutover is proven.

The Kafka MirrorMaker architecture is three Kafka Connect connectors run by one or more worker processes, coordinating cross-cluster replication through two distinct layers of internal Kafka topics. MirrorMaker 2 ships no bespoke server of its own — it inherits everything about how it runs from the Connect framework underneath it.

That inheritance explains most of what confuses people operating MirrorMaker 2 in production: why it needs a Connect cluster at all, what its internal topics actually are, how one config file scales past two clusters, and what changed when exactly-once support arrived.

Kafka mirroring vs replication is a vocabulary question before it is an architecture question. In Kafka's own terms, replication copies partitions between brokers inside a single cluster. Mirroring copies topics from one cluster to another — the job MirrorMaker is named for. Both mechanisms produce copies of your data. They run in different places, protect against different failures, and in production you usually need both.

This post pins down where each term comes from and compares the two mechanisms side by side. It also shows why the copy neither of them makes — immutable, outside every cluster — still has to come from somewhere else.

The standard way to run Kafka MirrorMaker on Kubernetes is MirrorMaker 2 managed by the Strimzi operator, declared as a KafkaMirrorMaker2 custom resource. Two alternatives exist — a plain Deployment running the dedicated connect-mirror-maker.sh driver, or MirrorMaker 2 connectors added to a Kafka Connect cluster you already operate — and each earns its place in a specific situation.

This guide compares the three patterns, walks through a current Strimzi manifest field by field, and covers the decisions Kubernetes forces on you: scaling, placement, replication policy, cross-cluster networking, and what to watch once it runs.

To change Kafka replication factor, there is no single alter command. You change it by reassigning partitions: write a JSON plan that lists the full target replica set for each partition, then execute and verify that plan with kafka-reassign-partitions.sh. Done casually, the same procedure can saturate your network, stall producers, or leave throttles choking replication for weeks.

This is the runbook for doing it safely. It covers the preflight checks, the JSON plan, throttled execution, progress monitoring, verification, rollback, and the failure modes at each step. If you are still deciding what value to run, the replication factor guide covers choosing; this post covers changing.

How Kafka replication works in one sentence: Kafka copies each topic partition to several brokers, one copy leads, and the others follow, ready to take over. When the broker holding the leader copy fails, a follower becomes the new leader and clients carry on. That single idea is why a Kafka cluster survives dead disks and crashed machines without losing acknowledged data.

This guide is for readers meeting replication for the first time. It builds the mental model piece by piece: the vocabulary, the journey of one message, what happens during a broker failure, and a topic you can create yourself to watch it all work.

Kafka multi region replication is not one decision — it is three layers. A multi-region Kafka architecture combines replica placement across availability zones inside a single cluster, asynchronous replication or replicated backups across regions, and an independent backup that can rewind past the failures every replica copies faithfully. Which layers you build depends on which failure domains you must survive, and at what recovery point and recovery time.

This guide climbs the failure-domain ladder from a dead broker to a lost region to a bad deploy, shows what each layer contributes with real configuration, and closes with a decision table that maps recovery objectives to the cheapest architecture that meets them.

Kafka to Kafka replication continuously copies topics from one cluster to another using a replication tool. MirrorMaker 2 ships with Apache Kafka and is the default choice; Confluent, AWS, and others sell managed or commercial alternatives that trade money for operations.

The tool matters less than matching the mechanism to the job. Disaster recovery, cluster migration, geo distribution, aggregation, and environment seeding each pull toward different choices. And one boundary holds for all of them: replication keeps a second cluster near the present — it cannot give you yesterday back.