40 posts tagged with "Kafka Backup"

Running Kafka in KRaft mode with Docker Compose takes one service and a handful of environment variables. The official apache/kafka image runs KRaft out of the box — no ZooKeeper container, no migration steps, and docker run -p 9092:9092 apache/kafka:4.3.1 alone gives you a working broker. A short Compose file makes that broker reproducible, and a slightly longer one gives you a 3-controller, 3-broker cluster that mirrors production topology.

This post walks through both files, the env-var conventions behind them, and how to keep data across restarts. For what KRaft mode actually is, see our KRaft guide.

The KRaft controller quorum is the small group of Kafka nodes that replicate the cluster metadata log and elect the active controller. A majority of these voters must acknowledge every metadata change before it commits. Each Kafka KRaft controller is either the active leader or a hot standby, and every broker follows the same log as a non-voting observer.

Our KRaft mode guide covers the architecture. This post covers running the quorum: how elections work, how to read kafka-metadata-quorum.sh output, how static and dynamic quorums differ, and how to add or remove controllers without breaking the majority.

Kafka 4.0, released on March 18, 2025, is the first Apache Kafka that runs entirely without ZooKeeper — and it raises the floor on almost everything else at the same time. Old clients, old message formats, old Java versions, and the original MirrorMaker all exit in the same release.

That makes 4.0 less a feature release than a floor-raising release. The interesting question is not "what shiny thing do I get?" but "what do I have to be running before I can get there?" This post covers both: what shipped, what was removed, and the upgrade path that connects your cluster to it.

Kafka KRaft (Kafka Raft) is Kafka's built-in metadata layer: a small quorum of controller nodes that stores cluster metadata in an internal, replicated Kafka log. It replaces Apache ZooKeeper entirely — no external coordination system, no second cluster to run.

KRaft now appears in every modern Kafka conversation: in release notes, in the process.roles line of every new config file, in vendor migration deadlines. This post explains the thing itself — what KRaft is, how the architecture works, which properties define a KRaft node, and where it stands on production readiness.

Yes — running Kafka without ZooKeeper is not just possible, it is now the only supported way forward. Kafka has run without ZooKeeper in production since version 3.3, ZooKeeper mode was deprecated in 3.5, and Kafka 4.0 removed it entirely. A 4.x broker will not even start against a ZooKeeper ensemble.

The question arrives in many forms: is ZooKeeper still required? Is it removed? Can I run Kafka without it today? The answer depends entirely on which Kafka version and platform you are on. This post gives the version-by-version answer, lists what still requires ZooKeeper, and shows how to start a cluster with no ZooKeeper anywhere.

So what is ZooKeeper in Kafka, exactly? ZooKeeper is the external coordination service that ZooKeeper-mode Kafka clusters use to elect the controller, track which brokers are alive, and store cluster metadata — topics, partition assignments, configs, and ACLs. It is also on its way out: Apache Kafka deprecated ZooKeeper mode in 3.5 and removed it entirely in 4.0, replacing it with KRaft, a metadata quorum built into Kafka itself.

You still see ZooKeeper everywhere. Old tutorials start it before the broker, inherited clusters have zookeeper.connect in every properties file, and interview questions treat it as core Kafka knowledge. This post explains what the service actually does, what it never stored, what breaks when it fails, and why the whole layer is being retired.

How you run a ZooKeeper to KRaft migration depends entirely on where your Kafka lives. Self-managed clusters migrate in place with a rolling, dual-write procedure (KIP-866) through the Kafka 3.9 bridge release. Strimzi drives that same procedure on Kubernetes with a single annotation, Confluent Platform runs it on versions 7.6.1 through 7.9 before the KRaft-only 8.0 upgrade — and AWS MSK has no in-place path at all, which turns the migration into a cluster move.

The deadline is upstream and fixed: ZooKeeper mode was deprecated in Apache Kafka 3.5 and removed in 4.0. If you are still weighing the two modes, the KRaft vs ZooKeeper comparison settles that question. This guide assumes the decision is made and covers the part that actually carries risk: moving every cluster you have, on each platform, without losing metadata, offsets, or your rollback option.

The standard way to run Kafka MirrorMaker on Kubernetes is MirrorMaker 2 managed by the Strimzi operator, declared as a KafkaMirrorMaker2 custom resource. Two alternatives exist — a plain Deployment running the dedicated connect-mirror-maker.sh driver, or MirrorMaker 2 connectors added to a Kafka Connect cluster you already operate — and each earns its place in a specific situation.

This guide compares the three patterns, walks through a current Strimzi manifest field by field, and covers the decisions Kubernetes forces on you: scaling, placement, replication policy, cross-cluster networking, and what to watch once it runs.

To change Kafka replication factor, there is no single alter command. You change it by reassigning partitions: write a JSON plan that lists the full target replica set for each partition, then execute and verify that plan with kafka-reassign-partitions.sh. Done casually, the same procedure can saturate your network, stall producers, or leave throttles choking replication for weeks.

This is the runbook for doing it safely. It covers the preflight checks, the JSON plan, throttled execution, progress monitoring, verification, rollback, and the failure modes at each step. If you are still deciding what value to run, the replication factor guide covers choosing; this post covers changing.

How Kafka replication works in one sentence: Kafka copies each topic partition to several brokers, one copy leads, and the others follow, ready to take over. When the broker holding the leader copy fails, a follower becomes the new leader and clients carry on. That single idea is why a Kafka cluster survives dead disks and crashed machines without losing acknowledged data.

This guide is for readers meeting replication for the first time. It builds the mental model piece by piece: the vocabulary, the journey of one message, what happens during a broker failure, and a topic you can create yourself to watch it all work.