38 posts tagged with "Kafka Backup"

Kafka 4.0, released on March 18, 2025, is the first Apache Kafka that runs entirely without ZooKeeper — and it raises the floor on almost everything else at the same time. Old clients, old message formats, old Java versions, and the original MirrorMaker all exit in the same release.

That makes 4.0 less a feature release than a floor-raising release. The interesting question is not "what shiny thing do I get?" but "what do I have to be running before I can get there?" This post covers both: what shipped, what was removed, and the upgrade path that connects your cluster to it.

Kafka KRaft (Kafka Raft) is Kafka's built-in metadata layer: a small quorum of controller nodes that stores cluster metadata in an internal, replicated Kafka log. It replaces Apache ZooKeeper entirely — no external coordination system, no second cluster to run.

KRaft now appears in every modern Kafka conversation: in release notes, in the process.roles line of every new config file, in vendor migration deadlines. This post explains the thing itself — what KRaft is, how the architecture works, which properties define a KRaft node, and where it stands on production readiness.

Yes — running Kafka without ZooKeeper is not just possible, it is now the only supported way forward. Kafka has run without ZooKeeper in production since version 3.3, ZooKeeper mode was deprecated in 3.5, and Kafka 4.0 removed it entirely. A 4.x broker will not even start against a ZooKeeper ensemble.

The question arrives in many forms: is ZooKeeper still required? Is it removed? Can I run Kafka without it today? The answer depends entirely on which Kafka version and platform you are on. This post gives the version-by-version answer, lists what still requires ZooKeeper, and shows how to start a cluster with no ZooKeeper anywhere.

So what is ZooKeeper in Kafka, exactly? ZooKeeper is the external coordination service that ZooKeeper-mode Kafka clusters use to elect the controller, track which brokers are alive, and store cluster metadata — topics, partition assignments, configs, and ACLs. It is also on its way out: Apache Kafka deprecated ZooKeeper mode in 3.5 and removed it entirely in 4.0, replacing it with KRaft, a metadata quorum built into Kafka itself.

You still see ZooKeeper everywhere. Old tutorials start it before the broker, inherited clusters have zookeeper.connect in every properties file, and interview questions treat it as core Kafka knowledge. This post explains what the service actually does, what it never stored, what breaks when it fails, and why the whole layer is being retired.

How you run a ZooKeeper to KRaft migration depends entirely on where your Kafka lives. Self-managed clusters migrate in place with a rolling, dual-write procedure (KIP-866) through the Kafka 3.9 bridge release. Strimzi drives that same procedure on Kubernetes with a single annotation, Confluent Platform runs it on versions 7.6.1 through 7.9 before the KRaft-only 8.0 upgrade — and AWS MSK has no in-place path at all, which turns the migration into a cluster move.

The deadline is upstream and fixed: ZooKeeper mode was deprecated in Apache Kafka 3.5 and removed in 4.0. If you are still weighing the two modes, the KRaft vs ZooKeeper comparison settles that question. This guide assumes the decision is made and covers the part that actually carries risk: moving every cluster you have, on each platform, without losing metadata, offsets, or your rollback option.

The standard way to run Kafka MirrorMaker on Kubernetes is MirrorMaker 2 managed by the Strimzi operator, declared as a KafkaMirrorMaker2 custom resource. Two alternatives exist — a plain Deployment running the dedicated connect-mirror-maker.sh driver, or MirrorMaker 2 connectors added to a Kafka Connect cluster you already operate — and each earns its place in a specific situation.

This guide compares the three patterns, walks through a current Strimzi manifest field by field, and covers the decisions Kubernetes forces on you: scaling, placement, replication policy, cross-cluster networking, and what to watch once it runs.

To change Kafka replication factor, there is no single alter command. You change it by reassigning partitions: write a JSON plan that lists the full target replica set for each partition, then execute and verify that plan with kafka-reassign-partitions.sh. Done casually, the same procedure can saturate your network, stall producers, or leave throttles choking replication for weeks.

This is the runbook for doing it safely. It covers the preflight checks, the JSON plan, throttled execution, progress monitoring, verification, rollback, and the failure modes at each step. If you are still deciding what value to run, the replication factor guide covers choosing; this post covers changing.

How Kafka replication works in one sentence: Kafka copies each topic partition to several brokers, one copy leads, and the others follow, ready to take over. When the broker holding the leader copy fails, a follower becomes the new leader and clients carry on. That single idea is why a Kafka cluster survives dead disks and crashed machines without losing acknowledged data.

This guide is for readers meeting replication for the first time. It builds the mental model piece by piece: the vocabulary, the journey of one message, what happens during a broker failure, and a topic you can create yourself to watch it all work.

Kafka backup compliance comes down to four obligations that appear, in different words, in almost every regulation: retain topic data for a defined period, restore it on demand, prove both with tamper-evident records, and erase personal data when the law requires it. No regulation names Kafka. Auditors simply treat the events in your topics as regulated records — and broker retention, replication factor, and disk durability satisfy none of the four obligations.

Most Kafka estates that fail an audit fail on process, not technology. Retention was set by disk budget instead of regulation, no copy exists outside the cluster, and nobody can produce evidence that a restore has ever worked. This guide maps GDPR, SOX, HIPAA, and PCI DSS to the specific backup controls that close those gaps.

Kafka backup cost comes down to one dominant term — stored gigabytes multiplied by how long you keep them — plus smaller charges for requests, data transfer, and the compute that runs the backup job. Three levers cut that dominant term, and they multiply: compression shrinks every byte you store, incremental backups stop you from re-storing bytes you already have, and storage-class lifecycle rules move aged bytes to tiers that cost up to 23x less.

A naive setup — uncompressed daily fulls parked in S3 Standard — routinely costs 10 to 50 times more than an optimized pipeline holding the same data with better recovery guarantees. This guide breaks down where the money goes, walks each lever with real configuration, and prices a worked example at the end.