Skip to main content

51 posts tagged with "Kafka Backup"

Backup strategies, tooling, and operational guidance for Apache Kafka.

View All Tags

Kafka Point-in-Time Recovery, Explained

· 10 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka point-in-time recovery (PITR) restores topic data to a specific moment — down to the millisecond — by filtering backed-up records on the timestamps they already carry. Kafka itself has no rewind button. Retention deletes forward, and replication copies corruption as faithfully as it copies good data. PITR therefore needs two things the platform does not provide: an immutable copy of the data outside the cluster, and a restore engine that filters that copy by time.

This post explains the mechanism end to end: why native Kafka features cannot rewind topic state, how timestamp filtering works, what happens to consumer groups, and the caveats — clock skew, transactions, compaction — that decide whether the restore you get matches the restore you planned.

Running Kafka in KRaft Mode with Docker Compose

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

Running Kafka in KRaft mode with Docker Compose takes one service and a handful of environment variables. The official apache/kafka image runs KRaft out of the box — no ZooKeeper container, no migration steps, and docker run -p 9092:9092 apache/kafka:4.3.1 alone gives you a working broker. A short Compose file makes that broker reproducible, and a slightly longer one gives you a 3-controller, 3-broker cluster that mirrors production topology.

This post walks through both files, the env-var conventions behind them, and how to keep data across restarts. For what KRaft mode actually is, see our KRaft guide.

KRaft Controller Quorum Explained: Voters, Observers, and kafka-metadata-quorum

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

The KRaft controller quorum is the small group of Kafka nodes that replicate the cluster metadata log and elect the active controller. Membership is controller.quorum.voters (static quorum) or controller.quorum.bootstrap.servers (dynamic quorum, KIP-853), and a majority of these voters must acknowledge every metadata change before it commits. Each Kafka KRaft controller is either the active leader or a hot standby, and every broker follows the same log as a non-voting observer.

Our KRaft mode guide covers the architecture. This post covers running the quorum: how elections work, how to read kafka-metadata-quorum.sh output, how static and dynamic quorums differ, and how to add or remove controllers without breaking the majority.

Kafka 4.0: What Changed and What It Means for Your Cluster

· 10 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka 4.0, released on March 18, 2025, is the first Apache Kafka that runs entirely without ZooKeeper — and it raises the floor on almost everything else at the same time. Old clients, old message formats, old Java versions, and the original MirrorMaker all exit in the same release.

That makes 4.0 less a feature release than a floor-raising release. The interesting question is not "what shiny thing do I get?" but "what do I have to be running before I can get there?" This post covers both: what shipped, what was removed, and the upgrade path that connects your cluster to it.

What Is Kafka KRaft Mode? Architecture, Configuration, and Readiness

· 9 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka KRaft (Kafka Raft) is Kafka's built-in metadata layer: a small quorum of controller nodes that stores cluster metadata in an internal, replicated Kafka log. It replaces Apache ZooKeeper entirely — no external coordination system, no second cluster to run.

KRaft now appears in every modern Kafka conversation: in release notes, in the process.roles line of every new config file, in vendor migration deadlines. This post explains the thing itself — what KRaft is, how the architecture works, which properties define a KRaft node, and where it stands on production readiness.

Kafka Without ZooKeeper: Is ZooKeeper Deprecated?

· 9 min read
OSO Engineering
The team behind OSO Kafka Backup

Yes — running Kafka without ZooKeeper is not just possible, it is now the only supported way forward. Kafka has run without ZooKeeper in production since version 3.3, ZooKeeper mode was deprecated in 3.5, and Kafka 4.0 removed it entirely. A 4.x broker will not even start against a ZooKeeper ensemble.

The question arrives in many forms: is ZooKeeper still required? Is it removed? Can I run Kafka without it today? The answer depends entirely on which Kafka version and platform you are on. This post gives the version-by-version answer, lists what still requires ZooKeeper, and shows how to start a cluster with no ZooKeeper anywhere.

What Is ZooKeeper in Kafka? (and Why It's Going Away)

· 10 min read
OSO Engineering
The team behind OSO Kafka Backup

So what is ZooKeeper in Kafka, exactly? ZooKeeper is the external coordination service that ZooKeeper-mode Kafka clusters use to elect the controller, track which brokers are alive, and store cluster metadata — topics, partition assignments, configs, and ACLs. It is also on its way out: Apache Kafka deprecated ZooKeeper mode in 3.5 and removed it entirely in 4.0, replacing it with KRaft, a metadata quorum built into Kafka itself.

You still see ZooKeeper everywhere. Old tutorials start it before the broker, inherited clusters have zookeeper.connect in every properties file, and interview questions treat it as core Kafka knowledge. This post explains what the service actually does, what it never stored, what breaks when it fails, and why the whole layer is being retired.

ZooKeeper to KRaft Migration: The Complete Guide (MSK, Strimzi, Confluent)

· 18 min read
OSO Engineering
The team behind OSO Kafka Backup

How you run a ZooKeeper to KRaft migration depends on where your Kafka lives. Self-managed clusters migrate in place with a rolling, dual-write procedure (KIP-866) through the Kafka 3.9 bridge release. Strimzi drives that same procedure on Kubernetes with a single annotation. Confluent Platform runs it on versions 7.6.1 through 7.9, before the KRaft-only 8.0 upgrade.

Eligible AWS MSK clusters migrate in place as well. The cluster stays available, and no topic data moves. The change cannot be reversed.

The deadline is upstream and fixed: ZooKeeper mode was deprecated in Apache Kafka 3.5 and removed in 4.0. If you are still weighing the two modes, the KRaft vs ZooKeeper comparison settles that question.

This guide assumes the decision is made. It covers the risky part: moving every cluster you have, on each platform, without losing metadata or offsets. It also names the last point where rollback still exists.

Kafka MirrorMaker 2 on Kubernetes: Deployment Patterns That Work

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

The standard way to run Kafka MirrorMaker on Kubernetes is MirrorMaker 2 managed by the Strimzi operator, declared as a KafkaMirrorMaker2 custom resource. Two alternatives exist — a plain Deployment running the dedicated connect-mirror-maker.sh driver, or MirrorMaker 2 connectors added to a Kafka Connect cluster you already operate — and each earns its place in a specific situation.

This guide compares the three patterns, walks through a current Strimzi manifest field by field, and covers the decisions Kubernetes forces on you: scaling, placement, replication policy, cross-cluster networking, and what to watch once it runs.

How to Change Kafka Replication Factor Safely (Step by Step)

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

To change Kafka replication factor, there is no single alter command. You change it by reassigning partitions: write a JSON plan that lists the full target replica set for each partition, then execute and verify that plan with kafka-reassign-partitions.sh. Done casually, the same procedure can saturate your network, stall producers, or leave throttles choking replication for weeks.

This is the runbook for doing it safely. It covers the preflight checks, the JSON plan, throttled execution, progress monitoring, verification, rollback, and the failure modes at each step. If you are still deciding what value to run, the replication factor guide covers choosing; this post covers changing.