The metadata question is settled
ZooKeeper coordinated Kafka reliably for over a decade, and it earned its place. But Apache Kafka deprecated it in 3.5 and removed it entirely in 4.0 — KRaft is now the only metadata layer. The real question left is not which mode to run. It is how to move every ZooKeeper cluster you still have without losing data, offsets, or sleep.
KRaft vs ZooKeeper at a glance
The KRaft vs ZooKeeper decision has been made upstream. KRaft (Kafka Raft) replaces the external ZooKeeper ensemble with a Raft quorum of controllers inside Kafka itself. ZooKeeper mode was deprecated in Apache Kafka 3.5 and removed in Kafka 4.0, so every ZooKeeper-mode cluster needs a migration plan before its next major upgrade.
What changes in practice:
| ZooKeeper mode | KRaft mode | |
|---|---|---|
| Metadata store | External ZooKeeper ensemble | Raft quorum inside Kafka |
| Systems to run, patch, and monitor | Two | One |
| Controller failover | Minutes on large clusters | Seconds |
| Partition scale per cluster | Constrained by ZooKeeper | Millions of partitions |
| Kafka versions | Up to 3.9 (final ZooKeeper release) | 3.3+ for new clusters; the only mode in 4.x |
| Status | Deprecated 3.5, removed 4.0 | Default and required going forward |
Credit where it is due
ZooKeeper was not a mistake. It gave Kafka — and half the distributed-systems ecosystem — dependable leader election, configuration storage, and membership tracking for years. Most of its operational pain was simply the cost of running a second consensus system next to the first one.
KRaft removes that second system. Metadata becomes an ordered event log replicated by a controller quorum, which is why failover drops from minutes to seconds and why partition counts can grow past ZooKeeper's practical limits. Fewer moving parts, one security model, one thing to page you at night.
The Kafka 4.0 deadline
Apache Kafka 4.0 will not start against a ZooKeeper ensemble. Self-managed clusters migrate through Kafka 3.9, the bridge release that supports the documented ZooKeeper-to-KRaft migration. New clusters should simply start in KRaft mode, which has been production-ready since Kafka 3.3.
Managed platforms have their own constraints. AWS MSK supports KRaft from Kafka 3.7.x onward — but MSK offers no in-place ZooKeeper-to-KRaft conversion. Moving means standing up a new KRaft cluster and migrating topic data, configuration, ACLs, and consumer group offsets across. Done by hand, the offsets are usually the casualty: consumers restart from earliest or latest, and you eat duplicates or data loss.
Migrating without losing your offsets
This is the gap OSO Kafka Backup's Enterprise
MSK KRaft migration pipeline closes: a controlled
cutover with a short producer freeze, validated offset translation so consumers resume
exactly where they left off, ACL migration, and a cryptographically signed evidence
bundle proving what moved. The plan and precheck commands are completely free — run
them against production today and get a generated runbook, cost estimate, IAM policies,
and a readiness report.
- Plan (free)
- Precheck (free)
- migration.yaml
kafka-backup migrate msk-kraft plan \
--config migration.yaml \
--format all \
--out-dir ./migration-plan
kafka-backup migrate msk-kraft precheck --config migration.yaml
enterprise:
msk_kraft_migration:
source:
cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/prod-zk-cluster/abc-def-123
auth:
mode: iam
target:
cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/prod-kraft-cluster/ghi-jkl-456
auth:
mode: iam
backup:
s3_bucket: prod-migration-segments
s3_prefix: zk-to-kraft/
evidence:
s3_bucket: prod-migration-evidence
s3_prefix: migrations/
retention: 7y
Whichever path you take — managed pipeline, MirrorMaker, or hand-rolled scripts — take a point-in-time backup before touching cluster metadata. A migration is exactly the kind of change you want an independent, restorable copy sitting outside of. The migration use cases cover the backup-and-restore route for non-MSK clusters too.
Frequently asked questions
Is ZooKeeper deprecated in Kafka?
Yes. ZooKeeper mode was deprecated in Apache Kafka 3.5 and removed entirely in Kafka 4.0. Kafka 3.9 is the final release that supports ZooKeeper and serves as the bridge release for migrating to KRaft.
Can Kafka run without ZooKeeper?
Yes. In KRaft mode, Kafka stores metadata in an internal Raft controller quorum instead of an external ZooKeeper ensemble. KRaft has been production-ready for new clusters since Kafka 3.3 and is the only mode available in Kafka 4.x.
What is the difference between KRaft and ZooKeeper?
ZooKeeper mode keeps cluster metadata in a separate ZooKeeper ensemble that you deploy and operate alongside Kafka. KRaft moves that metadata into a Raft quorum of controllers inside Kafka itself — one system instead of two, controller failover in seconds instead of minutes, and support for far higher partition counts.
Does AWS MSK support KRaft?
Yes, MSK supports KRaft mode from Kafka 3.7.x onward, and new MSK clusters should be provisioned in KRaft mode. MSK does not support in-place ZooKeeper-to-KRaft conversion, so existing ZooKeeper-mode clusters must migrate to a new KRaft cluster.
How do I migrate an MSK cluster from ZooKeeper to KRaft?
Create a new KRaft-mode MSK cluster, then move topic data, configurations, ACLs, and consumer group offsets across. The Enterprise MSK KRaft migration pipeline automates this as a controlled cutover with validated offset continuity and a signed evidence bundle; its plan and precheck stages are free to run without a license.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.