What Is ZooKeeper in Kafka? (and Why It's Going Away)
So what is ZooKeeper in Kafka, exactly? ZooKeeper is the external coordination service that ZooKeeper-mode Kafka clusters use to elect the controller, track which brokers are alive, and store cluster metadata — topics, partition assignments, configs, and ACLs. It is also on its way out: Apache Kafka deprecated ZooKeeper mode in 3.5 and removed it entirely in 4.0, replacing it with KRaft, a metadata quorum built into Kafka itself.
You still see ZooKeeper everywhere. Old tutorials start it before the broker, inherited clusters have zookeeper.connect in every properties file, and interview questions treat it as core Kafka knowledge. This post explains what the service actually does, what it never stored, what breaks when it fails, and why the whole layer is being retired.
ZooKeeper stores Kafka's coordination metadata: controller identity, broker liveness, topic definitions, configs, and ACLs. It never stores your message data, and since Kafka 0.9 it does not store consumer offsets either. The entire layer is deprecated, and Kafka 4.0 ships without it.
What is ZooKeeper in Kafka?
Apache ZooKeeper is not a Kafka component. It is a standalone Apache project — a centralized coordination service that predates Kafka's use of it — designed to give distributed systems a small, consistent, strictly ordered store for the facts they must agree on: naming, configuration, group membership, and leader election.
Its data model looks like a tiny filesystem. Data lives in a tree of nodes called znodes, each holding a small blob of bytes. Two primitives make it useful for coordination. An ephemeral znode is deleted automatically when the session of the client that created it ends. And any client can set a watch on a znode to be notified when it changes.
Those two primitives are exactly what Kafka built on. A broker proves it is alive by holding an ephemeral znode; everyone else finds out about changes through watches.
ZooKeeper runs as an ensemble — typically three or five servers. Every write commits only when a majority of the ensemble acknowledges it. That majority rule is why ensembles are odd-sized: three nodes tolerate one failure, five tolerate two.
What does ZooKeeper actually do for Kafka?
In a ZooKeeper-mode cluster, ZooKeeper carries six coordination jobs. Each one has a direct successor in KRaft mode, which is a useful way to see that the responsibilities survive even though the technology changes:
| Responsibility | ZooKeeper mechanism | Where it lives in KRaft |
|---|---|---|
| Controller election | First broker to create the ephemeral /controller znode wins | Raft leader election within the controller quorum |
| Broker liveness | Ephemeral znodes under /brokers/ids | Broker heartbeats to the controller |
| Topic metadata and replica assignments | Znodes under /brokers/topics | Records in the __cluster_metadata log |
| Dynamic configuration | Znodes under /config | Records in the metadata log |
| ACLs | Stored in ZooKeeper by the ZooKeeper-based AclAuthorizer | StandardAuthorizer backed by the metadata log |
| Cluster identity | The /cluster/id znode | meta.properties plus the metadata log |
The two jobs worth understanding in detail are election and liveness, because they explain most ZooKeeper-related incidents.
Controller election. One broker in every ZooKeeper-mode cluster acts as the controller: it decides partition leadership and propagates cluster state. The election is brutally simple — the first broker to create the ephemeral /controller znode is the controller. If its ZooKeeper session dies, the znode vanishes, every broker's watch fires, and the race runs again.
Broker membership. Each broker registers an ephemeral znode under /brokers/ids when it starts. If the broker crashes or is partitioned away long enough for its session to expire, the znode disappears and the controller treats the broker as dead. It then elects new leaders for every partition that broker led.
Everything in the tree is plainly inspectable. This command reads the cluster ID straight out of ZooKeeper — the same ID a KRaft migration must carry over:
bin/zookeeper-shell.sh localhost:2181 get /cluster/id
What ZooKeeper does not store
The most common misconception is that ZooKeeper holds Kafka's data. It holds pointers about the data — assignments, configs, identities — never the data itself.
Message data: never. Partition logs live on broker disks. ZooKeeper knows which brokers host which replicas; it has no idea what is inside them.
Consumer offsets: not since 2015. Only the old pre-0.9 consumer committed offsets to ZooKeeper. Modern Kafka stores them in __consumer_offsets, an internal topic replicated across the brokers like any other. That detail matters more than it looks: offsets are regular topic data, which is why they survive an in-place KRaft migration — and why a backup tool can capture them.
Client connections: none. Modern producers and consumers never talk to ZooKeeper. They connect to brokers via bootstrap.servers and learn cluster topology from the brokers themselves. Kafka 3.0 removed the --zookeeper flag from the main admin tools, so even operators go through --bootstrap-server now.
There is a data-protection corollary here. Because neither ZooKeeper nor the KRaft quorum holds topic data, a healthy metadata layer says nothing about whether your data is recoverable. Replication does not change that — replicas copy corruption and deletes as faithfully as they copy good writes, which is why disaster recovery needs an independent copy, not just more replicas.
What happens to Kafka when ZooKeeper goes down?
The cluster degrades; it does not instantly die. Existing partition leaders keep serving produces and fetches, because the data path runs broker-to-broker. What stops is everything that needs coordination: there is no controller, so no leader elections, no topic creation, no configuration changes, and no broker re-registration.
The danger is the compound failure. If a broker dies while ZooKeeper is unavailable, nobody can elect new leaders for its partitions. Those partitions stay unavailable until the ZooKeeper quorum returns — a broker failure that would normally cost seconds of leadership transfer instead lasts as long as the outage does.
This is the lived pain behind the architecture change. Running ZooKeeper-mode Kafka means operating two distributed systems, and the second one's quorum math gates the first one's ability to heal itself.
Why ZooKeeper is going away
The removal was proposed in KIP-500, accepted in 2019, and the reasoning comes down to three problems.
Metadata wanted to be a log. Kafka's own worldview is ordered, replicated event logs — yet its most critical state sat in a tree of znodes updated piecemeal. KRaft makes cluster metadata an ordered log, __cluster_metadata, replicated across a controller quorum with Raft consensus, and brokers simply tail it like consumers.
Controller failover was too slow at scale. A newly elected ZooKeeper-mode controller must load the full cluster state out of ZooKeeper before it can act, and that pause grows with partition count. KRaft standby controllers already hold the metadata log, so failover stops being a state-reload problem.
Two systems is one too many. Every ZooKeeper-mode deployment carries a second cluster to size, secure, patch, monitor, and page on — with its own security model that never quite matched Kafka's. KRaft collapses that to one system with one configuration and one authorizer.
How the controller quorum actually works — voters, observers, and the kafka-metadata-quorum tooling — deserves its own post. For the purposes of this one: the responsibilities in the table above all survive; only their home changes.
The removal timeline (and what to do about it)
The deprecation is not a warning; it is a completed process with dates attached:
| Kafka version | What changed |
|---|---|
| 2.8 (2021) | KRaft ships as early access |
| 3.3 | KRaft declared production-ready for new clusters |
| 3.5 | ZooKeeper mode officially deprecated |
| 3.6 | ZooKeeper-to-KRaft migration production-ready |
| 3.9 | The bridge release — last version with ZooKeeper support |
| 4.0 | ZooKeeper removed; Kafka will not start against it |
Managed services follow the same arc: AWS MSK offers KRaft on new clusters from Kafka 3.7.x and provides no in-place conversion, so ZooKeeper-based MSK clusters face a cluster move rather than a rolling migration.
If you are weighing what this means for your own clusters, the decision — and the deprecation questions that come with it — are covered on the KRaft vs ZooKeeper comparison. When the decision is made, the ZooKeeper to KRaft migration guide walks the procedure for self-managed Kafka, Strimzi, Confluent Platform, and MSK.
One position we hold firmly: back up before you touch the metadata layer. A migration is a planned, high-blast-radius change to the cluster's brain, executed through multiple rolling restarts. Kafka Backup takes an independent, point-in-time copy of topic data and consumer offsets to S3, Azure Blob, GCS, or filesystem storage before you start — see the migration use case and getting started guide.
FAQ
Frequently asked questions
What does Kafka store in ZooKeeper?
ZooKeeper-mode Kafka stores coordination metadata in ZooKeeper: the current controller (/controller), live broker registrations (/brokers/ids), topic definitions and partition replica assignments (/brokers/topics), dynamic configuration (/config), ACLs via the ZooKeeper-based authorizer, and the cluster ID (/cluster/id). Message data and modern consumer offsets are never stored there.
How many ZooKeeper nodes does a Kafka cluster need?
A production ensemble needs an odd number of nodes so a majority quorum survives failures: three nodes tolerate one failure, five tolerate two. Writes commit only when a majority acknowledges them, so an even-sized ensemble adds cost without adding failure tolerance.
What happens to Kafka when ZooKeeper goes down?
The cluster degrades rather than dying immediately. Existing partition leaders keep serving producers and consumers, but with no controller there are no leader elections, topic creations, config changes, or broker re-registrations. If a broker fails during the outage, its partitions stay unavailable until the ZooKeeper quorum returns.
Do Kafka producers and consumers connect to ZooKeeper?
No. Modern clients connect only to brokers via bootstrap.servers and discover cluster topology from them. Only brokers talk to ZooKeeper, and Kafka 3.0 removed the --zookeeper flag from the main admin tools, so operator tooling also goes through --bootstrap-server.
Does Kafka store consumer offsets in ZooKeeper?
Not since Kafka 0.9. Only the legacy pre-0.9 consumer committed offsets to ZooKeeper. Modern Kafka stores them in the internal __consumer_offsets topic on the brokers, which is why offsets survive an in-place KRaft migration and can be captured by backup tooling.
Conclusion
ZooKeeper was Kafka's external brain for over a decade: it elected the controller, tracked broker liveness, and held the metadata that defines a cluster. It did that job well — and its replacement moves the same responsibilities inside Kafka, as a Raft-replicated metadata log, removing the second distributed system from every deployment.
If you run ZooKeeper-mode Kafka today, the question is no longer whether to move but when and how. Kafka 4.0 has already shipped without ZooKeeper, and 3.9 is the last bridge.
Whatever metadata mode you run, topic data and consumer offsets are what you cannot lose. Kafka Backup captures both to S3, Azure, GCS, or filesystem storage with point-in-time recovery — take an independent backup before any migration. Get started.