KRaft Controller Quorum Explained: Voters, Observers, and kafka-metadata-quorum
The KRaft controller quorum is the small group of Kafka nodes that replicate the cluster metadata log and elect the active controller. A majority of these voters must acknowledge every metadata change before it commits. Each Kafka KRaft controller is either the active leader or a hot standby, and every broker follows the same log as a non-voting observer.
Our KRaft mode guide covers the architecture. This post covers running the quorum: how elections work, how to read kafka-metadata-quorum.sh output, how static and dynamic quorums differ, and how to add or remove controllers without breaking the majority.
The quorum is the voter set: controllers that replicate __cluster_metadata and elect a leader. Brokers are observers. Check health with kafka-metadata-quorum.sh describe. Membership is static (controller.quorum.voters) or dynamic (controller.quorum.bootstrap.servers, KIP-853, Kafka 3.9+) — and dynamic is the documented path for new clusters.
What is the KRaft controller quorum?
The quorum is the set of voters: the controller nodes that hold a replicated copy of the cluster metadata log, vote in leader elections, and count toward the commit majority. One voter at a time is the active controller — the Raft leader that writes to the log. The others are hot standbys that replicate it and stay ready to take over.
Majority rules everything here. A quorum of 2N + 1 voters tolerates N concurrent failures, so three controllers survive one loss and five survive two. The metadata log's high-watermark only advances once a majority of voters has replicated a record — that is the moment a topic creation, config change, or broker registration becomes durable.
Brokers participate too, but differently. They are observers: they fetch and replay the same metadata log to learn cluster state, and they never vote. When quorum tooling lists CurrentObservers, those are your brokers following along.
How the quorum elects a leader
Kafka's Raft implementation (KIP-595) is pull-based. Voters and observers fetch from the leader — the same direction data replication already flows in Kafka — instead of the leader pushing entries out as in classical Raft. Elections hang off that fetch traffic.
A voter that goes too long without a successful fetch assumes the leader is gone, becomes a candidate, and asks the other voters for their vote. The same clock runs in reverse on the leader: if it stops hearing fetch requests from a majority, it resigns. Three configurations control the timing:
| Config | Default | What it controls |
|---|---|---|
controller.quorum.fetch.timeout.ms | 2000 | How long a voter goes without a successful fetch from the leader before triggering an election — and how long a leader tolerates silence from a majority before resigning |
controller.quorum.election.timeout.ms | 1000 | Maximum wait, once fetching has stalled, before a voter starts a new election |
controller.quorum.election.backoff.max.ms | 1000 | Cap on the exponential backoff between failed election rounds — prevents gridlocked elections |
The defaults suit most clusters. Raise the fetch timeout only if controller network links are genuinely slow, because every millisecond you add extends metadata-plane downtime during a real failure.
Failover itself is fast. A standby that wins an election already holds the full replicated state in memory, so there is no rebuild step — the property that makes KRaft failover near-instant compared to the old ZooKeeper-backed controller.
Checking quorum health with kafka-metadata-quorum
The kafka-metadata-quorum.sh tool describes the runtime state of the metadata quorum. Start with the summary:
$ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
ClusterId: fMCL8kv1SWm87L_Md-I2hg
LeaderId: 3002
LeaderEpoch: 2
HighWatermark: 10
MaxFollowerLag: 0
MaxFollowerLagTimeMs: -1
CurrentVoters: [{"id": 3000, "directoryId": "ILZ5MPTeRWakmJu99uBJCA", "endpoints": ["CONTROLLER://localhost:9093"]}, ...]
CurrentObservers: [{"id": 0, "directoryId": "3Db5QLSqSZieL3rJBUUegA"}, ...]
LeaderId names the active controller and HighWatermark marks the last committed metadata offset. CurrentVoters lists the controllers with their directory IDs and endpoints; CurrentObservers lists the brokers. For per-node detail, use the replication view:
$ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication
NodeId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
3002 10 0 ... ... Leader
3000 10 0 ... ... Follower
3001 10 0 ... ... Observer
Healthy looks like this: exactly one Leader, followers at or near the leader's log end offset, and MaxFollowerLagTimeMs low. A follower whose lag keeps growing has a fetch problem brewing — the same signal that, left alone, eventually triggers an election. An observer falling behind means that broker is acting on stale cluster metadata.
Static vs dynamic quorums (KIP-853)
There are two ways to define quorum membership, and the difference decides how much operational freedom you have:
| Static quorum | Dynamic quorum | |
|---|---|---|
| Config key | controller.quorum.voters | controller.quorum.bootstrap.servers |
| Entry format | {id}@{host}:{port} | {host}:{port} |
| Membership | Fixed in configuration on every node | Add and remove controllers at runtime |
kraft.version feature level | 0 (or absent) | 1 |
| Introduced | Original KRaft configuration | Kafka 3.9 (KIP-853) |
| Documented path for new clusters | No | Yes |
Static quorums hard-code every controller's ID, host, and port into every broker and controller config. Changing a hostname, or adding a controller, means touching configuration across the whole cluster. Dynamic quorums, added by KIP-853 in Kafka 3.9, store membership in the metadata log itself. The controller.quorum.bootstrap.servers list works like a client's bootstrap.servers — it only needs enough entries to find the quorum.
Not sure which you are running? Ask the feature system:
$ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe
Feature: kraft.version ... FinalizedVersionLevel: 1 ...
Feature: metadata.version ... FinalizedVersionLevel: 4.0-IV3 ...
If kraft.version is finalized at 0 or missing, the quorum is static. Kafka 3.9 could not convert a static quorum to dynamic; that upgrade arrived in Kafka 4.1. On 4.1 or later, run bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature kraft.version=1, then replace controller.quorum.voters with controller.quorum.bootstrap.servers on every node. The version floor matters when planning: Kafka 4.0 raised the baselines that make these upgrades reachable.
Adding and removing controllers
Every node in a KRaft cluster is formatted before first start, and the quorum's membership mode is fixed at that moment. Generate one cluster ID and reuse it everywhere:
$ bin/kafka-storage.sh random-uuid
$ bin/kafka-storage.sh format --cluster-id <CLUSTER_ID> --standalone --config config/controller.properties
Kafka stopped auto-formatting blank directories deliberately. If a majority of controllers could start on empty logs, they could elect a leader that is missing committed data — explicit formatting turns that silent corruption into a loud startup error.
Three formatting flags cover the membership cases:
--standalone— bootstrap a new dynamic quorum with this node as the only voter. The documented starting point for new clusters.--initial-controllers— bootstrap with several voters at once, each described asid@host:port:directoryId. The value must be identical on every initial controller.--no-initial-controllers— format a node that will join an existing cluster.
Growing a dynamic quorum is a four-step flow:
- Format the new controller with
--no-initial-controllersand the existing cluster ID. - Start it — it joins as an observer and begins replicating the metadata log.
- Watch
kafka-metadata-quorum.sh describe --replicationuntil its lag reaches zero. - Promote it:
bin/kafka-metadata-quorum.sh --command-config config/controller.properties --bootstrap-server localhost:9092 add-controller
Shrinking runs through remove-controller, and order matters. Until KIP-996 pre-vote ships, the documentation recommends shutting the controller down before removing it, so a stale voter cannot disturb elections on its way out:
$ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 \
remove-controller --controller-id <id> --controller-directory-id <directory-id>
All of this assumes you are on KRaft already. Clusters still on ZooKeeper get none of the membership tooling until they migrate to KRaft — our migration guide covers that path end to end.
Inspecting the metadata log: kafka-dump-log and kafka-metadata-shell
The quorum's whole world is one single-partition internal topic: __cluster_metadata. It lives in metadata.log.dir (by default, the first directory in log.dirs) under __cluster_metadata-0. Kafka compacts it with snapshots — a new snapshot after 20 MB of records (metadata.log.max.record.bytes.between.snapshots) or one hour (metadata.log.max.snapshot.interval.ms), whichever comes first.
Two read-only tools decode it. kafka-dump-log.sh prints the raw records from a segment or a snapshot:
$ bin/kafka-dump-log.sh --cluster-metadata-decoder \
--files metadata_log_dir/__cluster_metadata-0/00000000000000000000.log
kafka-metadata-shell.sh loads a snapshot into an interactive tree you can browse like a filesystem:
$ bin/kafka-metadata-shell.sh --snapshot \
metadata_log_dir/__cluster_metadata-0/00000000000000007228-0000000001.checkpoint
>> ls /
brokers local metadataQuorum topicIds topics
>> cat /topics/foo/0/data
{
"partitionId" : 0,
"topicId" : "5zoAlv-xEh9xRANKXt1Lbg",
"replicas" : [ 1 ],
"isr" : [ 1 ],
...
}
One trap from the docs: the all-zeros bootstrap file 00000000000000000000-0000000000.checkpoint contains membership records, not cluster metadata. Point the shell at a real snapshot. And treat both tools as debuggers — they inspect state, they do not edit it.
Sizing is modest. The controllers keep all metadata in memory and on disk, and the documentation calls 5 GB of memory plus 5 GB of disk on the metadata log directory sufficient for a typical cluster. Run dedicated controller nodes in production — combined broker,controller processes cannot be rolled or scaled separately.
What the quorum means for your backup and DR posture
The quorum replicates metadata, not data. Topic data lives in partition logs on the brokers, and consumer positions live in the internal __consumer_offsets topic — also on the brokers. Losing a quorum majority is an availability incident: metadata writes stop until voters return. Losing broker data is a durability incident, and the quorum does nothing to prevent it. Replication is not backup, on either plane.
That split has practical consequences. Monitor quorum lag with describe --replication, but protect the data plane independently: Kafka Backup writes topic data and consumer group offsets to S3, Azure Blob, GCS, or filesystem storage, with point-in-time recovery when something destructive replicates faithfully across the cluster.
Quorum surgery deserves the same respect as any change window. Adding controllers, removing them, or upgrading kraft.version all mutate the metadata plane of a running cluster — take a verified backup before you start, so a bad afternoon stays recoverable. On Kubernetes, the operator is Strimzi compatible and works with dedicated controller pools.
FAQ
Frequently asked questions
How do I check the status of a Kafka KRaft quorum?
Run bin/kafka-metadata-quorum.sh describe --status for a summary showing the leader, high-watermark, voters, and observers, or describe --replication for per-node log end offsets, lag, and fetch timestamps. A healthy quorum shows one Leader with followers at or near its offset.
What is the __cluster_metadata topic in Kafka?
It is the single-partition internal topic that holds the KRaft metadata log. Controllers replicate it through Raft voting, brokers replay it as observers to learn cluster state, and Kafka compacts it with periodic snapshots. It lives in metadata.log.dir under the directory __cluster_metadata-0.
What is the difference between a KRaft voter and an observer?
Voters are controllers: they replicate the metadata log, vote in leader elections, and count toward the commit majority. Observers are brokers: they fetch and replay the same log to stay current but never vote. The kafka-metadata-quorum tool lists both as CurrentVoters and CurrentObservers.
Can I add or remove KRaft controllers without downtime?
Yes, on a dynamic quorum. KIP-853, shipped in Kafka 3.9, lets you add a caught-up controller with kafka-metadata-quorum.sh add-controller and remove one with remove-controller. Static quorums cannot do this — converting a static quorum to dynamic requires Kafka 4.1 and a kraft.version upgrade to level 1.
What is the kafka-metadata-shell tool?
kafka-metadata-shell.sh is a read-only interactive inspector for KRaft metadata snapshots. It loads a checkpoint file and presents cluster state as a browsable tree with brokers, topics, and quorum information. It needs a real snapshot — the bootstrap checkpoint ending in -0000000000 contains no cluster metadata.
Conclusion
The KRaft controller quorum is a small Raft cluster living inside Kafka: voters replicating a metadata log, one leader writing to it, brokers observing it. Once you know your membership mode — static config or KIP-853 dynamic — the operational surface is pleasantly small. Watch describe --replication, keep a majority healthy, and treat every controller change as a real change window.
And keep the division of labor straight: the quorum guards metadata consistency, nothing more. Your topic data and consumer offsets need their own protection plan.
KRaft controllers replicate cluster metadata, not topic data. Kafka Backup writes topic data and consumer group offsets to S3, Azure, GCS, or filesystem storage with point-in-time recovery — protection the quorum was never designed to give. Get started.