Kafka Under-Replicated Partitions: Diagnosis and Fixes
Kafka under-replicated partitions (URPs) are partitions whose in-sync replica (ISR) count has dropped below their replication factor. One or more follower replicas has either died or fallen behind the leader for longer than replica.lag.time.max.ms — 30 seconds by default. Every broker reports the count through the UnderReplicatedPartitions JMX gauge, and any value above zero means the cluster is running with less redundancy than you configured.
URP is the most-watched health metric in Kafka operations for a simple reason: it measures how much failure margin remains before an outage becomes data loss. This guide covers what the metric means, the commands that find the failing broker, a fix for each root cause, and the alert thresholds that separate routine maintenance from a real incident.
Diagnose URPs with kafka-topics.sh --describe --under-replicated-partitions and look for the broker ID missing from the ISR across many partitions. Transient URPs during rolling restarts are normal; sustained URPs mean a dead or slow follower. Alert on duration, not on the first nonzero sample.
What under-replicated partitions actually mean
Replication factor is intent; the ISR is reality. When you create a topic with replication factor 3, Kafka places three replicas of each partition on three brokers. The ISR is the subset of those replicas that is currently caught up with the leader. A partition is under-replicated the moment its ISR is smaller than its replica list.
A follower leaves the ISR when it stops keeping up. Specifically, the leader removes a follower that has not sent a fetch request, or has not caught up to the leader's log end offset, within replica.lag.time.max.ms. The default is 30 seconds (raised from 10 seconds in Kafka 2.5 by KIP-537, to stop routine GC pauses from causing spurious ISR shrinks).
Why the count matters: each replica missing from the ISR is one broker failure the partition can no longer survive. A replication factor 3 partition with two in-sync replicas tolerates one more failure, not two. With acks=all, the durability guarantee also weakens — the leader only waits for the replicas still in the ISR, so writes land on fewer copies than the replication factor promises.
One distinction keeps on-call sane: transient versus sustained. A rolling broker restart legitimately produces URPs while the restarted broker catches back up, and they clear on their own. URPs that persist beyond a few minutes, or that appear without any maintenance in progress, are an incident.
URP vs under-min-ISR: three severity levels
Under-replicated partitions is actually the mildest of three related conditions. Kafka exposes a separate metric for each, and they deserve different responses.
| Condition | Metric (JMX MBean) | Producer impact | Severity |
|---|---|---|---|
| Under-replicated | kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions | None yet — reduced redundancy | Investigate |
| Below min ISR | kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount | acks=all producers fail with NotEnoughReplicas | Page |
| Offline | kafka.controller:type=KafkaController,name=OfflinePartitionsCount | No leader — partition fully unavailable | Page |
The middle row is where degradation becomes an outage. When the ISR shrinks below the topic's min.insync.replicas, every producer using acks=all gets NotEnoughReplicas errors and writes stop. With the common configuration of replication factor 3 and min.insync.replicas=2, that means two of the three replicas are already gone — one more failure than a plain URP.
Two supporting rates round out the picture: IsrShrinksPerSec and IsrExpandsPerSec (both under kafka.server:type=ReplicaManager). A healthy cluster shows near-zero on both. Constant shrink-expand flapping means a follower is chronically on the edge — usually a saturated disk or network, not a dead broker.
Diagnose: find the affected partitions and the common broker
Start with one command. It lists every partition whose ISR is smaller than its replica set:
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --under-replicated-partitions
The output shows each affected partition with its leader, replica list, and current ISR:
Topic: orders Partition: 3 Leader: 1 Replicas: 1,2,3 Isr: 1,3
Topic: orders Partition: 7 Leader: 3 Replicas: 3,2,1 Isr: 3,1
Topic: payments Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,3
Read it for the pattern, not the individual rows. Here broker 2 appears in every Replicas list but no Isr list — the problem is broker 2, not the topics. When one broker ID is missing from the ISR across many partitions, you have found the culprit, and the remaining diagnosis is about that broker.
From there, work down a short tree:
- Is the broker down? Check the process and the broker's availability. A crashed or partitioned broker is the most common cause and the most obvious fix.
- Is the broker up but slow? A running follower that cannot keep up points to a resource bottleneck: saturated disk I/O, a congested network link, or long garbage-collection pauses. Check disk latency and GC logs on that broker first.
- Is it spread across brokers? URPs with no common broker suggest a cluster-level cause: overall load has outgrown capacity, replication fetchers are starved, or partition placement is skewed so a few brokers carry most of the leadership.
Also check how close you are to the next severity level. The same tool reports partitions whose ISR has shrunk exactly to min.insync.replicas — one more failure blocks producers:
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --at-min-isr-partitions
If that list is non-empty, treat the incident with page-level urgency even though producers still work. The margin is one broker.
Fix each cause
The fix follows the diagnosis. This table maps the common root causes to their remedies:
| Root cause | Signal | Fix |
|---|---|---|
| Broker crashed or restarted | Broker ID absent from ISR everywhere, process down | Restart the broker; followers rejoin the ISR automatically once caught up |
| Disk failure | Broker up, log directory errors | Replace the disk or broker; reassign replicas if the broker is not coming back |
| Slow follower: disk or network saturation | Broker up, ISR flapping, high I/O wait | Remove the bottleneck; consider raising num.replica.fetchers (default 1) |
| Long GC pauses | ISR shrink-expand cycles matching GC log pauses | Tune heap size and collector settings on that broker |
| Skewed partition placement | A few brokers persistently hot, URPs cluster there | Rebalance with kafka-reassign-partitions.sh, always throttled |
| Broker permanently lost | Hardware gone, not returning | Reassign its replicas to surviving brokers |
Three of these deserve more detail.
A restarted broker heals itself. Its followers re-fetch from each partition leader and rejoin the ISR as soon as they reach the log end offset. URPs during this catch-up window are expected — watch IsrExpandsPerSec climb as replicas rejoin, and only escalate if the count stops falling.
Slow followers often need more fetcher threads. num.replica.fetchers controls how many threads a broker uses to replicate from each source broker, and the default is 1. On brokers hosting many partitions, a single fetcher thread per source can become the ceiling that keeps followers permanently behind. Raising it increases replication parallelism at the cost of more connections and I/O concurrency — change it deliberately, not reflexively.
Reassignment must be throttled. Moving replicas copies whole partitions between brokers, and unthrottled reassignment traffic can saturate the same disks and network links that caused the URPs. Set a limit when you execute:
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
--reassignment-json-file reassign.json \
--execute --throttle 50000000
That caps replication traffic for the moved partitions at 50 MB/s. When the move completes, run the same command with --verify — it confirms completion and removes the throttle. A forgotten throttle silently limits normal replication traffic and can itself cause under-replicated partitions later.
After a broker recovers, leadership is often left unbalanced, because leaders failed over during the outage and did not move back. Restore the preferred leaders so load spreads evenly again:
kafka-leader-election.sh --bootstrap-server localhost:9092 \
--election-type preferred --all-topic-partitions
Alerting on under-replicated partitions
The three severity levels map directly to an alerting policy:
| Alert | Condition | Response |
|---|---|---|
| Sustained URPs | UnderReplicatedPartitions > 0 for 10 minutes | Investigate within the hour |
| Below min ISR | UnderMinIsrPartitionCount > 0 | Page immediately — producers are failing |
| Offline partitions | OfflinePartitionsCount > 0 | Page immediately — data is unavailable |
The duration clause on the first rule is the important part. An alert on the first nonzero URP sample fires during every rolling restart and every scheduled deploy, and a rule that fires during routine maintenance trains the on-call rotation to ignore it. Ten minutes rides out a normal restart-and-catch-up cycle; tune the window to how long your brokers actually take to rejoin the ISR.
This is the same alert philosophy that applies to data protection pipelines: alert on conditions that persist, page on conditions that block writes, and never rely on a signal that maintenance routinely trips. The Kafka backup monitoring guide applies the same model to backup lag and stalls.
URPs are a replication signal, not a backup
It is tempting to read UnderReplicatedPartitions: 0 as "our data is safe." It means something narrower: every partition currently has its full set of in-sync copies inside this cluster. Replication protects against broker failure — and only against broker failure.
A full ISR replicates everything, including damage. A bad deploy that writes corrupt records, an accidental topic deletion, a misconfigured retention policy that expires data early: all of these propagate to every replica in milliseconds, with URP reading zero throughout. The replication factor guide covers this boundary in depth — replication factor is one protection layer, not the whole strategy.
There is also a quieter implication in the URP metric itself. Every minute a cluster spends under-replicated is a minute of reduced failure margin, and clusters under sustained load pressure spend more of those minutes than their dashboards suggest. An independent backup in object storage — S3, GCS, or Azure Blob — is the layer whose durability does not depend on the cluster's own health. OSO Kafka Backup provides that layer with point-in-time recovery and consumer group offset preservation, so a restore puts both the data and the consumers back where they were.
For the full picture of recovering a Kafka estate after the failures replication cannot absorb, see the disaster recovery use case.
Zero is normal, sustained is an incident
UnderReplicatedPartitions is the fastest-reading health gauge Kafka gives you. Zero is normal. Brief spikes accompany restarts and reassignments and clear on their own. Sustained values mean a dead or slow follower, and one kafka-topics.sh --describe --under-replicated-partitions run usually names the broker. Fix by cause, throttle every reassignment, alert on duration — and keep an independent backup for the failure modes that a perfectly replicated cluster copies faithfully to every replica.
Frequently asked questions
What is under replicated partitions Kafka?
An under-replicated partition is one whose in-sync replica count is below its replication factor. A follower replica has died or fallen more than replica.lag.time.max.ms (default 30 seconds) behind the leader. The broker reports the count as the UnderReplicatedPartitions JMX metric, and any value above zero means reduced redundancy.
How to fix under replicated partitions in Kafka?
Run kafka-topics.sh --describe --under-replicated-partitions and find the broker ID missing from the ISR lists. If that broker is down, restart it and its replicas rejoin automatically. If it is up but slow, fix the disk, network, or GC bottleneck, or raise num.replica.fetchers. If the broker is permanently lost, reassign its replicas with a throttled kafka-reassign-partitions.sh run.
What is in sync replica in Kafka?
An in-sync replica (ISR) is a replica that is currently caught up with the partition leader — it has fetched up to the leader log end offset within replica.lag.time.max.ms. The ISR is the set of replicas eligible to become leader without data loss, and acks=all writes wait for it.
How does Kafka replicate partitions?
Each partition has one leader replica and zero or more followers on other brokers. Producers and consumers work through the leader while followers continuously fetch new records from it. Followers that keep up stay in the ISR; if the leader fails, a follower from the ISR takes over. Replication protects against broker failure but copies deletions and corruption too, which is why it complements rather than replaces backups.