Kafka Retention Policy: retention.ms and retention.bytes
A Kafka retention policy decides when brokers delete old log data. The time limit is retention.ms. The size limit is retention.bytes.
Apache Kafka 4.4 documents both on the topic configs page. Broker defaults are on the broker configs page.
The default Kafka retention period is 7 days. Brokers ship with log.retention.hours set to 168. Two topic settings carry that policy.
retention.ms is the time limit, and it defaults to 604800000 milliseconds. retention.bytes is the size limit on each partition, and it defaults to -1, which means no size cap. Under cleanup.policy=delete, the first limit reached discards old segments.
What is retention.ms in Kafka?
retention.ms is the topic time limit. Apache describes it as the maximum time the log is retained before old segments are discarded, when the cleanup policy is delete. The default is 604800000, which the same page labels as 7 days.
Set it to -1 and no time limit is applied. Valid values start at -1.
The topic setting uses the broker time default when you do not override it. The server default property is log.retention.ms. Its default is null. When it is unset, Kafka uses log.retention.minutes.
That minutes default is also null. When minutes are unset, Kafka uses log.retention.hours. The hours default is 168. 168 hours is 7 days, which is why the topic default is 604800000.
The topic page states the rule for overrides. A server default applies only when the topic has no explicit config override. Set retention.ms on one topic and that topic stops using the broker time chain.
Hours and minutes are read-only broker settings, so a change needs a process restart. log.retention.ms is cluster-wide. The topic override is a separate path, shown later on this page.
retention.bytes is the other knob. It caps how large a partition log may grow before old segments are discarded, again under the delete policy. The default is -1. The page says that means no size limit, only a time limit, until you set a size.
| Setting | Default | Role in the policy |
|---|---|---|
retention.ms | 604800000 (7 days) | Topic time limit. -1 means no time limit |
retention.bytes | -1 | Topic size limit per partition. -1 means none |
log.retention.ms | null | Broker time. Wins when set. -1 means no time limit |
log.retention.minutes | null | Broker time used when log.retention.ms is unset |
log.retention.hours | 168 | Broker time used when minutes are unset |
log.retention.bytes | -1 | Broker size default for topics without an override |
segment.ms | 604800000 (7 days) | Rolls the active segment by time |
segment.bytes | 1073741824 (1 gibibyte) | Rolls the active segment by size |
log.retention.check.interval.ms | 300000 (5 minutes) | How often the cleaner looks for deletable logs |
cleanup.policy | delete | delete, compact, or both |
local.retention.ms | -2 | Local time limit. -2 means use retention.ms |
local.retention.bytes | -2 | Local size limit. -2 means use retention.bytes |
How do retention.ms and retention.bytes interact?
Under the delete policy, old segments go when their retention time or size limit has been reached. That is the cleanup.policy wording. The first limit reached is the one that discards.
A short retention.ms can delete a small partition. A small retention.bytes can delete a busy partition before the time limit.
retention.bytes is enforced at the partition level. Apache says to multiply by the partition count to get the topic total in bytes. Six partitions at 1073741824 bytes each hold 6 gibibytes, not 1.
The two limits do not replace the segment settings. The topic page says retention.ms and retention.bytes operate independently of segment.ms and segment.bytes. Segment size decides when a file rolls. Retention decides when old files are eligible to go.
One edge case is easy to misread. -1 on retention.bytes means no size limit. A value of 0 does not. The page says retention.bytes configured to zero triggers the rolling of a new segment.
Does tiered storage change the limits?
It adds a local pair. It does not replace the total pair.
local.retention.ms defaults to -2, which means use retention.ms. local.retention.bytes defaults to -2, which means use retention.bytes. The effective local value must be less than or equal to the total value.
The broker copies of those settings are log.local.retention.ms and log.local.retention.bytes. Both default to -2 and follow the same rule. With remote storage enabled, those local limits still apply to local segments. The tiered storage guide separates that local window from a backup.
How do I change retention for one topic?
The topic configs page shows two moments. At create time, pass --config. Later, change the live topic with kafka-configs.sh --alter --add-config.
This create command follows that example and sets a 1 day time limit. 86400000 milliseconds is 24 hours. Apache uses that same number, labeled 1 day, for delete.retention.ms.
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 \
--replication-factor 1 --config retention.ms=86400000
Change an existing topic with the alter form from the same page. Replace my-topic and point --bootstrap-server at a broker.
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name my-topic \
--alter --add-config retention.ms=86400000
Set a per-partition size cap with the same flags and retention.bytes. This example is 1 gibibyte, the same unit Apache uses for the segment.bytes default.
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name my-topic \
--alter --add-config retention.bytes=1073741824
Confirm the override. The page uses --describe for that check.
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name my-topic --describe
Remove the override and the topic falls back to the server default. The documented flag is --delete-config.
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name my-topic \
--alter --delete-config retention.ms
--describe prints topic overrides. It does not print a broker default the topic never overrode. If the key is absent, the broker chain above is what applies.
Why is old data not deleted?
Retention does not delete single records inside an open file. segment.bytes says retention and cleaning are always done a file at a time.
The active segment is the file still being written. It rolls once it reaches segment.bytes. The default is 1073741824, labeled 1 gibibyte. The server default property is log.segment.bytes, with the same default.
The active segment has not rolled
segment.ms forces a roll even when the file is not full, so retention can delete or compact old data. The default is 604800000 (7 days). Its server default property is log.roll.ms, which defaults to null.
Null means use log.roll.hours, and that default is 168. A quiet topic can keep one open segment for days. Records in that file stay until the file rolls.
retention.ms can roll a segment too. The topic page says it triggers a new segment when the retention.ms condition is satisfied. A segment whose newest record is still inside the window has not met that condition.
Older records in that same file wait with it. retention.bytes does not use that same rule, except at zero, which rolls a new segment.
The cleaner has not checked yet
log.retention.check.interval.ms is how often the log cleaner checks whether any log is eligible for deletion. The default is 300000, labeled 5 minutes. The update mode is read-only, so changing it takes a broker restart. A segment can be eligible and still sit until the next check.
After the log drops a segment, the bytes can remain on disk for a short wait. file.delete.delay.ms defaults to 60000 (1 minute). The server default is log.segment.delete.delay.ms, also 60000.
That delay is the unlink. The segment is already out of the log.
The policy is not delete
cleanup.policy has to include delete before time and size discard segments. compact alone keeps the latest value for each key. An empty cleanup list means infinite retention. The next section is that choice.
How do delete and compact cleanup policies differ?
cleanup.policy chooses what happens to old segments. The default is delete. Valid values are compact and delete. The broker default property is log.cleanup.policy, also delete.
| Policy | What Kafka keeps |
|---|---|
delete | Discards old segments once the time or size limit is reached |
compact | Keeps the latest value for each key |
delete,compact | Compacts retained segments and also discards by time and size |
| empty list | Keeps segments. No cleanup policy is applied |
Apache gives delete,compact as the comma-separated example. In that mode, old segments are discarded per the retention time and size configuration, and retained segments are compacted.
An empty list is the infinite case. No cleanup policy runs, and segments are retained indefinitely. The same paragraph adds a remote-storage note. Local limits (log.local.retention.ms and log.local.retention.bytes) are still applied to local segments.
delete.retention.ms is a different timer. It is how long delete tombstones stay on a compacted topic. The default is 86400000 (1 day).
Do not use it as the topic time limit. That job belongs to retention.ms.
Is a long Kafka retention policy a backup?
A longer window is not a backup. It keeps the live log on the same cluster. A topic delete, a bad write, or a shorter policy removes that log. Replicas store the same log, so they are not a restore point.
There is no point-in-time restore once the segment is gone. The topic retention comparison is the full boundary. A regulatory schedule is a different problem, and the compliance guide maps it.
How long a backup set is kept is separate too. Retention and Erasure is that control. The feature comparison lists replication and backup as different jobs.
FAQ
Frequently asked questions
What is the default Kafka retention period?
The default is 7 days. log.retention.hours is 168, and retention.ms defaults to 604800000 milliseconds. retention.bytes defaults to -1, so there is no size limit until you set one.
Can Kafka retain data forever?
Yes. Set retention.ms to -1 and retention.bytes to -1. Apache documents -1 as no time limit and no size limit. An empty cleanup.policy also keeps segments indefinitely. With remote storage enabled, local retention limits still apply to local segments.
How do I set retention to 1 day?
Set retention.ms to 86400000. That is 24 hours in milliseconds. On a live topic, run kafka-configs.sh with --bootstrap-server, --entity-type topics, --entity-name, --alter, and --add-config retention.ms=86400000. Confirm the override with --describe.
Does retention apply per partition?
The size limit does. retention.bytes is enforced on each partition. Multiply by the partition count for the topic total. retention.ms is one time limit on the log, not a size budget split across partitions.
Does tiered storage change retention?
It adds local limits. It does not replace the total limits. local.retention.ms and local.retention.bytes default to -2, which means use retention.ms and retention.bytes. The local value must be less than or equal to the total value.
Conclusion
A Kafka retention policy is a time limit, a per-partition size limit, and a cleanup policy. The default time is 7 days, from log.retention.hours=168 and retention.ms=604800000. The default size limit is none.
Change one live topic with kafka-configs.sh --alter, then --describe to confirm. Deletion still waits for a closed segment and the next cleaner check. A longer window remains the live log, not a backup.
Topic retention keeps records readable while the cluster is healthy. Compare it with backup, or read how backup retention and erasure bound that separate copy.