Kafka Log Compaction Explained (and Backing It Up)
Kafka log compaction keeps the latest value for each key in a partition. Older values for that key can be removed. The record that stays is the current state, not the full change history.
Apache Kafka 4.4 describes this on the log compaction page. Topic settings are on the topic configs page. Broker defaults are on the broker configs page.
Kafka log compaction retains the latest value for each key. The log cleaner drops older duplicates in closed segments. A tombstone is a key with a null value, kept for delete.retention.ms (default 1 day), then removed. A backup copies that log, not values the cleaner already dropped.
What is log compaction in Kafka?
Log compaction guarantees that Kafka retains at least the last known value for each message key inside one partition. The design page uses an email topic as the example. Each update uses the user id as the key. Compaction can keep bill@gmail.com and drop the older addresses for user 123.
An event record often stands alone. A changelog does not. The same key is written again when the row changes.
Time or size retention can throw away an old update, and a reader from the start then misses the current value. Compaction removes a record only when a newer record has the same key.
The choice is per topic. cleanup.policy defaults to delete. Set it to compact to enable compaction, which retains the latest value for each key.
The broker default is log.cleanup.policy, also delete. A topic override wins when you set one.
The design page shows the broker line when compaction should be the default for topics that do not set a policy:
log.cleanup.policy=compact
Create one compacted topic with the flags from the topic configs page. This example sets cleanup.policy=compact.
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 \
--replication-factor 1 --config cleanup.policy=compact
Change a live topic with the alter form from that same page.
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name my-topic \
--alter --add-config cleanup.policy=compact
Confirm the override with --describe. The topic page uses that command to print overrides. If the key is absent, the broker default applies.
The design page splits the log into a head and a tail. The head is the recent part. It keeps every record, and the offsets are dense.
The tail is the older part. Compaction rewrites that tail. Offsets of records that remain never change. A read at a removed offset is the same position as the next offset that still exists.
That split is why a compacted topic is not a history. The tail holds the latest value per key. The head still holds every write the cleaner has not rewritten, including the active segment.
How does the log cleaner decide what to remove?
The log cleaner is a pool of background threads. It recopies segment files and drops records whose key appears later in the log. log.cleaner.threads defaults to 1. Cleaning does not block reads.
The cleaner chooses the log with the highest ratio of head to tail. It runs only for topics whose cleanup policy includes compact.
log.cleaner.enable defaults to true. Kafka 4.4 marks that setting deprecated and says it will be removed in Kafka 5.0. Leave it on. If it is off, compact topics, including the internal offsets topic, are not compacted and grow without bound.
There is no fixed compaction interval. min.cleanable.dirty.ratio defaults to 0.5. The page says Kafka avoids cleaning a log where more than 50% of the log has been compacted.
At 0.5, at most half the log could be duplicates. A higher ratio means fewer cleanings and more wasted space.
min.compaction.lag.ms defaults to 0. It is the minimum time a message stays uncompacted. At 0, age does not hold a closed segment back. The dirty ratio still has to be met.
The broker property is log.cleaner.min.compaction.lag.ms, also 0. The active segment stays uncompacted even when every message in it is older than the lag.
max.compaction.lag.ms defaults to 9223372036854775807. It is the maximum time a message stays ineligible for compaction. The design page says this bound stops a quiet log from waiting forever when it never crosses the dirty ratio.
With that default, the deadline does not arrive in practice. The dirty ratio is what makes the log eligible.
The active segment is never compacted. It rolls once it reaches segment.bytes. The default is 1073741824 (1 gibibyte). It also rolls by time.
For a compacted topic, that time is the smaller of segment.ms and max.compaction.lag.ms. segment.ms defaults to 604800000 (7 days). The default max lag is larger, so the time roll is 7 days unless you lower the max lag. After the roll, the max lag can make those records eligible even when the dirty ratio is not met.
The design page says this deadline is not a hard guarantee. The copy still waits for a cleaner thread and for the time the recopy takes.
When no log is eligible, the cleaner sleeps. log.cleaner.backoff.ms defaults to 15000 (15 seconds). That sleep is not a compaction schedule.
log.retention.check.interval.ms defaults to 300000 (5 minutes). That check looks for logs eligible for deletion. It is not the compaction clock. The Kafka retention policy post covers the delete path.
Cleaning can be throttled so it does not take all disk throughput. log.cleaner.io.max.bytes.per.second defaults to 1.7976931348623157E308. Until you set a lower number, that throttle does not bind.
| Setting | Default | What it controls |
|---|---|---|
cleanup.policy | delete | compact, delete, or both |
min.cleanable.dirty.ratio | 0.5 | Dirty share that makes a log eligible |
min.compaction.lag.ms | 0 | Minimum time a message stays uncompacted |
max.compaction.lag.ms | 9223372036854775807 | Maximum time a message stays ineligible |
segment.bytes | 1073741824 (1 gibibyte) | Size roll of the active segment |
segment.ms | 604800000 (7 days) | Time roll of the active segment |
log.cleaner.threads | 1 | Background threads that recopy segments |
log.cleaner.backoff.ms | 15000 (15 seconds) | Sleep when nothing is eligible |
log.cleaner.enable | true | Deprecated. Leave it on. Removed in Kafka 5.0 |
What are tombstones and delete.retention.ms?
A delete is a message with a key and a null payload. The design page calls that record a tombstone. It causes any prior message with that key to be removed, as a newer value would.
The marker itself is special. The cleaner drops it later, at the delete retention point, so the key can leave the log.
delete.retention.ms is that wait. The default is 86400000 (1 day). Valid values start at 0. The broker property is log.cleaner.delete.retention.ms, with the same default.
The design page states the consumer side of that timer. A reader that reaches the head in less than delete.retention.ms sees the delete markers. The page gives the default as 24 hours. A reader that lags longer can miss a marker, because removal runs while reads are in progress.
The topic page states the same bound for a scan from offset 0. Finish that read inside delete.retention.ms, or tombstones may be collected before the scan ends. A backup that starts at the beginning of the log is that scan. A slow read can miss a tombstone and keep the previous value, as if the delete never happened.
This timer is not the topic time limit. retention.ms is that limit, and only when the policy includes delete. The retention policy guide is the page for retention.ms and retention.bytes.
When should you use compact vs delete (and compact,delete)?
cleanup.policy accepts compact and delete. You can set both as a comma-separated list. Apache's example is delete,compact. An empty list means no cleanup policy runs.
| Policy | What Kafka keeps |
|---|---|
delete | Discards old segments when the time or size limit is reached |
compact | Latest value for each key. The head stays complete until it is cleaned |
delete,compact | Compacts retained segments, and also discards by time and size |
| empty list | Keeps segments. No cleanup policy is applied |
Use delete when each record stands alone. The design page points at logging for that case. Use compact when a consumer must rebuild current state from the log. The same page names a database changelog, a cache reload, and a local store that replays updates after a crash.
Use delete,compact when you want both behaviors. Old segments go by the retention time and size configuration. Segments that remain are compacted.
retention.ms and retention.bytes do that discard only when delete is in the policy. compact alone does not drop a segment because it is old or large. An empty policy keeps segments indefinitely.
With remote storage enabled, local retention limits still apply to local segments. That note is on the cleanup.policy entry.
A compacted topic can still be time-bounded. Add delete and set retention.ms. The window you keep is compacted.
Data older than the window is discarded, latest value included. That window is still not a full history of the key.
How do you back up and restore a compacted topic?
A compacted topic is a poor archive if you need every past value. The cleaner has already dropped older values in the tail. The head, including the active segment, still holds recent duplicates and tombstones.
A backup copies the records still in the log. It does not put back values the cleaner removed before the copy started.
OSO Kafka Backup reads the topic as a consumer. One-shot mode copies from start_offset to the current high watermark, then exits. start_offset defaults to earliest in the configuration reference.
On a compacted topic, earliest is the start of the log that still exists. That is the compacted tail plus the uncompacted head.
The storage format stores each record's key and value. A length of -1 means null, which is different from an empty value. A tombstone is a key plus a null payload, so both belong in the segment.
Restore writes those archived records back. Lose the key, or turn the null into an empty value, and the target will not treat the record as a delete.
This file follows the first backup tutorial. It names one compacted topic and reads from earliest.
mode: backup
backup_id: "production-backup-001"
source:
bootstrap_servers:
- broker-1.kafka.svc:9092
topics:
include:
- user-email
storage:
backend: filesystem
path: "/var/lib/kafka-backup/data"
backup:
compression: zstd
compression_level: 3
start_offset: earliest
kafka-backup backup --config backup.yaml
Filesystem, S3, Azure Blob, and GCS are the storage backends in that tutorial. Swap the storage block for the backend you use. compression accepts zstd, lz4, or none.
The stored copy does not follow later deletes on the cluster. The architecture FAQ says backups are immutable snapshots. A later compaction, a bad produce, or a topic delete leaves that copy in place.
MirrorMaker 2 is replication. It copies new writes, including a delete, into another live log. That live copy cannot rewind the topic. The backup and restore guide is the general sequence for a topic that is not compacted.
Later runs can be incremental. The first backup tutorial shows an offset_storage block for that. Runs with the same backup_id then start at the last saved offset and copy new records only. They still cannot fetch a value the cleaner removed before the first read.
Check cleanup.policy on the target before you restore. The first backup report says an existing topic is appended to. The migration guide says restore uses the topic configuration already on the target.
create_topics defaults to false. A target left on delete will age the restored records out by retention.ms. The framework definitions include topic configuration in a backup. Confirm the live policy anyway, because an existing topic keeps the configuration you gave it.
Validate the restore, then run it. Both commands are in the first backup tutorial.
mode: restore
backup_id: "production-backup-001"
target:
bootstrap_servers:
- dr-broker-1.kafka.svc:9092
storage:
backend: filesystem
path: "/var/lib/kafka-backup/data"
restore:
consumer_group_strategy: skip
kafka-backup validate-restore --config restore.yaml
kafka-backup restore --config restore.yaml
Point-in-time restore filters that backup by timestamp. The PITR notes say a compacted topic is restored from the backup data, and the compaction state can differ from the source. time_window_end is the optional bound shown in the restore file. It cannot return a value the cleaner removed before the backup ran.
Enterprise erasure is a separate filter on restore. The default on_match is tombstone: the record is produced with its key and a null value, so compaction can retire older copies. drop skips the record.
On a compacted topic, drop leaves older values of that key in place. The retention and erasure guide states that pairing.
__consumer_offsets is a different compacted topic. The monitoring guide names it as the log the group coordinator reads. A normal backup does not include it. include_internal_topics defaults to false.
Group positions belong in consumer_group_snapshot. It defaults to false. When it is set, the backup writes consumer-groups-snapshot.json. The consumer offsets backup runbook is that path.
FAQ
Frequently asked questions
Does log compaction delete data?
Yes. It removes an older record when a newer record with the same key is in the log. The latest value stays. A tombstone, a key with a null value, deletes that key. delete.retention.ms then bounds how long the tombstone itself stays. The default is 86400000 (1 day).
How often does Kafka compact?
There is no fixed compaction interval. The log cleaner is a background thread pool, and log.cleaner.threads defaults to 1. It cleans a log when the dirty ratio or max.compaction.lag.ms says the log is eligible. When nothing is eligible it sleeps log.cleaner.backoff.ms, which defaults to 15000 (15 seconds). The active segment is never compacted.
Can a topic be both compacted and time-retained?
Yes. Set cleanup.policy to delete,compact. Old segments are discarded by the retention time and size settings, and segments that are still retained are compacted. compact alone does not discard segments by retention.ms.
Are __consumer_offsets compacted?
Yes. The internal offsets topic uses cleanup.policy=compact, and broker monitoring names that topic __consumer_offsets. offsets.topic.segment.bytes defaults to 104857600 (100 mebibytes) to help that compaction. offsets.retention.minutes defaults to 10080 and can expire a committed offset after the group is empty. That expiry is not the compaction policy.
Conclusion
Kafka log compaction keeps the latest value for each key, plus the records still sitting in the uncompacted head. The cleaner uses the dirty ratio, the two lag settings, and delete.retention.ms. A backup of that topic is a copy of this log, not of every value the key ever had.
Restore the archived records with keys and null values intact. Set cleanup.policy=compact on the target if new writes should be compacted too. A replica or a MirrorMaker copy is a second live log. It is not that backup.
Take the backup from the first backup tutorial. The feature comparison separates that copy from replication.