Kafka Tiered Storage Explained: How KIP-405 Works and When to Use It
Kafka tiered storage (KIP-405) splits each partition's log into two tiers: a local tier on broker disk and a remote tier in external storage such as S3. When a segment rolls, the broker copies it to remote storage and can delete the local copy once local retention expires — so how long you keep data stops being a question of how much broker disk you can afford.
The feature has been production-ready since Apache Kafka 3.9 and is transparent to producers and consumers. This guide covers how the two-tier architecture works, which versions and platforms support it, the exact configs to enable and disable it, the documented limitations, and the line between what tiered storage protects and what it does not.
Tiered storage is retention infrastructure, not data protection. It makes a long live log affordable by offloading older segments to object storage, but those segments remain part of the live log: topic deletion, retention expiry, and corrupt writes all propagate to the remote tier.
What is Kafka tiered storage?
Kafka traffic is dominated by tail reads: consumers fetching records written moments ago, served from the operating system's page cache rather than disk. Reads of older data — backfills, replays, recovery — are comparatively rare. Yet before tiered storage, every byte of retention lived on broker volumes, replicated three ways, priced as premium block storage, and dragged along during every partition reassignment.
KIP-405 breaks that coupling by giving each topic two tiers. The local tier is the broker log directory Kafka has always used. The remote tier is an external system — S3, HDFS, or another object store — that holds completed log segments and their indexes.
The lifecycle is simple. Only rolled (closed) segments are offloaded; the active segment always stays on broker disk. After a segment uploads successfully, the broker may delete the local copy once the local retention window passes — never before the upload completes.
That gives every tiered topic two retention windows. local.retention.ms and local.retention.bytes bound the local copy, while retention.ms and retention.bytes bound total retention including the remote tier. Local retention can drop to a few hours while total retention stretches to months.
Reads stay transparent to clients. Consumers fetch from the broker exactly as before, and when a requested offset has left local disk, the broker fetches the segment from remote storage and serves it. No client changes are needed — but note the dependency this creates: reading remote data requires running brokers that own the log.
The architecture: two plugins and a metadata topic
Tiered storage is built around two plugin interfaces. RemoteStorageManager handles the data plane: copying segments and indexes to remote storage, fetching them back for reads, and deleting them when retention expires. RemoteLogMetadataManager handles the control plane: tracking which segments live where, with strongly consistent semantics.
Apache Kafka ships no production RemoteStorageManager. The in-tree LocalTieredStorage class exists for integration tests only. To run tiered storage on self-managed Kafka you deploy a plugin — the widely used open-source implementation is Aiven's tiered-storage-for-apache-kafka, which supports S3, GCS, and Azure Blob Storage. Managed platforms bundle their own implementations.
For metadata, Kafka does provide a default: a topic-based RemoteLogMetadataManager that stores segment metadata in the internal topic __remote_log_metadata. Because this manager runs its own internal clients, the broker config remote.log.metadata.manager.listener.name must point at a listener those clients can reach.
Version timeline and platform support
Tiered storage spent three years maturing in the open before Apache Kafka called it done. The milestones:
| Milestone | Version |
|---|---|
| KIP-405 early access release | Apache Kafka 3.6.0 |
| Production-ready | Apache Kafka 3.9.0 |
| Per-topic disablement (KIP-950) | Apache Kafka 3.9.0, KRaft mode |
| Upload/download quotas (KIP-956) | Apache Kafka 3.9.0 |
| Remote offsets exposed to clients (KIP-1005) | Apache Kafka 3.9.0 |
Platform support varies more than version support:
| Platform | Tiered storage |
|---|---|
| Self-managed Apache Kafka 3.9+ | Yes — bring your own RemoteStorageManager plugin |
| Confluent Platform / Cloud | Yes — Confluent's own implementation |
| Amazon MSK | Provisioned clusters only; not MSK Serverless |
| Older self-managed Kafka (< 3.6) | No |
On MSK the feature is a checkbox rather than a plugin deployment, but it follows the same retention model. The Kafka on AWS guide covers where tiered storage sits in the MSK architecture and what the platform does and does not protect.
How to enable Kafka tiered storage
Enabling the feature takes broker configuration plus per-topic enablement. On the broker side, four settings matter — all from the Apache Kafka documentation:
# Turn the tiered storage subsystem on (rolling restart required)
remote.log.storage.system.enable=true
# Listener the default metadata manager's internal clients use
remote.log.metadata.manager.listener.name=PLAINTEXT
# Your RemoteStorageManager plugin — class and classpath
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/kafka/plugins/tiered-storage/*
The class name and path above point at a real plugin deployment. The LocalTieredStorage implementation you will see in the official quick start simulates remote storage in a local directory and is for testing only.
With brokers ready, enable the feature per topic and set the two retention windows:
bin/kafka-topics.sh --create --topic payments-archive \
--bootstrap-server localhost:9092 \
--config remote.storage.enable=true \
--config local.retention.ms=86400000 \
--config retention.ms=31536000000
This keeps one day of data on broker disk and a year in remote storage. If local.retention.ms or local.retention.bytes is unset, Kafka falls back to the total retention values — meaning nothing offloads early, so setting the local window explicitly is the point of the exercise.
Turning it off again (KIP-950)
Until Kafka 3.9, enabling tiered storage on a topic was permanent. KIP-950 added two documented exit paths, both KRaft-mode only.
The first freezes the remote tier. Setting remote.log.copy.disable=true stops new segments from uploading while existing remote data stays readable:
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name payments-archive \
--add-config 'remote.storage.enable=true,remote.log.copy.disable=true,local.retention.ms=-2,local.retention.bytes=-2'
The -2 values align local retention with total retention. The Apache docs call this out explicitly: once copying stops, local retention rules no longer apply, and leaving short local windows configured invites a disk-full surprise.
The second path disables the feature and deletes all remote data for the topic:
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name payments-archive \
--add-config 'remote.storage.enable=false,remote.log.delete.on.disable=true'
One one-way door remains at the cluster level. Disabling remote.log.storage.system.enable on brokers requires deleting every tiered topic first; skipping that step fails broker startup.
Documented limitations
The Apache Kafka documentation lists four limitations worth planning around:
- No compacted topics. Tiered storage does not support topics with log compaction, and a tiered topic cannot switch its cleanup policy to compact.
- Broker-level disablement requires topic deletion. Every tiered topic must be deleted before turning the subsystem off cluster-wide.
- Admin tooling needs clients 3.0+. Tiered-storage admin actions are only supported from client version 3.0 onwards.
- Old segments without producer snapshots. Segments missing a producer snapshot file — possible on topics created before Kafka 2.8 — are not supported.
Beyond the official list, budget for the operational surface. A plugin, an internal metadata topic, and an object store now sit on the read path for historical data. Object-store throttling or an unavailable bucket does not affect tail reads, but it does affect exactly the backfill and recovery reads you enabled the feature for.
What tiered storage protects — and what it doesn't
Remote segments are still the live log. They belong to one cluster, obey that cluster's retention rules, and hold whatever the producers wrote — including the writes you regret. Nothing about moving a segment to S3 makes it a recovery point.
| Scenario | Does tiered storage help? |
|---|---|
| Broker disk pressure from long retention | Yes |
| Storage cost of keeping months of data | Yes |
| Consumers replaying old data | Yes — transparent reads |
| Accidental topic deletion | No — remote data is deleted with the topic |
| Rolling back bad or corrupt writes | No — the remote tier holds them too |
| Cluster loss (metadata, region, ransomware) | No — remote reads need the owning cluster |
| Restoring data into a different cluster | No mechanism exists |
This is the same boundary that makes topic retention a poor backup strategy: the data's lifecycle is still governed by the system you are trying to protect. The pairing rule follows. Use tiered storage for retention economics, and keep an independent point-in-time backup for recoverability — the tiered storage comparison walks that line item by item, and the disaster recovery use cases show the failure scenarios only a backup covers.
If you already archive topics to S3 for other reasons, note that tiered storage does not replace that either — segments in the remote tier are Kafka-internal files, not a consumable archive. The backup-to-S3 tutorial covers building an independent copy in the same bucket family for an entirely different job.
Tiered storage keeps a long log affordable. OSO Kafka Backup keeps it recoverable — independent point-in-time copies of topic data and consumer group offsets, restorable to any cluster with millisecond precision. Get started.
Good retention infrastructure, adopted with open eyes
Kafka tiered storage earns its place: production-ready since 3.9, transparent to clients, and the sane way to hold months of log without buying months of broker disk. Adopt it where retention cost or disk pressure hurts. Respect the edges — no compacted topics, KRaft-only disablement, a plugin to operate — and be clear about the boundary. A cheaper live log is still a live log. For deletion, corruption, and cluster loss, the full comparison of alternatives shows why an independent backup remains the other half of the design.
Frequently asked questions
Is Kafka tiered storage a backup?
No. Tiered storage offloads older log segments to object storage, but they remain part of the live topic, owned by one cluster. Topic deletion, retention expiry, and corrupted writes all propagate to the remote tier, and there is no point-in-time restore. A backup is an independent copy with its own lifecycle.
Which Kafka version supports tiered storage?
KIP-405 shipped as early access in Apache Kafka 3.6.0 and became production-ready in 3.9.0. Kafka 3.9 also added per-topic disablement (KIP-950) and upload/download quotas (KIP-956). You still need a RemoteStorageManager plugin, since Apache Kafka does not ship a production implementation.
Can I disable tiered storage on a topic once it is enabled?
Since Kafka 3.9, in KRaft mode, yes. Setting remote.log.copy.disable=true stops new uploads while keeping remote data readable, and setting remote.storage.enable=false with remote.log.delete.on.disable=true removes the remote data entirely. Disabling the feature at the broker level still requires deleting every tiered topic first.
Does Kafka tiered storage work with compacted topics?
No. The Apache Kafka documentation lists compacted topics as unsupported, and a topic with tiered storage enabled cannot change its cleanup policy from delete to compact. Compacted topics need a different protection strategy, such as an independent backup tool that supports them.
Does Amazon MSK support tiered storage?
Yes, on provisioned clusters only — MSK Serverless does not offer it. MSK manages the remote tier for you, so there is no plugin to deploy, but the retention model and its limits are the same: tiered data still expires with retention and is deleted with the topic.