Skip to main content

Kafka Log Compaction Explained (and Backing It Up)

· 14 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka log compaction keeps the latest value for each key in a partition. Older values for that key can be removed. The record that stays is the current state, not the full change history.

Apache Kafka 4.4 describes this on the log compaction page. Topic settings are on the topic configs page. Broker defaults are on the broker configs page.

Quick Answer

Kafka log compaction retains the latest value for each key. The log cleaner drops older duplicates in closed segments. A tombstone is a key with a null value, kept for delete.retention.ms (default 1 day), then removed. A backup copies that log, not values the cleaner already dropped.

Kafka Dead Letter Queue: Patterns and Config

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

A Kafka dead letter queue holds records that failed processing. Kafka has no broker setting that creates one for a classic consumer group. The pattern is a separate topic.

Apache Kafka 4.4 documents that topic in Kafka Connect, in Kafka Streams, and on share groups. Each path has its own config, and each one defaults to off.

Quick Answer

Kafka has no broker-level dead letter queue. A DLQ is a separate topic for records a consumer or connector could not process. Kafka Connect includes one for sink connectors. Set errors.tolerance to all and set errors.deadletterqueue.topic.name. Source connectors do not get that topic. Plain consumers produce to their own topic.

Kafka Retention Policy: retention.ms and retention.bytes

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

A Kafka retention policy decides when brokers delete old log data. The time limit is retention.ms. The size limit is retention.bytes.

Apache Kafka 4.4 documents both on the topic configs page. Broker defaults are on the broker configs page.

Quick Answer

The default Kafka retention period is 7 days. Brokers ship with log.retention.hours set to 168. Two topic settings carry that policy.

retention.ms is the time limit, and it defaults to 604800000 milliseconds. retention.bytes is the size limit on each partition, and it defaults to -1, which means no size cap. Under cleanup.policy=delete, the first limit reached discards old segments.

Amazon MSK Pricing Explained (2026)

· 17 min read
OSO Engineering
The team behind OSO Kafka Backup

MSK pricing is a stack of metered lines. A provisioned cluster bills broker instance hours and storage. MSK Serverless bills cluster hours, partition hours, and the bytes producers write and consumers read. Data transfer, tiered storage, MSK Replicator, and MSK Connect are extra lines when you use them.

Rates below come from the published examples on the Amazon MSK pricing page. Each example names its Region. Prices as of 21 October 2026. Other Regions use the pricing tables on that page.

Quick Answer

Amazon MSK pricing has two cluster shapes. Provisioned clusters bill broker instance hours plus storage per GB-month. MSK Serverless bills cluster hours, partition hours, and data in and out. Tiered storage, data transfer, MSK Replicator, and MSK Connect are separate lines, and rates differ by Region.

How to Delete a Kafka Topic (and Recover It)

· 14 min read
OSO Engineering
The team behind OSO Kafka Backup

A kafka delete topic request is a how-to with a hard ending. The admin command removes the topic. The cluster then drops its replicas and the committed offsets that pointed at it. There is no undo inside Kafka.

Quick Answer

Delete a topic with bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic <name>. The broker config delete.topic.enable is the switch that blocks it. Kafka 4.3 broker configs default that setting to true. When it is false, the broker rejects the delete. The command does not ask for confirmation, and the cluster cannot undo it.

From Kafka PoC to Production: The Checklist

· 9 min read
OSO Engineering
The team behind OSO Kafka Backup

A Kafka PoC and a Kafka production cluster frequently run the exact same broker version, on the exact same defaults. The gap between them isn't code — it's everything nobody explicitly configured. Replication factor, backup, DR targets, monitoring, security, cluster topology, and retention all ship with defaults that are harmless for a three-week proof of concept and dangerous for a system carrying real traffic.

This is the checklist for closing that gap. It doesn't teach you anything new — it maps each silent default to the guide that already covers the fix.

Kafka Cross-Region Replication Patterns: Hub-and-Spoke, Mesh, and When to Use Neither

· 11 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka cross-region replication comes down to a small set of named topologies: active-passive, active-active, hub-and-spoke, and mesh. Most teams need one of the first two, and both already have a full implementation guide on this site. This post covers the other two — the ones that show up as a single sentence in every comparison article and never get built out: hub-and-spoke fan-in with real MirrorMaker 2 configuration, and an honest account of what mesh actually costs before you reach for it.

Kafka for Database Replication (CDC) — and What It Means for Backup

· 10 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka database replication almost never means what the words suggest. Searchers using that phrase are usually not asking about Kafka's own broker-to-broker replication — they're asking how to get changes out of a database and into Kafka, so downstream systems can react to them. That's change data capture (CDC), and Kafka Connect with a connector like Debezium is the standard way to do it.

This post untangles the terms, walks through the CDC architecture that "Kafka database replication" actually describes, and covers something that architecture's own documentation rarely does: what happens to those topics when you need to recover from a failure.

Kafka Streams Disaster Recovery: State Stores and Changelogs

· 9 min read
OSO Engineering
The team behind OSO Kafka Backup

Kafka Streams disaster recovery means recovering state stores, not just topics. Every Streams task keeps its aggregation, join, or window state on local disk — Kafka's replication never touches that disk. The only durable copy of that state Kafka manages is the changelog topic behind it, and that is the thing your recovery plan actually needs to protect.

This post explains why a Streams app needs a different recovery plan than a plain consumer, compares the three ways to get its state back, and works out what state-store size does to your recovery time.

MirrorMaker 1 Is Gone: How to Migrate Legacy Mirroring to MirrorMaker 2

· 10 min read
OSO Engineering
The team behind OSO Kafka Backup

The original Kafka MirrorMaker (kafka.tools.MirrorMaker, run through the kafka-mirror-maker.sh script) was deprecated in Kafka 3.0.0 under KIP-720 and fully removed in Kafka 4.0.0. If a cluster is still running it, moving that job onto MirrorMaker 2 isn't a config patch — it's a new config file format, a topic-naming decision, and, for the first time, real consumer-offset translation between clusters.

This post is the flag-by-flag migration guide: what changes, what the old options map to, and the order of operations that keeps a rollback path open until the cutover is proven.