What "backing up a Kafka topic" actually means​

A real topic backup captures four things:

1. Records — keys, values, headers, and timestamps
2. Partition layout — which records lived on which partition, in what order
3. Offsets — both record offsets and consumer group positions
4. Topic configuration — partition count, retention, cleanup policy

Two things that are not backups, despite being treated as such:

• Retention is scheduled deletion. When retention.ms expires, the data is gone regardless of whether anyone still needs it.
• Replication (in-cluster RF=3, or MirrorMaker 2 across clusters) copies every write — including the accidental delete and the poisoned deploy — within milliseconds.

Backups exist so you can go backwards in time. Replication only goes forwards.

Method 1 — kafka-backup CLI (production topics)​

The OSO Kafka Backup CLI backs up topics with offset preservation and restores them with millisecond-precision time windows. It is the right default for production.

Step 1: Install and configure​

Follow the installation guide for your platform, then write a backup config:

mode: backup

source:
  bootstrap_servers:
    - broker-1:9092

topics:
  include:
    - orders
    - payments

storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-west-2
  prefix: backups/production

backup:
  compression: zstd
  start_offset: earliest

Filesystem, Azure Blob, and GCS backends use the same shape — see the configuration reference and the S3 integration guide.

Step 2: Run the backup​

kafka-backup backup --config backup.yaml

Progress is checkpointed as it runs, so an interrupted backup resumes rather than restarting. For continuously changing topics, set continuous: true to stream changes instead of taking discrete snapshots.

Step 3: Restore — to anywhere, at any point in time​

Restore to the original cluster, a new cluster, or a renamed topic:

mode: restore
backup_id: "prod-backup-20260701"

target:
  bootstrap_servers:
    - dr-broker-1:9092

storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-west-2
  prefix: backups/production

restore:
  create_topics: true
  topic_mapping:
    orders: orders-restored
  # Point-in-time: only records up to the moment before the incident
  time_window_end: 1751500800000   # Unix ms

kafka-backup restore --config restore.yaml

The time_window_end option is what makes this a genuine undo button: restore the topic to 14:03:27.451, the millisecond before the bad deploy started writing. Set dry_run: true first to validate the whole plan without producing a record.

Use this method when: you need point-in-time recovery, consumer offsets must survive the restore, or the backup must live outside the cluster's failure domain.

Method 2 — Kafka Connect S3 sink (existing Connect estates)​

If you already operate Kafka Connect, an S3 sink connector can archive topic records to a bucket:

{
  "name": "orders-s3-backup",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "topics": "orders",
    "s3.bucket.name": "my-kafka-archive",
    "s3.region": "us-west-2",
    "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
    "flush.size": "10000"
  }
}

Restoring means running the matching S3 source connector to replay the records into a topic.

The trade-offs are real, though: consumer group offsets are not captured, restores replay records with new offsets (breaking offset-based consumers), and there is no point-in-time selection beyond whatever partitioning your sink wrote. It is archiving, not recovery tooling.

Use this method when: you need raw record archives for analytics or compliance, offsets do not matter, and Connect is already running.

Method 3 — Consumer script (small topics, one-offs)​

For a small topic in a dev environment, a script can be enough:

# Backup: dump records with key, timestamp, and partition
kafka-console-consumer --bootstrap-server broker-1:9092 \
  --topic orders --from-beginning \
  --property print.key=true \
  --property print.timestamp=true \
  --property print.partition=true \
  --timeout-ms 10000 > orders-backup.txt

# Restore: replay values into a new topic
kafka-console-producer --bootstrap-server broker-1:9092 \
  --topic orders-restored < orders-backup.txt

Be honest about the limits: no offset preservation, no header capture in older tooling, timestamps become produce-time on restore, and nothing about this is incremental. It is a photocopy, not a backup system.

Use this method when: the topic is small, the moment is now, and the stakes are low.

Verifying the backup (whichever method you chose)​

An unverified backup is a guess. After every backup — and on a weekly schedule — check:

• Record counts match between source and restored topic (kafka-run-class kafka.tools.GetOffsetShell on both sides)
• Offset continuity — no gaps at segment boundaries
• Schema compatibility — restored records deserialize with the current schema
• Consumer resume — a consumer group restored with the data picks up where it left off instead of reprocessing from zero

The backup best practices guide covers turning this checklist into automated, alerting-backed verification.

Next steps: the first backup tutorial walks this end to end, and the CLI reference documents every flag used above.

1. Treat backup as code​

Backup configuration belongs in version control next to the applications it protects. A backup.yaml reviewed in a pull request is auditable; a config hand-edited on a VM is a mystery six months later.

• Store backup configurations in Git
• Provision backup infrastructure with Terraform or Helm, not the console
• Ship config changes through CI so a typo cannot silently disable a nightly job

If you run Kafka on Kubernetes, the backup operator takes this further: backup schedules become custom resources that GitOps tools reconcile like any other manifest.

2. Verify backups automatically — trust, but verify​

The most common backup failure is discovered during an outage: the backup ran for months, but nobody ever restored one. Schedule automated restore tests — weekly at minimum — into a scratch cluster or isolated namespace.

Validate three things on every test:

1. Record counts match between the source topic and the restored topic
2. Offsets are continuous — no gaps or overlaps at segment boundaries
3. Consumers can resume from restored consumer group offsets

The restore mode supports dry_run: true, which validates a backup against the target cluster without producing a single record — cheap enough to run daily:

mode: restore
backup_id: "prod-backup-latest"

restore:
  dry_run: true

3. Monitor backup lag and health continuously​

A backup job that dies on Friday night should page someone before Monday. OSO Kafka Backup exposes Prometheus metrics for exactly this:

The full list is in the metrics reference. Wire the lag metric to your alerting with a threshold derived from your RPO — not a number picked in a hurry.

4. Define recovery objectives before you need them​

Two numbers drive every backup decision:

• RPO (recovery point objective) — how much data you can afford to lose. This sets backup frequency, or pushes you to continuous mode.
• RTO (recovery time objective) — how long you can be down. This sets your restore method, parallelism, and where backups physically live.

Map every critical topic to an RPO/RTO tier, write the mapping down, and review it quarterly. A payments topic and a clickstream topic should not share a policy. For the architecture side of this decision, see the disaster recovery use cases.

5. Back up offsets and metadata, not just messages​

A topic restore that loses consumer group offsets forces every consumer to choose between reprocessing everything and skipping to latest — both are incidents of their own. Message data alone is roughly half a backup. Capture:

• Consumer group offsets, so processing resumes where it stopped
• Topic configurations — partitions, retention, cleanup policy
• Schemas, so downstream consumers can still deserialize what you restored
• ACLs, so security posture survives the restore

OSO Kafka Backup captures offsets and topic configuration as part of every backup, and keeps offsets consistent with the restored data during point-in-time recovery.

6. Encrypt backup data at rest and in transit​

Backups concentrate months of your most valuable data into one bucket — treat them with at least the rigor of the cluster itself.

• Enable server-side encryption on the storage target (SSE-S3 or SSE-KMS on Amazon S3, service-managed keys on Azure and GCS)
• Use TLS between the backup process and both the brokers and the object store
• Keep encryption keys in a KMS with its own access policy, so a Kafka credential leak does not also expose the archive

7. Control costs without weakening the safety net​

Backup storage costs are predictable and controllable — unlike the cost of losing the data.

• Compress. Backups are compressed with Zstandard or LZ4 before upload (compression: zstd in the backup config), independent of producer-side compression.
• Tier. Keep recent backups hot for fast restore; move older ones to infrequent-access or archive classes with bucket lifecycle rules.
• Expire. Retention policies should delete what compliance no longer requires — storage you forgot about is pure waste.

Storage layout details are in the storage format reference, which is what lifecycle rules operate against.

8. Meet data retention regulations deliberately​

If your topics carry regulated data, backups are in scope too:

• Map topics to their regimes (GDPR, HIPAA, SOX, PCI-DSS) and set backup retention to match — both minimums and maximums
• Use object-lock or immutable storage for audit-relevant backups
• Automate deletion when retention windows close; manual cleanup does not survive staff turnover
• Keep restore procedures documented — auditors ask for evidence that recovery works, not just that backups exist

9. Test disaster recovery quarterly​

Backup verification (rule 2) proves the data is restorable. A DR test proves your organization can restore it: the runbook is current, the credentials work, the on-call engineer knows which cluster to target, and the application teams can validate their services afterward.

Run a full failover drill quarterly. Measure the RTO and RPO you actually achieved against the targets from rule 4, and fix the gap — in tooling or in targets — after every drill.

10. Write runbooks for the engineer at 3 a.m.​

The person running a restore under pressure should never compose a config from memory. Good runbooks contain:

• Pre-validated, copy-paste commands: kafka-backup restore --config restore-payments.yaml, with the config already in Git
• A decision tree: partial topic restore vs. full recovery vs. point-in-time rollback
• Escalation paths and the list of application owners to notify
• Links to the dashboards from rule 3, so progress is observable

Start with the first backup tutorial as a template and extend it with your environment's specifics.

Where to start​

Do not attempt all ten at once. Add backup lag alerting today (rule 3), schedule a weekly automated restore test this week (rule 2), and write the RPO/RTO map next sprint (rule 4). The rest layer on from there.

A backup you test is a backup you can trust — everything else on this list exists to make that testing routine instead of heroic.

Ready to put these into practice? Take your first backup in minutes, or see how backup fits alongside replication in our MirrorMaker 2 comparison.

What geo replication is (and is not)​

Setting replication.factor=3 puts three copies of each partition on three brokers — in the same cluster. Rack awareness can spread those replicas across availability zones, but the cluster is still one blast radius: one control plane, one region, one set of humans with admin rights.

Geo replication runs a second (or third) Kafka cluster elsewhere and copies topics between them, cluster to cluster. The use cases:

• Disaster recovery — survive a region failure with a warm standby
• Data locality — serve consumers from the nearest region
• Compliance — keep regional data in-region while sharing what is allowed
• Migration — move workloads between datacenters or clouds without a big bang (see the migration use cases)

The metrics that govern every design below: end-to-end replication lag, cross-region latency, and network transfer cost.

The four replication patterns​

Active-passive is the honest default. One cluster serves traffic; a standby in another region receives a continuous copy. Failover means repointing clients — the hard part is offset translation, not data movement.

Active-active lets both regions produce and consume. It halves your wasted standby capacity but introduces bidirectional flows, loop prevention, and topic naming discipline. Choose it when both regions must serve writes, not because idle standby feels wasteful.

Hub-and-spoke fits aggregation topologies: regional clusters replicate into a central hub for analytics, or a hub fans reference data out to the edges.

Mesh — everyone replicates to everyone — multiplies links, monitoring, and failure modes quadratically. It is listed here mostly so you can decline it deliberately.

Implementing geo replication with MirrorMaker 2​

MirrorMaker 2 (MM2) ships with Apache Kafka and runs on the Connect framework. A minimal active-passive setup:

clusters = primary, dr
primary.bootstrap.servers = kafka-us-east.example.com:9092
dr.bootstrap.servers = kafka-us-west.example.com:9092

# Replicate everything except internals from primary to DR
primary->dr.enabled = true
primary->dr.topics = .*

# Consumer group offset sync, so consumers can fail over
emit.checkpoints.enabled = true
sync.group.offsets.enabled = true
sync.group.offsets.interval.seconds = 60

Three things bite teams on real WAN links:

1. Remote topic prefixes. MM2 replicates orders as primary.orders on the DR cluster by default. Consumers failing over must subscribe accordingly, or you must override the replication policy.
2. Offset translation. Offsets differ between source and target. MM2's checkpoints translate consumer group positions — verify translated offsets in a drill before an outage forces the issue.
3. WAN tuning. Raise producer batch.size and linger.ms on the connectors, enable compression, and monitor end-to-end lag (record timestamp delta at the consumer), not just Connect task lag.

Our MirrorMaker 2 comparison covers where MM2 shines and where it stops.

A reference architecture that holds up​

The pattern we see work repeatedly for cross-datacenter DR:

• Two regions, active-passive, MM2 running in the target region (pull model — the DR site keeps working if the primary degrades)
• Dedicated replication bandwidth sized at peak produce throughput plus headroom, compressed on the wire
• Health-checked DNS failover for client bootstrap servers, with TTLs low enough to matter during an incident
• Quarterly failover drills that measure achieved RTO/RPO against targets — untested DR is a diagram, not a capability
• Cost line items reviewed explicitly: standby compute, cross-region transfer (usually the surprise), and duplicated storage

The failure mode replication cannot solve​

Geo replication is built to copy everything, quickly. That is precisely why it cannot protect you from:

• A producer bug writing poisoned records — replicated in milliseconds
• An accidental topic deletion — propagated to the standby
• A compliance request to reconstruct data as of last quarter

For those you need an immutable copy that lives outside both clusters and can be restored to a moment in time. That is what point-in-time backup provides: records, consumer offsets, and topic configuration in object storage, restorable to any cluster — including the DR cluster you just failed over to.

Mature Kafka estates run both layers: replication for availability, backups for recoverability. The best practices guide covers operating that second layer well.

Related reading: OSO Kafka Backup vs MirrorMaker 2, disaster recovery use cases, and how to backup and restore Kafka topics.