Assemble a Kafka disaster recovery plan

What a Kafka disaster recovery plan covers​

A Kafka disaster recovery plan answers one question per failure: how does the cluster get back to serving traffic, and what data survives? Replication alone cannot answer it. Replication copies corruption, propagates deletions, and offers no way back to "before the incident."

A backup-based plan does. It needs four parts: a scenario table naming each failure and its recovery action, a backup schedule that sets your data-loss window, a runbook the on-call engineer can run at 3 AM, and targets that get tested on a calendar — not during the outage.

Part 1 — Name the failures before they happen​

Each row maps to the same three commands — validate, validate-restore, three-phase-restore — with different configs. That is the point of the plan: no improvisation during the incident.

Part 2 — Schedule the backups that fund the plan​

mode: backup
backup_id: "production-${TIMESTAMP}"

source:
bootstrap_servers:
  - kafka-prod-1:9092
  - kafka-prod-2:9092
  - kafka-prod-3:9092
topics:
  include:
    - "*"
  exclude:
    - "__consumer_offsets"
    - "_schemas"

storage:
backend: s3
bucket: kafka-dr-backups
region: us-east-1  # DR region, not the production region
prefix: production/hourly

backup:
compression: zstd
compression_level: 3
checkpoint_interval_secs: 30

# Both required for consumer group recovery after restore
include_offset_headers: true
source_cluster_id: "prod-us-west-2"

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
name: dr-hourly-backup
spec:
schedule: "0 * * * *"  # Hourly
kafkaCluster:
  bootstrapServers:
    - kafka-prod-1:9092
topics:
  - "*"
storage:
  storageType: s3
  s3:
    bucket: kafka-dr-backups
    region: us-east-1
    prefix: production/hourly
compression: zstd
includeOffsetHeaders: true
sourceClusterId: "prod-us-west-2"

#!/bin/bash
# Kafka DR runbook: pass the backup ID approved for recovery
BACKUP_ID="${1:?Usage: dr-runbook.sh <backup-id>}"

# 1. Prove the backup is intact before touching the cluster
kafka-backup validate \
--path s3://kafka-dr-backups/production/hourly \
--backup-id "$BACKUP_ID" \
--deep

# 2. Prove the restore config is valid
kafka-backup validate-restore --config dr-restore.yaml

# 3. Restore data, then reposition consumer groups
kafka-backup three-phase-restore --config dr-restore.yaml

Part 3 — Set targets the schedule can meet​

Backup frequency sets your recovery point objective. Data volume sets your recovery time objective. Put both numbers in the plan and defend them.

Setting targets per workload — not one number for the whole cluster — is covered in the RTO/RPO planning guide on the blog.

Part 4 — Put the tests on a calendar​

A plan nobody has executed is a document, not a capability. Schedule the proof:

The DR testing playbook turns this table into a full drill program, with pass/fail evidence for each level.