Run a warm standby Kafka cluster from backups

How a backup-fed Kafka active-passive topology works​

In a Kafka active passive setup, one cluster serves all traffic while a standby waits in another region. The design question is how the standby gets its data. Continuous replication is one answer. Feeding the standby from scheduled backups is the other — and it needs no network path between the clusters.

The loop is short. The primary writes compressed backups to a bucket in the DR region on a schedule. The standby stays small and idle. On a regional failure, you validate the latest backup, restore it into the standby, reposition consumer groups, and redirect clients.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
name: primary-hourly-backup
spec:
schedule: "0 * * * *"  # Hourly — this sets your RPO
kafkaCluster:
  bootstrapServers:
    - kafka-0.us-west-2.example.com:9092
topics:
  - "*"
storage:
  storageType: s3
  s3:
    bucket: kafka-backups-dr
    region: us-east-1  # standby region
    prefix: us-west-2/hourly
compression: zstd
includeOffsetHeaders: true
sourceClusterId: "prod-us-west-2"

#!/bin/bash
# Promote the passive cluster. Choose the restore point explicitly —
# never restore an unconfirmed backup during an incident.
BACKUP_ID="${1:?Usage: promote-standby.sh <backup-id>}"
BACKUP_PATH="s3://kafka-backups-dr/us-west-2/hourly"

# 1. Confirm the backup exists and is the one you mean
kafka-backup list --path "$BACKUP_PATH" --backup-id "$BACKUP_ID"

# 2. Deep-validate it before promotion
kafka-backup validate --path "$BACKUP_PATH" \
--backup-id "$BACKUP_ID" --deep

# 3. Restore data and reposition consumer groups
kafka-backup three-phase-restore --config standby-restore.yaml

# 4. Verify, then redirect clients to the standby
kafka-topics --bootstrap-server kafka-0.us-east-1.example.com:9092 --list

mode: restore
backup_id: "${BACKUP_ID}"

target:
bootstrap_servers:
  - kafka-0.us-east-1.example.com:9092
  - kafka-1.us-east-1.example.com:9092

storage:
backend: s3
bucket: kafka-backups-dr
region: us-east-1
prefix: us-west-2/hourly

restore:
# Map each group's committed offset onto the restored log
consumer_group_strategy: header-based
reset_consumer_offsets: true
consumer_groups:
  - order-service
  - payment-processor
  - inventory-manager

Warm standby from backups vs hot standby from replication​

A replication-fed hot standby is the right call when you need failover in seconds and an RPO near zero — MirrorMaker and similar tools keep the standby continuously current. The backup-fed warm standby trades some recovery time for cost and safety.

Many teams run both: replication for the seconds-matter workloads, scheduled backups behind everything as the recovery layer. The active-passive architecture deep dive on the blog works through standby tiers, client cutover, and failback; the DR architecture pillar compares active-passive with active-active.

Sizing the standby window​

The backup schedule is the RPO dial. Hourly backups mean the standby can lose up to an hour of records on promotion; a 15-minute schedule cuts that to 15 minutes at higher storage and compute cost. Restore time is the RTO floor — rehearse a full restore into the standby quarterly and time it, because the estimate only counts if it is measured.

Promotion is half the story. Returning to the primary afterward is a second restore in the opposite direction — back up the promoted standby, restore the delta to the recovered primary, and redirect clients again. The multi-cluster DR example includes both the failover and failback scripts.