Kafka DR Testing Playbook: Prove Your Disaster Recovery Plan Works
A Kafka disaster recovery plan is only credible after it survives a test. This playbook turns the written plan into a scheduled program of drills that climbs four levels: backup validation, restore drills, failover simulation, and a full game day.
Most Kafka DR plans are documents nobody has executed. The first real execution happens during a real incident, with the clock running and half the team asleep. Testing moves that first execution to a Tuesday afternoon, where mistakes cost nothing.
An untested DR plan is a hypothesis. Climb the test ladder in order — validate backups continuously, restore monthly, simulate failover quarterly, and run a game day once or twice a year. Write the pass criteria before each drill, and keep the evidence.
What a Kafka disaster recovery plan must contain
Before testing anything, confirm the plan itself is testable. A written Kafka disaster recovery plan needs six parts.
First, a recovery boundary: which clusters, topics, schemas, and consumer groups are in scope. Second, declared RTO and RPO targets per workload tier — the Kafka RTO/RPO planning guide covers how to set them. Third, trigger criteria: who declares a disaster, on what evidence, within what time budget.
Fourth, runbooks with named owners for each recovery path. Fifth, a dependency order, because Kafka rarely recovers alone — Schema Registry, connectors, and downstream databases have their own sequence. Sixth, communication paths for the incident channel, stakeholders, and customers.
Each section is a claim. Testing converts claims into evidence, one drill at a time. If any section is missing, write it before scheduling the first drill — you cannot test a runbook that does not exist.
The four-level Kafka DR test ladder
DR tests differ enormously in cost and in what they prove. Treat them as a ladder, not a menu.
| Level | Test | What it proves | What it cannot prove | Typical cadence |
|---|---|---|---|---|
| 1 | Backup validation | Stored data is complete and readable | That a restore works end to end | Daily / continuous |
| 2 | Restore drill | Data comes back correctly in an isolated target | That failover meets the RTO | Monthly per critical workload |
| 3 | Failover simulation | The promotion runbook works, offsets translate, stages fit the RTO budget | That people execute well under pressure | Quarterly |
| 4 | Game day | The team, the communication paths, and the full plan work together | Nothing — this is the top | Once or twice a year |
Each level assumes the one below it passes. Never schedule a game day on backups that fail deep validation — you would rehearse a recovery that cannot succeed. Climbing in order also keeps early failures cheap: a damaged segment surfaces in a Level 1 alert, not halfway through a timed failover.
Level 1 — validate backups continuously
Backup validation is the only level that runs without human involvement, so it should run all the time.
Quick validation checks stored metadata and the expected segment inventory. It catches missing artifacts in seconds:
kafka-backup validate \
--path s3://company-kafka-backups/production/orders-cluster \
--backup-id production-orders-daily
Deep validation reads and verifies every segment, and reports checked, valid, missing, and damaged counts:
kafka-backup validate \
--path s3://company-kafka-backups/production/orders-cluster \
--backup-id production-orders-daily \
--deep
Schedule quick validation after every backup run and deep validation at least weekly. Alert on any failure, and treat a validation alert with the same urgency as a failed backup — from the DR plan's point of view, they are the same event.
The Kafka backup verification guide covers the full mechanics, including manifest inspection and acceptance criteria. This playbook only insists that it runs on a schedule, not on good intentions.
Level 2 — run restore drills on a schedule
A restore drill proves that stored data becomes usable Kafka topics again. It runs against an isolated scratch target — never production.
Write a restore configuration for the drill target and validate the plan before writing any records:
kafka-backup validate-restore \
--config restore-drill.yaml \
--format json
Review the reported topics, segments, record estimate, and consumer offset actions. Then execute:
kafka-backup restore --config restore-drill.yaml
After the restore, compare the target against the backup manifest rather than eyeballing topic sizes. The validation run command checks message counts, offset ranges, and consumer group offsets against the stored manifest:
kafka-backup validation run --config validation.yaml
Record four things for every drill: the backup ID, the recovery point, the elapsed time, and the result. Those records make drills comparable across quarters, which is how you notice a restore that has quietly grown from twenty minutes to two hours. The restore and PITR guide documents the restore options in depth.
Level 3 — simulate a regional failover
Level 3 executes the real promotion runbook against the DR region, without touching production clients. The scenario: the primary region is declared lost.
Run the runbook steps in order. Stop the backup feed or replication flow into the DR region. Note the data boundary — the recovery point the drill will certify. Restore or finalize data on the DR cluster. Reset consumer groups. Run a controlled acceptance consumer. Verify schemas resolve.
The promotion restore uses header-based offset translation so consumer groups resume at meaningful positions:
mode: restore
backup_id: "orders-dr-feed"
target:
bootstrap_servers:
- kafka-0.dr.svc:9092
storage:
backend: s3
bucket: company-kafka-dr
region: us-east-1
prefix: backups/prod
restore:
create_topics: true
default_replication_factor: 3
consumer_group_strategy: header-based
auto_consumer_groups: true
offset_report: /var/log/kafka-backup/promotion-offsets.json
Time every stage separately: data finalization, topic creation, offset translation, acceptance checks. This is where the RTO budget meets reality, and the slowest stage is the one to engineer next. The full configuration schema is in the configuration reference, and the active-passive DR architecture guide covers how to design the runbook this drill executes.
Keep production clients out of the drill entirely. Client redirection gets tested at Level 4, in a test environment, where a mistake cannot split production traffic.
Level 4 — run a game day
A game day rehearses the plan with the people who would run the real incident. The declaration, the incident channel, the runbook execution, the verification, and the retrospective all happen, timed, in a test environment.
Inject realistic friction. Run the drill with the secondary on-call, not the engineer who wrote the runbook. Expire one credential on purpose in the test environment and watch how long the workaround takes. Require the written runbook only — no tribal knowledge, no asking the author.
Measure observed RTO and RPO against the declared targets, stage by stage. Every gap becomes a tracked engineering task with an owner, not a bullet in a retrospective document that nobody reopens. A game day that finds nothing was probably too gentle; tighten the scenario next time.
Build a scenario library, not one scenario
Teams that test only "the region went down" get good at exactly one recovery. Real incidents are more varied, and some of them break the replication-based recovery path entirely.
| Scenario | Trigger example | Recovery path | Recovery point implication |
|---|---|---|---|
| Region outage | Cloud region unavailable | Promote DR cluster from replicated or backup-fed standby | Bounded by feed lag |
| Broker / storage loss | Volume failure beyond replication factor | Restore affected topics from backup | Last validated backup point |
| Accidental topic deletion | Operator deletes a production topic | Restore the topic from an independent backup | Point in time before deletion |
| Corrupting deploy | Bad producer writes garbage for hours | Point-in-time restore to before the bad deploy | Chosen timestamp, millisecond precision |
| Credential / secret loss | Compromised or expired cluster credentials | Re-issue secrets, verify access paths from runbook | None, if caught early |
| Schema Registry loss | Registry data lost or corrupted | Restore schemas alongside topic data | Must align with topic recovery point |
The middle rows are the reason this table exists. Replication copies deletes and corruption faithfully, at replication speed — a deleted topic disappears from every replica, and poisoned records arrive on the standby moments after the primary. Those scenarios can only be recovered from an independent backup with point-in-time restore. A DR test program that never drills deletion or corruption is testing replication, not disaster recovery.
Each scenario in the library names its trigger, its expected recovery path, its recovery point implication, and the runbook that owns it. Drill a different scenario each quarter rather than repeating the comfortable one.
Cadence, ownership, and evidence
A test that is not scheduled does not happen. Put the ladder on a calendar with named owners.
| Level | Cadence | Owner | Evidence produced |
|---|---|---|---|
| 1 — Backup validation | After every backup; deep weekly | Automation, alerting to platform on-call | Validation reports, alert history |
| 2 — Restore drill | Monthly per Tier 1 workload | Platform engineer on rotation | Drill record: backup ID, recovery point, elapsed time, result |
| 3 — Failover simulation | Quarterly | DR runbook owner | Stage timings vs RTO budget, offset translation report |
| 4 — Game day | Once or twice a year | Incident commander | Observed RTO/RPO vs targets, gap list with owners |
Two rules make the evidence worth keeping. First, write the pass/fail criteria before the drill — "restore completes within 45 minutes with exact message counts" is a test, while "restore looked fine" is a feeling. Second, store the evidence outside the failure domain being tested. A drill report that lives in the region you are simulating the loss of proves less than nothing during a real event.
Evidence has an audience beyond the team. Auditors, customers, and your own leadership ask the same question after every public Kafka outage: would we survive that? A folder of dated drill records with measured recovery times answers it. The disaster recovery use case shows how OSO Kafka Backup fits that evidence chain, and the CLI basics guide covers the commands the drills are built from.
Kafka DR testing checklist
- Written DR plan with scope, targets, triggers, runbooks, dependency order, and owners
- RTO/RPO targets declared per workload tier and approved
- Level 1 validation automated, with alerts treated as incidents
- Level 2 restore drills scheduled monthly for Tier 1 workloads
- Level 3 failover simulation dated for this quarter
- Level 4 game day on this year's calendar
- Scenario library covers deletion and corruption, not just region loss
- Pass/fail criteria written before each drill
- Evidence stored outside the tested failure domain
- Every gap tracked to closure with an owner
A disaster recovery plan that has never been tested is a wish. The ladder makes testing tractable: start at Level 1 this week — it is automation, not a meeting. Schedule one restore drill this month. Put a failover simulation on next quarter's calendar. By the time the game day arrives, it should feel boring, and boring is exactly what you want a disaster to be.
Frequently asked questions
How to backup Kafka?
Run scheduled or continuous backups of the required topics to object storage with kafka-backup, including consumer group offsets and schemas in scope. A backup only counts toward a DR plan once it passes deep validation and a restore drill into an isolated target.
How to take Kafka backup?
Create a reviewed backup configuration listing the source topics, storage backend, and backup ID, then run kafka-backup backup with that config. For DR readiness, validate the stored data on a schedule and record the backup ID and recovery point so drills can reference exact restore points.
Why replication is required in Kafka?
Intra-cluster replication keeps partitions available when individual brokers fail, which protects everyday availability. It does not replace DR testing, because replication faithfully copies deletions and corrupt records to every replica — recovering from those scenarios requires an independent backup and a tested restore path.