Kafka Schema Registry Backup: A Safe Restore Runbook
A Kafka Schema Registry backup must protect registry state alongside the Kafka records that depend on it. Capture subjects, versions, IDs, references, compatibility settings, and modes through the supported Registry API. Restore that state before starting consumers against recovered records.
Backing up records alone can leave valid bytes that applications cannot decode. Treat Kafka data and Schema Registry as one recovery set with one tested recovery point.
Do not treat the internal _schemas topic as a portable backup. Use the Schema
Registry API, restore referenced schemas first, and prove recovery with a
controlled consumer.
What a Kafka Schema Registry backup must capture
Schema Registry holds more than schema text. A recovery-grade backup preserves the relationships and policies that let producers and consumers interpret records consistently.
Capture these items for every registry in scope:
- Subject names and every required schema version
- Schema type, definition, ID, and cross-schema references
- Global compatibility and subject-specific overrides
- Global mode and subject-specific mode overrides
- Soft-deleted subjects when policy requires their recovery
- The dependency order for referenced schemas
Confluent serializers commonly write a registry identifier with each record. The exact framing depends on the serializer and platform version. Confluent's wire-format documentation describes the current payload and header formats.
The important recovery fact stays constant: the identifier in a record must resolve to the intended schema. Restoring the definition under a different ID without rewriting affected records can break deserialization.
Definitions alone are also insufficient. A missing Protobuf import or Avro reference can block registration. Lost compatibility settings can allow an unexpected producer change after failover.
Why copying _schemas is the wrong recovery boundary
Confluent Schema Registry uses _schemas as an internal Kafka store. That does
not make raw topic copying a stable application backup interface.
A topic-level copy does not express the restore decisions operators need. It cannot resolve existing subjects, reference order, or environment mappings. It also cannot rewrite record IDs when a target assigns new ones.
Use the supported REST API boundary instead. It exposes subjects, versions, references, compatibility, and modes as registry concepts. The Schema Registry API reference documents those resources.
Keep _schemas excluded from ordinary topic selection when using the dedicated
registry backup. OSO Kafka Backup captures registry state separately through
the API and stores it beside the Kafka backup.
This separation also makes the recovery order explicit. Registry metadata can be restored and checked before Kafka records reach consumers.
Choose scope and acceptance criteria
Start with an inventory. List the topics, serializer formats, subjects, schema versions, references, compatibility overrides, and consumer groups required by the workload.
Then define a single recovery point. A topic backup from one hour and a registry snapshot from another can create an ID gap. Use one backup run for both unless a written procedure proves that separate snapshots remain consistent.
Choose the workflow from the recovery goal:
| Goal | Registry scope | Kafka data | Main restore choice |
|---|---|---|---|
| Full disaster recovery | All active subjects, versions, settings, and references | All required topics and consumer positions | Clean target with original IDs |
| Partial application recovery | Application subjects and referenced dependencies | Selected application topics | Preserve existing target state |
| Cross-environment migration | Mapped subjects and all dependencies | Selected topics | Accept new IDs and rewrite records |
| Compliance archive | Policy-defined subjects and versions | None when not required | Schema-only snapshot |
Write pass conditions before running the backup. Require every selected subject and version, the expected compatibility settings, resolved references, and a successful test consumer for each serialization format in scope.
Include soft-deleted subjects only when policy or incident recovery requires them. Their inclusion changes both backup scope and restore review.
Configure Kafka Schema Registry backup
Schema Registry backup and restore is an OSO Kafka Backup Enterprise feature. It requires HTTP or HTTPS access to Confluent Schema Registry and credentials that can read the selected resources.
Add enterprise.schema_registry to the normal backup configuration:
mode: backup
backup_id: production-daily
source:
bootstrap_servers:
- kafka-0:9092
- kafka-1:9092
topics:
include:
- "orders-*"
- "payments-*"
exclude:
- "__consumer_offsets"
- "_schemas"
storage:
backend: s3
bucket: company-kafka-backups
region: us-west-2
prefix: production/daily
enterprise:
schema_registry:
url: "https://schema-registry:8081"
auth:
type: basic
username: ${SR_USERNAME}
password: ${SR_PASSWORD}
backup:
subjects:
- "orders-*"
- "payments-*"
exclude:
- "*-test"
include_soft_deleted: false
include_versions: all
include_references: true
Environment variables keep credentials out of the file. Use a secret manager or workload identity to inject them at runtime. Do not place registry passwords in source control or command history.
Subject filters and topic filters are independent. Confirm that every topic in scope has its key and value subjects included. Reference inclusion should stay enabled unless the registry model guarantees there are no external dependencies.
For a custom certificate authority, configure tls.ca_cert. The
product Schema Registry guide documents basic
authentication, mutual TLS, filtering, and connection tuning.
Run and inspect the backup
Run one backup for Kafka records and registry state:
kafka-backup backup --config backup-config.yaml
Use schema-only mode when the approved scope intentionally excludes records:
kafka-backup backup --config backup-config.yaml --schema-only
A schema-only snapshot cannot recover Kafka messages. Label it accordingly so an operator does not mistake it for a complete application recovery point.
Inspect the stored backup and run a deep integrity check:
kafka-backup describe \
--path s3://company-kafka-backups/production/daily \
--backup-id production-daily \
--format json
kafka-backup validate \
--path s3://company-kafka-backups/production/daily \
--backup-id production-daily \
--deep
Review the schema manifest beside the topic segments. Confirm the subject and version counts, compatibility overrides, modes, references, and recorded dependency order against the inventory.
An integrity pass proves that expected backup artifacts are readable. It does not prove that the target registry accepts them or that consumers can decode restored records. The Kafka backup verification runbook covers the wider restore drill.
Select the Schema Registry restore strategy
The target registry determines the safest restore behavior. Inspect it before choosing a strategy.
| Strategy | Existing subjects | Backup subjects | Suitable use |
|---|---|---|---|
preserve | Kept | Missing versions are added | Partial recovery or a shared target |
overwrite | Replaced from backup | Restored | Controlled full recovery |
skip | Left unchanged | Only absent subjects are restored | Adding missing applications |
preserve is the default. It has the smallest collision risk, but operators
must still check that existing versions and IDs match application expectations.
Use force_ids: true only for an empty target where original schema IDs must be
retained. Confluent Schema Registry must allow mode changes with
mode.mutability=true, because importing explicit IDs uses IMPORT mode. Review
the target and maintenance plan before enabling it.
For a migration where the target assigns different IDs, use
rewrite_ids: true. OSO Kafka Backup then updates the schema identifiers in
Kafka message bytes during restore. Test every serializer format in scope.
This restore block targets a clean disaster-recovery registry:
mode: restore
backup_id: production-daily
storage:
backend: s3
bucket: company-kafka-backups
region: us-west-2
prefix: production/daily
target:
bootstrap_servers:
- dr-kafka-0:9092
- dr-kafka-1:9092
enterprise:
schema_registry:
url: "https://dr-schema-registry:8081"
auth:
type: basic
username: ${DR_SR_USERNAME}
password: ${DR_SR_PASSWORD}
restore:
strategy: preserve
force_ids: true
rewrite_ids: false
dry_run: false
Validate the complete restore configuration before it writes data:
kafka-backup validate-restore \
--config restore-config.yaml \
--format json
The CLI basics guide lists the supported backup, restore, describe, and validation commands.
Restore schemas, records, then consumer positions
Stop producers and consumers that could change or read the target during the restore. Isolate outbound side effects such as payment calls, email, and writes to production databases.
Run the reviewed restore:
kafka-backup restore --config restore-config.yaml
OSO Kafka Backup restores referenced schemas before schemas that depend on them. It restores Schema Registry before Kafka records, so registry identifiers are ready before consumers inspect the data.
Keep the full dependency order clear:
- Restore referenced schemas and registry settings.
- Restore dependent subjects and versions.
- Restore Kafka topic records.
- Verify registry and topic state.
- Reset or restore consumer-group positions from the matching snapshot.
- Start one controlled consumer, then expand service recovery.
Original record offsets stored with records are not the same as committed consumer-group positions. The consumer offsets backup runbook explains how to restore records before resetting groups.
Do not start normal consumers after schema restore alone. They also need the matching topic records and valid group positions.
Verify Schema Registry recovery
Compare three inventories: the source before backup, the stored manifest, and the restored registry. Counts provide a quick signal, but inspect identity and settings too.
These read-only API calls check the target registry:
curl --silent --show-error \
--user "$DR_SR_USERNAME:$DR_SR_PASSWORD" \
"$DR_SR_URL/subjects?deleted=true"
curl --silent --show-error \
--user "$DR_SR_USERNAME:$DR_SR_PASSWORD" \
"$DR_SR_URL/subjects/orders-value/versions"
curl --silent --show-error \
--user "$DR_SR_USERNAME:$DR_SR_PASSWORD" \
"$DR_SR_URL/config"
curl --silent --show-error \
--user "$DR_SR_USERNAME:$DR_SR_PASSWORD" \
"$DR_SR_URL/mode"
For each critical subject, compare version numbers, IDs, schema types, definitions, and references. Check subject-level compatibility and mode when an override exists.
Then run one controlled producer and consumer for every serialization format in scope. Verify that the consumer resolves schemas, deserializes keys and values, and processes representative records without calling production systems.
Test referenced schemas directly. A correct subject count can hide a missing dependency or an incorrect version reference.
Record the backup ID, recovery point, software versions, target, result, and failed checks. Keep this evidence with the recovery runbook.
Troubleshoot Schema Registry restore failures
Stop recovery at the first unexplained mismatch. Preserve the backup and test evidence while investigating.
| Symptom | Likely cause | Safe response |
|---|---|---|
Schema ID not found | Schema was not restored or an ID changed | Compare the record ID, manifest, and target registry; restore or rewrite before retrying |
| Authentication failure | Missing or expired registry credentials | Rotate the secret and retest with read-only API calls first |
| HTTP 429 or server errors | Registry request pressure or instability | Lower request rate, increase backoff, and retry the isolated restore |
| Reference registration error | Dependency missing or cyclic | Inspect the reference graph and correct the source before a new backup |
| Subject already exists | Target state conflicts with the chosen strategy | Compare versions and choose preserve, overwrite, or skip deliberately |
| Compatibility rejection | Target compatibility differs from the source | Restore the approved setting or resolve the schema conflict |
| Consumer deserialization error | Wrong ID, schema, format, or record rewrite | Test the failing record and serializer in isolation before cutover |
Do not delete a failed recovery point during cleanup. Correct the source or configuration, create a new recovery point when needed, and rerun the entire drill.
Secure and operate the recovery path
Give the backup identity read access to required subjects and settings. Give the restore identity only the writes required by the chosen strategy. Keep these credentials separate from application credentials.
Verify server certificates and protect any mounted client keys. Apply storage encryption, access logging, retention controls, and immutability according to the data policy.
Schedule registry checks with Kafka restore drills. Run an extra drill after a serializer change, compatibility change, registry upgrade, credential change, subject-filter change, or new schema reference.
The broader Kafka backup strategy guide explains how object-storage backups complement replication. Replication is not a backup because it can copy bad writes and deletions to the target.
Build one tested recovery set
A usable Schema Registry backup preserves the metadata and identifiers needed to decode recovered records. It also preserves settings and references that govern future schema changes.
Back up registry state and Kafka data together. Restore schemas in dependency order, restore records, reset consumer positions, and prove the result with a controlled consumer before production cutover.
Frequently asked questions
How do you back up Kafka _schemas?
Do not copy the internal _schemas topic as a portable application backup. Capture subjects, versions, IDs, references, compatibility settings, and modes through the supported Schema Registry API.
How do you back up Kafka with Schema Registry?
Run one backup configuration that selects the required Kafka topics and enables enterprise.schema_registry. This keeps records and registry state in the same recovery set.
How do you take a backup of a Kafka topic that uses schemas?
Back up the topic records plus every key and value subject, required version, schema reference, and registry setting. Verify the set with an isolated restore and a controlled consumer.
Should you restore schemas before Kafka records?
Yes. Restore referenced schemas first, then dependent schemas, Kafka records, and consumer-group positions. Consumers should start only after registry and topic checks pass.