Schema Registry Backup & Restore

Enterprise 0.6.1 backs up Confluent self-hosted and Apicurio v2/v3 registries alongside Kafka data. Examples use the Homebrew Enterprise command kafka-backup-enterprise; use ./kafka-backup for an unpacked release binary. It captures exact reference versions, registry settings, metadata and lifecycle state in a versioned snapshot with integrity hashes.

For original-ID preservation, ID allocation, cross-provider migration, restore --schema-only and read-only validation, follow the Schema Registry restore guide.

The Problem

Registry serializers identify schemas using numeric prefixes, GUID headers or native ID headers. Those identifiers must resolve on the destination registry. Without backing up schemas, restored messages are useless:

Without Schema Backup:
  Kafka data ──── Backed up      ✓
  Schema IDs ──── NOT backed up  ✗
  Result: "Schema ID 42 not found"

With Schema Backup:
  Kafka data ──── Backed up      ✓
  Schema IDs ──── Backed up      ✓
  Result: Consumers work correctly

Quick Start

Add the enterprise.schema_registry section to your existing backup config:

backup-config.yaml

mode: backup
backup_id: daily-2026-04-06
source:
  bootstrap_servers: ["kafka:9092"]
storage:
  backend: s3
  bucket: kafka-backups

enterprise:
  schema_registry:
    url: "https://schema-registry:8081"
    auth:
      type: basic
      username: ${SR_USERNAME}
      password: ${SR_PASSWORD}

Run the backup — schemas are captured automatically alongside Kafka data:

kafka-backup-enterprise backup --config backup-config.yaml

Or back up schemas only (no Kafka data):

kafka-backup-enterprise backup --config backup-config.yaml --schema-only

What Gets Backed Up

Item	Description
Subjects	All subject names matching filter patterns
Schema versions	Every version of every subject: schema definition, type, ID, references
Global compatibility	Global compatibility level (BACKWARD, FORWARD, FULL, NONE, etc.)
Per-subject compatibility	Subject-specific compatibility overrides
Global mode	Registry mode (READWRITE, READONLY, IMPORT)
Per-subject mode	Subject-specific mode overrides
Schema references	Exact referenced versions, recursively including dependencies outside filters
Schema envelopes	Metadata, rule sets, GUIDs when supplied and normalization settings
Snapshot integrity	Version-2 completeness index, object hashes and unresolved dependencies

Configuration Reference

Authentication

Three authentication methods are supported:

Basic Auth (most common)

enterprise:
  schema_registry:
    url: "https://schema-registry:8081"
    auth:
      type: basic
      username: ${SR_USERNAME}
      password: ${SR_PASSWORD}

Mutual TLS (mTLS)

enterprise:
  schema_registry:
    url: "https://schema-registry:8081"
    auth:
      type: mtls
      ca_cert: /certs/ca.crt
      client_cert: /certs/client.crt
      client_key: /certs/client.key

No Authentication

enterprise:
  schema_registry:
    url: "http://schema-registry:8081"
    # auth section omitted = no authentication

TLS Server Verification

For HTTPS endpoints with custom CA certificates:

enterprise:
  schema_registry:
    url: "https://schema-registry:8081"
    tls:
      ca_cert: /certs/custom-ca.crt

Subject Filtering

Control which subjects are backed up using glob patterns:

enterprise:
  schema_registry:
    backup:
      # Include patterns (default: ["*"] = all subjects)
      subjects:
        - "orders-*"
        - "payments-*"
      # Exclude patterns (applied after include)
      exclude:
        - "*-test"
        - "*-internal"
      # Include soft-deleted subjects (default: false)
      include_soft_deleted: false
      # Back up all versions or only latest (default: all)
      include_versions: all  # or "latest"
      # Auto-include referenced subjects outside filter (default: true)
      include_references: true

Restore options

Use enterprise.registry_restore for source/destination providers, ID policy, coordinate mappings and independent key/value serializer profiles. Original-ID preservation is the default. The restore guide includes complete configurations and command examples.

The legacy enterprise.schema_registry.restore fields remain aliases:

Field	Behavior in 0.6.1
force_ids: true	Requests original IDs; ID preservation is already the default
rewrite_ids: true	Allocates destination IDs and enables record rewriting; explicit serializer profiles are still required
subject_mapping	Maps Confluent subject names; do not duplicate the same mapping in registry_restore.coordinates
strategy: skip	Reuses entries only after verifying equivalence
strategy: overwrite	Rejected; destructive registry replacement is unsupported
dry_run: true	Makes the entire invocation read-only, including Kafka

force_ids: true cannot be combined with ID rewriting. Unmapped IDs, conflicting entries and verification/cleanup failures stop recovery before Kafka replay.

Connection Tuning

enterprise:
  schema_registry:
    connection:
      timeout_ms: 30000        # Request timeout (default: 30s)
      max_retries: 3           # Safe-read retries on retryable responses
      retry_backoff_ms: 1000   # Initial retry backoff
      rate_limit_rps: 25       # Max requests per second
      concurrent_requests: 4   # Max parallel API calls

Apicurio v2/v3 backup

Select the Core Registry API generation explicitly. The URL is the registry base URL; api_version selects v2 or v3.

enterprise:
  apicurio_registry:
    url: https://apicurio:8080
    api_version: v3  # Use v2 for Apicurio 2.x
    backup:
      groups: ["orders"]
      artifacts: ["*"]
      include_versions: all
      include_references: true
      include_export: true

The structured snapshot retains original global/content IDs, content hashes, references, metadata, rules and version states. The native export ZIP records its own scope. A whole-registry export can contain artifacts outside the structured filters; it is never imported for a filtered restore.

Original-ID Apicurio recovery requires a complete, unfiltered archive and an empty target using the same API generation. Filtered recovery and supported cross-generation migrations allocate IDs through the normal APIs. See the restore guide for settings and state limits.

Storage Format

Schema backups are stored alongside Kafka data in the same storage backend:

{backup_id}/
  manifest.json              # Main backup manifest
  topics/...                 # Kafka data segments
  schema-registry/           # Schema Registry backup
    _manifest.json           # Schema manifest (subjects, counts, dependency order)
    _global_config.json      # Global compatibility and mode
    subjects/
      orders-value/
        _metadata.json       # Subject config, mode, version list
        v1.json              # Schema version 1 (full definition)
        v2.json              # Schema version 2
      payments-value/
        _metadata.json
        v1.json

Required object failures leave an explicitly incomplete snapshot. Restoration rejects missing dependencies or fidelity instead of inventing it. Older backups remain inspectable, but cannot restore information they did not record.

Schema References & Dependencies

Schemas can reference other schemas (e.g., Protobuf imports). The backup engine automatically:

1. Discovers all cross-subject references
2. Builds a dependency graph (DAG)
3. Performs a topological sort
4. Records the correct restore order in the manifest

Restore orders exact versions and references. A subject-level cycle can be satisfiable at version level; genuinely unsatisfiable dependency/history cycles are rejected. See preservation and recovery.

Use Cases

Disaster recovery and migration

Use original IDs for a compatible recovery target, or allocate IDs and rewrite records for a supported migration. Confluent self-hosted and Apicurio v2/v3 are in scope. Confluent Cloud registry restore and MSK SEED/TAIL registry conversion are excluded in 0.6.1.

See complete restore examples, serializer profiles and tested versions.

Compliance Archive (Schema-Only)

Periodic schema snapshots for regulatory requirements:

kafka-backup-enterprise backup --config schema-archive.yaml --schema-only

Troubleshooting

Issue	Cause	Solution
Schema ID not found after restore	Schemas not restored before data	Schema restore runs automatically before data restore
Subject not found during backup	Subject was deleted	Enable include_soft_deleted: true
Dependency/history conflict	Exact versions cannot be registered faithfully	Inspect the version-level plan and include the required dependencies
429 rate limit errors	Too many API calls	Reduce rate_limit_rps or increase retry_backoff_ms
Authentication failed	Invalid credentials	Check SR_USERNAME/SR_PASSWORD env vars

Requirements

• Confluent self-hosted or Apicurio v2/v3 for the released restore workflow
• HTTP(S) access to Schema Registry REST API
• Enterprise license with schema_registry feature enabled