Back up Apache Kafka through native client protocols

Apache Kafka backup through standard client settings​

Apache Kafka backup starts with the broker addresses and security settings your clients already use. OSO Kafka Backup connects as a Kafka client and selects topics through exact names or patterns.

The backup writes topic records, consumer group offsets, and topic configuration to independent storage. Supported targets include S3, S3-compatible storage, Azure Blob, GCS, and a filesystem.

Kafka provides replica durability for the live log. A separate backup adds recovery from unwanted writes, topic deletion, and other logical incidents. See the first backup tutorial for the full backup and restore procedure.

Configure a production Kafka backup​

This example uses a TLS-encrypted SCRAM connection and Amazon S3. Match the broker addresses, credentials, topic patterns, and storage settings to your environment.

mode: backup
backup_id: kafka-prod-daily

source:
bootstrap_servers:
  - broker-1.kafka.example.com:9093
  - broker-2.kafka.example.com:9093
security:
  security_protocol: SASL_SSL
  sasl_mechanism: SCRAM-SHA256
  sasl_username: backup-user
  sasl_password: ${KAFKA_PASSWORD}
  ssl_ca_location: /etc/kafka/ca.crt
topics:
  include:
    - orders
    - "payments-*"
  exclude:
    - "*-internal"

storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: apache-kafka/production

backup:
compression: zstd
continuous: true
include_offset_headers: true

mode: backup
backup_id: kafka-local-test

source:
bootstrap_servers:
  - localhost:9092
topics:
  include:
    - orders

storage:
backend: filesystem
path: /var/lib/kafka-backup/data

backup:
compression: lz4

kafka-backup backup --config apache-kafka-backup.yaml

What an Apache Kafka backup preserves​

Each backup keeps records with their timestamps and headers, plus consumer group offsets and topic settings. Compression uses Zstd or LZ4 before data reaches the storage backend.

Restores can target the original cluster or another reachable Kafka cluster. A millisecond restore timestamp lets operators recover the state before an incident. The configuration reference documents the available backup and restore keys.