Back up Apache Kafka to Ceph

How Kafka backup to Ceph works​

Kafka backup to Ceph uses the S3-compatible API exposed by Ceph Object Gateway. OSO Kafka Backup connects through your gateway endpoint, so no separate Ceph storage adapter is required.

Ceph keeps backup objects outside Kafka brokers and their attached volumes. Your team controls the storage cluster, network path, placement, and retention policy. This can suit private cloud and on-premises recovery designs.

OSO compresses topic records with Zstd or LZ4 before upload. It also preserves consumer group offsets and topic configuration. The storage format reference explains the objects written to Ceph.

Ceph Kafka backup configuration​

Create a Ceph Object Gateway user with access to the backup bucket. Configure the s3 backend with the gateway endpoint and its access credentials.

This example assumes virtual-hosted bucket access. Configure gateway DNS for bucket subdomains and use a matching TLS certificate. Ceph also supports path-style requests, but client support must be verified for your deployment.

mode: backup
backup_id: "ceph-daily"

source:
bootstrap_servers:
  - kafka-1.internal:9092
  - kafka-2.internal:9092
topics:
  include:
    - "*"
  exclude:
    - "__consumer_offsets"

storage:
backend: s3
bucket: kafka-backups
region: us-east-1
endpoint: https://s3.ceph.example.com
access_key: ${CEPH_ACCESS_KEY_ID}
secret_key: ${CEPH_SECRET_ACCESS_KEY}
prefix: backups/production

backup:
compression: zstd
compression_level: 5
include_offset_headers: true

mode: restore
backup_id: "ceph-daily"

target:
bootstrap_servers:
  - kafka-dr-1.internal:9092

storage:
backend: s3
bucket: kafka-backups
region: us-east-1
endpoint: https://s3.ceph.example.com
access_key: ${CEPH_ACCESS_KEY_ID}
secret_key: ${CEPH_SECRET_ACCESS_KEY}
prefix: backups/production

restore:
# Restore through the moment before the incident (Unix ms)
time_window_end: 1784160000000
consumer_group_strategy: header-based

kafka-backup backup --config backup.yaml
kafka-backup restore --config restore.yaml

Ceph deployment checks​

Use the S3 deployment guide for credential and connection patterns. Test a full restore before relying on the bucket for disaster recovery.