Back up Apache Kafka to Cloudflare R2

Why use Cloudflare R2 for Kafka backup​

Kafka backup for Cloudflare R2 uses the storage service's S3-compatible API. OSO Kafka Backup connects through the account endpoint, so no separate R2 adapter is required.

R2 provides object storage outside your Kafka brokers and their attached disks. Cloudflare does not charge data-transfer egress for direct R2 access. Storage, operation, and some retrieval charges still apply, so model backup and restore traffic together.

OSO compresses topic records with Zstd or LZ4 before upload. It also preserves consumer group offsets and topic configuration. The storage format reference explains the portable objects written to the bucket.

Cloudflare R2 Kafka backup configuration​

Create an R2 API token with access to the backup bucket. Use its access key ID and secret access key with the account-scoped S3 endpoint. Cloudflare documents auto as the region value for S3 clients.

mode: backup
backup_id: "r2-daily"

source:
bootstrap_servers:
  - kafka-1.internal:9092
  - kafka-2.internal:9092
topics:
  include:
    - "*"
  exclude:
    - "__consumer_offsets"

storage:
backend: s3
bucket: kafka-backups
region: auto
endpoint: https://your-account-id.r2.cloudflarestorage.com
access_key: ${R2_ACCESS_KEY_ID}
secret_key: ${R2_SECRET_ACCESS_KEY}
prefix: backups/production

backup:
compression: zstd
compression_level: 5
include_offset_headers: true

mode: restore
backup_id: "r2-daily"

target:
bootstrap_servers:
  - kafka-dr-1.internal:9092

storage:
backend: s3
bucket: kafka-backups
region: auto
endpoint: https://your-account-id.r2.cloudflarestorage.com
access_key: ${R2_ACCESS_KEY_ID}
secret_key: ${R2_SECRET_ACCESS_KEY}
prefix: backups/production

restore:
# Restore through the moment before the incident (Unix ms)
time_window_end: 1783947600000
consumer_group_strategy: header-based

kafka-backup backup --config backup.yaml
kafka-backup restore --config restore.yaml

What the R2 backup contains​

Each backup contains topic records with timestamps and headers, consumer group offsets, and topic configuration. Backups remain separate from the source cluster's retention and broker storage.

Restores can target the source cluster or a new cluster. A millisecond timestamp lets you stop before a bad deployment or harmful record. See the configuration reference for topic filters, compression, and restore controls.