Scheduled Kafka backups as a Kubernetes resource
No cron boxes, no wrapper scripts. Apply one KafkaBackup manifest and the operator runs incremental point-in-time backups on schedule, snapshots consumer groups, and prunes old sets under an opt-in retention policy.
Kafka backup on Kubernetes, the production pattern
Kafka backup on Kubernetes comes down to one custom resource. The
OSO Kafka Backup Operator watches KafkaBackup objects in the
kafka.oso.sh/v1alpha1 API group and turns each schedule into point-in-time
backup runs against any Kafka the pod can reach.
The recommended production shape is the incremental schedule. Each run captures the current high watermarks, backs up only messages that arrived since the last checkpoint, and exits. The manifest merges across runs, so any run restores the full history. A failed run simply resumes from the last successful checkpoint.
| Mode | Key fields | When to use it |
|---|---|---|
| Incremental scheduled | schedule + stopAtCurrentOffsets: true + checkpoint.enabled: true | Default for production — fast runs, full history |
| Full scheduled snapshot | schedule + stopAtCurrentOffsets: true | Each backup must stand alone, e.g. compliance evidence |
| Continuous | continuous: true | Streaming backup with checkpoints, minimal lag |
| One-time | omit schedule | Manual or pre-change backups |
- Install
- KafkaBackup
- Operate
helm repo add oso https://osodevops.github.io/helm-charts/
helm repo update
kubectl create namespace kafka-backup
helm install kafka-backup-operator oso/kafka-backup-operator \
--namespace kafka-backup
apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
name: incremental-hourly
namespace: kafka-backup
spec:
# 7-field cron, seconds first: minute 0 of every hour
schedule: "0 0 * * * * *"
stopAtCurrentOffsets: true
kafkaCluster:
bootstrapServers:
- kafka:9092
topics:
- orders
- payments
storage:
storageType: s3
s3:
bucket: kafka-backups
region: us-west-2
prefix: incremental-hourly
credentialsSecret:
name: s3-credentials
# Incremental: each run continues from the last checkpoint (v0.13.5+)
checkpoint:
enabled: true
intervalSecs: 30
compression: zstd
includeOffsetHeaders: true
consumerGroupSnapshot: true
# Opt-in pruning — keep dryRun: true until the report looks right
retention:
enabled: true
maxAgeDays: 30
keepLast: 3
dryRun: true
kubectl get kafkabackup incremental-hourly \
-o jsonpath='{.status.nextScheduledBackup}'
kubectl get kafkabackup incremental-hourly \
-o jsonpath='{.status.lastBackupTime}'
# What would retention delete? (while dryRun: true)
kubectl get kafkabackup incremental-hourly \
-o jsonpath='{.status.retentionEligibleBackups}'
Restore, offsets, and the rest of the CRD family
Backups are half the job. The same operator ships KafkaRestore for restores with
a target time, KafkaOffsetReset and KafkaOffsetRollback for consumer group
positions, and KafkaBackupValidation for evidence reports — all declarative, all
in Git next to your platform config.
Retention stays disabled unless you opt in, and dryRun: true reports what would
be deleted before anything is. The current backup set is always kept. Details and
edge cases live in the backup retention guide.
If Strimzi manages your cluster, use the Strimzi-aware operator described on the Strimzi integration page — it resolves bootstrap servers and credentials from your existing Strimzi resources. On EKS, pair the operator with IRSA so backup pods reach S3 without static keys, as shown in the AWS backup architecture.
Frequently asked questions
How do I run scheduled Kafka backups on Kubernetes?
Install the OSO Kafka Backup Operator with Helm, then apply a KafkaBackup resource with a schedule and stopAtCurrentOffsets: true. The operator runs each backup as a Kubernetes Job at the scheduled time. Note the cron format has seven fields, starting with seconds.
What is an incremental Kafka backup on Kubernetes?
A scheduled KafkaBackup with checkpoint.enabled: true (available from v0.13.5). The first run backs up everything; each later run loads the checkpoint and backs up only new messages. Segment manifests merge across runs, so the backup set always restores the full history.
How are old backups cleaned up?
Through opt-in retention on the KafkaBackup spec: maxAgeDays and keepLast control pruning of whole backup sets after successful runs. Retention is disabled by default and supports dryRun: true, which reports eligible sets in the resource status without deleting anything. For GCS, use bucket lifecycle policies instead.
Does this work with Strimzi-managed Kafka clusters?
Yes, two ways. The generic operator backs up any Kafka the pod can reach, including Strimzi brokers. Alternatively, the Strimzi Backup Operator resolves bootstrap servers and credentials directly from Strimzi resources — the better fit when Strimzi owns your cluster definitions.
Can the operator restore consumer group offsets too?
Yes. Set consumerGroupSnapshot: true and includeOffsetHeaders: true on the backup, then use KafkaRestore with automatic consumer group handling, or the KafkaOffsetReset resource to reposition groups explicitly after a restore.
Ready to protect your Kafka data?
Take your first backup in minutes with the open source CLI, or talk to us about Enterprise features like encryption, RBAC, and audit logging.