Skip to main content

Scheduled Backups

Configure recurring backup schedules with the KafkaBackup CRD.

Cron Schedule Format​

The operator uses the Rust cron parser. Schedules include seconds:

┌───────────── second (0 - 59)
│ ┌───────────── minute (0 - 59)
│ │ ┌───────────── hour (0 - 23)
│ │ │ ┌───────────── day of month (1 - 31)
│ │ │ │ ┌───────────── month (1 - 12)
│ │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
│ │ │ │ │ │ ┌───────────── year (optional)
│ │ │ │ │ │ │
* * * * * * *

Common Schedules​

ScheduleCron ExpressionDescription
Every hour0 0 * * * * *At minute 0 of every hour
Every 15 minutes0 */15 * * * * *Every 15 minutes
Every 6 hours0 0 */6 * * * *At minute 0 past every 6th hour
Daily at midnight0 0 0 * * * *Every day at 00:00
Daily at 2 AM0 0 2 * * * *Every day at 02:00
Weekly on Sunday0 0 0 * * 0 *Every Sunday at 00:00
Monthly0 0 0 1 * * *First day of month at 00:00

Scheduled Snapshot Backup​

Set stopAtCurrentOffsets: true for scheduled point-in-time backups. The backup captures the current high watermarks when it starts and exits after all partitions reach them.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: hourly-backup
  namespace: kafka-backup
spec:
  schedule: "0 0 * * * * *"
  stopAtCurrentOffsets: true

  kafkaCluster:
    bootstrapServers:
      - kafka:9092

  topics:
    - orders
    - payments

  storage:
    storageType: s3
    s3:
      bucket: kafka-backups
      region: us-west-2
      prefix: hourly
      credentialsSecret:
        name: s3-credentials

  compression: zstd
  compressionLevel: 3
  includeOffsetHeaders: true
  sourceClusterId: production

Continuous Backup​

Set continuous: true for streaming backups. In v1.0.0 and later, checkpoint.enabled does not imply continuous mode.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: continuous-backup
  namespace: kafka-backup
spec:
  continuous: true

  kafkaCluster:
    bootstrapServers:
      - kafka:9092

  topics:
    - "*"

  storage:
    storageType: azure
    azure:
      accountName: kafkabackups123456
      container: kafka-backups
      prefix: continuous
      useWorkloadIdentity: true

  checkpoint:
    enabled: true
    intervalSecs: 30
  consumerGroupSnapshot: true

Incremental Scheduled Backup​

Available from v0.13.5

Set checkpoint.enabled: true with stopAtCurrentOffsets: true to create incremental scheduled backups. Each scheduled run picks up where the previous one stopped, backing up only new messages. This is the recommended pattern for most scheduled backup workloads — it combines the consistency guarantees of snapshot mode with the efficiency of incremental backups.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: incremental-hourly
  namespace: kafka-backup
spec:
  schedule: "0 0 * * * * *"
  stopAtCurrentOffsets: true

  kafkaCluster:
    bootstrapServers:
      - kafka:9092

  topics:
    - orders
    - payments

  storage:
    storageType: s3
    s3:
      bucket: kafka-backups
      region: us-west-2
      prefix: incremental-hourly
      credentialsSecret:
        name: s3-credentials

  checkpoint:
    enabled: true
    intervalSecs: 30
  compression: zstd
  compressionLevel: 3
  includeOffsetHeaders: true

With this configuration:

  • First run backs up all existing data, saves the offset checkpoint, exits
  • Subsequent runs load the checkpoint, skip already-backed-up data, and back up only new messages
  • The manifest is merged across runs — all segments from all runs are preserved
  • If a run fails, the next run resumes from the last successful checkpoint
When to use incremental vs full scheduled backups

Use incremental when your topics have high throughput and you want fast, efficient scheduled backups. Use full (without checkpoint.enabled) when you want each backup to be a standalone, self-contained snapshot — for example, for compliance evidence where each backup must independently cover a specific time window.

Multi-Tier Backup Strategy​

# Tier 1: frequent critical-topic snapshots
---
apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: tier1-frequent
spec:
  schedule: "0 */15 * * * * *"
  stopAtCurrentOffsets: true
  kafkaCluster:
    bootstrapServers:
      - kafka:9092
  topics:
    - orders
    - payments
  storage:
    storageType: s3
    s3:
      bucket: kafka-backups
      prefix: tier1-frequent
      region: us-west-2
      credentialsSecret:
        name: s3-credentials
  compression: lz4

---
# Tier 2: hourly all-topic snapshots
apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: tier2-hourly
spec:
  schedule: "0 0 * * * * *"
  stopAtCurrentOffsets: true
  kafkaCluster:
    bootstrapServers:
      - kafka:9092
  topics:
    - "*"
  storage:
    storageType: s3
    s3:
      bucket: kafka-backups
      prefix: tier2-hourly
      region: us-west-2
      credentialsSecret:
        name: s3-credentials
  compression: zstd

One-Time Backup​

For manual or on-demand backups, omit schedule.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: manual-backup-20260413
spec:
  kafkaCluster:
    bootstrapServers:
      - kafka:9092
  topics:
    - orders
    - payments
  storage:
    storageType: s3
    s3:
      bucket: kafka-backups
      region: us-west-2
      prefix: manual
      credentialsSecret:
        name: s3-credentials
  stopAtCurrentOffsets: true

Pausing a Schedule​

spec:
  suspend: true

Monitoring Schedules​

kubectl get kafkabackup hourly-backup \
  -o jsonpath='{.status.nextScheduledBackup}'
kubectl get kafkabackup hourly-backup \
  -o jsonpath='{.status.lastBackupTime}'

Best Practices​

  1. Set stopAtCurrentOffsets: true for recurring point-in-time backups.
  2. Set continuous: true only for streaming backups.
  3. Keep includeOffsetHeaders: true when you plan to restore and reset offsets.
  4. Use object-store lifecycle policies for long-term retention.
  5. Enable consumerGroupSnapshot: true when consumer group recovery matters.

Next Steps​