Incremental Backups

By default, one-shot and snapshot backups start from the configured start_offset (typically earliest) on every run, producing a full backup each time. Starting from v0.13.5, you can make these backups incremental by adding offset_storage to your configuration. Each run then resumes from where the previous one stopped, backing up only new messages.

How It Works

OSO Kafka Backup tracks progress using a local SQLite database (the offset store). After each partition is backed up, the last consumed offset is saved. On the next run:

1. The offset store is loaded (from local disk or remote storage)
2. Each partition resumes from last_saved_offset + 1
3. Only new messages are fetched and written to storage
4. The manifest is merged with the existing one — segments from all runs are preserved
5. The updated offset store is synced back to remote storage

This means the backup grows incrementally over time, and each run is fast because it only processes new data.

Configuration

Add the offset_storage section to your backup config:

incremental-backup.yaml

mode: backup
backup_id: "production-incremental"

source:
  bootstrap_servers:
    - broker-1.kafka.svc:9092
  topics:
    include:
      - "orders"
      - "payments"

storage:
  backend: s3
  bucket: kafka-backups
  region: us-west-2
  prefix: incremental

backup:
  compression: zstd
  stop_at_current_offsets: true  # Exit after catching up
  include_offset_headers: true

# This section enables incremental backups
offset_storage:
  db_path: /data/offsets.db       # Local path for SQLite database
  sync_interval_secs: 30          # Sync to remote storage every 30s

Key fields

Field	Default	Description
offset_storage.db_path	$TMPDIR/{backup_id}-offsets.db	Local path for the SQLite offset database
offset_storage.sync_interval_secs	30	How often the local DB is synced to remote storage

The offset database is also synced to remote storage at {backup_id}/offsets.db, so it survives pod restarts and machine changes. On startup, the engine checks remote storage for an existing offset database and loads it if the local one is empty.

Backup Modes Comparison

Mode	Config	Incremental?	Exits?	Best for
Full one-shot	continuous: false (default)	No — always starts from start_offset	Yes	Initial full backup, one-off snapshots
Incremental one-shot	continuous: false + offset_storage	Yes — resumes from last checkpoint	Yes	Scheduled cron backups, hourly/daily DR
Snapshot	stop_at_current_offsets: true	No (without offset_storage)	Yes	Consistent point-in-time snapshots
Incremental snapshot	stop_at_current_offsets: true + offset_storage	Yes — resumes from last checkpoint	Yes	Scheduled incremental DR snapshots
Continuous	continuous: true	Yes (automatic)	No	Streaming replication, zero-loss

Example: Scheduled Hourly Incremental Backup

A common pattern is running an incremental backup every hour via cron:

# Crontab entry
0 * * * * kafka-backup backup --config /etc/kafka-backup/incremental.yaml

First run (e.g. Monday 00:00):

[INFO] No saved offsets found, starting from earliest
[INFO] Backed up 1,200,000 records across 3 topics
[INFO] Offset database synced to s3://kafka-backups/incremental/production-incremental/offsets.db

Second run (Monday 01:00):

[INFO] Loaded offset database from remote storage
[INFO] Resuming from saved offsets
[INFO] orders:0 — starting at offset 450001 (saved: 450000)
[INFO] Backed up 45,000 new records across 3 topics

Third run (Monday 02:00):

[INFO] Loaded offset database from remote storage
[INFO] Resuming from saved offsets
[INFO] Backed up 38,000 new records across 3 topics

Each subsequent run takes seconds instead of minutes, and the manifest accumulates all segments from every run.

Verifying Incremental Behavior

After running an incremental backup, you can verify the manifest contains segments from multiple runs:

kafka-backup describe \
  --path s3://kafka-backups/incremental \
  --backup-id production-incremental \
  --format json

Look for multiple segments per partition with non-overlapping offset ranges — each segment corresponds to a single backup run's output.

Recovery and Fault Tolerance

If the offset database is lost locally: The engine loads it from remote storage on startup. As long as the remote copy exists, no data is lost.

If both local and remote offset databases are lost: The backup falls back to start_offset (default: earliest). This produces a full backup, but duplicate segments are deduplicated during manifest merging — existing segments with the same key or start offset are preserved, and the new run fills in any gaps.

If a backup fails mid-run: Offsets are saved incrementally during the run (not only at the end). The next run picks up from the last checkpointed offset, so at most a few seconds of work is repeated.

Kubernetes Operator

For Kubernetes deployments, enable incremental backups using the checkpoint section in the KafkaBackup CRD:

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: incremental-hourly
spec:
  schedule: "0 0 * * * * *"
  stopAtCurrentOffsets: true
  checkpoint:
    enabled: true
    intervalSecs: 30
  # ... rest of config

See Scheduled Backups for complete examples.