Configuration Reference

OSO Kafka Backup uses YAML configuration files for backup and restore operations. This reference documents all available options.

Configuration Structure

# Required: Operation mode
mode: backup  # or "restore"

# Required: Unique backup identifier
backup_id: "my-backup-001"

# Source/Target Kafka cluster configuration
source:  # For backup mode
target:  # For restore mode
  bootstrap_servers: []
  security: {}
  topics: {}

# Storage backend configuration
storage:
  backend: filesystem  # or s3, azure, gcs
  # Backend-specific options...

# Mode-specific options
backup: {}   # Backup options
restore: {}  # Restore options

Common Configuration

mode

Required. Operation mode.

mode: backup   # Run a backup operation
mode: restore  # Run a restore operation

backup_id

Required. Unique identifier for the backup.

backup_id: "production-daily-001"
backup_id: "dr-backup-$(date +%Y%m%d)"

Kafka Cluster Configuration

Used as source in backup mode and target in restore mode.

bootstrap_servers

Required. List of Kafka broker addresses.

source:
  bootstrap_servers:
    - broker-1.kafka.svc:9092
    - broker-2.kafka.svc:9092
    - broker-3.kafka.svc:9092

security

Optional security configuration for Kafka connection.

source:
  security:
    # Security protocol
    security_protocol: SASL_SSL  # PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL

    # SASL configuration
    sasl_mechanism: SCRAM-SHA256  # PLAIN, SCRAM-SHA256, SCRAM-SHA512
    sasl_username: backup-user
    sasl_password: ${KAFKA_PASSWORD}  # Environment variable substitution

    # SSL/TLS configuration
    ssl_ca_location: /etc/kafka/ca.crt
    ssl_certificate_location: /etc/kafka/client.crt
    ssl_key_location: /etc/kafka/client.key
    ssl_key_password: ${SSL_KEY_PASSWORD}

Security Protocol Options

Protocol	Description
`PLAINTEXT`	No encryption, no authentication
`SSL`	TLS encryption, optional mTLS
`SASL_PLAINTEXT`	SASL authentication, no encryption
`SASL_SSL`	SASL authentication with TLS encryption

SASL Mechanism Options

Mechanism	Description	Feature gate
`PLAIN`	Simple username/password	Built-in
`SCRAM-SHA256`	SCRAM-SHA-256 — Salted Challenge Response (SHA-256)	Built-in
`SCRAM-SHA512`	SCRAM-SHA-512 — Salted Challenge Response (SHA-512)	Built-in
`GSSAPI`	Kerberos authentication (RFC 4752). Uses a keytab for non-interactive credential acquisition.	Requires CLI built with `--features gssapi`

SASL/GSSAPI opt-in

GSSAPI is gated behind the gssapi cargo feature. The default oso/kafka-backup release binary and Docker image do not include it. A binary without the feature that loads a sasl_mechanism: GSSAPI config fails fast at startup with a clear error. See the Security Setup guide for build instructions.

SASL/GSSAPI (Kerberos) Fields

Required only when sasl_mechanism: GSSAPI. Ignored for other mechanisms.

source:
  security:
    security_protocol: SASL_PLAINTEXT   # or SASL_SSL
    sasl_mechanism: GSSAPI
    sasl_kerberos_service_name: kafka
    sasl_keytab_path: /etc/kafka-backup/client.keytab
    sasl_krb5_config_path: /etc/krb5.conf

Field	Type	Required	Description
`sasl_kerberos_service_name`	string	Yes	The service-name portion of the broker's service principal. For `kafka/broker.host@REALM`, this is `kafka`.
`sasl_keytab_path`	path	Yes	Absolute path to the keytab holding the client principal's long-term key. Must be readable by the kafka-backup process; lock to `0400` in production.
`sasl_krb5_config_path`	path	No	Override the default `/etc/krb5.conf`. Useful inside containers that don't share the host's krb5 config.

Session re-authentication (KIP-368) is handled automatically — if the broker advertises a session lifetime, kafka-backup re-handshakes at ~80 % of the window (30 s floor, ±5 s jitter) before the broker forcibly disconnects.

connection

Optional TCP connection settings. These are particularly important for cloud-hosted Kafka services like Confluent Cloud that terminate idle connections.

source:
  connection:
    # Enable TCP keepalive to prevent idle connection termination
    tcp_keepalive: true           # Default: true

    # Time in seconds before first keepalive probe is sent
    keepalive_time_secs: 60       # Default: 60

    # Interval in seconds between keepalive probes
    keepalive_interval_secs: 20   # Default: 20

    # Enable TCP_NODELAY (disable Nagle's algorithm) for lower latency
    tcp_nodelay: true             # Default: true

Connection Options Reference

Option	Type	Default	Description
`tcp_keepalive`	bool	`true`	Enable TCP keepalive probes
`keepalive_time_secs`	int	`60`	Seconds idle before first probe
`keepalive_interval_secs`	int	`20`	Seconds between probes
`tcp_nodelay`	bool	`true`	Disable Nagle's algorithm

Confluent Cloud

Confluent Cloud terminates idle TCP connections after ~5 minutes. The default keepalive settings (60s time, 20s interval) prevent this. If you're experiencing "Broken pipe" errors with Confluent Cloud, ensure TCP keepalive is enabled.

topics

Topic selection for backup or restore.

source:
  topics:
    # Include specific topics or patterns
    include:
      - orders              # Exact topic name
      - payments            # Another exact name
      - "events-*"          # Wildcard pattern
      - "logs-2024-*"       # Date-based pattern

    # Exclude topics (applied after include)
    exclude:
      - "__consumer_offsets"  # Internal Kafka topic
      - "_schemas"            # Schema Registry topic
      - "*-internal"          # Pattern exclusion

Storage Configuration

backend

Required. Storage backend type.

storage:
  backend: filesystem  # Local filesystem or mounted volume
  backend: s3          # Amazon S3 or S3-compatible storage
  backend: azure       # Azure Blob Storage
  backend: gcs         # Google Cloud Storage

Filesystem Storage

storage:
  backend: filesystem
  path: "/var/lib/kafka-backup/data"
  prefix: "cluster-prod"  # Optional subdirectory

S3 Storage

storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-west-2
  prefix: backups/production  # Optional key prefix

  # Optional: Custom endpoint for MinIO, Ceph, etc.
  endpoint: https://minio.example.com:9000

  # Credentials (optional - uses AWS credential chain if not specified)
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}

Azure Blob Storage

storage:
  backend: azure
  container: kafka-backups
  account_name: mystorageaccount
  prefix: backups/production

  # Credentials
  account_key: ${AZURE_STORAGE_KEY}
  # Or use connection string:
  # connection_string: ${AZURE_STORAGE_CONNECTION_STRING}

Google Cloud Storage

storage:
  backend: gcs
  bucket: my-kafka-backups
  prefix: backups/production

  # Credentials (uses GOOGLE_APPLICATION_CREDENTIALS if not specified)
  service_account_json: /etc/gcp/service-account.json

Backup Configuration

Options specific to backup mode.

backup:
  # Compression settings
  compression: zstd           # Options: zstd, lz4, none
  compression_level: 3        # 1-22 for zstd (default: 3)

  # Starting offset
  start_offset: earliest      # earliest, latest, or specific offset

  # Segment settings
  segment_max_bytes: 134217728     # 128 MB - roll segment after this size
  segment_max_interval_ms: 60000   # 60 sec - roll segment after this time

  # Continuous backup mode
  continuous: false           # true for streaming backup

  # Internal topics
  include_internal_topics: false  # Include __consumer_offsets, etc.

  # Checkpointing
  checkpoint_interval_secs: 30    # Save progress every 30 seconds
  sync_interval_secs: 60          # Sync to storage every 60 seconds

  # Offset headers (required for consumer offset reset)
  include_offset_headers: true

  # Source cluster identifier
  source_cluster_id: "prod-cluster-east"

  # Snapshot backup mode (v0.8.0+)
  # Captures high watermarks at start and exits when all partitions reach them.
  # Incompatible with continuous: true.
  stop_at_current_offsets: false

  # Performance tuning (v0.8.0+)
  max_concurrent_partitions: 8    # Parallel partition processing (default: 8)
  poll_interval_ms: 100           # Delay between poll attempts in ms (default: 100)

  # Consumer group snapshot (v0.12.0+)
  # After each backup cycle, queries every broker for consumer groups (KRaft-safe),
  # fetches their committed offsets, filters to groups with offsets on backed-up
  # topics, and writes {backup_id}/consumer-groups-snapshot.json to storage.
  # Use kafka-backup snapshot-groups --config backup.yaml to run on demand instead.
  consumer_group_snapshot: false

Backup Options Reference

Option	Type	Default	Description
`compression`	string	`zstd`	Compression algorithm
`compression_level`	int	`3`	Compression level (1-22 for zstd)
`start_offset`	string	`earliest`	Starting offset: `earliest`, `latest`
`segment_max_bytes`	int	`134217728`	Max segment size in bytes
`segment_max_interval_ms`	int	`60000`	Max segment duration in ms
`continuous`	bool	`false`	Enable continuous/streaming backup
`include_internal_topics`	bool	`false`	Include internal Kafka topics
`checkpoint_interval_secs`	int	`30`	Checkpoint frequency
`sync_interval_secs`	int	`60`	Storage sync frequency
`include_offset_headers`	bool	`true`	Store original offsets
`source_cluster_id`	string	-	Cluster identifier for tracking
`stop_at_current_offsets`	bool	`false`	Snapshot mode: stop after reaching current high watermark (v0.8.0+)
`max_concurrent_partitions`	int	`8`	Maximum parallel partition processing (v0.8.0+)
`poll_interval_ms`	int	`100`	Delay in ms between consumer poll attempts (v0.8.0+)
`consumer_group_snapshot`	bool	`false`	Write consumer group offsets to storage after each cycle — KRaft-safe, queries all brokers (v0.12.0+)

Offset Storage Configuration

Configuration for the local SQLite database used to track backup progress. When present, this enables resumable incremental backups in any mode — one-shot, snapshot, or continuous. If the process exits and restarts with the same backup_id, it picks up where it left off.

For continuous mode (continuous: true), the offset store is created automatically even without this section. For one-shot or snapshot mode, add this section explicitly to enable incremental behavior.

New in v0.13.5

Prior to v0.13.5, the offset store was only created in continuous mode. One-shot and snapshot backups always started from start_offset (default: earliest), producing a full backup every time. From v0.13.5, adding offset_storage to your config enables incremental behavior in all modes.

offset_storage:
  backend: sqlite              # sqlite (default) or memory
  db_path: /data/offsets.db    # Path to local SQLite database
  sync_interval_secs: 30       # How often to sync to remote storage

Offset Storage Options Reference

Option	Type	Default	Description
`backend`	string	`sqlite`	Storage backend: `sqlite` or `memory`
`db_path`	string	`$TMPDIR/{backup_id}-offsets.db`	Path to local SQLite database file
`s3_key`	string	-	Remote storage key for syncing
`sync_interval_secs`	int	`30`	How often to sync local DB to remote storage

Default path

By default, the offset database is created in the system temp directory ($TMPDIR, typically /tmp). This works out of the box on most systems, including Kubernetes pods with readOnlyRootFilesystem: true (as long as /tmp is mounted as an emptyDir).

For persistent local state across pod restarts, configure db_path to point to a mounted volume (e.g. /data/offsets.db). Note that offsets are also synced to remote storage, so local persistence is optional.

Restore Configuration

Options specific to restore mode.

restore:
  # Point-in-Time Recovery (PITR)
  time_window_start: 1701417600000  # Unix milliseconds (optional)
  time_window_end: 1701504000000    # Unix milliseconds (optional)

  # Auto-create topics if they don't exist (v0.3.0+)
  create_topics: true
  default_replication_factor: 3     # Replication factor for new topics

  # Partition filtering
  source_partitions:              # Only restore specific partitions
    - 0
    - 1
    - 2

  # Partition mapping (remap partitions during restore)
  partition_mapping:
    0: 0
    1: 2    # Source partition 1 -> target partition 2

  # Topic remapping
  topic_mapping:
    orders: orders_restored       # orders -> orders_restored
    payments: payments_dr         # payments -> payments_dr

  # Consumer offset strategy
  consumer_group_strategy: skip   # skip, header-based, timestamp-based, manual

  # Dry run mode
  dry_run: false                  # Validate without executing

  # Include original offset in headers
  include_original_offset_header: true

  # Rate limiting
  rate_limit_records_per_sec: null  # null for unlimited
  rate_limit_bytes_per_sec: null    # null for unlimited

  # Performance tuning
  max_concurrent_partitions: 4      # Parallel partition processing
  produce_batch_size: 1000          # Records per produce batch

  # Resumable restores
  checkpoint_state: null            # Path to checkpoint file
  checkpoint_interval_secs: 60      # Checkpoint frequency

  # Offset mapping report
  offset_report: /tmp/offset-mapping.json  # Save offset mapping

  # Consumer group offset reset
  reset_consumer_offsets: false     # Reset offsets after restore
  consumer_groups:                  # Groups to reset
    - my-consumer-group
    - analytics-consumer

  # Strimzi / pre-populated topic support (v0.12.0+)
  # Purge target topics via DeleteRecords before restoring.
  # Advances log-start-offset to the current end — makes the topic appear empty
  # without deleting it (safe for Strimzi-managed KafkaTopic resources).
  purge_topics: false

  # Automatically load consumer-groups-snapshot.json from storage (v0.12.0+)
  # Reads {backup_id}/consumer-groups-snapshot.json and uses those group IDs
  # to drive offset reset after restore. Sets reset_consumer_offsets: true
  # automatically. Silently ignored when the snapshot file is absent.
  auto_consumer_groups: false

  # Producer reliability tuning (v0.12.0+)
  # -1 = all in-sync replicas (default, highest durability)
  #  1 = leader only (faster, reduced durability — use when ISR is healthy)
  #  0 = fire-and-forget (fastest, no delivery guarantee)
  produce_acks: -1
  produce_timeout_ms: 30000         # Broker-side produce timeout in milliseconds

Restore Options Reference

Option	Type	Default	Description
`time_window_start`	int	-	PITR start timestamp (Unix ms)
`time_window_end`	int	-	PITR end timestamp (Unix ms)
`create_topics`	bool	`false`	Auto-create topics if they don't exist (v0.3.0+)
`default_replication_factor`	int	`1`	Replication factor for auto-created topics
`source_partitions`	list	-	Partitions to restore
`partition_mapping`	map	-	Partition remapping
`topic_mapping`	map	-	Topic remapping
`consumer_group_strategy`	string	`skip`	Offset handling strategy
`dry_run`	bool	`false`	Validate without executing
`include_original_offset_header`	bool	`true`	Add original offset header
`rate_limit_records_per_sec`	int	-	Rate limit (records/sec)
`rate_limit_bytes_per_sec`	int	-	Rate limit (bytes/sec)
`max_concurrent_partitions`	int	`4`	Parallel partitions
`produce_batch_size`	int	`1000`	Batch size
`reset_consumer_offsets`	bool	`false`	Reset consumer offsets
`consumer_groups`	list	-	Consumer groups to reset
`purge_topics`	bool	`false`	Delete all records from target topic partitions before restoring via `DeleteRecords` API — safe for Strimzi (v0.12.0+)
`auto_consumer_groups`	bool	`false`	Load consumer group IDs from `consumer-groups-snapshot.json` and reset their offsets after restore (v0.12.0+)
`produce_acks`	int	`-1`	Producer ack level: `-1` all ISR (default, safest), `1` leader only (faster), `0` fire-and-forget (v0.12.0+)
`produce_timeout_ms`	int	`30000`	Broker-side produce timeout in milliseconds (v0.12.0+)

Consumer Group Strategies

Strategy	Description
`skip`	Don't modify consumer offsets
`header-based`	Use offset mapping from backup headers
`timestamp-based`	Reset to timestamp-based offsets
`cluster-scan`	Scan target cluster for offset mapping
`manual`	Generate script for manual reset

Complete Examples

Basic Backup

mode: backup
backup_id: "daily-backup"

source:
  bootstrap_servers:
    - kafka:9092
  topics:
    include:
      - orders
      - payments

storage:
  backend: filesystem
  path: "/data/backups"

backup:
  compression: zstd

Production S3 Backup

mode: backup
backup_id: "prod-backup-${BACKUP_DATE}"

source:
  bootstrap_servers:
    - broker-1.prod.kafka:9092
    - broker-2.prod.kafka:9092
    - broker-3.prod.kafka:9092
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA256
    sasl_username: backup-service
    sasl_password: ${KAFKA_PASSWORD}
    ssl_ca_location: /etc/kafka/ca.crt
  # Connection settings (recommended for cloud Kafka services)
  connection:
    tcp_keepalive: true
    keepalive_time_secs: 60
    keepalive_interval_secs: 20
    tcp_nodelay: true
  topics:
    include:
      - "*"
    exclude:
      - "__consumer_offsets"
      - "_schemas"
      - "*-internal"

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-west-2
  prefix: production/${CLUSTER_NAME}

backup:
  compression: zstd
  compression_level: 5
  checkpoint_interval_secs: 60
  include_offset_headers: true
  source_cluster_id: "prod-us-west-2"

Point-in-Time Restore

mode: restore
backup_id: "prod-backup-20241201"

target:
  bootstrap_servers:
    - dr-broker-1:9092
    - dr-broker-2:9092

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-west-2
  prefix: production/prod-cluster

restore:
  # Restore only data from Dec 1, 2024 10:00 to 14:00 UTC
  time_window_start: 1701424800000
  time_window_end: 1701439200000

  topic_mapping:
    orders: orders_restored

  consumer_group_strategy: header-based
  reset_consumer_offsets: true
  consumer_groups:
    - order-processor
    - analytics-service

Disaster Recovery Restore

mode: restore
backup_id: "prod-backup-latest"

target:
  bootstrap_servers:
    - dr-broker-1.dr.kafka:9092
    - dr-broker-2.dr.kafka:9092
    - dr-broker-3.dr.kafka:9092
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA256
    sasl_username: restore-service
    sasl_password: ${DR_KAFKA_PASSWORD}
    ssl_ca_location: /etc/kafka/ca.crt

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-east-1  # DR region
  prefix: production/prod-cluster

restore:
  dry_run: false
  max_concurrent_partitions: 8
  produce_batch_size: 5000

  consumer_group_strategy: header-based
  reset_consumer_offsets: true
  consumer_groups:
    - order-service
    - payment-service
    - notification-service
    - analytics-pipeline

Environment Variable Substitution

Configuration files support ${VAR_NAME} substitution. All ${...} patterns are expanded from the process environment before YAML parsing. This works in any config value — passwords, paths, bucket names, backup IDs, etc.

source:
  security:
    sasl_password: ${KAFKA_PASSWORD}

storage:
  backend: s3
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}

Set variables before running:

export KAFKA_PASSWORD="secret123"
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."

kafka-backup backup --config backup.yaml

Behavior notes:

Unset variables are replaced with an empty string and produce a warning in the log
A bare $ without { is left unchanged (e.g., $5 stays as $5)
Expansion happens at all config loading sites: backup, restore, three-phase restore, validate-restore, and status watch

Configuration Structure​

Common Configuration​

mode​

backup_id​

Kafka Cluster Configuration​

bootstrap_servers​

security​

Security Protocol Options​

SASL Mechanism Options​

SASL/GSSAPI (Kerberos) Fields​

connection​

Connection Options Reference​

topics​

Storage Configuration​

backend​

Filesystem Storage​

S3 Storage​

Azure Blob Storage​

Google Cloud Storage​

Backup Configuration​

Backup Options Reference​

Offset Storage Configuration​

Offset Storage Options Reference​

Restore Configuration​

Restore Options Reference​

Consumer Group Strategies​

Complete Examples​

Basic Backup​

Production S3 Backup​

Point-in-Time Restore​

Disaster Recovery Restore​

Environment Variable Substitution​