Configuration Reference

OSO Kafka Backup uses YAML configuration files for backup and restore operations. This reference documents all available options.

Configuration Structure

# Required: Operation mode
mode: backup  # or "restore"

# Required: Unique backup identifier
backup_id: "my-backup-001"

# Source/Target Kafka cluster configuration
source:  # For backup mode
target:  # For restore mode
  bootstrap_servers: []
  security: {}
  topics: {}

# Storage backend configuration
storage:
  backend: filesystem  # or s3, azure, gcs
  # Backend-specific options...

# Mode-specific options
backup: {}   # Backup options
restore: {}  # Restore options

# Optional Prometheus endpoint
metrics: {}

Common Configuration

mode

Required. Operation mode.

mode: backup   # Run a backup operation
mode: restore  # Run a restore operation

backup_id

Required. Unique identifier for the backup.

backup_id: "production-daily-001"
backup_id: "dr-backup-$(date +%Y%m%d)"

Kafka Cluster Configuration

Used as source in backup mode and target in restore mode.

bootstrap_servers

Required. List of Kafka broker addresses.

source:
  bootstrap_servers:
    - broker-1.kafka.svc:9092
    - broker-2.kafka.svc:9092
    - broker-3.kafka.svc:9092

security

Optional security configuration for Kafka connection.

source:
  security:
    # Security protocol
    security_protocol: SASL_SSL  # PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL

    # SASL configuration
    sasl_mechanism: SCRAM-SHA256  # PLAIN, SCRAM-SHA256, SCRAM-SHA512
    sasl_username: backup-user
    sasl_password: ${KAFKA_PASSWORD}  # Environment variable substitution

    # SSL/TLS configuration
    ssl_ca_location: /etc/kafka/ca.crt
    ssl_certificate_location: /etc/kafka/client.crt
    ssl_key_location: /etc/kafka/client.key
    ssl_key_password: ${SSL_KEY_PASSWORD}

Security Protocol Options

Protocol	Description
`PLAINTEXT`	No encryption, no authentication
`SSL`	TLS encryption, optional mTLS
`SASL_PLAINTEXT`	SASL authentication, no encryption
`SASL_SSL`	SASL authentication with TLS encryption

SASL Mechanism Options

Mechanism	Description	Feature gate
`PLAIN`	Simple username/password	Built-in
`SCRAM-SHA256`	SCRAM-SHA-256 — Salted Challenge Response (SHA-256)	Built-in
`SCRAM-SHA512`	SCRAM-SHA-512 — Salted Challenge Response (SHA-512)	Built-in
`GSSAPI`	Kerberos authentication (RFC 4752). Uses a keytab for non-interactive credential acquisition.	Requires CLI built with `--features gssapi`

SASL/GSSAPI opt-in

GSSAPI is gated behind the gssapi cargo feature. The default oso/kafka-backup release binary and Docker image do not include it. A binary without the feature that loads a sasl_mechanism: GSSAPI config fails fast at startup with a clear error. See the Security Setup guide for build instructions.

SASL/GSSAPI (Kerberos) Fields

Required only when sasl_mechanism: GSSAPI. Ignored for other mechanisms.

source:
  security:
    security_protocol: SASL_PLAINTEXT   # or SASL_SSL
    sasl_mechanism: GSSAPI
    sasl_kerberos_service_name: kafka
    sasl_keytab_path: /etc/kafka-backup/client.keytab
    sasl_krb5_config_path: /etc/krb5.conf

Field	Type	Required	Description
`sasl_kerberos_service_name`	string	Yes	The service-name portion of the broker's service principal. For `kafka/broker.host@REALM`, this is `kafka`.
`sasl_keytab_path`	path	Yes	Absolute path to the keytab holding the client principal's long-term key. Must be readable by the kafka-backup process; lock to `0400` in production.
`sasl_krb5_config_path`	path	No	Override the default `/etc/krb5.conf`. Useful inside containers that don't share the host's krb5 config.

Session re-authentication (KIP-368) is handled automatically — if the broker advertises a session lifetime, kafka-backup re-handshakes at ~80 % of the window (30 s floor, ±5 s jitter) before the broker forcibly disconnects.

connection

Optional TCP connection settings. These are particularly important for cloud-hosted Kafka services like Confluent Cloud that terminate idle connections.

source:
  connection:
    # Enable TCP keepalive to prevent idle connection termination
    tcp_keepalive: true           # Default: true

    # Time in seconds before first keepalive probe is sent
    keepalive_time_secs: 60       # Default: 60

    # Interval in seconds between keepalive probes
    keepalive_interval_secs: 20   # Default: 20

    # Enable TCP_NODELAY (disable Nagle's algorithm) for lower latency
    tcp_nodelay: true             # Default: true

    fetch_max_wait_ms: 500         # 0–30000 ms; v0.28.0+

Connection Options Reference

Option	Type	Default	Description
`tcp_keepalive`	bool	`true`	Enable TCP keepalive probes
`keepalive_time_secs`	int	`60`	Seconds idle before first probe
`keepalive_interval_secs`	int	`20`	Seconds between probes
`tcp_nodelay`	bool	`true`	Disable Nagle's algorithm
`fetch_max_wait_ms`	int	`500`	Broker Fetch wait in ms, 0–30000 (v0.28.0+). Does not extend client timeouts or the no-progress budget

Confluent Cloud

Confluent Cloud terminates idle TCP connections after ~5 minutes. The default keepalive settings (60s time, 20s interval) prevent this. If you're experiencing "Broken pipe" errors with Confluent Cloud, ensure TCP keepalive is enabled.

topics

Topic selection for backup or restore.

source:
  topics:
    # Include specific topics or patterns
    include:
      - orders              # Exact topic name
      - payments            # Another exact name
      - "events-*"          # Wildcard pattern
      - "logs-2024-*"       # Date-based pattern

    # Exclude topics (applied after include)
    exclude:
      - "__consumer_offsets"  # Internal Kafka topic
      - "_schemas"            # Schema Registry topic
      - "*-internal"          # Pattern exclusion

Storage Configuration

backend

Required. Storage backend type.

storage:
  backend: filesystem  # Local filesystem or mounted volume
  backend: s3          # Amazon S3 or S3-compatible storage
  backend: azure       # Azure Blob Storage
  backend: gcs         # Google Cloud Storage

Filesystem Storage

storage:
  backend: filesystem
  path: "/var/lib/kafka-backup/data"
  prefix: "cluster-prod"  # Optional subdirectory

S3 Storage

storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-west-2
  prefix: backups/production  # Optional key prefix

  # Optional: Custom endpoint for MinIO, Ceph RGW, etc. A custom endpoint
  # implies path-style requests; an http:// endpoint implies allow_http (v0.22.0+)
  endpoint: https://minio.example.com:9000
  # path_style: true   # force path-style without a custom endpoint
  # allow_http: true   # plain-HTTP endpoints (in-cluster stores)

  # Credentials (optional - uses AWS credential chain if not specified).
  # access_key_id / secret_access_key are accepted as aliases (v0.22.0+).
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}

Azure Blob Storage

storage:
  backend: azure
  account_name: mystorageaccount
  container_name: kafka-backups
  prefix: backups/production
  # endpoint: https://mystorageaccount.blob.core.usgovcloudapi.net   # sovereign clouds

  # Credentials — precedence: sas_token → account_key → service principal
  # (client_secret) → Workload Identity → DefaultAzureCredential chain
  account_key: ${AZURE_STORAGE_KEY}
  # sas_token: ${AZURE_SAS_TOKEN}
  # client_id: ${AZURE_CLIENT_ID}
  # tenant_id: ${AZURE_TENANT_ID}
  # client_secret: ${AZURE_CLIENT_SECRET}
  # use_workload_identity: true   # AKS; auto-enabled when AZURE_FEDERATED_TOKEN_FILE is set

Google Cloud Storage

storage:
  backend: gcs
  bucket: my-kafka-backups
  prefix: backups/production

  # Credentials (uses GOOGLE_APPLICATION_CREDENTIALS if not specified)
  service_account_path: /etc/gcp/service-account.json

Backup Configuration

Options specific to backup mode.

backup:
  # Compression settings
  compression: zstd           # Options: zstd, lz4, none
  compression_level: 3        # 1-22 for zstd (default: 3)

  # Starting offset
  start_offset: earliest      # earliest, latest, or specific offset

  # Segment settings
  segment_max_bytes: 134217728     # 128 MB - roll segment after this size
  segment_max_interval_ms: 60000   # 60 sec - roll segment after this time
  segment_max_records: 2000000     # Roll segment after this many records (v0.16.0+, default: unlimited)

  # Fetch sizing (v0.16.0+)
  # Maximum bytes requested per Kafka Fetch request. When unset, fetches
  # request min(segment_max_bytes, 16 MB).
  fetch_max_bytes: 16777216
  empty_fetch_retry_secs: 30      # 0–3600 seconds; 0 fails immediately (v0.28.0+)

  # Continuous backup mode
  continuous: false           # true for streaming backup

  # Internal topics
  include_internal_topics: false  # Include __consumer_offsets, etc.

  # Checkpointing
  on_missing_topic: fail          # or warn — skip absent literal topics and record them (v0.22.0+)
  sync_interval_secs: 30          # Sync offsets.db to storage every 30 seconds (default)

  # Offset-tracking headers (default: true). Adds x-original-offset and
  # x-original-timestamp to every archived record; needed for header-based
  # consumer offset recovery. Set false for a verbatim archive.
  include_offset_headers: true

  # Source cluster identifier
  source_cluster_id: "prod-cluster-east"

  # Snapshot backup mode (v0.8.0+)
  # Captures high watermarks at start and exits when all partitions reach them.
  # Incompatible with continuous: true.
  stop_at_current_offsets: false

  # Performance tuning (v0.8.0+)
  max_concurrent_partitions: 8    # Parallel partition processing (default: 8)
  poll_interval_ms: 100           # Delay between poll attempts in ms (default: 100)

  # Rate limits for the whole backup, shared by all partitions (v0.27.0+).
  # Unset = unlimited; before 0.27.0 these keys were ignored with a warning.
  rate_limit_records_per_sec: null
  rate_limit_bytes_per_sec: null

  # Consumer group snapshot (v0.12.0+)
  # After each backup cycle, queries every broker for consumer groups (KRaft-safe),
  # fetches their committed offsets, filters to groups with offsets on backed-up
  # topics, and writes {backup_id}/consumer-groups-snapshot.json to storage.
  # Use kafka-backup snapshot-groups --config backup.yaml to run on demand instead.
  consumer_group_snapshot: false

Backup Options Reference

Option	Type	Default	Description
`compression`	string	`zstd`	Compression algorithm
`compression_level`	int	`3`	Compression level (1-22 for zstd)
`start_offset`	string	`earliest`	Starting offset: `earliest`, `latest`
`segment_max_bytes`	int	`134217728`	Max segment size in bytes
`segment_max_interval_ms`	int	`60000`	Max segment duration in ms
`segment_max_records`	int	unlimited	Roll segment after this many records (v0.16.0+)
`empty_fetch_retry_secs`	int	`30`	Per-partition no-progress budget, 0–3600 seconds (v0.28.0+). Starts at the first empty/no-progress Fetch below the captured end, includes backoff and subsequent requests, and resets when offsets advance. Zero fails immediately
`fetch_max_bytes`	int	`min(segment_max_bytes, 16777216)`	Max bytes requested per Kafka Fetch request (v0.16.0+)
`continuous`	bool	`false`	Enable continuous/streaming backup
`checkpoint_interval_secs`	int	`5`	Deprecated (v0.22.0) — no effect. Offsets are checkpointed at the end of every cycle; a warning is logged if set
`sync_interval_secs`	int	`30`	How often the manifest and offset store are synced to storage (`offset_storage.sync_interval_secs` overrides it for the offset store, v0.22.0+)
`include_internal_topics`	bool	`false`	Include internal Kafka topics
`internal_topics`	list	`[]`	Internal topics to include when `include_internal_topics` is set (e.g. `__consumer_offsets`)
`on_missing_topic`	string	`fail`	Literal `topics.include` entry absent from the cluster: `fail` errors; `warn` logs, records the names in the manifest's `missing_topics`, exposes `kafka_backup_missing_topics`, and continues (v0.22.0+). Globs matching nothing are always skipped
`include_offset_headers`	bool	`true`	Add `x-original-offset` / `x-original-timestamp` headers to every archived record — see Offset-tracking headers
`source_cluster_id`	string	-	Recorded in the `x-source-cluster` header (only with `include_offset_headers`)
`stop_at_current_offsets`	bool	`false`	Snapshot mode: stop after reaching current high watermark (v0.8.0+)
`max_concurrent_partitions`	int	`8`	Maximum parallel partition processing (v0.8.0+)
`rate_limit_records_per_sec`	int	-	Rate limit for the whole backup (records/sec), shared by every partition; must be > 0. An average: the next fetch waits until the previous one fits the rate (v0.27.0+)
`rate_limit_bytes_per_sec`	int	-	Rate limit for the whole backup (bytes/sec, each record's encoded size), shared by every partition; must be > 0 (v0.27.0+)
`poll_interval_ms`	int	`100`	Delay in ms between consumer poll attempts (v0.8.0+)
`consumer_group_snapshot`	bool	`false`	Write consumer group offsets to storage after each cycle — KRaft-safe, queries all brokers (v0.12.0+)

Empty-fetch retries

An empty Fetch below the captured backup end does not prove that the range is complete. Tiered-storage reads can return empty while remote data is unavailable. From 0.28.0, the engine retries for empty_fetch_retry_secs per no-progress streak and fails explicitly on expiry, retaining buffered records and the durable prefix. It never checkpoints unread offsets, and retry responses stay inside the captured snapshot end. Empty responses that advance offsets, such as compacted or filtered transactional batches, count as progress.

The budget starts at the first successful empty response; it does not replace the normal timeout for the initial RPC. After that response it covers backoff, reconnects and subsequent requests. Setting it to 0 fails on the first stalled empty response; it does not restore the earlier silent-success behavior.

connection.fetch_max_wait_ms is separate. Older Kafka remote reads may use the Fetch wait; Kafka with KIP-1018 uses broker remote.fetch.max.wait.ms for remote reads. Raising the client wait does not guarantee a remote read will finish. Re-run affected backups after upgrading and validate record counts and manifests; upgrading alone does not repair an existing incomplete archive.

Offset-tracking headers

include_offset_headers is on by default. Every record is archived with two extra headers appended after the record's own headers:

Header	Value	Added when
`x-original-offset`	source offset, little-endian `i64`	always
`x-original-timestamp`	source timestamp (epoch ms), little-endian `i64`	always
`x-source-cluster`	`source_cluster_id`, UTF-8	`source_cluster_id` is set

They are what makes consumer_group_strategy: header-based recovery possible after a restore. The trade-off is that an archived — and therefore a restored — record is not header-for-header identical to its source. The backup logs which headers it is adding at startup (include_offset_headers=true (default): ..., v0.19.0+).

To get a verbatim copy, either set backup.include_offset_headers: false at backup time, or set restore.strip_offset_headers: true (v0.19.0+) at restore time — that works for archives that already carry the headers and does not affect offset mapping, because the source offset is stored natively in the segment.

Record fidelity

Keys, values, timestamps and headers are restored verbatim, including the difference between a null and an empty key, value or header value. Null header values were archived as empty by versions before v0.18.0 (#155); archives taken with those versions cannot be repaired and must be re-taken where the distinction matters. Duplicate header keys on one record are currently collapsed to the last one (#156).

Unknown keys are warned about (v0.16.0+)

Config keys that kafka-backup does not recognize are logged as warnings at startup, for example:

WARN Ignoring unknown config key `backup.fetch_max_bytez` — check for typos; see https://kafkabackup.com/reference/config-yaml

Older versions silently ignored unrecognized keys, so a typo — or an option newer than the running binary — simply did nothing. If an option does not seem to take effect, check the job logs for this warning and confirm your kafka-backup version supports the option.

Offset Storage Configuration

Configuration for the local SQLite database used to track backup progress. When present, this enables resumable incremental backups in any mode — one-shot, snapshot, or continuous. If the process exits and restarts with the same backup_id, it picks up where it left off.

For continuous mode (continuous: true), the offset store is created automatically even without this section. For one-shot or snapshot mode, add this section explicitly to enable incremental behavior.

New in v0.13.5

Prior to v0.13.5, the offset store was only created in continuous mode. One-shot and snapshot backups always started from start_offset (default: earliest), producing a full backup every time. From v0.13.5, adding offset_storage to your config enables incremental behavior in all modes.

offset_storage:
  backend: sqlite              # sqlite (default) or memory
  db_path: /data/offsets.db    # Path to local SQLite database
  sync_interval_secs: 30       # How often to sync to remote storage

Offset Storage Options Reference

Option	Type	Default	Description
`backend`	string	`sqlite`	Storage backend: `sqlite` or `memory`
`db_path`	string	`./offsets.db`	Path to local SQLite database file when `offset_storage` is present
`s3_key`	string	-	Remote storage key for syncing
`sync_interval_secs`	int	`30`	How often to sync local DB to remote storage

Default path

When offset_storage is omitted, continuous mode creates its database in the system temporary directory as {backup_id}-offsets.db. When the section is present and db_path is omitted, the field default is ./offsets.db.

For persistent local state across pod restarts, configure db_path to point to a mounted volume (e.g. /data/offsets.db). Note that offsets are also synced to remote storage, so local persistence is optional.

Metrics Configuration

metrics:
  enabled: true
  port: 8080
  bind_address: "0.0.0.0"
  path: /metrics
  update_interval_ms: 500
  keep_alive_seconds: 60
  max_partition_labels: 100

Option	Type	Default	Description
`enabled`	bool	`true`	Start the Prometheus HTTP server
`port`	int	`8080`	Listener port
`bind_address`	string	`0.0.0.0`	Listener address
`path`	string	`/metrics`	Metrics endpoint path
`update_interval_ms`	int	`500`	Metrics recalculation interval
`keep_alive_seconds`	int	`0`	Keep serving after a one-shot operation completes
`max_partition_labels`	int	`100`	Maximum unique per-partition label sets; `0` is unlimited

For a short-lived Kubernetes Job, set keep_alive_seconds to at least twice the Prometheus scrape interval. See the metrics reference for cardinality behavior and metric names.

Restore Configuration

Options specific to restore mode.

restore:
  # Point-in-Time Recovery (PITR)
  time_window_start: 1701417600000  # Unix milliseconds (optional)
  time_window_end: 1701504000000    # Unix milliseconds (optional)

  # Auto-create topics if they don't exist (v0.3.0+)
  create_topics: true
  default_replication_factor: 3     # Replication factor for new topics

  # Partition filtering
  source_partitions:              # Only restore specific partitions
    - 0
    - 1
    - 2

  # Partition mapping (remap partitions during restore)
  partition_mapping:
    0: 0
    1: 2    # Source partition 1 -> target partition 2

  # Topic remapping
  topic_mapping:
    orders: orders_restored       # orders -> orders_restored
    payments: payments_dr         # payments -> payments_dr

  # Consumer offset strategy
  consumer_group_strategy: skip   # skip, header-based, timestamp-based, cluster-scan, manual

  # Dry run mode
  dry_run: false                  # Validate without executing

  # Add x-original-offset / x-original-timestamp / x-source-partition to every
  # restored record (default: false; implied by consumer_group_strategy: header-based)
  include_original_offset_header: true

  # Remove the headers the backup added (x-original-*, x-source-*) before
  # producing, for a header-for-header identical restore (v0.19.0+, default: false)
  strip_offset_headers: false

  # Rate limiting
  rate_limit_records_per_sec: null  # null for unlimited
  rate_limit_bytes_per_sec: null    # null for unlimited

  # Performance tuning
  max_concurrent_partitions: 4      # Parallel partition processing
  produce_batch_size: 1000          # Records per produce batch

  # Resumable restores
  checkpoint_state: null            # Path to checkpoint file
  checkpoint_interval_secs: 60      # Checkpoint frequency

  # Offset mapping report
  offset_report: /tmp/offset-mapping.json  # Save offset mapping

  # Consumer group offset reset
  reset_consumer_offsets: false     # Reset offsets after restore
  consumer_groups:                  # Groups to reset
    - my-consumer-group
    - analytics-consumer

  # Strimzi / pre-populated topic support (v0.12.0+)
  # Purge target topics via DeleteRecords before restoring.
  # Advances log-start-offset to the current end — makes the topic appear empty
  # without deleting it (safe for Strimzi-managed KafkaTopic resources).
  purge_topics: false

  # Automatically load consumer-groups-snapshot.json from storage (v0.12.0+)
  # Reads {backup_id}/consumer-groups-snapshot.json and uses those group IDs
  # to drive offset reset after restore. Sets reset_consumer_offsets: true
  # automatically. Silently ignored when the snapshot file is absent.
  auto_consumer_groups: false

  # Producer reliability tuning (v0.12.0+)
  # -1 = all in-sync replicas (default, highest durability)
  #  1 = leader only (faster, reduced durability — use when ISR is healthy)
  #  0 = fire-and-forget (fastest, no delivery guarantee)
  produce_acks: -1
  produce_timeout_ms: 30000         # Broker-side produce timeout in milliseconds

Restore Options Reference

Option	Type	Default	Description
`time_window_start`	int	-	PITR start timestamp (Unix ms)
`time_window_end`	int	-	PITR end timestamp (Unix ms)
`create_topics`	bool	`false`	Auto-create topics if they don't exist (v0.3.0+)
`default_replication_factor`	int	`1`	Replication factor for auto-created topics
`source_partitions`	list	-	Partitions to restore
`partition_mapping`	map	-	Partition remapping
`topic_mapping`	map	-	Topic remapping
`consumer_group_strategy`	string	`skip`	Offset handling strategy
`dry_run`	bool	`false`	Validate without executing
`include_original_offset_header`	bool	`false`	Add `x-original-offset` / `x-original-timestamp` / `x-source-partition` headers to every restored record (also implied by `consumer_group_strategy: header-based`)
`strip_offset_headers`	bool	`false`	Remove the headers kafka-backup added at backup time (`x-original-`, `x-source-`) before producing, for a header-for-header identical restore — see Offset-tracking headers (v0.19.0+)
`checkpoint_state`	string	-	Path to the restore checkpoint file (resumable restores)
`checkpoint_interval_secs`	int	`60`	Restore checkpoint frequency
`offset_report`	string	-	Path to write the offset-mapping report
`repartitioning`	map	-	Per target topic: `{strategy: murmur2 \| automatic, target_partitions: N}`; mutually exclusive with `partition_mapping`
`rate_limit_records_per_sec`	int	-	Rate limit (records/sec)
`rate_limit_bytes_per_sec`	int	-	Rate limit (bytes/sec)
`max_concurrent_partitions`	int	`4`	Parallel partitions
`produce_batch_size`	int	`1000`	Batch size
`reset_consumer_offsets`	bool	`false`	Reset consumer offsets
`consumer_groups`	list	-	Consumer groups to reset
`purge_topics`	bool	`false`	Delete all records from target topic partitions before restoring via `DeleteRecords` API — safe for Strimzi (v0.12.0+)
`auto_consumer_groups`	bool	`false`	Load consumer group IDs from `consumer-groups-snapshot.json` and reset their offsets after restore (v0.12.0+)
`produce_acks`	int	`-1`	Producer ack level: `-1` all ISR (default, safest), `1` leader only (faster), `0` fire-and-forget (v0.12.0+)
`produce_timeout_ms`	int	`30000`	Broker-side produce timeout in milliseconds (v0.12.0+)

Consumer Group Strategies

Strategy	Description
`skip`	Don't modify consumer offsets
`header-based`	Use offset mapping from backup headers
`timestamp-based`	Reset to timestamp-based offsets
`cluster-scan`	Scan target cluster for offset mapping
`manual`	Generate script for manual reset

Complete Examples

Basic Backup

mode: backup
backup_id: "daily-backup"

source:
  bootstrap_servers:
    - kafka:9092
  topics:
    include:
      - orders
      - payments

storage:
  backend: filesystem
  path: "/data/backups"

backup:
  compression: zstd

Production S3 Backup

mode: backup
backup_id: "prod-backup-${BACKUP_DATE}"

source:
  bootstrap_servers:
    - broker-1.prod.kafka:9092
    - broker-2.prod.kafka:9092
    - broker-3.prod.kafka:9092
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA256
    sasl_username: backup-service
    sasl_password: ${KAFKA_PASSWORD}
    ssl_ca_location: /etc/kafka/ca.crt
  # Connection settings (recommended for cloud Kafka services)
  connection:
    tcp_keepalive: true
    keepalive_time_secs: 60
    keepalive_interval_secs: 20
    tcp_nodelay: true
  topics:
    include:
      - "*"
    exclude:
      - "__consumer_offsets"
      - "_schemas"
      - "*-internal"

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-west-2
  prefix: production/${CLUSTER_NAME}

backup:
  compression: zstd
  compression_level: 5
  checkpoint_interval_secs: 60
  include_offset_headers: true
  source_cluster_id: "prod-us-west-2"

metrics:
  enabled: true
  keep_alive_seconds: 60
  max_partition_labels: 100

Point-in-Time Restore

mode: restore
backup_id: "prod-backup-20241201"

target:
  bootstrap_servers:
    - dr-broker-1:9092
    - dr-broker-2:9092

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-west-2
  prefix: production/prod-cluster

restore:
  # Restore only data from Dec 1, 2024 10:00 to 14:00 UTC
  time_window_start: 1701424800000
  time_window_end: 1701439200000

  topic_mapping:
    orders: orders_restored

  consumer_group_strategy: header-based
  reset_consumer_offsets: true
  consumer_groups:
    - order-processor
    - analytics-service

Disaster Recovery Restore

mode: restore
backup_id: "prod-backup-latest"

target:
  bootstrap_servers:
    - dr-broker-1.dr.kafka:9092
    - dr-broker-2.dr.kafka:9092
    - dr-broker-3.dr.kafka:9092
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA256
    sasl_username: restore-service
    sasl_password: ${DR_KAFKA_PASSWORD}
    ssl_ca_location: /etc/kafka/ca.crt

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-east-1  # DR region
  prefix: production/prod-cluster

restore:
  dry_run: false
  max_concurrent_partitions: 8
  produce_batch_size: 5000

  consumer_group_strategy: header-based
  reset_consumer_offsets: true
  consumer_groups:
    - order-service
    - payment-service
    - notification-service
    - analytics-pipeline

Environment Variable Substitution

Configuration files support ${VAR_NAME} substitution. All ${...} patterns are expanded from the process environment before YAML parsing. This works in any config value — passwords, paths, bucket names, backup IDs, etc.

source:
  security:
    sasl_password: ${KAFKA_PASSWORD}

storage:
  backend: s3
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}

Set variables before running:

export KAFKA_PASSWORD="secret123"
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."

kafka-backup backup --config backup.yaml

Behavior notes:

Unset variables are replaced with an empty string and produce a warning in the log
A bare $ without { is left unchanged (e.g., $5 stays as $5)
Expansion happens at all config loading sites: backup, restore, three-phase restore, validate-restore, and status watch

Configuration Structure​

Common Configuration​

mode​

backup_id​

Kafka Cluster Configuration​

bootstrap_servers​

security​

Security Protocol Options​

SASL Mechanism Options​

SASL/GSSAPI (Kerberos) Fields​

connection​

Connection Options Reference​

topics​

Storage Configuration​

backend​

Filesystem Storage​

S3 Storage​

Azure Blob Storage​

Google Cloud Storage​

Backup Configuration​

Backup Options Reference​

Empty-fetch retries​

Offset-tracking headers​

Offset Storage Configuration​

Offset Storage Options Reference​

Metrics Configuration​

Restore Configuration​

Restore Options Reference​

Consumer Group Strategies​

Complete Examples​

Basic Backup​

Production S3 Backup​

Point-in-Time Restore​

Disaster Recovery Restore​

Environment Variable Substitution​