How to Backup Kafka Topics to S3: A Production Runbook

To backup Kafka to S3, prepare an independent bucket, grant scoped access, select the required topics, and set storage.backend: s3. Then run kafka-backup backup --config s3-backup.yaml and validate the stored result.

This workflow keeps topic data and recovery metadata outside the Kafka cluster. It creates a recovery path that broker replication alone cannot provide.

Key takeaway

A successful upload is not recovery proof. Run integrity validation after each backup, then rehearse a restore into an isolated topic or cluster.

Choose the right Kafka-to-S3 pattern

Several tools can move Kafka records into S3. They do not produce the same recovery outcome.

OSO Kafka Backup captures records, topic configuration, original offsets, and consumer group positions. It also supports millisecond-precision restore windows. These features make it suitable for operational recovery.

Kafka Connect S3 Sink is strong at exporting records into analysis-friendly objects. It fits teams that already run Connect and need an archive or data-lake feed. However, it does not preserve consumer group positions as a backup tool.

Native Kafka replication keeps partitions available across brokers. A cross-cluster replicator can also keep another cluster current. Both approaches can copy a bad write or deletion before an operator responds.

Pattern	Primary outcome	Consumer positions	Historical restore point	Good fit
OSO Kafka Backup to S3	Recoverable backup	Built in	Yes	Topic recovery and disaster recovery
Kafka Connect S3 Sink	Record archive	No	Based on exported object layout	Analytics and long-term record access
Kafka replication	Live redundant copy	Tool dependent	No	Availability and warm standby
Console consumer export	One-off record dump	No	Manual	Small development tasks

The Kafka backup strategies guide compares these patterns across wider recovery requirements. This runbook focuses on a recoverable S3 backup.

Plan the S3 recovery boundary

Start with the incident that the backup must survive. Cluster loss requires a copy outside the brokers. Account compromise may require a separate AWS account and tighter deletion controls.

Choose a bucket region close to the backup process. This reduces transfer delay and avoids unnecessary cross-region traffic. Confirm any residency requirements before records leave the Kafka environment.

Use a dedicated key prefix for each source cluster and environment. A structure such as production/orders-cluster makes access policies and lifecycle rules easier to review.

Enable bucket versioning and default encryption through your normal AWS controls. Versioning can protect an earlier object version from an accidental overwrite. It does not replace a tested backup retention policy.

Define the following before the first run:

• The exact topics and consumer groups in scope
• The maximum acceptable time between recovery points
• The time allowed to retrieve and restore the data
• The owner of backup failures and restore drills
• The retention period for each backup class

S3 lifecycle rules can move older objects into archive storage. Those tiers may require retrieval before the backup can be read. Include that delay in recovery tests and runbooks.

Prepare S3 access without static credentials

Prefer short-lived AWS credentials. Use an EC2 instance profile for an EC2 host, or IAM Roles for Service Accounts for an EKS workload.

OSO Kafka Backup uses the AWS credential chain when keys are absent from the YAML file. This keeps long-lived credentials out of source control and deployed configuration.

The backup identity needs access to the intended bucket and objects. The repository's AWS setup policy includes these actions:

Scope	Required actions	Purpose
Bucket	s3:ListBucket, s3:GetBucketLocation	Find and inventory stored backups
Objects	s3:GetObject, s3:PutObject, s3:DeleteObject	Write, validate, read, and manage backup objects

Restrict the policy resources to the backup bucket and intended prefix. Do not grant broad S3 access when the process serves one backup location.

Confirm the active identity and bucket access before testing Kafka:

aws sts get-caller-identity
aws s3 ls s3://company-kafka-backups/production/

The AWS S3 setup guide includes bucket, policy, instance-role, and IRSA examples. Apply the controls required by your organization around those examples.

Configure the Kafka backup to S3

Create s3-backup.yaml. Begin with explicit topic names so the first run has a predictable scope.

s3-backup.yaml

mode: backup
backup_id: "production-orders-daily"

source:
  bootstrap_servers:
    - kafka-0.kafka.svc:9092
    - kafka-1.kafka.svc:9092
    - kafka-2.kafka.svc:9092
  topics:
    include:
      - orders
      - payments
    exclude:
      - "*-internal"

storage:
  backend: s3
  bucket: company-kafka-backups
  region: us-west-2
  prefix: production/orders-cluster

backup:
  compression: zstd
  compression_level: 3
  start_offset: earliest
  checkpoint_interval_secs: 30
  include_offset_headers: true
  consumer_group_snapshot: true

offset_storage:
  backend: sqlite
  db_path: /data/production-orders-offsets.db
  sync_interval_secs: 30

The topic list belongs under source.topics. Include patterns are supported, but review every match before using a broad wildcard in production. Exclusions are applied after inclusions.

start_offset: earliest captures retained records from the beginning of each selected partition. Kafka records that already expired cannot be recovered by a new backup.

Zstandard compression reduces the bytes written to S3. Original offset headers support later position mapping. The consumer group snapshot records positions for groups that consume the backed-up topics.

The offset_storage section enables resumable incremental behavior. Reusing the same backup ID and state lets an interrupted run continue from its tracked positions.

Mount db_path on persistent storage when local state must survive a pod replacement. The state is also synced to remote storage at the configured interval.

For separate full snapshots, use unique backup IDs and manage each snapshot as its own retention unit. For a continuous backup, set continuous: true and operate it as a long-running workload.

Review every supported field in the YAML configuration reference. Add Kafka TLS or SASL settings there when the source cluster requires them.

Run and inspect the first S3 backup

Run the backup with the reviewed configuration:

kafka-backup backup --config s3-backup.yaml

Use verbose output only when operators need more diagnostic detail:

kafka-backup -v backup --config s3-backup.yaml

A zero exit status is necessary, but it is not enough. Confirm that the backup inventory contains the expected ID:

kafka-backup list \
  --path s3://company-kafka-backups/production/orders-cluster

kafka-backup describe \
  --path s3://company-kafka-backups/production/orders-cluster \
  --backup-id production-orders-daily \
  --format json

Check topic names, partition counts, record totals, time bounds, and source cluster identity in the description. Investigate any unexpected omission before scheduling another run.

You can also inspect the object inventory:

aws s3 ls \
  s3://company-kafka-backups/production/orders-cluster/ \
  --recursive

Objects prove that bytes reached S3. They do not prove that every segment is readable or that applications can resume after a restore.

Validate backup integrity and restoreability

Run quick validation after each completed backup. It checks stored metadata and the expected segment inventory.

kafka-backup validate \
  --path s3://company-kafka-backups/production/orders-cluster \
  --backup-id production-orders-daily

Run deep validation on a schedule that fits the data size and recovery policy. It reads and verifies all segments, so it provides stronger integrity evidence.

kafka-backup validate \
  --path s3://company-kafka-backups/production/orders-cluster \
  --backup-id production-orders-daily \
  --deep

Integrity validation still does not test Kafka permissions, target capacity, or application behavior. Only a restore drill exercises those dependencies.

Restore into a scratch cluster or mapped topic. Then verify:

1. Every expected topic and partition exists.
2. Record counts and time bounds match the chosen recovery point.
3. Keys, headers, timestamps, and schemas remain usable.
4. Selected consumer groups resume at tested positions.
5. Applications process restored records without hidden dependencies.

Keep the source application isolated during the drill. Never let a test target publish into production systems by accident.

The backup and restore tutorial covers the restore configuration. The CLI basics guide documents list, describe, validate, and restore commands.

Schedule backups and manage retention

Run the configuration through a production scheduler. A Kubernetes CronJob, the Kafka Backup operator, or a controlled system scheduler can own discrete runs.

Treat scheduling and retention as one design. A daily job with seven-day retention cannot meet a requirement for monthly recovery points.

For discrete full backups, generate a unique backup ID before each run. Keep the generated value visible in logs, alerts, and restore documentation.

For incremental or continuous work, preserve the offset database and reuse the intended backup ID. Do not silently change either value during a redeployment.

Monitor these conditions:

Signal	Failure it reveals	Response
Last successful completion	Missing or late backup	Page the run owner after the accepted interval
Validation result	Missing or damaged segments	Preserve evidence and create a new verified recovery point
S3 access errors	Identity, policy, network, or bucket change	Check the active identity and scoped policy
Backup bytes and records	Unexpected topic or data change	Compare the topic inventory and source traffic
Restore drill age	Recovery path has gone untested	Run the isolated drill and record its outcome

Apply lifecycle transitions only after testing data retrieval from each target storage class. A cheaper retained object is not useful if retrieval misses the required recovery time.

Troubleshoot common Kafka S3 backup failures

Start with identity, topic selection, and stored state. These checks are more useful than changing several performance settings at once.

Symptom	Likely cause	Corrective check
AccessDenied before upload	Wrong identity or bucket policy	Run aws sts get-caller-identity and test the exact prefix
Backup completes with no expected topic	Include pattern or Kafka ACL mismatch	List matched topics and verify source describe/read access
Inventory exists but validation fails	Missing, unreadable, or damaged segment	Preserve the failed ID, run deep validation, and create a new backup
Restart reads old records again	Offset state or backup ID changed	Restore the intended SQLite state and stable ID
Upload throughput is low	Distant region, network limits, or small segments	Check placement, network path, and measured segment behavior
Restore waits on S3 retrieval	Lifecycle moved objects into an archive tier	Start retrieval and revise the tested recovery-time plan

Avoid deleting the failed backup until the cause is understood. Its manifest, logs, and object inventory can explain where the run stopped.

Keep replication and S3 backup separate

Replication and backup should have separate objectives. Replication provides a current copy for availability. S3 backup provides retained recovery points.

A replicated cluster may reduce failover time after a regional outage. It may also receive the same poisoned event, tombstone, or administrative deletion.

An independent S3 backup can preserve the state from before that incident. Its restore takes longer than switching to a healthy warm cluster.

Many production systems need both controls. Monitor replication lag for the standby path. Monitor backup completion, integrity, and restore drills for the historical path.

The Kafka Backup for Amazon S3 page summarizes supported S3 and S3-compatible storage. The backup best practices guide places verification inside a broader operating model.

Finish with a tested recovery point

Backing up Kafka topics to S3 requires more than a bucket and scheduled command. The recovery point needs scoped access, explicit topic selection, verified segments, and a rehearsed restore.

Start with two noncritical topics. Run quick and deep validation, restore them into an isolated target, and record the results. Expand the scope only after that path works.

Frequently asked questions

How to backup Kafka to S3?

Create an independent S3 bucket, grant scoped access to an AWS identity, select topics under source.topics, set storage.backend to s3, and run kafka-backup backup --config s3-backup.yaml. Validate the resulting backup and rehearse a restore before relying on it.

How to backup Kafka?

Copy Kafka records and recovery metadata into storage outside the cluster. A recoverable backup should preserve topic configuration, original offsets, and consumer group positions, then pass integrity checks and an isolated restore drill.

How to take backup of Kafka topic?

List the topic under source.topics.include in a backup YAML file, choose an independent storage backend, and run the backup command. Confirm the topic and all partitions appear in the backup description, then validate the stored segments.

How to backup Kafka to a file system?

Use the same topic selection and backup command, but set storage.backend to filesystem and provide storage.path. A mounted file system must still sit outside the Kafka broker failure boundary and support the required retention and restore tests.