Kafka backup on AWS, built from parts you already run

The Kafka backup on AWS architecture​

Kafka backup on AWS is three decisions, not thirty: where the backup process runs, how it authenticates to the cluster, and how the S3 bucket is laid out. The pattern below works for Amazon MSK and for self-managed Kafka on EC2 alike.

Run the backup on EC2 or EKS inside the same VPC as the brokers. Reads stay on the private network, and the process picks up credentials from the instance profile or IRSA service account — no static AWS keys to rotate or leak. Backups land in an S3 bucket in the same region, outside the cluster's failure domain.

mode: backup
backup_id: "msk-prod-aws"

source:
bootstrap_servers:
  - b-1.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
  - b-2.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
security:
  security_protocol: SASL_SSL
  sasl_mechanism: SCRAM-SHA512
  sasl_username: ${MSK_SCRAM_USER}
  sasl_password: ${MSK_SCRAM_PASSWORD}
# Recommended for cloud Kafka services
connection:
  tcp_keepalive: true
  keepalive_time_secs: 60
  keepalive_interval_secs: 20
topics:
  include:
    - "*"
  exclude:
    - "__consumer_offsets"

storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: msk/production
# No access_key/secret_key — instance profile or IRSA supplies credentials

backup:
compression: zstd
include_offset_headers: true
consumer_group_snapshot: true

{
"Version": "2012-10-17",
"Statement": [
  {
    "Sid": "ListBucket",
    "Effect": "Allow",
    "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
    "Resource": "arn:aws:s3:::my-kafka-backups"
  },
  {
    "Sid": "ObjectOperations",
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
    "Resource": "arn:aws:s3:::my-kafka-backups/*"
  }
]
}

kafka-backup backup --config backup.yaml

kafka-backup list --path s3://my-kafka-backups/msk/production

Why replication inside AWS is not the backup​

MSK replicates every partition across brokers, and MSK Replicator can mirror whole clusters between regions. Both copy faithfully — including the topic a bad deploy just corrupted or the one someone deleted. Point-in-time backup is the layer that remembers the state from before the incident; the MSK Replicator comparison walks the distinction.

The backup captures records with timestamps and headers, consumer group offsets, and topic configuration, compressed with Zstd. Restores target the original cluster or a new one — another region or another account — and stop at a precise millisecond. For durability past the first month, S3 lifecycle rules tier old backups to STANDARD_IA and GLACIER, and cross-region replication keeps a second-region copy without any extra backup infrastructure. The AWS S3 setup guide has the exact bucket, lifecycle, and replication commands.