On-prem to MSK: S3 is the bridge, not a VPN

Why migrate to MSK through object storage​

Most plans to migrate to MSK start with MirrorMaker 2 and a site-to-site VPN or Direct Connect. That works, but it makes the network link the project's critical path. The link must stay up for the whole sync, carry your full produce rate, and clear every firewall review between your data center and AWS.

One tool is off the table entirely: AWS MSK Replicator only copies topics between MSK clusters, so it cannot pull from a self-managed source. For the on-prem leg you are choosing between MirrorMaker over a private link and a backup-based move through S3.

The S3 route needs no connection between the clusters. The on-prem cluster writes a compressed backup to a bucket. MSK restores from the same bucket, in the same region, so the heavy transfer happens inside AWS.

The move, end to end​

Take the backup with include_offset_headers: true — that single key is what lets consumer groups resume at the right position on MSK. Restore over SASL/SCRAM on port 9096 or mutual TLS on 9094, whichever your MSK cluster exposes. The bucket and IAM setup is covered in the AWS S3 setup guide.

mode: backup
backup_id: "onprem-to-msk-seed"

source:
bootstrap_servers:
  - kafka-1.dc.internal:9092
  - kafka-2.dc.internal:9092
topics:
  include:
    - "*"
  exclude:
    - "__consumer_offsets"

storage:
backend: s3
bucket: kafka-msk-migration
region: eu-west-1
prefix: transfer/onprem

backup:
compression: zstd
include_offset_headers: true
source_cluster_id: "dc-prod"

kafka-backup backup --config onprem-backup.yaml

kafka-backup validate \
--path s3://kafka-msk-migration/transfer/onprem \
--backup-id "onprem-to-msk-seed" \
--deep

mode: restore
backup_id: "onprem-to-msk-seed"

target:
bootstrap_servers:
  - b-1.mymsk.abcd12.c2.kafka.eu-west-1.amazonaws.com:9096
  - b-2.mymsk.abcd12.c2.kafka.eu-west-1.amazonaws.com:9096
security:
  security_protocol: SASL_SSL
  sasl_mechanism: SCRAM-SHA512
  sasl_username: ${MSK_SCRAM_USER}
  sasl_password: ${MSK_SCRAM_PASSWORD}

storage:
backend: s3
bucket: kafka-msk-migration
region: eu-west-1
prefix: transfer/onprem

restore:
create_topics: true
default_replication_factor: 3
consumer_group_strategy: header-based
reset_consumer_offsets: true
consumer_groups:
  - order-processor
  - billing-service
offset_report: /tmp/msk-migration-offsets.json

kafka-backup validate-restore --config msk-restore.yaml

kafka-backup restore --config msk-restore.yaml

Cutting over without losing writes​

The seed backup moves the bulk of the data while producers keep running on-prem. Cutover then only has to cover the delta. Stop producers, take a second backup of the gap, restore it to MSK, and repoint applications. The freeze window scales with the delta, not with the full data volume.

Keep the offset_report file from each restore — it records where every consumer group landed, which is the artifact your cutover review will ask for. The full procedure, including the rollback backup of the target, is in the migration guide. Every key used above is documented in the configuration reference.

One decision before you provision anything: create the MSK cluster in KRaft mode. MSK supports KRaft from version 3.7.x, and there is no in-place conversion later — a ZooKeeper-mode cluster eventually needs the MSK ZooKeeper to KRaft move as a second migration.