Skip to main content

Example: MSK KRaft Migration (SCRAM to IAM)

This example covers a cross-auth migration: SCRAM-SHA-512 on the source ZooKeeper cluster to IAM on the target KRaft cluster. This is common when modernizing legacy authentication alongside the ZK→KRaft migration.

Scenario​

SourceTarget
Metadata modeZooKeeperKRaft
Kafka version3.5.03.9.0
AuthenticationSCRAM-SHA-512IAM
SCRAM usersadmin, order-svc, analytics-svc, billing-svc— (IAM roles)
ACLsPer-user topic/group ACLsIAM policies (generated)
Topics250 (empty target)

The Cross-Auth Challenge​

When migrating from SCRAM to IAM:

  • Kafka ACLs (based on SCRAM usernames) don't apply on IAM-auth clusters
  • Each SCRAM user's permissions must be translated to an IAM policy
  • The tool generates an access-map.json that maps SCRAM principals to required IAM actions

Migration Config​

migration-cross-auth.yaml
enterprise:
  msk_kraft_migration:
    source:
      cluster_arn: arn:aws:kafka:eu-west-1:123456789012:cluster/legacy-scram/abc-123
      auth:
        mode: scram-sha-512
        username: ${KAFKA_ADMIN_USER}
        password: ${KAFKA_ADMIN_PASS}
    target:
      cluster_arn: arn:aws:kafka:eu-west-1:123456789012:cluster/modern-kraft/def-456
      auth:
        mode: iam
    backup:
      s3_bucket: migration-segments-eu
      s3_prefix: scram-to-iam/
    evidence:
      s3_bucket: migration-evidence-eu
      s3_prefix: migrations/
      retention: 7y
    cutover:
      drain_timeout: 30m
      max_producer_freeze: 120s
    acl:
      on_drift: merge
Environment variable interpolation

SCRAM credentials support ${ENV_VAR} interpolation. Set KAFKA_ADMIN_USER and KAFKA_ADMIN_PASS in your environment before running.

Run the Migration​

Step 1: Plan and precheck​

export KAFKA_ADMIN_USER=admin
export KAFKA_ADMIN_PASS=<your-admin-password>

kafka-backup migrate msk-kraft plan \
  --config migration-cross-auth.yaml \
  --format all \
  --out-dir ./migration-plan

kafka-backup migrate msk-kraft precheck --config migration-cross-auth.yaml

Expected precheck findings:

  • I01: Target is IAM-auth — ACLs emitted as access-map.json
  • W09: MSK internal ACLs will be filtered (User:ANONYMOUS, etc.)

These are expected and do not block migration.

Step 2: Execute​

kafka-backup migrate msk-kraft execute \
  --config migration-cross-auth.yaml \
  --journal-dir ./journal

Step 3: Cutover​

kafka-backup migrate msk-kraft cutover \
  --config migration-cross-auth.yaml \
  --migration-id <ID> \
  --journal-dir ./journal

ACL Translation: access-map.json​

After cutover, the evidence bundle includes an access-map.json that maps each SCRAM principal's Kafka ACLs to the IAM actions needed:

{
  "principals": {
    "User:order-svc": {
      "topics": {
        "orders": ["kafka-cluster:ReadData", "kafka-cluster:WriteData", "kafka-cluster:DescribeTopic"],
        "order-events": ["kafka-cluster:ReadData", "kafka-cluster:DescribeTopic"]
      },
      "groups": {
        "order-processing-cg": ["kafka-cluster:ReadGroup", "kafka-cluster:DescribeGroup"]
      }
    },
    "User:analytics-svc": {
      "topics": {
        "orders": ["kafka-cluster:ReadData", "kafka-cluster:DescribeTopic"],
        "events": ["kafka-cluster:ReadData", "kafka-cluster:DescribeTopic"]
      },
      "groups": {
        "analytics-cg": ["kafka-cluster:ReadGroup", "kafka-cluster:DescribeGroup"]
      }
    }
  }
}

Post-Migration IAM Setup​

Use the access map to create IAM policies for each service:

order-svc-kafka-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster"
      ],
      "Resource": "arn:aws:kafka:eu-west-1:123456789012:cluster/modern-kraft/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:ReadData",
        "kafka-cluster:WriteData",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": "arn:aws:kafka:eu-west-1:123456789012:topic/modern-kraft/*/orders"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:ReadData",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": "arn:aws:kafka:eu-west-1:123456789012:topic/modern-kraft/*/order-events"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:ReadGroup",
        "kafka-cluster:DescribeGroup"
      ],
      "Resource": "arn:aws:kafka:eu-west-1:123456789012:group/modern-kraft/*/order-processing-cg"
    }
  ]
}
# Attach to each service's IAM role
aws iam put-role-policy \
  --role-name order-svc-role \
  --policy-name kafka-access \
  --policy-document file://order-svc-kafka-policy.json

Update application authentication​

Each service needs to switch from SCRAM to IAM authentication:

Before (SCRAM):

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="order-svc" password="secret";

After (IAM):

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Finalize​

kafka-backup migrate msk-kraft cutover-ack \
  --config migration-cross-auth.yaml \
  --migration-id <ID> \
  --journal-dir ./journal

kafka-backup migrate msk-kraft finalize \
  --config migration-cross-auth.yaml \
  --migration-id <ID> \
  --journal-dir ./journal

Next Steps​