Kafka TopicAuthorizationException: Fix It Fast

Quick answer​

Kafka topic authorization failed means the broker refused a topic operation for the principal it authenticated. The client exception is TopicAuthorizationException. Its text is Not authorized to access topics.

Read that principal in the broker log. Grant the same name the missing operation. Kafka ACLs, MSK IAM actions, and Confluent roles are separate grants.

What does Kafka topic authorization failed mean?​

org.apache.kafka.common.errors.TopicAuthorizationException:
Not authorized to access topics: [orders]

Protocol error 29 is TOPIC_AUTHORIZATION_FAILED. The Kafka 4.4 protocol errors describe it as "Topic authorization failed." The client class builds the longer text. On the 4.4.0 tag, TopicAuthorizationException uses Not authorized to access topics: plus the topic set.

The authorization guide returns that code when a topic resource is denied. A group denial is error 30, group authorization failed. A cluster denial is error 31. Do not grant a group ACL for a topic error.

A failed login is a different failure. SASL authentication failed means the broker never accepted the credentials. Fix that in the security setup guide before you edit ACLs.

Those rows are the operations table in the Apache Kafka 4.4 ACL guide. Fetch needs Read on the topic. Metadata needs Describe on the topic. A Read allow does not cover a Describe check.

Offset commit checks the group first, then the topic. The topic side of that call needs Read. A group allow can succeed and the topic Read can still throw TopicAuthorizationException.

Which principal did the broker deny?​

The authorizer log is the name that must match the ACL. Search the broker log for is Denied Operation. The line names the principal, the operation, the client host, and the resource pattern. Copy that principal. Do not grant the username from a ticket or a config file until it matches.

ACL principals use PrincipalType:name. The default type string User is case sensitive. user:backup-user is not User:backup-user. The ACL command reference states that rule for --allow-principal and for super.users.

How the broker builds the name depends on the listener.

SASL PLAIN and SCRAM. The name is the username that authenticated. In Kafka Backup the configuration reference sets that with sasl_username. The ACL principal is User: plus that username, unless a custom principal builder rewrites it.

SASL GSSAPI. Apache Kafka says the default SASL user name is the primary part of the Kerberos principal. sasl.kerberos.principal.to.local.rules changes that mapping. The security setup guide shows one Kerberos identity as User:kafka-backup/host.corp@CORP.EXAMPLE. Treat that as the documented example. Grant the principal the authorizer log printed.

mTLS. The default SSL user name is the certificate distinguished name. ssl.principal.mapping.rules maps that DN to a short name. Rules run in order. The first match wins. Later rules are ignored. DEFAULT returns the DN string. The ACL guide documents the RULE: format and the /L and /U case flags.

An allow on the DN does nothing after a rule shortens the name to the CN. An allow on the CN does nothing while the default rule is still in force.

How do I check topic ACLs with kafka-acls?​

List every pattern that can affect the topic. A plain list shows the literal name only. --resource-pattern-type match includes literal, wildcard, and prefixed ACLs that apply to that name.

bin/kafka-acls.sh --bootstrap-server localhost:9092 \
--command-config admin.properties \
--list --topic orders --resource-pattern-type match

bin/kafka-acls.sh --bootstrap-server localhost:9092 \
--command-config admin.properties \
--list --principal User:backup-user

source:
bootstrap_servers:
  - broker-1.kafka.svc:9092
security:
  security_protocol: SASL_SSL
  sasl_mechanism: SCRAM-SHA256
  sasl_username: backup-user
  sasl_password: ${KAFKA_PASSWORD}

{
"Version": "2012-10-17",
"Statement": [
  {
    "Effect": "Allow",
    "Action": ["kafka-cluster:Connect"],
    "Resource": "arn:aws:kafka:us-east-1:123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
  },
  {
    "Effect": "Allow",
    "Action": [
      "kafka-cluster:DescribeTopic",
      "kafka-cluster:ReadData"
    ],
    "Resource": "arn:aws:kafka:us-east-1:123456789012:topic/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1/orders"
  },
  {
    "Effect": "Allow",
    "Action": [
      "kafka-cluster:DescribeGroup",
      "kafka-cluster:AlterGroup"
    ],
    "Resource": "arn:aws:kafka:us-east-1:123456789012:group/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1/backup-reader"
  }
]
}

--command-config points kafka-acls.sh at an Admin client properties file. Use it when the cluster requires SASL or TLS. Listing ACLs needs Describe on the Cluster resource. That is the operator's grant, not the client's topic grant.

Check three mismatches before you add another allow.

Literal versus prefixed. --resource-pattern-type defaults to literal. A literal ACL hits that exact name. A prefixed ACL hits names that start with the prefix. The ACL guide shows prefixed on --topic Test- for topics such as Test-topic. Literal orders does not cover orders.eu. Prefixed orders covers orders and orders.eu. Prefixed orders. does not cover orders.

Wildcard resource. The literal resource name * matches any name of that type. It is not a prefixed pattern. User:* is a different wildcard. That principal matches every user. The topic name * matches every topic.

Host and deny. --allow-host defaults to * when you set --allow-principal. The value must be an IP address. Hostnames are not accepted. A deny for that principal and operation blocks the allow. List denies before you add a second allow.

--consumer is a shortcut. It writes Read and Describe on the topic, and Read on the group. --producer writes Write, Describe, and Create on the topic. Pass --operation yourself when you want a smaller set. The default operation, if you omit it, is All.

Do consumer group ACLs fix a topic authorization error?​

No. The exception names topics. A missing group allow produces group authorization failed, not this exception.

You still need the group ACL for the consume path. Join, heartbeat, and offset commit need Read on the group. Offset fetch and group describe need Describe on the group. Those checks fail on their own when the topic allow is already present.

Grant the group on the group resource. Grant Read or Write on the topic resource. One command with --group does not authorize --topic.

What does allow.everyone.if.no.acl.found change?​

The default is false. If no ACL matches the resource, only super.users may use it. Everyone else is denied. The security model describes the same default.

Set allow.everyone.if.no.acl.found=true and a resource with no matching ACL is open to every principal. The moment any ACL matches that resource, normal allow and deny rules apply. The flag does not skip those rules. One allow on orders for a different user removes the open behavior for orders.

super.users bypass ACL checks. The list uses the same User: form, separated by semicolons. A principal in that list will not hit this error. That is a broker setting, not an ACL you will see from --list.

If no authorizer is configured, authenticated principals are not checked. This exception means an authorizer, or a cloud control such as MSK IAM, denied the call.

How do I fix topic authorization failed on Amazon MSK?​

MSK IAM access control does not consult Kafka ACLs. kafka-acls.sh will not fix a deny on an IAM-auth listener. The IAM policy on the caller is the grant. AWS documents the action names and the ARN shapes in IAM authorization actions and the kafka-cluster service reference.

The topic ARN format is arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/topic-name. The group ARN uses group in that same position. The cluster ARN is arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid. The documented topic ARN includes the cluster name and the cluster UUID. A resource that drops the UUID, and does not use a wildcard in its place, does not match that topic. AWS sample policies also use a wildcard segment, such as topic/MyTestCluster/*.

kafka-cluster:ReadData is the IAM name for a topic Read. AWS lists kafka-cluster:Connect, kafka-cluster:DescribeTopic, and kafka-cluster:AlterGroup as required with it. Connect belongs on the cluster ARN. DescribeTopic and ReadData belong on the topic ARN. AlterGroup belongs on the group ARN. The JSON tab above follows that split.

kafka-cluster:*Topic* in an AWS sample is an IAM action wildcard. It is not the Kafka topic name *.

Policy updates show up in IAM quickly. AWS says the broker can take noticeable time to enforce them, often under a minute. Retry after that delay before you add a broader resource.

Kafka Backup's docs do not publish an IAM policy for the backup or restore principal. The next section lists the Kafka ACL operations the product does document. On MSK IAM, translate those operations with the table above. Do not paste the sample policy as a product requirement.

How does Confluent Cloud RBAC deny a topic?​

Confluent Cloud can deny the same client call with a missing role binding. DeveloperRead is read-only on the resource you bind. Consuming also needs DeveloperRead on the consumer group, unless the consumer assigns partitions itself. DeveloperWrite is write-only on the topic. It does not grant Read.

Granular topic, group, and transactional-id roles are supported on Standard, Enterprise, Dedicated, and Freight clusters. Service accounts show up as User:sa-… in ACL and role-binding output. Bind the role to that principal, not to the API key string.

Cloud clusters can also use Kafka ACLs. A role binding does not replace an ACL deny, and an ACL allow does not replace a missing role. Check both when the cluster has both.

Permissions and RBAC is a different control. Kafka Backup uses the Kafka credentials you configure. It does not ship a backup-admin role that grants topic access. Enterprise permission capture records Confluent bindings. It does not replace the client's topic allow.

Which ACLs does the Kafka Backup user need?​

The ACL Requirements section is the documented grant. The backup principal and the restore principal are different users.

Backup, from that guide:

kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Read --operation Describe \
  --topic '*'

kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Read --operation Describe \
  --group '*'

Optional cluster Describe, also from that guide:

kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Describe \
  --cluster

Restore, from that guide:

kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:restore-user \
  --operation Write --operation Create --operation Describe \
  --topic '*'

kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:restore-user \
  --operation Read --operation Write \
  --group '*'

Replace User:backup-user and User:restore-user with the principal the broker logged. The error catalog lists a shorter backup set. It names Topic Read and Describe, Group Read, and optional Cluster DescribeConfigs. Describe and DescribeConfigs are different operations. The security setup commands are the grant to run. They include Group Describe, which the catalog line does not. The common errors page shows the same topic Read and Describe for a backup user that cannot read one topic.

A replicator that is denied a topic is not a backup. Replication copies records to another Kafka cluster. The copy stays inside Kafka. A backup is a separate copy in external storage, read by the backup principal above.