Security Setup

Secure your Kafka backup operations with TLS encryption and SASL authentication.

Security Overview

OSO Kafka Backup supports multiple security configurations:

Feature	Description
TLS/SSL	Encrypt data in transit
SASL	Authenticate to Kafka
mTLS	Mutual TLS authentication
Storage Encryption	Encrypt data at rest

TLS Configuration

TLS Only (Encryption)

Encrypt communication without authentication:

source:
  bootstrap_servers:
    - kafka:9093
  security:
    security_protocol: SSL
    ssl_ca_location: /certs/ca.crt

TLS with Client Certificate (mTLS)

Mutual TLS for encryption and authentication:

source:
  bootstrap_servers:
    - kafka:9093
  security:
    security_protocol: SSL
    ssl_ca_location: /certs/ca.crt
    ssl_certificate_location: /certs/client.crt
    ssl_key_location: /certs/client.key
    ssl_key_password: ${SSL_KEY_PASSWORD}  # If key is encrypted

Certificate Requirements

File	Purpose	Format
ca.crt	CA certificate to verify broker	PEM
client.crt	Client certificate	PEM
client.key	Client private key	PEM (PKCS#8)

Creating Certificates

Using OpenSSL:

# Generate CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 365 -key ca.key -out ca.crt \
  -subj "/CN=Kafka-CA"

# Generate client key and CSR
openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr \
  -subj "/CN=kafka-backup"

# Sign client certificate
openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt

SASL Configuration

SASL/PLAIN

Simple username/password authentication:

source:
  bootstrap_servers:
    - kafka:9092
  security:
    security_protocol: SASL_PLAINTEXT  # Or SASL_SSL
    sasl_mechanism: PLAIN
    sasl_username: backup-user
    sasl_password: ${KAFKA_PASSWORD}

SASL/SCRAM-SHA-256

SCRAM authentication using SHA-256 (RFC 5802). Uses PBKDF2 key derivation and mutual server signature verification to prevent MITM attacks:

source:
  bootstrap_servers:
    - kafka:9092
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA256
    sasl_username: backup-user
    sasl_password: ${KAFKA_PASSWORD}
    ssl_ca_location: /certs/ca.crt

SASL/SCRAM-SHA-512

SCRAM authentication using SHA-512. Same protocol as SCRAM-SHA-256 but with a stronger hash. Recommended for production:

source:
  bootstrap_servers:
    - kafka:9092
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA512
    sasl_username: backup-user
    sasl_password: ${KAFKA_PASSWORD}
    ssl_ca_location: /certs/ca.crt

SASL/GSSAPI (Kerberos)

Kerberos authentication via the GSSAPI mechanism (RFC 4752). Typical in enterprise environments that already run an MIT Kerberos KDC.

Opt-in feature

GSSAPI is gated behind the gssapi cargo feature. Release binaries and the default Docker image do not include it. You need to either build from source with --features gssapi or build a Docker image with --build-arg FEATURES=gssapi. See Build & runtime prerequisites below.

kerberos-backup.yaml

source:
  bootstrap_servers:
    - kafka.prod.corp:9098
  security:
    security_protocol: SASL_PLAINTEXT   # or SASL_SSL if the broker advertises both
    sasl_mechanism: GSSAPI
    sasl_kerberos_service_name: kafka    # must match the broker's service principal
    sasl_keytab_path: /etc/kafka-backup/client.keytab
    sasl_krb5_config_path: /etc/krb5.conf

The three GSSAPI-specific fields:

Field	Required	Description
sasl_kerberos_service_name	Yes	The service name portion of the broker's service principal (e.g. kafka for kafka/broker.host@REALM).
sasl_keytab_path	Yes	Absolute path to a keytab containing the client principal's long-term key. Read-only to the process.
sasl_krb5_config_path	No	Override the default /etc/krb5.conf. Useful when packaging kafka-backup inside a container that doesn't share the host's krb5 config.

Build & runtime prerequisites

GSSAPI dynamically links against the system MIT Kerberos library — the same libgssapi_krb5.so.2 your Kafka clients and brokers already load.

At build time (the machine running cargo build):

Platform	Install
macOS	brew install krb5 and export PKG_CONFIG_PATH="$(brew --prefix krb5)/lib/pkgconfig:$PKG_CONFIG_PATH" before building. Apple's bundled Heimdal does not expose the symbols libgssapi needs.
Debian/Ubuntu	apt-get install libkrb5-dev
Fedora/RHEL	dnf install krb5-devel

Then:

cargo build --release --features gssapi -p kafka-backup-cli

At runtime (the machine the binary runs on):

Platform	Package
Debian/Ubuntu	libkrb5-3 (almost always already present if any Kerberos client is installed)
RHEL/Fedora	krb5-libs
Alpine	krb5-libs

If the host can already SASL-authenticate to Kafka with any Kerberos tool, libgssapi_krb5.so.2 is already there.

Keytab layout

A keytab is a binary file holding the client principal and its long-term Kerberos key. Create and scope one per kafka-backup identity:

# On a machine with kadmin access to the KDC
kadmin -p admin/admin
> addprinc -randkey kafka-backup/host.corp@CORP.EXAMPLE
> ktadd -k /tmp/kafka-backup.keytab kafka-backup/host.corp@CORP.EXAMPLE

Move the keytab to the backup host and lock it down:

install -m 0400 -o kafka-backup /tmp/kafka-backup.keytab \
  /etc/kafka-backup/client.keytab

Grant the principal the Kafka ACLs listed under ACL Requirements; Kerberos identities appear on the Kafka side as User:kafka-backup/host.corp@CORP.EXAMPLE.

Hostname / krb5.conf expectations

Kerberos service-principal matching is strict. The hostname in your bootstrap_servers must match the FQDN in the broker's service principal. If the broker runs as kafka/broker.prod.corp@CORP.EXAMPLE, your config must reach the broker via broker.prod.corp — not localhost, not a load-balancer DNS name, not a container alias. Mismatches surface as KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN.

Your krb5.conf needs the realm's KDC and its hostname resolution. Minimal example:

/etc/krb5.conf

[libdefaults]
    default_realm = CORP.EXAMPLE
    rdns = false                # disable reverse DNS — spn matching must be honest

[realms]
    CORP.EXAMPLE = {
        kdc = kdc.corp.example:88
    }

[domain_realm]
    .corp.example = CORP.EXAMPLE
    corp.example = CORP.EXAMPLE

Automatic re-authentication (KIP-368)

If the broker advertises a session lifetime (connections.max.reauth.ms), kafka-backup schedules a Kerberos re-handshake at ~80 % of that window (with ±5 s jitter and a 30 s floor) before the broker forcibly drops the connection. This happens transparently — no operator action required.

You don't need to set anything. The scheduler is on the SaslMechanismPlugin trait that both built-in mechanisms and GSSAPI share.

Offline / airgapped builds

For dev or production hosts without crates.io access:

# On an internet-connected machine, from the kafka-backup repo root
cargo vendor > .cargo/config-vendor.toml
git add vendor .cargo/config-vendor.toml
git commit -m "chore: vendor cargo deps for airgapped builds"

# Transfer the repo to the airgapped host, pre-install libkrb5-dev from your
# internal apt/rpm mirror, then build with:
cargo build --release --features gssapi -p kafka-backup-cli --offline

The vendored libgssapi, libgssapi-sys, and all transitive crates live inside the repo. No network access is needed after cargo vendor. The only remaining system prerequisite is the krb5 development headers, which every enterprise Linux distribution ships.

Docker image with GSSAPI

The default oso/kafka-backup:<version> image does not include GSSAPI. Build a variant yourself:

FROM rust:1.82-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
        libkrb5-dev pkg-config build-essential && \
    rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY . .
RUN cargo build --release --features gssapi -p kafka-backup-cli

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
        libkrb5-3 libgssapi-krb5-2 ca-certificates && \
    rm -rf /var/lib/apt/lists/*
COPY --from=builder /src/target/release/kafka-backup /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/kafka-backup"]

Then mount the keytab and krb5.conf at runtime:

docker run --rm \
  -v /etc/krb5.conf:/etc/krb5.conf:ro \
  -v /etc/kafka-backup/client.keytab:/etc/kafka-backup/client.keytab:ro \
  -v $(pwd)/config:/config:ro \
  my-registry/kafka-backup:gssapi \
  backup --config /config/kerberos-backup.yaml

Pluggable SASL mechanisms

kafka-backup exposes a SaslMechanismPlugin Rust trait so downstream crates can add mechanisms (OAUTHBEARER, MSK IAM, custom) without forking core. GSSAPI is the first in-tree consumer; OAUTHBEARER and MSK IAM are expected community additions.

If you are an operator, this section is informational — you do not need to interact with the trait directly. You configure a mechanism via sasl_mechanism: in YAML; the plugin is wired automatically.

If you are building a downstream mechanism, see examples/custom_sasl_plugin.rs in the source repository for a minimal OAUTHBEARER reference implementation, and the PRD at docs/PRD-sasl-mechanism-plugin.md for the trait contract (handshake dispatch, KIP-368 re-auth scheduler, RFC 7628 error parsing).

Security Protocol Matrix

Protocol	Encryption	Authentication
PLAINTEXT	No	No
SSL	Yes (TLS)	Optional (mTLS)
SASL_PLAINTEXT	No	Yes (SASL)
SASL_SSL	Yes (TLS)	Yes (SASL)

Supported SASL Mechanisms

Mechanism	sasl_mechanism: value	Built-in	Feature gate
SASL/PLAIN	PLAIN	Yes	—
SASL/SCRAM-SHA-256	SCRAM-SHA256	Yes	—
SASL/SCRAM-SHA-512	SCRAM-SHA512	Yes	—
SASL/GSSAPI (Kerberos)	GSSAPI	Opt-in	--features gssapi at build time

A CLI built without --features gssapi that loads a config with sasl_mechanism: GSSAPI fails fast at startup with a clear error pointing at the missing feature flag.

Kubernetes Secret Management

Create Secrets

kafka-credentials.yaml

apiVersion: v1
kind: Secret
metadata:
  name: kafka-credentials
  namespace: kafka-backup
type: Opaque
stringData:
  username: backup-user
  password: your-secure-password

kafka-tls.yaml

apiVersion: v1
kind: Secret
metadata:
  name: kafka-tls
  namespace: kafka-backup
type: Opaque
data:
  ca.crt: <base64-encoded-ca>
  client.crt: <base64-encoded-cert>
  client.key: <base64-encoded-key>

Use in Deployment

spec:
  containers:
    - name: kafka-backup
      env:
        - name: KAFKA_USERNAME
          valueFrom:
            secretKeyRef:
              name: kafka-credentials
              key: username
        - name: KAFKA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: kafka-credentials
              key: password
      volumeMounts:
        - name: tls
          mountPath: /certs
          readOnly: true
  volumes:
    - name: tls
      secret:
        secretName: kafka-tls

Kubernetes Operator Configuration

The operator handles secrets automatically:

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: secure-backup
spec:
  kafkaCluster:
    bootstrapServers:
      - kafka:9093
    securityProtocol: SASL_SSL
    tlsSecret:
      name: kafka-tls
      caKey: ca.crt
    saslSecret:
      name: kafka-credentials
      mechanism: SCRAM-SHA256
      usernameKey: username
      passwordKey: password

Strimzi Integration

Strimzi stores the cluster CA certificate and per-user client certificates in separate secrets. Use caSecret to reference the CA independently:

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: strimzi-backup
spec:
  kafkaCluster:
    bootstrapServers:
      - my-cluster-kafka-bootstrap:9093
    securityProtocol: SSL
    caSecret:
      name: my-cluster-cluster-ca-cert    # Strimzi cluster CA
      caKey: ca.crt
    tlsSecret:
      name: my-kafka-user                 # Strimzi KafkaUser secret
      certKey: user.crt
      keyKey: user.key

This preserves Strimzi's automatic certificate rotation without requiring a manually combined secret.

Storage Encryption

S3 Server-Side Encryption

storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-west-2
  # S3 encrypts at rest automatically with SSE-S3
  # Or configure SSE-KMS in bucket settings

Enable in bucket:

aws s3api put-bucket-encryption \
  --bucket my-kafka-backups \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "arn:aws:kms:us-west-2:123456789:key/12345"
      }
    }]
  }'

Azure Encryption

Azure encrypts all data at rest by default. For customer-managed keys:

az storage account update \
  --name mystorageaccount \
  --resource-group mygroup \
  --encryption-key-source Microsoft.Keyvault \
  --encryption-key-vault https://myvault.vault.azure.net \
  --encryption-key-name mykey

GCS Encryption

GCS encrypts all data at rest by default. For customer-managed keys:

gcloud storage buckets update gs://my-bucket \
  --default-encryption-key=projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY

ACL Requirements

Minimum Kafka ACLs for Backup

# Allow reading from all topics
kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Read --operation Describe \
  --topic '*'

# Allow reading consumer group offsets
kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Read --operation Describe \
  --group '*'

# Optional: Cluster describe for metadata
kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Describe \
  --cluster

Minimum Kafka ACLs for Restore

# Allow writing to topics
kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:restore-user \
  --operation Write --operation Create --operation Describe \
  --topic '*'

# For offset reset
kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:restore-user \
  --operation Read --operation Write \
  --group '*'

Environment Variables

Secure credentials using environment variables:

export KAFKA_PASSWORD="your-password"
export SSL_KEY_PASSWORD="key-password"
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."

Reference in config:

source:
  security:
    sasl_password: ${KAFKA_PASSWORD}
    ssl_key_password: ${SSL_KEY_PASSWORD}

storage:
  backend: s3
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}

Best Practices

Credential Management

1. Never commit credentials - Use secrets management
2. Rotate credentials - Regular rotation policy
3. Least privilege - Minimal required permissions
4. Audit access - Log and monitor credential usage

Certificate Management

1. Automate renewal - Use cert-manager or similar
2. Monitor expiry - Alert before certificates expire
3. Secure storage - Protect private keys
4. Use short-lived certs - 90 days or less

Network Security

1. Use TLS - Always encrypt in transit
2. Private networks - Keep Kafka on private subnets
3. Firewall rules - Restrict access to backup service
4. VPC endpoints - Use private connectivity to cloud storage

Troubleshooting

SSL Handshake Failed

Error: SSL handshake failed

Causes:

• Certificate mismatch
• Expired certificate
• Wrong CA certificate

Solution:

# Verify certificate
openssl s_client -connect kafka:9093 -CAfile ca.crt

# Check expiry
openssl x509 -in client.crt -noout -dates

SASL Authentication Failed

Error: SASL authentication failed

Causes:

• Wrong username/password
• User not created in Kafka
• Wrong SASL mechanism

Solution:

# Test with kafka-console-consumer
kafka-console-consumer \
  --bootstrap-server kafka:9092 \
  --consumer.config client.properties \
  --topic test

Permission Denied

Error: Not authorized to access topic

Solution: Check and update ACLs:

kafka-acls --bootstrap-server kafka:9092 \
  --list --principal User:backup-user

GSSAPI: gssapi.h' file not found at build time

fatal error: 'gssapi.h' file not found
failed to generate gssapi bindings

Cause: krb5 development headers are missing on the build host. Dynamic link needs the header files at compile time even though the .so ships in the runtime OS package.

Solution: Install the development package for your distribution:

# Debian/Ubuntu
sudo apt-get install libkrb5-dev

# Fedora/RHEL
sudo dnf install krb5-devel

# macOS (Apple Heimdal does not work — must be MIT krb5)
brew install krb5
export PKG_CONFIG_PATH="$(brew --prefix krb5)/lib/pkgconfig:$PKG_CONFIG_PATH"

GSSAPI: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN

Error: acquire_cred failed: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN

Cause: the hostname you're connecting to doesn't match the broker's service principal. If the broker runs as kafka/broker.prod.corp@CORP.EXAMPLE and you connect via localhost:9098 or a load balancer DNS name, the KDC refuses to issue a ticket for the service.

Solution: connect via the exact FQDN that appears in the broker's service principal. Verify:

kvno kafka/broker.prod.corp@CORP.EXAMPLE

If you're testing against a local Docker fixture, add a /etc/hosts entry pointing that FQDN at 127.0.0.1.

GSSAPI: Authentication failed due to invalid credentials (broker-side)

Error: Authentication failed due to invalid credentials with SASL mechanism GSSAPI

Cause: MIT Kerberos preferred a stale ticket in the OS default credential cache over a fresh TGT from your keytab. Most common on macOS, where the default cache is API:<UUID> and persists across logins — and when the broker's service key has rotated since that cache was populated, AP-REQ decryption fails.

Solution: kafka-backup isolates its credential cache automatically when a keytab is configured (KRB5CCNAME=MEMORY:<ptr> per plugin instance). If you still hit this, confirm:

• The keytab you configured is fresh (klist -kte /path/to/keytab — check the KVNO column matches what the KDC knows about your principal).
• Nothing on the host is kinit-ing the same principal into the default ccache with a stale key.
• rdns = false is set in your krb5.conf — reverse-DNS fuzz on SPN matching is a common silent failure.

GSSAPI: Configured but CLI rejects it at startup

Error: sasl_mechanism: GSSAPI is configured but this binary was built without the `gssapi` feature

Cause: you're running the default (non-gssapi) CLI against a Kerberized broker.

Solution: rebuild with the feature (cargo build --release --features gssapi -p kafka-backup-cli) or use a Docker image built with --build-arg FEATURES=gssapi. See Build & runtime prerequisites.

Complete Secure Configuration Example

secure-backup.yaml

mode: backup
backup_id: "secure-production-backup"

source:
  bootstrap_servers:
    - kafka-0.kafka.svc:9093
    - kafka-1.kafka.svc:9093
    - kafka-2.kafka.svc:9093
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA512
    sasl_username: backup-service
    sasl_password: ${KAFKA_PASSWORD}
    ssl_ca_location: /certs/ca.crt
    ssl_certificate_location: /certs/client.crt
    ssl_key_location: /certs/client.key
  topics:
    include:
      - "*"
    exclude:
      - "__consumer_offsets"

storage:
  backend: s3
  bucket: secure-kafka-backups
  region: us-west-2
  # Uses IAM role - no static credentials

backup:
  compression: zstd
  compression_level: 3
  include_offset_headers: true

Complete GSSAPI (Kerberos) Configuration Example

For enterprises running an MIT Kerberos KDC. Assumes the CLI was built with --features gssapi and libkrb5-3 is installed on the runtime host.

kerberos-backup.yaml

mode: backup
backup_id: "kerberos-production-backup"

source:
  bootstrap_servers:
    - broker-0.prod.corp:9098
    - broker-1.prod.corp:9098
    - broker-2.prod.corp:9098
  security:
    security_protocol: SASL_PLAINTEXT          # SASL_SSL if broker advertises TLS too
    sasl_mechanism: GSSAPI
    sasl_kerberos_service_name: kafka
    sasl_keytab_path: /etc/kafka-backup/client.keytab
    sasl_krb5_config_path: /etc/krb5.conf
  topics:
    include:
      - "*"
    exclude:
      - "__consumer_offsets"

storage:
  backend: s3
  bucket: secure-kafka-backups
  region: us-west-2

backup:
  compression: zstd
  compression_level: 3
  include_offset_headers: true

Next Steps

• Deployment Guide - Production deployment
• Kubernetes Operator - Secure K8s setup
• Enterprise Features - Field-level encryption