Skip to main content

Azure Blob Storage Setup

Configure Azure Blob Storage for Kafka backups.

Prerequisites​

  • Azure subscription
  • Azure CLI installed
  • Permissions to create storage accounts

Create Storage Account​

Using Azure CLI​

# Set variables
RESOURCE_GROUP="kafka-backup-rg"
STORAGE_ACCOUNT="kafkabackups$(date +%s)" # Must be globally unique
LOCATION="westus2"

# Create resource group
az group create --name $RESOURCE_GROUP --location $LOCATION

# Create storage account
az storage account create \
--name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--location $LOCATION \
--sku Standard_LRS \
--kind StorageV2 \
--min-tls-version TLS1_2 \
--allow-blob-public-access false

# Create container
az storage container create \
--name kafka-backups \
--account-name $STORAGE_ACCOUNT \
--auth-mode login

Using Terraform​

azure-storage.tf
resource "azurerm_resource_group" "kafka_backup" {
name = "kafka-backup-rg"
location = "West US 2"
}

resource "azurerm_storage_account" "kafka_backup" {
name = "kafkabackups${random_string.suffix.result}"
resource_group_name = azurerm_resource_group.kafka_backup.name
location = azurerm_resource_group.kafka_backup.location
account_tier = "Standard"
account_replication_type = "LRS"
min_tls_version = "TLS1_2"

blob_properties {
versioning_enabled = true

delete_retention_policy {
days = 30
}
}
}

resource "azurerm_storage_container" "kafka_backups" {
name = "kafka-backups"
storage_account_name = azurerm_storage_account.kafka_backup.name
container_access_type = "private"
}

resource "random_string" "suffix" {
length = 8
special = false
upper = false
}

Authentication Methods​

Storage Account Key​

# Get storage account key
az storage account keys list \
--account-name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--query "[0].value" -o tsv

Managed Identity (Azure VMs/AKS)​

# Enable system-assigned managed identity on VM
az vm identity assign \
--resource-group $RESOURCE_GROUP \
--name my-vm

# Get principal ID
PRINCIPAL_ID=$(az vm show \
--resource-group $RESOURCE_GROUP \
--name my-vm \
--query identity.principalId -o tsv)

# Assign Storage Blob Data Contributor role
az role assignment create \
--role "Storage Blob Data Contributor" \
--assignee $PRINCIPAL_ID \
--scope "/subscriptions/<sub-id>/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$STORAGE_ACCOUNT"

Service Principal​

# Create service principal
az ad sp create-for-rbac \
--name kafka-backup-sp \
--role "Storage Blob Data Contributor" \
--scopes "/subscriptions/<sub-id>/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$STORAGE_ACCOUNT"

Configuration​

Kafka Backup core 0.25.0 selects Azure credentials in this order: SAS token, account key, service principal, workload identity, then managed identity through IMDS. It uses Rust HTTP clients and does not use the Azure SDK or Azure CLI credentials.

Authentication Methods​

MethodUse CaseRequired Fields
Account KeyDevelopment/testingaccount_key
SAS TokenTime-limited accesssas_token
Service PrincipalCI/CD pipelinesclient_id, tenant_id, client_secret
Workload IdentityAKS with managed identityuse_workload_identity: true (or auto-detected)
Managed identityAzure VM or VMSSOmit secret/key/SAS fields; client_id selects a user-assigned identity

With Storage Account Key​

backup.yaml
storage:
backend: azure
account_name: kafkabackups123456
container_name: kafka-backups
account_key: ${AZURE_STORAGE_KEY}
prefix: production/daily

With SAS Token​

backup.yaml
storage:
backend: azure
account_name: kafkabackups123456
container_name: kafka-backups
sas_token: ${AZURE_SAS_TOKEN}
prefix: production/daily

With Service Principal​

backup.yaml
storage:
backend: azure
account_name: kafkabackups123456
container_name: kafka-backups
client_id: ${AZURE_CLIENT_ID}
tenant_id: ${AZURE_TENANT_ID}
client_secret: ${AZURE_CLIENT_SECRET}
prefix: production/daily

With Workload Identity (AKS)​

backup.yaml
storage:
backend: azure
account_name: kafkabackups123456
container_name: kafka-backups
use_workload_identity: true # Or auto-detected via AZURE_FEDERATED_TOKEN_FILE
prefix: production/daily
# client_id / tenant_id are optional overrides of the webhook-injected values (v0.22.0+)

This works for the CLI running as a plain Kubernetes Job — no operator needed. The AKS Workload Identity webhook injects AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_AUTHORITY_HOST and AZURE_FEDERATED_TOKEN_FILE into a pod when:

  1. its ServiceAccount is annotated azure.workload.identity/client-id: <managed-identity-client-id>, and
  2. the pod carries the label azure.workload.identity/use: "true".

On the Azure side the managed identity (or app registration) needs a federated credential with issuer = the cluster's OIDC issuer (az aks show … --query oidcIssuerProfile.issuerUrl), subject = system:serviceaccount:<namespace>:<serviceaccount>, audience api://AzureADTokenExchange, and the Storage Blob Data Contributor role on the container. Run with RUST_LOG=debug to see Azure authentication: Workload Identity (client_id=…, tenant_id=…, token_file=…); a not set token file means the label or annotation is missing. Credential precedence: sas_token → account_key → service principal → Workload Identity → DefaultAzureCredential chain.

With DefaultAzureCredential (Auto-detect)​

backup.yaml
storage:
backend: azure
account_name: kafkabackups123456
container_name: kafka-backups
# No credentials - uses Azure SDK's DefaultAzureCredential chain
prefix: production/daily

Custom Endpoint (Sovereign Clouds)​

For Azure Government, China, or private endpoints:

backup.yaml
storage:
backend: azure
account_name: kafkabackups123456
container_name: kafka-backups
endpoint: https://kafkabackups123456.blob.core.usgovcloudapi.net
account_key: ${AZURE_STORAGE_KEY}
prefix: production/daily

Environment Variables​

VariableDescription
AZURE_STORAGE_ACCOUNTStorage account name
AZURE_STORAGE_KEYStorage account key
AZURE_SAS_TOKENSAS token for time-limited access
AZURE_CLIENT_IDService principal or managed identity client ID
AZURE_CLIENT_SECRETService principal secret
AZURE_TENANT_IDAzure AD tenant ID
AZURE_FEDERATED_TOKEN_FILEAuto-injected by Workload Identity webhook
AZURE_AUTHORITY_HOSTAzure AD authority (auto-injected)

Generate SAS Token​

# Generate SAS token with 1-year expiry
END_DATE=$(date -u -d "+1 year" '+%Y-%m-%dT%H:%MZ')

az storage container generate-sas \
--account-name $STORAGE_ACCOUNT \
--name kafka-backups \
--permissions rwdl \
--expiry $END_DATE \
--auth-mode login \
--as-user \
-o tsv

AKS Workload Identity​

For AKS, use workload identity:

# Enable workload identity on cluster
az aks update \
--resource-group $RESOURCE_GROUP \
--name my-aks-cluster \
--enable-oidc-issuer \
--enable-workload-identity

# Create managed identity
az identity create \
--name kafka-backup-identity \
--resource-group $RESOURCE_GROUP

# Get identity client ID
CLIENT_ID=$(az identity show \
--name kafka-backup-identity \
--resource-group $RESOURCE_GROUP \
--query clientId -o tsv)

# Assign role
az role assignment create \
--role "Storage Blob Data Contributor" \
--assignee $CLIENT_ID \
--scope "/subscriptions/<sub-id>/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$STORAGE_ACCOUNT"

# Create federated credential
az identity federated-credential create \
--name kafka-backup-federated \
--identity-name kafka-backup-identity \
--resource-group $RESOURCE_GROUP \
--issuer $(az aks show --name my-aks-cluster --resource-group $RESOURCE_GROUP --query oidcIssuerProfile.issuerUrl -o tsv) \
--subject system:serviceaccount:kafka-backup:kafka-backup \
--audience api://AzureADTokenExchange

Kubernetes ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
name: kafka-backup
namespace: kafka-backup
annotations:
azure.workload.identity/client-id: <client-id>
labels:
azure.workload.identity/use: "true"

Azure Key Vault Integration​

Use Azure Key Vault with the CSI Secrets Store Driver to securely manage credentials for Kafka and storage access.

Prerequisites​

  • AKS cluster with Workload Identity enabled
  • Azure Key Vault created
  • Azure CSI Secrets Store Driver installed

Install CSI Secrets Store Driver​

# Add the Helm repo
helm repo add csi-secrets-store-provider-azure \
https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm repo update

# Install the Azure provider
helm install csi-secrets-store-provider-azure \
csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
--namespace kube-system

Grant Key Vault Access​

# Get the managed identity principal ID
PRINCIPAL_ID=$(az identity show \
--name kafka-backup-identity \
--resource-group $RESOURCE_GROUP \
--query principalId -o tsv)

# Grant secret access to Key Vault
az keyvault set-policy \
--name $KEY_VAULT_NAME \
--object-id $PRINCIPAL_ID \
--secret-permissions get list

Store Secrets in Key Vault​

# Store Kafka credentials
az keyvault secret set \
--vault-name $KEY_VAULT_NAME \
--name kafka-sasl-username \
--value "<kafka-api-key>"

az keyvault secret set \
--vault-name $KEY_VAULT_NAME \
--name kafka-sasl-password \
--value "<kafka-api-secret>"

# Store storage account key (if not using Workload Identity for storage)
az keyvault secret set \
--vault-name $KEY_VAULT_NAME \
--name azure-storage-account-key \
--value "<storage-account-key>"

Create SecretProviderClass​

The SecretProviderClass syncs secrets from Azure Key Vault to Kubernetes secrets:

secretproviderclass.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: kafka-backup-secrets
namespace: kafka-backup
spec:
provider: azure
parameters:
usePodIdentity: "false"
clientID: <managed-identity-client-id>
keyvaultName: <key-vault-name>
tenantId: <azure-tenant-id>
objects: |
array:
- |
objectName: kafka-sasl-username
objectType: secret
- |
objectName: kafka-sasl-password
objectType: secret
- |
objectName: azure-storage-account-key
objectType: secret
secretObjects:
- secretName: kafka-backup-secrets
type: Opaque
data:
- objectName: kafka-sasl-username
key: KAFKA_SASL_USERNAME
- objectName: kafka-sasl-password
key: KAFKA_SASL_PASSWORD
- objectName: azure-storage-account-key
key: AZURE_STORAGE_KEY

Using Secrets in Pods​

Mount the CSI volume and reference the synced Kubernetes secret:

pod-with-secrets.yaml
apiVersion: v1
kind: Pod
metadata:
name: kafka-backup
namespace: kafka-backup
labels:
azure.workload.identity/use: "true"
spec:
serviceAccountName: kafka-backup
containers:
- name: kafka-backup
image: ghcr.io/osodevops/kafka-backup:latest
env:
- name: KAFKA_SASL_USERNAME
valueFrom:
secretKeyRef:
name: kafka-backup-secrets
key: KAFKA_SASL_USERNAME
- name: KAFKA_SASL_PASSWORD
valueFrom:
secretKeyRef:
name: kafka-backup-secrets
key: KAFKA_SASL_PASSWORD
- name: AZURE_STORAGE_KEY
valueFrom:
secretKeyRef:
name: kafka-backup-secrets
key: AZURE_STORAGE_KEY
volumeMounts:
- name: secrets-store
mountPath: /mnt/secrets-store
readOnly: true
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: kafka-backup-secrets

Helm Values for Key Vault Integration​

When using the Kafka Backup Operator Helm chart, configure Key Vault integration:

values-azure-keyvault.yaml
deployment:
tenantId: <azure-tenant-id>
workloadIdentityClientId: <managed-identity-client-id>
serviceAccountName: kafka-backup
syncSecrets:
keyVaultName: <key-vault-name>
envSecrets:
kafka-sasl-username: KAFKA_SASL_USERNAME
kafka-sasl-password: KAFKA_SASL_PASSWORD
azure-storage-account-key: AZURE_STORAGE_KEY

This configuration:

  • Uses Workload Identity for authentication (no stored credentials)
  • Syncs secrets from Azure Key Vault to Kubernetes secrets
  • Maps Key Vault secret names to environment variable names

Lifecycle Management​

Set Up Lifecycle Policy​

# Create lifecycle policy JSON
cat > lifecycle-policy.json << 'EOF'
{
"rules": [
{
"enabled": true,
"name": "archive-old-backups",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"tierToCool": {
"daysAfterModificationGreaterThan": 30
},
"tierToArchive": {
"daysAfterModificationGreaterThan": 90
},
"delete": {
"daysAfterModificationGreaterThan": 365
}
}
},
"filters": {
"blobTypes": ["blockBlob"],
"prefixMatch": ["kafka-backups/production/"]
}
}
}
]
}
EOF

# Apply policy
az storage account management-policy create \
--account-name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--policy @lifecycle-policy.json

Access Tiers​

TierUse CaseAccess TimeCost
HotFrequent accessInstant$$$
CoolInfrequent (30+ days)Instant$$
ArchiveRare access (180+ days)Hours$

Geo-Replication​

For disaster recovery:

# Create storage account with geo-redundancy
az storage account create \
--name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--location $LOCATION \
--sku Standard_GRS \
--kind StorageV2

Redundancy options:

SKUDescription
Standard_LRSLocally redundant
Standard_ZRSZone redundant
Standard_GRSGeo-redundant
Standard_RAGRSRead-access geo-redundant

Security Best Practices​

Enable Encryption​

# Enable infrastructure encryption
az storage account update \
--name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--require-infrastructure-encryption

Configure Firewall​

# Restrict to specific VNet
az storage account network-rule add \
--account-name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--vnet-name my-vnet \
--subnet my-subnet

# Set default action to deny
az storage account update \
--name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP \
--default-action Deny

Enable Soft Delete​

az storage blob service-properties delete-policy update \
--account-name $STORAGE_ACCOUNT \
--enable true \
--days-retained 30

Testing​

Verify Access​

# Test with Azure CLI
az storage blob list \
--account-name $STORAGE_ACCOUNT \
--container-name kafka-backups \
--auth-mode login

# Test write
echo "test" | az storage blob upload \
--account-name $STORAGE_ACCOUNT \
--container-name kafka-backups \
--name test.txt \
--data @- \
--auth-mode login

# Clean up
az storage blob delete \
--account-name $STORAGE_ACCOUNT \
--container-name kafka-backups \
--name test.txt \
--auth-mode login

Test from Application​

# Run backup
kafka-backup backup --config backup.yaml

# List backups
kafka-backup list --path azure://$STORAGE_ACCOUNT/kafka-backups/production/daily

Troubleshooting​

Authentication Errors​

# Check credentials
az storage account keys list --account-name $STORAGE_ACCOUNT

# Test with connection string
az storage container list --connection-string "$AZURE_STORAGE_CONNECTION_STRING"

Network Errors​

# Check firewall rules
az storage account network-rule list \
--account-name $STORAGE_ACCOUNT \
--resource-group $RESOURCE_GROUP

# Test connectivity
curl -v https://$STORAGE_ACCOUNT.blob.core.windows.net/

Next Steps​