Common Errors

This guide covers the most frequently encountered errors and their solutions.

Connection Errors

Failed to Connect to Kafka

Error: Failed to connect to Kafka broker
Cause: Connection refused (os error 111)
Broker: kafka:9092

Causes:

• Broker is not running
• Wrong hostname/port
• Network/firewall issue

Solutions:

# Test connectivity
nc -zv kafka 9092

# Check if Kafka is running
kafka-broker-api-versions --bootstrap-server kafka:9092

# Verify DNS resolution
nslookup kafka

# Fix: Correct broker addresses
source:
  bootstrap_servers:
    - kafka-0.kafka.svc.cluster.local:9092
    - kafka-1.kafka.svc.cluster.local:9092

SSL Handshake Failed

Error: SSL handshake failed
Cause: certificate verify failed

Causes:

• Missing CA certificate
• Expired certificate
• Hostname mismatch

Fixed in v0.4.0

Prior to v0.4.0, custom CA certificates configured via ssl_ca_location were not being used during TLS connections. This caused "UnknownIssuer" errors when connecting to Kafka brokers with self-signed or internal CA certificates. Upgrade to v0.4.0 or later to resolve this issue. See GitHub Issue #3 for details.

Solutions:

# Verify certificate
openssl s_client -connect kafka:9093 -CAfile ca.crt

# Check certificate expiry
openssl x509 -in ca.crt -noout -dates

# Check hostname in certificate
openssl x509 -in server.crt -noout -text | grep -A1 "Subject Alternative Name"

# Fix: Provide correct CA certificate
source:
  security:
    security_protocol: SSL
    ssl_ca_location: /certs/ca.crt

SASL Authentication Failed

Error: SASL authentication failed
Cause: Authentication failed during SASL handshake

Causes:

• Wrong username/password
• User doesn't exist
• Wrong SASL mechanism

Solutions:

# Test authentication with kafka-console-consumer
kafka-console-consumer \
  --bootstrap-server kafka:9092 \
  --consumer.config client.properties \
  --topic test --max-messages 1

# Fix: Verify credentials
source:
  security:
    security_protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA256  # Ensure correct mechanism
    sasl_username: backup-user
    sasl_password: ${KAFKA_PASSWORD}  # Check environment variable

Broken Pipe / Idle Connection Timeout

Error: Broken pipe (os error 32)
Cause: Connection terminated by broker

Or:

Error: Failed to send request: Connection reset by peer

Causes:

• Cloud Kafka service terminated idle connection (Confluent Cloud, AWS MSK, etc.)
• Network idle timeout
• TCP keepalive not enabled

Symptoms:

• Errors appear after ~5 minutes of low topic activity
• Topics with infrequent messages fail while active topics work
• Restarting the backup temporarily fixes it

Solutions:

# Fix: Enable TCP keepalive (enabled by default in v0.6.0+)
source:
  bootstrap_servers:
    - kafka.confluent.cloud:9092
  connection:
    tcp_keepalive: true           # Keep connections alive
    keepalive_time_secs: 60       # Probe after 60s idle
    keepalive_interval_secs: 20   # Probe every 20s
    tcp_nodelay: true             # Reduce latency

Confluent Cloud Users

Confluent Cloud terminates idle TCP connections after approximately 5 minutes. The default connection settings (enabled since v0.6.0) prevent this. If you're upgrading from an earlier version, ensure tcp_keepalive: true is set.

Storage Errors

Unable to Open Database File

Error: Storage error: Backend error: error returned from database: (code: 14) unable to open database file

Causes:

• Running with readOnlyRootFilesystem: true in Kubernetes without a writable /tmp volume
• The working directory is not writable (common in container environments)

Solutions:

Mount /tmp as an emptyDir in your Kubernetes pod spec:

volumeMounts:
  - name: tmp
    mountPath: /tmp
volumes:
  - name: tmp
    emptyDir: {}

Or configure a custom offset database path via offset_storage.db_path pointing to a writable volume:

offset_storage:
  db_path: /data/offsets.db

note

Since v0.11.4, the automatically created continuous-mode offset store uses $TMPDIR. An explicit offset_storage block still defaults db_path to ./offsets.db, so set a writable path when the container filesystem is read only. See GitHub Issue #62.

Access Denied to S3

Error: Access denied to S3 bucket
Bucket: my-kafka-backups
Operation: PutObject

Causes:

• Missing IAM permissions
• Wrong credentials
• Bucket policy restriction

Solutions:

# Test AWS credentials
aws sts get-caller-identity

# Test bucket access
aws s3 ls s3://my-kafka-backups/

# Check bucket policy
aws s3api get-bucket-policy --bucket my-kafka-backups

// Fix: Add IAM policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-kafka-backups",
        "arn:aws:s3:::my-kafka-backups/*"
      ]
    }
  ]
}

Bucket Not Found

Error: Bucket not found
Bucket: my-kafka-backups
Region: us-west-2

Solutions:

# Verify bucket exists
aws s3api head-bucket --bucket my-kafka-backups

# Check region
aws s3api get-bucket-location --bucket my-kafka-backups

# Fix: Correct bucket and region
storage:
  backend: s3
  bucket: my-kafka-backups
  region: us-east-1  # Correct region

Azure Storage Error

Error: Azure storage operation failed
Container: kafka-backups
Status: 403 Forbidden

Solutions:

# Test Azure credentials
az storage container list --account-name mystorageaccount

# Check connection string
az storage container show --name kafka-backups --account-name mystorageaccount

# Fix: Verify connection string
storage:
  backend: azure
  container: kafka-backups
  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}

Backup Errors

Topic Not Found

Error: Topic not found
Topic: orders
Cluster: kafka:9092

Causes:

• Topic doesn't exist
• ACL restriction
• Typo in topic name

Solutions:

# List topics
kafka-topics --bootstrap-server kafka:9092 --list

# Check if topic exists
kafka-topics --bootstrap-server kafka:9092 --describe --topic orders

# Fix: Use correct topic pattern
source:
  topics:
    include:
      - orders  # Exact name
      - "orders-*"  # Or pattern

Not Authorized to Access Topic

Error: Not authorized to access topic
Topic: production-orders
User: backup-user

Solutions:

# Check ACLs
kafka-acls --bootstrap-server kafka:9092 \
  --list --topic production-orders

# Add read permission
kafka-acls --bootstrap-server kafka:9092 \
  --add --allow-principal User:backup-user \
  --operation Read --operation Describe \
  --topic production-orders

The topic authorization failed runbook covers the principal the broker logs, literal versus prefixed ACLs, and MSK IAM.

Offset Store Sync Failed

WARN Best-effort offset sync failed after fetch error: ...
Path: s3://bucket/backups/<backup-id>/offsets.db

Causes:

• Storage permission issue
• Disk full
• Network timeout

Solutions:

# Test write access
echo "test" | aws s3 cp - s3://bucket/backups/test.txt
aws s3 rm s3://bucket/backups/test.txt

# Fix: Adjust checkpoint settings
backup:
  checkpoint_interval_secs: 60  # Less frequent
  checkpoint_retries: 3

Restore Errors

Partition Not Available

Error: Produce error: Kafka error: Partition 0 not available for topic test-backup_restored

Causes:

• Topic doesn't exist when using topic_mapping to restore to a new topic name
• Topic was just created but Kafka hasn't finished propagating metadata
• Insufficient replication factor for the cluster

Solutions:

# Fix: Enable auto topic creation (v0.3.0+)
restore:
  create_topics: true
  default_replication_factor: 3  # Match your cluster's requirements
  topic_mapping:
    source-topic: target-topic

# Alternative: Pre-create the topic before restore
kafka-topics --bootstrap-server kafka:9092 \
  --create --topic test-backup_restored \
  --partitions 6 --replication-factor 3

tip

When using topic_mapping, always enable create_topics: true to ensure target topics are created automatically before the restore begins. This was fixed in v0.3.0.

Backup Not Found

Error: Backup not found
Backup ID: production-backup-20241201
Path: s3://bucket/backups

Solutions:

# List available backups
kafka-backup list --path s3://bucket/backups

# Check exact backup ID
kafka-backup list --path s3://bucket/backups --backup-id my-backup-id

Invalid Time Range

Error: Invalid time range for PITR
Start: 1701388800000
End: 1701302400000
Reason: Start time is after end time

Solution:

# Fix: Correct time order
restore:
  time_window_start: 1701302400000  # Earlier time
  time_window_end: 1701388800000    # Later time

Topic Already Exists

Error: Topic already exists with different configuration
Topic: orders
Existing partitions: 12
Backup partitions: 6

Solutions:

# Option 1: Use topic mapping
restore:
  topic_mapping:
    orders: restored-orders

# Option 2: Delete existing topic first
# (manual step before restore)

# Option 3: Allow partition mismatch
restore:
  allow_partition_mismatch: true

Schema ID Not Found

Error: Schema ID not found
Schema ID: 42
Topic: orders

Causes:

• Schema Registry not backed up
• Schema deleted
• Wrong Schema Registry URL

Solutions:

# Check if schema exists
curl https://schema-registry:8081/schemas/ids/42

# List all schemas
curl https://schema-registry:8081/subjects

# Fix: Enable Schema Registry sync (Enterprise)
enterprise:
  schema_registry:
    enabled: true
    url: https://schema-registry:8081

Offset Errors

Consumer Group Not Found

Error: Consumer group not found
Group: order-processor

Solutions:

# List consumer groups
kafka-consumer-groups --bootstrap-server kafka:9092 --list

# Describe group
kafka-consumer-groups --bootstrap-server kafka:9092 \
  --describe --group order-processor

Offset Out of Range

Error: Offset out of range
Topic: orders
Partition: 0
Requested: 50000
Available: 0-45000

Causes:

• Data retention cleaned old messages
• PITR window mismatch
• Wrong offset mapping

Solutions:

# Check available offsets
kafka-run-class kafka.tools.GetOffsetShell \
  --broker-list kafka:9092 \
  --topic orders

# Fix: Use timestamp-based reset
offset_reset:
  strategy: timestamp
  timestamp: 1701388800000  # Within available range

Offset Header Not Found

Error: Original offset header not found
Topic: orders
Expected header: x-original-offset

Causes:

• Backup was taken with include_offset_headers: false (the default is true)
• Headers stripped during processing (e.g. restore.strip_offset_headers: true)

Solutions:

# Fix: Re-backup with headers enabled (the default)
backup:
  include_offset_headers: true

# Alternative: Use timestamp-based offset reset
offset_reset:
  strategy: timestamp

Configuration Errors

Invalid Configuration

Error: Invalid configuration
Field: compression_level
Value: 25
Valid range: 1-22

Solution: Check the Configuration Reference for valid values.

Missing Required Field

Error: Missing required field
Field: source.bootstrap_servers

Solution:

# Fix: Add required field
source:
  bootstrap_servers:
    - kafka:9092

Environment Variable Not Set

Error: Environment variable not set
Variable: KAFKA_PASSWORD
Location: source.security.sasl_password

Solution:

# Set the environment variable
export KAFKA_PASSWORD="your-password"

# Or use a different approach
source:
  security:
    sasl_password: "direct-value"  # Not recommended

Validation Errors

Backup Validation Failed

Error: Backup validation failed
Backup ID: my-backup
Reason: Checksum mismatch in segment-0005.dat

Causes:

• Corrupted storage
• Incomplete upload
• Storage system issue

Solutions:

# Run deep validation
kafka-backup validate \
  --path s3://bucket/backups \
  --backup-id my-backup \
  --deep

# Re-run backup
kafka-backup backup --config backup.yaml --force

Restore Validation Failed

Error: Restore validation failed
Reason: Target cluster unreachable

Solution:

# Validate configuration first
kafka-backup validate-restore --config restore.yaml

# Check target connectivity
kafka-broker-api-versions --bootstrap-server target-kafka:9092

Quick Reference

Error	Likely Cause	First Step
Connection refused	Broker down	Check broker status
SSL handshake failed	Certificate issue	Verify certificates
SASL auth failed	Wrong credentials	Test with console tools
Broken pipe	Idle connection timeout	Enable TCP keepalive
Access denied (S3)	IAM permissions	Check IAM policy
Topic not found	ACL or typo	List topics
Partition not available	Topic doesn't exist	Enable create_topics
Backup not found	Wrong path/ID	List backups
Schema ID not found	SR not backed up	Enable SR sync

Next Steps

• Kafka message too large - RecordTooLargeException when a record batch exceeds the size limits
• Kafka topic authorization failed - TopicAuthorizationException when the broker denies a topic operation
• Performance Issues - Slow backup/restore
• Offset Discontinuity - Offset problems
• Debug Mode - Enable verbose logging
• Support - Get help

Empty Fetch below the captured backup end

If an older backup completes with fewer records than its captured range, upgrade to CLI 0.28.0, Kafka Backup operator 1.8.0, or Strimzi Backup operator 0.6.0. The fixed engine retries successful empty/no-progress Fetch responses for 30 seconds by default, then fails with an incomplete-backup error while retaining its durable prefix. A zero budget fails immediately. This can occur during remote-tier reads; an empty response alone does not establish retention loss.

For Strimzi, check both spec.image and the Helm backupJobs.image override: an older pin keeps the older engine. Update the CRDs before using new typed settings. Re-run the affected backup from its last durable checkpoint, or take a new full backup, then run kafka-backup validate --config backup.yaml --deep and compare source/snapshot offsets, manifest coverage and restored record counts. Expect either recovery of the remaining range or an explicit failed run; a Completed status alone is not enough evidence for an archive created by an older engine.

See configuration and remote-fetch wait semantics, generic operator fields, and Strimzi fields and image overrides. Kafka with KIP-1018 uses broker remote.fetch.max.wait.ms independently of client Fetch wait.