Back up Cloudera Kafka through its existing listeners

Kafka backup for Cloudera and Hortonworks environments​

Cloudera-managed Kafka exposes standard Kafka client listeners. OSO Kafka Backup uses those listeners without a Cloudera Manager plugin. It reads selected topics and writes records, topic configuration, and consumer offset metadata to independent storage.

That copy can use S3, Azure Blob, GCS, or filesystem storage. It remains available when the source brokers do not. You can restore selected topics to the original cluster or another reachable Kafka deployment.

Use the security protocol configured for your client listener. Cloudera documents TLS, SASL/PLAIN, and Kerberos client patterns in its Kafka client configuration reference. OSO maps those settings into its YAML configuration.

Connect OSO Kafka Backup to Cloudera​

The first example uses a TLS-protected SASL/PLAIN listener. The second uses GSSAPI with a keytab. GSSAPI requires an OSO binary built with the gssapi feature and is absent from the default release binary and Docker image.

mode: backup
backup_id: cloudera-prod-daily

source:
bootstrap_servers:
  - kafka-1.cdp.internal:9093
  - kafka-2.cdp.internal:9093
security:
  security_protocol: SASL_SSL
  sasl_mechanism: PLAIN
  sasl_username: ${KAFKA_BACKUP_USER}
  sasl_password: ${KAFKA_BACKUP_PASSWORD}
  ssl_ca_location: /etc/kafka/ca.crt
topics:
  include:
    - orders
    - payments

storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: cloudera/production

backup:
compression: zstd
include_offset_headers: true
consumer_group_snapshot: true

mode: backup
backup_id: cloudera-prod-daily

source:
bootstrap_servers:
  - kafka-1.cdp.internal:9093
  - kafka-2.cdp.internal:9093
security:
  security_protocol: SASL_SSL
  sasl_mechanism: GSSAPI
  sasl_kerberos_service_name: kafka
  sasl_keytab_path: /etc/kafka-backup/client.keytab
  sasl_krb5_config_path: /etc/krb5.conf
  ssl_ca_location: /etc/kafka/ca.crt
topics:
  include:
    - orders
    - payments

storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: cloudera/production

backup:
compression: zstd
include_offset_headers: true

kafka-backup backup --config cloudera-backup.yaml

Keep recovery separate from the Cloudera cluster​

Broker replication protects Kafka availability during ordinary node failures. It does not preserve an earlier state after a valid delete or corrupt producer write. Replication is not backup because every current replica can receive the same change.

OSO stores backup data beyond that broker failure boundary. Restore controls can select topics, remap names, and stop at a precise millisecond timestamp. Consumer offset metadata helps applications resume from the restored position.

The disaster recovery guide shows how independent storage fits cluster recovery. The security guide covers certificates, keytabs, and file permissions for production use.