Kubernetes Operator

The OSO Kafka Backup Operator provides Kubernetes-native management of Kafka backup and restore operations.

Overview

The operator extends Kubernetes with Custom Resource Definitions (CRDs) that allow you to:

• Schedule automated backups
• Enforce opt-in backup retention
• Perform point-in-time restores
• Manage consumer group offsets
• Snapshot and rollback consumer positions
• Validate backups and generate evidence reports

Custom Resource Definitions

CRD	Short Name	Description
KafkaBackup	kb	Backup Kafka topics to object storage
KafkaRestore	kr	Restore data from backups
KafkaOffsetReset	kor	Reset consumer group offsets
KafkaOffsetRollback	korb	Snapshot and rollback consumer offsets
KafkaBackupValidation	kbv	Validate backups and produce evidence reports

Current release: v1.3.0

Operator v1.3.0 uses kafka-backup-core v0.19.2 and adds KafkaRestore.spec.includeOriginalOffsetHeader / stripOffsetHeaders. It includes the opt-in retention support introduced in v1.1.0 and the following backup-mode behavior introduced in v1.0.0.

Backup mode behavior

Operator v1.0.0 aligns the CRDs with kafka-backup-core v0.12.0.

• checkpoint.enabled no longer implies continuous backup mode. Set continuous: true explicitly for streaming backups.
• Scheduled point-in-time backups should set stopAtCurrentOffsets: true.
• KafkaBackup supports consumer group snapshots with consumerGroupSnapshot: true.
• KafkaRestore supports autoConsumerGroups, purgeTopics, produceAcks, produceBatchSize, produceTimeoutMs, and per-topic repartitioning.
• S3-compatible storage supports pathStyle and allowHttp.
• kafkaCluster.connection supports TCP tuning, including connectionsPerBroker, across backup, restore, validation, offset reset, and rollback resources.
• Backup and restore pod templates support hostAliases for split-DNS or private endpoint storage access.

Retention

• KafkaBackup.spec.retention supports opt-in backup-set pruning for PVC/local, S3/S3-compatible, and Azure Blob Storage.
• Retention is disabled by default and supports dry-run reporting before active deletion.
• GCS users should continue to use bucket lifecycle policies for retention.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Kubernetes Cluster                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │                   Kafka Backup Operator                              │   │
│   │                                                                      │   │
│   │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │   │
│   │  │   Backup     │  │   Restore    │  │   Offset     │              │   │
│   │  │  Controller  │  │  Controller  │  │  Controller  │              │   │
│   │  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘              │   │
│   │         │                 │                 │                       │   │
│   └─────────┼─────────────────┼─────────────────┼───────────────────────┘   │
│             │                 │                 │                            │
│             ▼                 ▼                 ▼                            │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │                      Custom Resources                                │   │
│   │                                                                      │   │
│   │   KafkaBackup      KafkaRestore    KafkaOffsetReset                 │   │
│   │   KafkaOffsetRollback                                                │   │
│   │                                                                      │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
│                                                                              │
│   ┌──────────────────┐            ┌──────────────────┐                      │
│   │  Kafka Cluster   │            │  Object Storage  │                      │
│   │                  │◀──────────▶│  (S3/Azure/GCS)  │                      │
│   └──────────────────┘            └──────────────────┘                      │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Features

Scheduled Backups

Create automated backup schedules using cron expressions:

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: hourly-backup
spec:
  schedule: "0 0 * * * * *"  # Every hour; cron format includes seconds
  stopAtCurrentOffsets: true
  kafkaCluster:
    bootstrapServers:
      - kafka:9092
  topics:
    - orders
    - payments
  storage:
    storageType: s3
    s3:
      bucket: kafka-backups
      region: us-west-2
      credentialsSecret:
        name: s3-credentials

Point-in-Time Recovery

Restore data to any specific moment:

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaRestore
metadata:
  name: pitr-restore
spec:
  backupRef:
    name: production-backup
  pitr:
    endTime: "2024-12-01T12:00:00Z"
  kafkaCluster:
    bootstrapServers:
      - kafka:9092

Offset Management

Manage consumer group offsets across clusters:

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaOffsetReset
metadata:
  name: reset-after-restore
spec:
  resetStrategy: from-mapping
  consumerGroups:
    - order-processor
  kafkaCluster:
    bootstrapServers:
      - kafka:9092

GitOps Integration

All resources are declarative and version-controllable:

# backup.yaml - Store in Git
apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: production-backup
spec:
  # ...

Pod Template Customisation

Use spec.template.pod.hostAliases when backup or restore pods must resolve an external name to an internal IP address, such as an S3-compatible endpoint reached over VPN or split DNS.

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: private-s3-backup
spec:
  # ...
  template:
    pod:
      hostAliases:
        - ip: "10.10.0.5"
          hostnames:
            - s3.internal
            - minio.internal

Quick Start

1. Install the Operator

# Add Helm repository
helm repo add oso https://osodevops.github.io/helm-charts/
helm repo update

# Install operator
helm install kafka-backup-operator oso/kafka-backup-operator \
  --namespace kafka-backup \
  --create-namespace

2. Create a Backup

apiVersion: kafka.oso.sh/v1alpha1
kind: KafkaBackup
metadata:
  name: my-first-backup
  namespace: kafka-backup
spec:
  kafkaCluster:
    bootstrapServers:
      - kafka.default.svc:9092
  topics:
    - my-topic
  storage:
    storageType: s3
    s3:
      bucket: my-kafka-backups
      region: us-west-2
      credentialsSecret:
        name: s3-credentials
  stopAtCurrentOffsets: true

kubectl apply -f backup.yaml

3. Monitor Progress

kubectl get kafkabackup my-first-backup -o yaml

# Or watch status
kubectl get kafkabackup -w

API Reference

API Group and Version

• API Group: kafka.oso.sh
• API Version: v1alpha1

Resource Types

Kind	Description
KafkaBackup	Defines backup configuration and schedule
KafkaRestore	Defines restore operation
KafkaOffsetReset	Defines offset reset operation
KafkaOffsetRollback	Defines offset snapshot/rollback
KafkaBackupValidation	Defines backup validation and evidence reporting

Comparison with CLI

Feature	CLI	Operator
Scheduled backups	External scheduler (cron)	Built-in schedule reconciliation
Declarative config	YAML files	Kubernetes CRDs
GitOps	Manual	Native
Monitoring	Prometheus endpoint	ServiceMonitor
HA deployment	Manual	Built-in
Secret management	Environment variables	Kubernetes Secrets

Prerequisites

• Kubernetes 1.26+
• Helm 3.0+ (for installation)
• Access to Kafka cluster
• Access to object storage (S3, Azure, GCS)

Next Steps

• Installation - Install the operator
• Configuration - Configure Helm values
• KafkaBackup CRD - Backup resource reference
• Scheduled Backups Guide - Set up automated backups