Skip to main content

Monitoring Setup

OSO Kafka Backup serves Prometheus metrics while a backup or restore is running. This guide creates a small local Prometheus and Grafana stack and then shows the Kubernetes setup.

Enable the endpoint​

Add a metrics block to the backup or restore configuration:

mode: backup
backup_id: production-backup

source:
  bootstrap_servers: ["kafka:9092"]
  topics:
    include: ["orders", "events"]

storage:
  backend: filesystem
  path: /var/lib/kafka-backup/data

backup:
  stop_at_current_offsets: true

metrics:
  enabled: true
  port: 8080
  bind_address: "0.0.0.0"
  path: /metrics
  update_interval_ms: 500
  keep_alive_seconds: 60
  max_partition_labels: 100

keep_alive_seconds lets Prometheus collect final values after a one-shot operation completes. Set it to at least twice your scrape interval.

kafka-backup backup --config backup.yaml
curl --fail http://127.0.0.1:8080/metrics

Local Docker Compose stack​

Download compose.yaml and prometheus.yml, or create the following files. First, compose.yaml:

services:
  prometheus:
    image: prom/prometheus:v2.54.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    extra_hosts:
      - host.docker.internal:host-gateway

  grafana:
    image: grafana/grafana:11.2.0
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_USERS_ALLOW_SIGN_UP: "false"
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - prometheus

volumes:
  prometheus-data: {}
  grafana-data: {}

Create prometheus.yml beside it:

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: kafka-backup
    metrics_path: /metrics
    static_configs:
      - targets: ["host.docker.internal:8080"]

Start the stack:

docker compose up -d
docker compose ps

On Linux, host-gateway maps host.docker.internal to the host. Docker Desktop provides the same hostname on macOS and Windows.

In Grafana, add http://prometheus:9090 as a Prometheus data source.

Finite backup progress:

100 * (
  1 - kafka_backup_snapshot_records_remaining
      / clamp_min(kafka_backup_snapshot_records_target, 1)
)

Aggregate lag:

kafka_backup_lag_records_sum

Backup throughput:

rate(kafka_backup_records_total_total[5m])

Storage write latency:

histogram_quantile(
  0.99,
  sum by (le, backend, operation) (
    rate(kafka_backup_storage_write_latency_seconds_bucket[5m])
  )
)

Errors:

sum by (backup_id, error_type) (
  increase(kafka_backup_errors_total_total[5m])
)

See the metrics reference for the complete list and label sets.

Kubernetes​

The two operators expose metrics differently:

  • The OSO Kafka Backup Operator runs the core library in its own process and exposes controller/completion metrics from one Service.
  • The Strimzi Backup Operator creates separate Job pods. Enable both its ServiceMonitor and Job PodMonitor to collect controller health and live backup progress.

For short-lived Jobs, a PodMonitor is preferable to a ServiceMonitor because it discovers the pods directly.

Alerts​

kafka-backup-alerts.yaml
groups:
  - name: kafka-backup
    rules:
      - alert: KafkaBackupStalled
        expr: >-
          increase(kafka_backup_records_total_total[10m]) == 0
          and kafka_backup_lag_records_sum > 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: Kafka backup is not making progress

      - alert: KafkaBackupErrors
        expr: increase(kafka_backup_errors_total_total[5m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: Kafka backup emitted errors

Metrics from a one-shot process are ephemeral. Use Job/CR status or an external batch-success metric for durable last-success alerting.