Kafka Backup Monitoring with Prometheus: Metrics, Alerts, and Dashboards

Kafka backup monitoring starts with one fact: OSO Kafka Backup serves Prometheus text-format metrics on port 8080 while a backup or restore is running. Scrape that endpoint, watch lag, progress, throughput, and errors, and alert when the pipeline stalls — then add a durable success signal, because one-shot jobs take their metrics with them when they exit.

An unmonitored backup fails silently. Nobody notices the stalled consumer or the storage errors, and the failure is discovered months later, during a restore, when it can no longer be fixed. This guide covers the endpoint configuration, the metrics that matter, copy-paste PromQL, two alerts, and how scraping changes on Kubernetes.

Key takeaway

Scrape the metrics endpoint on every backup process, alert on stalled progress and on any error increase, and never rely on in-process metrics for "did the backup run" — a one-shot job's metrics disappear when its pod exits. Use operator outcome counters or Job status for durable last-success alerting.

Why Kafka backup monitoring is not cluster monitoring

Most Kafka teams already have broker dashboards: under-replicated partitions, in-sync replica counts, request latency. None of those panels can answer the questions that matter for data protection. Did last night's backup complete? How far behind the live topics is the continuous backup right now? Are writes to object storage slowing down?

Backup monitoring is also structurally harder than broker monitoring, because backup processes are often short-lived. A broker is always running, so a missing metric means trouble. A one-shot backup job runs for minutes and exits — silence is what success looks like. Without deliberate design, silence is also what total failure looks like.

One distinction keeps the scope of this post honest: monitoring tells you the pipeline is moving; it does not prove the stored data is restorable. That second property needs scheduled validation and restore drills, covered in the Kafka backup verification guide. Run both. A green dashboard over a corrupted backup is worse than no dashboard, because it manufactures confidence.

Enable the metrics endpoint

The metrics server is configured in the same YAML as the backup itself and is enabled by default. A complete metrics block looks like this:

metrics:
  enabled: true
  port: 8080
  bind_address: "0.0.0.0"
  path: "/metrics"
  update_interval_ms: 500
  keep_alive_seconds: 60
  max_partition_labels: 100

Field	Default	What it controls
enabled	true	Start the HTTP metrics server
port	8080	Listener port
bind_address	0.0.0.0	Listener address
path	/metrics	Metrics endpoint path
update_interval_ms	500	Metrics recalculation interval
keep_alive_seconds	0	Keep serving after a one-shot operation completes
max_partition_labels	100	Cap on unique per-partition label sets; 0 is unlimited

Two fields deserve attention. keep_alive_seconds exists because Prometheus scrapes on an interval: a job that finishes between scrapes would otherwise vanish before its final values are collected. Set it to at least twice your scrape interval — with a 30-second scrape, keep_alive_seconds: 60 guarantees the completed totals survive to a scrape.

max_partition_labels bounds cardinality. It caps the unique topic/partition/backup_id label sets emitted by the per-partition lag gauges, so a backup covering thousands of partitions cannot flood Prometheus with series. Aggregate lag and snapshot progress always include every partition, even those the cap excludes, so the totals stay trustworthy.

Start the backup and smoke-test the endpoint:

kafka-backup backup --config backup.yaml
curl --fail http://127.0.0.1:8080/metrics

Then point Prometheus at it:

scrape_configs:
  - job_name: kafka-backup
    static_configs:
      - targets: ["kafka-backup-host:8080"]
    metrics_path: /metrics
    scrape_interval: 30s

The full field reference, including cardinality behavior, is in the metrics reference, and the configuration reference shows the block in the context of a complete backup config.

The metrics that matter

The endpoint publishes several dozen metric families. Organize them by the operational question each one answers, and the dashboard designs itself.

Question	Metric	Type
How far behind is the continuous backup?	kafka_backup_lag_records_sum, kafka_backup_lag_records_max	Gauge
How stale is a specific partition?	kafka_backup_lag_records, kafka_backup_lag_seconds	Gauge
How close is a finite backup to done?	kafka_backup_snapshot_records_target, kafka_backup_snapshot_records_remaining	Gauge
Is data actually flowing?	kafka_backup_records_total, kafka_backup_bytes_total	Counter
How fast is it flowing right now?	kafka_backup_throughput_records_per_sec	Gauge
Is anything failing?	kafka_backup_errors_total, kafka_backup_retries_total	Counter
Is object storage healthy?	kafka_backup_storage_write_latency_seconds, kafka_backup_storage_errors_total	Histogram, Counter
When did a partition last commit?	kafka_backup_last_successful_commit	Gauge

The error counter is labelled by category, which turns a generic "errors happened" panel into a diagnosis: broker_connection, consumer_timeout, storage_io, auth, and quota each point at a different on-call action. Storage metrics carry a backend label, so S3 latency and filesystem latency chart separately.

One version note: kafka-backup releases before v0.15.12 emitted counters with a doubled suffix, such as kafka_backup_records_total_total. If your queries return nothing on an older version, append the extra _total.

PromQL for dashboards

Five queries cover the core panels. Finite backup progress as a percentage:

100 * (
  1 - kafka_backup_snapshot_records_remaining
      / clamp_min(kafka_backup_snapshot_records_target, 1)
)

Aggregate lag, without summing thousands of per-partition series:

kafka_backup_lag_records_sum

Throughput over the last five minutes:

rate(kafka_backup_records_total[5m])

99th-percentile storage write latency, split by backend and operation:

histogram_quantile(
  0.99,
  sum by (le, backend, operation) (
    rate(kafka_backup_storage_write_latency_seconds_bucket[5m])
  )
)

Errors in the last five minutes, grouped for triage:

sum by (backup_id, error_type) (
  increase(kafka_backup_errors_total[5m])
)

For a working local stack — Prometheus and Grafana in Docker Compose, wired to a running backup — follow the monitoring setup guide and build these panels against live data.

Alerts that catch real failures

Two alert rules catch the failure modes that matter most in practice: a backup that stops making progress, and a backup that starts erroring.

kafka-backup-alerts.yaml

groups:
  - name: kafka-backup
    rules:
      - alert: KafkaBackupStalled
        expr: >-
          increase(kafka_backup_records_total[10m]) == 0
          and kafka_backup_lag_records_sum > 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: Kafka backup is not making progress

      - alert: KafkaBackupErrors
        expr: increase(kafka_backup_errors_total[5m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: Kafka backup emitted errors

The stall rule is deliberately two-sided: zero new records is only a problem while lag exists. A continuous backup that has fully caught up with an idle topic is healthy, not stalled.

Now the trap. Both rules watch in-process metrics, and in-process metrics are ephemeral: when a one-shot job's pod exits, its series go stale and disappear. An absent()-based rule on these metrics cannot distinguish "backup finished cleanly an hour ago" from "backup never started." Do not build your "did the backup run today" alert on them. Use the Kubernetes Job or custom resource status, an operator outcome counter (next section), or an external batch-success signal with its own lifetime. Treat the in-process metrics as a flight recorder, not a heartbeat.

Monitoring backups on Kubernetes

On Kubernetes the scraping model depends on which operator runs the backups, because the two operators expose metrics differently.

	OSO Kafka Backup Operator	Strimzi Backup Operator
Where backups run	Inside the operator process	Separate Job pods
Endpoints to scrape	One Service (:8080/metrics)	Controller (:9090) plus each Job pod (:8080)
Discovery	ServiceMonitor	ServiceMonitor and Job PodMonitor
Live progress metrics	Not exposed	Yes, from Job pods
Durable outcome counters	Yes (kafka_backup_operator_*)	Use CR status

The OSO Kafka Backup Operator runs the backup engine in its own long-lived process and publishes controller and completed-operation metrics from a single Service. That includes the durable counters that solve the last-success problem cleanly — for example, failed backups over the last hour:

sum by (namespace, name) (
  increase(kafka_backup_operator_backups_total{outcome="failure"}[1h])
)

The Strimzi Backup Operator creates a separate pod per backup or restore Job, so live progress metrics come from those pods. Enable both its ServiceMonitor and its Job PodMonitor: the ServiceMonitor alone collects controller health but no backup progress. For short-lived Job pods, a PodMonitor is the right tool — a ServiceMonitor cannot discover a Job pod unless a matching Service also exists, and Job pods rarely have one.

Kafka backup monitoring checklist

• metrics block enabled on every backup and restore config
• keep_alive_seconds set to at least twice the Prometheus scrape interval for one-shot jobs
• Scrape job (or ServiceMonitor/PodMonitor) deployed and targets showing as up
• Dashboard panels for lag, snapshot progress, throughput, storage latency, and errors
• KafkaBackupStalled and KafkaBackupErrors alerts routed to the platform on-call
• Durable "backup succeeded" signal independent of process lifetime — operator counters, Job status, or a batch-success metric
• max_partition_labels left at a bounded value unless partition count is known and small
• Scheduled deep validation and restore drills alongside the dashboards

The last item is the one teams skip. Monitoring and verification answer different questions, and the operational excellence guidance treats them as separate review items for good reason: a backup can scrape clean, chart green, and still fail a restore.

Backup failures are inevitable; silent backup failures are a choice. Enable the endpoint today — it is one YAML block. Adopt the two alerts this week. Then pair the dashboards with scheduled verification, so that the next time a panel goes red you are fixing a backup, not explaining a lost dataset. The complete metric list and label sets are in the metrics reference.

Frequently asked questions

How to backup Kafka?

Define a backup config listing the source topics and a storage backend, then run kafka-backup backup with that config. Keep the metrics block enabled so Prometheus can scrape lag, progress, throughput, and errors from port 8080 while the backup runs, and alert on stalls so failures surface immediately.

How to backup Kafka to S3?

Set the storage backend to S3 in the backup config and run kafka-backup backup. Once monitoring is enabled, watch kafka_backup_storage_write_latency_seconds and kafka_backup_storage_errors_total with the backend label to catch S3 throttling, credential, or connectivity problems before they stall the backup.

How to take backup of Kafka topic?

Scope the backup config to the topics you need with an include list, then run the backup. Per-topic monitoring comes free: kafka_backup_lag_records carries a topic label, so a dashboard can show exactly how far behind the backup is for each included topic and alert if one stops progressing.