Prometheus Metrics

Operator v0.4.0 exposes two independent classes of metrics. They must be scraped separately:

Target	Default endpoint	Discovery
Operator controller	:9090/metrics	ServiceMonitor or pod annotations
Backup/restore Job pod	:8080/metrics	PodMonitor

The operator does not proxy Job metrics. Enabling only serviceMonitor does not collect backup progress.

Enable Prometheus Operator discovery

The ServiceMonitor and PodMonitor require the Prometheus Operator CRDs.

values.yaml

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s
  jobPodMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s

helm upgrade --install strimzi-backup-operator \
  oso-devops/strimzi-backup-operator \
  --namespace kafka \
  --create-namespace \
  --version 0.4.0 \
  --values values.yaml

The Job PodMonitor selects pods labelled app.kubernetes.io/managed-by=kafka-backup-operator and kafkabackup.com/metrics=enabled. Its default namespaceSelector.any: true is intentional: custom resources can live outside the Helm release namespace.

If you set a custom spec.metrics.path, also set metrics.jobPodMonitor.path to the same value.

Configure a backup or restore

Metrics are enabled by default for Job pods. For one-shot operations, keep the endpoint alive long enough for Prometheus to scrape the final values. A useful minimum is twice the scrape interval:

spec:
  metrics:
    enabled: true
    port: 8080
    bindAddress: "0.0.0.0"
    path: /metrics
    updateIntervalMs: 500
    keepAliveSeconds: 60
    maxPartitionLabels: 100

maxPartitionLabels limits unique topic/partition/backup_id label sets, not metric updates. Set it to 0 only if unlimited per-partition cardinality is acceptable. Aggregate lag and snapshot progress remain available when the per-partition limit is reached.

Operator metrics

The controller endpoint currently publishes these populated metric families:

Metric	Labels	Description
strimzi_backup_operator_build_info	version	Running operator build
strimzi_backup_operator_reconciliations_total	controller, result	Reconciliations by controller and result
strimzi_backup_operator_reconciliation_duration_seconds	controller, result	Reconciliation duration histogram
strimzi_backup_operator_engine_image_info	image, source	1 for the default kafka-backup image Jobs run when spec.image is unset; source is compiled-in or env (Helm backupJobs.image). Operator v0.2.25+
strimzi_backup_operator_engine_version_unsupported_total	controller	Reconciliations that resolved an engine image older than the minimum supported kafka-backup release (see Compatibility). Operator v0.2.25+

Check it without Prometheus:

kubectl port-forward -n kafka \
  svc/strimzi-backup-operator-metrics 9090:9090
curl --fail http://127.0.0.1:9090/metrics

Job progress metrics

The Job endpoint uses the kafka-backup runtime metrics. Useful v0.19.0 families include:

Metric	Description
kafka_backup_lag_records	Current lag for an admitted topic/partition series
kafka_backup_lag_records_sum	Current lag summed across all partitions
kafka_backup_snapshot_records_target	Records in the captured finite-backup offset range
kafka_backup_snapshot_records_remaining	Records left before the finite backup completes
kafka_backup_records_total	Records backed up
kafka_backup_bytes_total	Uncompressed bytes backed up
kafka_backup_errors_total	Errors by backup and category
kafka_restore_progress_percent	Restore completion percentage

Example progress query:

100 * (
  1 - kafka_backup_snapshot_records_remaining
      / clamp_min(kafka_backup_snapshot_records_target, 1)
)

Use CR status for durable success/failure history. A completed one-shot Job's in-process metrics disappear when its pod exits, even when Prometheus scraped the final sample successfully.