Skip to main content

Retention and Erasure

A backup taken before a record was deleted still contains that record, and a restore reinstates it. For personal data this is the question every data protection officer asks: what happens to an erasure request once the data is in a backup? This guide gives the answer kafka-backup supports today, in the order regulators expect it: bounded retention first, erasure re-applied on restore second, and removal from the archive itself as the enterprise layer.

The regulatory position​

Both the UK ICO's right-to-erasure guidance and the EDPB's 2025/26 coordinated enforcement report on Article 17 accept that personal data may remain in a passive, time-bounded backup that is put beyond use, provided that:

  1. the backup is not used to make decisions about, or otherwise affect, the individual;
  2. it is deleted on its normal rotation — retention is bounded and documented;
  3. the erasure is re-applied if the backup is ever restored, so the data never re-enters an active system.

kafka-backup implements the mechanics for all three. What you document in your DPIA is the retention window, the restore procedure that re-applies erasure, and the evidence each restore produces.

Layer 1 — bounded retention (prune, backup.retention)​

Backup sets grow: incremental and continuous backups append segments under one stable backup_id and rewrite manifest.json on every run. Two safe ways to bound them, both available since kafka-backup 0.21.0:

Prune on a schedule — plan-only by default, --execute to delete:

kafka-backup prune --config backup.yaml --older-than 30d            # plan
kafka-backup prune --config backup.yaml --older-than 30d --execute  # delete

Retention in the backup config — applied at the end of every run:

backup:
  retention:
    max_age: 30d        # delete segments older than this
    keep_segments: 1    # never remove a partition's newest segment

Either way the manifest is rewritten before any object is deleted, the removed offset range is recorded as a pruned range (so validate stays green and describe shows what was removed and when), and a later run can never resurrect a pruned segment. The resume position of an incremental set is protected: segments the next run still needs are never pruned.

Do not use bucket lifecycle rules on incremental sets

An age-based S3/Azure/GCS lifecycle rule deletes segments the manifest still references and never expires the manifest itself (it is rewritten every run). The result is a backup that reports healthy and fails at restore. Lifecycle rules are only safe when every run has its own backup_id (for example backup_id: "daily-${DATE}"). See the storage guide.

A runnable walk-through is in the demos repository: cli/retention-prune.

Layer 2 — erasure re-applied on restore (Enterprise)​

The restore engine has a per-record filter hook (Keep | Drop | Tombstone, kafka-backup 0.21.0+). The Enterprise binary uses it for restore-time key suppression: an erasure register — one erased record key per line — is re-applied while restoring, so records for those keys are produced as tombstones (compacted topics) or skipped (plain topics). The restore output reports the list's SHA-256 and the counts, which is the audit evidence a DPO needs:

Records suppressed: 2 (0 dropped, 2 tombstoned; keys_file=/etc/kafka-backup/erasure/suppressed-keys.txt, entries=1, sha256=d4f0…a9fa)

If the feature is not licensed, or the list cannot be read, the restore is refused — it never silently runs without the list. Configuration, keys-file format and the tombstone-versus-drop rule are on the Enterprise erasure page; the demo cli/gdpr-erasure runs the whole flow under the built-in 14-day trial.

Without the Enterprise feature, the equivalent manual procedure is: keep an erasure register (key + timestamp); after any restore and before consumers are unfrozen, re-produce tombstones for every key erased since the backup was taken. Compaction then removes them again.

Layer 3 — removal from the archive (Enterprise roadmap)​

In-place redaction of an existing backup set (rewriting the affected segments and re-signing the evidence) and crypto-shredding via per-set data-encryption keys are the next erasure layers. They are designed so that a backup set is declared immutable (cyber-recovery, WORM) or erasable — never both. Track progress on the enterprise erasure issue.

What to write in the DPIA​

  • Retention window for backups (prune --older-than / retention.max_age), and that lifecycle rules are not used on incremental sets.
  • Restore procedure: restores of topics containing personal data run with the erasure register applied (enterprise.erasure.suppression); the register is version-controlled and its SHA-256 is recorded with each restore ticket.
  • Evidence: the restore output (and validate-restore --format json, which carries the same summary under enterprise.erasure_suppression).
  • Access control: backups live in object storage with its own IAM; the backup is passive and never read by application code.