GDPR Erasure (restore-time key suppression)

Enterprise 0.4.0+ · licence feature erasure · included in the 14-day built-in trial

A backup taken before an erasure request still holds the erased records. This feature re-applies the erasure while restoring: every record whose key is on your erasure register is produced as a tombstone or skipped, and the restore output records which register was applied (SHA-256) and how many records it affected. It is the "erasure on restore" half of the posture described in Retention and Erasure.

Configuration

restore.yaml

mode: restore
backup_id: gdpr-demo
target:
  bootstrap_servers: ["kafka:9092"]
  topics:
    include: ["customers"]
storage:
  backend: s3
  bucket: kafka-backups

enterprise:
  erasure:
    suppression:
      keys_file: /etc/kafka-backup/erasure/suppressed-keys.txt
      topics: ["customers", "customer-*"]   # optional; source-topic globs, default: all
      match: exact                           # exact | prefix | regex
      on_match: tombstone                    # tombstone | drop

Key	Default	Description
keys_file	required	An absolute path is read from the local filesystem; anything else is a key inside the configured backup storage (so the register can live next to the backups).
topics	all	Source-topic names or globs the register applies to.
match	exact	exact byte-for-byte; prefix — the key starts with the entry; regex — the entry is a regular expression evaluated over the raw key bytes.
on_match	tombstone	What happens to a matching record (below).

Keys file format

One entry per line, UTF-8:

# Erasure register — comments and blank lines are ignored
U2                       # plain key
base64:AAEC/w==          # binary key
orders<TAB>U9            # entry scoped to one source topic

Records with a null key never match. A malformed line (bad base64:, invalid regex) fails the restore and names the line.

tombstone vs drop

• tombstone (default) — the record is produced with its key and a null value. Use it for compacted topics: compaction retires every earlier copy of the key, and source→target offsets stay aligned one-to-one.
• drop — the record is not produced. Use it for non-compacted topics where the record must not exist on the target. Consumer-group offset mapping stays exact: each dropped source offset maps to the next surviving record's target offset.

Pairing them the other way round leaves data behind: drop on a compacted topic keeps older values of the key; tombstone on a plain topic leaves the earlier records intact.

What the restore reports

Records suppressed: 2 (0 dropped, 2 tombstoned; keys_file=/etc/kafka-backup/erasure/suppressed-keys.txt, entries=1, sha256=d4f0f9e4…a9fa)

three-phase-restore prints the same line in its Phase 2 block. validate-restore prints an Erasure Suppression (enterprise) block (keys file, entries, SHA-256, match, on_match, topics) and, with --format json, adds it under enterprise.erasure_suppression. The dry run loads and reports the register but does not evaluate records; counts appear on the real restore.

Fail-loud guarantees

The one thing this feature must never do is silently restore erased data. So:

• If enterprise.erasure.suppression is configured and the erasure feature is not licensed, the command is refused: Feature 'erasure' is not licensed … refused rather than run without it.
• If the keys file is missing, unreadable or malformed, the command is refused before any engine is constructed — no topic is created, nothing is restored. validate-restore fails the same way, so a broken register is caught in the dry run.
• The register's digest and the match configuration are folded into the restore checkpoint: resuming an interrupted restore with a changed register warns and restarts rather than mixing lists.

Evaluating it

The Enterprise binary ships a 14-day auto-trial that unlocks every feature, including erasure — no licence file needed. The demo cli/gdpr-erasure runs nine asserted proof points against Kafka + MinIO: backup before the request, tombstone restore, drop restore, the missing-register refusal, the licence-gate refusal (KAFKA_BACKUP_NO_TRIAL=1) and the validate-restore summary.

Roadmap

In-place redaction of existing backup sets and crypto-shredding via per-set keys follow, designed so a set is declared immutable (cyber-recovery) or erasable, never both — see the enterprise erasure issue.