Kafka Backup Cost: The Three Levers That Actually Cut the Bill
Kafka backup cost comes down to one dominant term — stored gigabytes multiplied by how long you keep them — plus smaller charges for requests, data transfer, and the compute that runs the backup job. Three levers cut that dominant term, and they multiply: compression shrinks every byte you store, incremental backups stop you from re-storing bytes you already have, and storage-class lifecycle rules move aged bytes to tiers that cost up to 23x less.
A naive setup — uncompressed daily fulls parked in S3 Standard — routinely costs 10 to 50 times more than an optimized pipeline holding the same data with better recovery guarantees. This guide breaks down where the money goes, walks each lever with real configuration, and prices a worked example at the end.
Backup spend is data × time × storage class. Compress with zstd, back up only new messages with incremental checkpoints, and tier aged backups to archive classes with a lifecycle policy. The levers multiply — together they turn hundreds of dollars a month into tens.
What drives Kafka backup cost
Object storage bills arrive in four categories. For backups, one of them dwarfs the rest.
| Charge category | What drives it | The lever |
|---|---|---|
| Storage (GB-months) | Data volume × retention × storage class | Compression, incrementals, lifecycle tiering |
| Requests (PUT/GET) | Number of segments written and read | Segment sizing |
| Data transfer | Restores, cross-region copies, NAT routing | Same-region buckets, VPC endpoints |
| Compute | How long the backup job runs | Incrementals shrink runtime too |
The monthly bill is roughly: stored GB × class rate + request count × request rate + GB transferred × transfer rate. Storage dominates because it compounds with retention. A 1 TB topic backed up in full every day and kept for 30 days is not 1 TB of storage — it is 30 TB, billed every month, forever.
That compounding is why the levers below matter more than any discount program. Each one attacks a different factor in the multiplication.
Lever 1 — compression: 2–7x off every byte
OSO Kafka Backup compresses segments before they leave the pipeline, with Zstd or LZ4. The ratios below come from the performance-tuning guide:
| Algorithm | Speed | Typical ratio | Fits |
|---|---|---|---|
none | Fastest | 1x | Payloads already compressed |
lz4 | Very fast | 2–3x | Throughput priority |
zstd level 1–3 | Fast | 3–4x | Balanced — the default |
zstd level 4–9 | Moderate | 4–6x | Storage priority |
zstd level 10+ | Slow | 5–7x | Archival |
Be honest about what your data will do. Text-heavy JSON and Avro streams compress well — the example run in the tuning guide reports a 4.9x ratio. Payloads that are already compressed do not: our published 800 GiB benchmark, built from 1 MiB records, achieved 1.67x at zstd level 3. Your ratio lives somewhere in that bracket, and the backup summary prints it after every run, so measure rather than assume.
The configuration is two lines in the config.yaml reference:
backup:
compression: zstd
compression_level: 3 # 1-22; 3 is the balanced default
Level 3 is the right default for scheduled backups. For long-term archival sets, level 9 trades CPU time you spend once for storage you would otherwise pay for over years. Higher levels cost real CPU — the performance tuning guide covers when that tradeoff stops paying.
Lever 2 — incremental backups: pay for churn, not the topic
By default, a one-shot backup starts from the earliest offset every run, producing a full copy each time. That is the 30-copies problem from the cost model. Since v0.13.5, adding an offset_storage section makes runs incremental: each run resumes from the last checkpointed offset and backs up only new messages.
mode: backup
backup_id: "production-incremental"
source:
bootstrap_servers:
- broker-1.kafka.svc:9092
topics:
include:
- "orders"
- "payments"
storage:
backend: s3
bucket: kafka-backups
region: us-west-2
prefix: incremental
backup:
compression: zstd
stop_at_current_offsets: true # Exit after catching up
offset_storage:
db_path: /data/offsets.db # Local SQLite offset store
sync_interval_secs: 30 # Synced to remote storage too
The offset store also syncs to remote storage, so checkpoints survive pod restarts. Manifests merge across runs, which means point-in-time recovery still works over the accumulated history — you are not trading recovery capability for cost.
The cost profile per mode:
| Mode | What each run stores | Cost profile |
|---|---|---|
| Full one-shot | The whole topic, every run | Storage scales with runs × topic size |
| Incremental one-shot | New messages since last checkpoint | Storage scales with churn |
| Incremental snapshot | New messages, then exits at current offsets | Churn-priced, cron-friendly |
| Continuous | New messages, streaming | Churn-priced, near-zero RPO |
For a topic with 2% daily churn, incremental runs store 2% of what daily fulls store. That is a 50x reduction on the growth rate by itself. On Kubernetes, the operator's checkpoint section in the KafkaBackup resource enables the same behavior.
Keep one periodic full backup in the schedule as a clean restore baseline — the worked example below prices exactly that pattern.
Lever 3 — storage-class lifecycle: 23x between hottest and coldest
Everything so far shrinks what you write. Lifecycle policies shrink what you keep. S3 prices per GB-month vary 23x between Standard and Deep Archive (us-east-1 rates as of publication — check the AWS S3 pricing page for current figures):
| Storage class | $/GB-month | Minimum duration | Retrieval | Fits backups that are |
|---|---|---|---|---|
| S3 Standard | $0.023 | None | Instant, free | In the active restore window |
| S3 Standard-IA | $0.0125 | 30 days | Instant, $0.01/GB | Restored rarely, needed fast |
| Glacier Instant | $0.004 | 90 days | Instant, $0.03/GB | Quarterly-access archives |
| Glacier Flexible | $0.0036 | 90 days | Hours (bulk free) | Compliance sets, planned pulls |
| Deep Archive | $0.00099 | 180 days | ~12 hours, $0.02/GB | Multi-year retention, rarely touched |
A lifecycle rule moves backups through the tiers automatically as they age. This pattern comes from our compliance and audit documentation:
{
"Rules": [
{
"ID": "backup-retention",
"Status": "Enabled",
"Transitions": [
{ "Days": 90, "StorageClass": "GLACIER" },
{ "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
],
"Expiration": { "Days": 2555 }
}
]
}
Recent backups stay in Standard where restores are instant. At 90 days, sets move to Glacier Flexible. At a year, Deep Archive. At seven years, they expire. Scope rules by prefix — the storage.prefix field in your backup config exists partly for this.
Two caveats bite teams regularly. First, minimum durations: transitioning or deleting objects before the class minimum (30, 90, or 180 days) incurs early-delete charges, so match transition days to your actual retention schedule — which should come from regulation, not the disk budget, as the Kafka backup compliance guide covers. Second, retrieval time: a Deep Archive restore takes around 12 hours. Backups inside your recovery time objective must stay in a class you can read immediately — price the restore path against your disaster recovery targets, not just the write path.
One boundary worth stating: this is lifecycle on the backup bucket. Kafka's own tiered storage (KIP-405) moves broker segments to object storage for retention economics inside the cluster — it is not a backup, and the two solve different problems.
The quiet line items: transfer and requests
Transfer and request charges rarely dominate, but they are the ones that surprise.
Writes are free; reads are not. Data into S3 costs nothing in transfer. Restores pay GET requests plus egress if the data leaves the region — internet egress starts around $0.09/GB. Keep the bucket in the same region as the cluster, and budget restore egress into your DR plan rather than discovering it during an incident.
The NAT gateway trap. In AWS, backup traffic that reaches S3 through a NAT gateway pays per-GB data processing on every byte. An S3 gateway VPC endpoint carries the same traffic with no per-GB charge. This one routing change has cut real backup bills by more than the storage line.
Requests scale with segment count. Each segment is a PUT ($0.005 per thousand). The segment_max_bytes setting (default 128 MB) controls how many segments a backup produces — larger segments mean fewer PUTs at the price of more memory, per the performance tuning guide.
Egress-free alternatives exist. For S3-compatible backends, Cloudflare R2 charges no data-transfer egress for direct access — storage and operations still bill, but cross-cloud or frequent restore reads do not. The R2 integration guide covers the setup. Azure Blob and GCS follow their own rate cards; the levers in this guide apply the same way.
A worked example: pricing 1 TB of topic data
Take a concrete estate: 1 TB of uncompressed topic data, roughly 2% daily churn (~20 GB of new messages a day), a 30-day operational restore window, and a one-year archive requirement. Rates are us-east-1, as of publication; arithmetic is rounded.
| Line item | Naive: daily fulls, no compression, Standard only | Optimized: zstd (~4x) + incrementals + lifecycle |
|---|---|---|
| Backup pattern | 30 × 1 TB full copies | 1 × 250 GB full + ~5 GB/day increments |
| Warm storage (Standard) | 30,000 GB × $0.023 ≈ $690/mo | ~400 GB × $0.023 ≈ $9/mo |
| Archive (Glacier Flexible) | None — nothing survives 30 days | ~1,700 GB × $0.0036 ≈ $6/mo |
| Requests + transfer | ~$5/mo | ~$1/mo |
| Monthly total | ≈ $695 | ≈ $16 |
The optimized column assumes a measured 4x compression ratio, so the baseline full is 250 GB and daily churn compresses to ~5 GB. Thirty days of increments plus the baseline is about 400 GB warm. A year of aged increments — around 1,700 GB — sits in Glacier Flexible.
Notice what the optimized pipeline buys: it is roughly 40x cheaper and it retains a full year of history where the naive setup kept 30 days. The levers multiply — 4x from compression, ~12x from incremental design at this churn rate, and 6x on every gigabyte that ages into Glacier. That multiplication is how two orders of magnitude happen without touching the recovery guarantees.
Your numbers will differ with churn rate and compression ratio. The arithmetic will not: measure your ratio from the backup summary, count your churn, and the model above prices your estate in ten minutes.
Kafka backup cost anti-patterns
Five patterns account for most oversized bills:
- Daily fulls kept forever. No incremental checkpoints, no lifecycle expiry — storage grows linearly with time and never stops.
- Archiving the entire restore window. Moving everything to Deep Archive looks cheap until a 12-hour retrieval meets a 1-hour RTO. Keep the operational window warm.
- Retention set by disk budget. Retention periods belong to regulation and recovery objectives, not to what the bucket happens to cost this quarter — the compliance guide maps the periods.
- A standby cluster doing backup's job. Replication is not backup: it copies corruption and deletes within milliseconds, and it bills broker compute 24/7. Object storage bills GB-months. The S3 backup runbook shows what the isolated copy costs to set up instead.
- Never measuring the compression ratio. The run summary prints it. Teams budgeting on assumed ratios miss by 2–3x in either direction.
Compression, incremental checkpoints, and the storage prefixes that lifecycle rules key on are all one YAML file. Start with the getting started guide and the config.yaml reference, run one backup, and read your measured ratio from the summary.
Frequently asked questions
How much does it cost to back up Kafka to S3?
Storage dominates: stored gigabytes times the S3 class rate, plus small request and transfer charges. As a reference point, 1 TB of topic data with 2% daily churn costs roughly $700 per month as uncompressed daily fulls in S3 Standard, and under $20 per month with zstd compression, incremental backups, and lifecycle tiering to Glacier.
Does compression reduce Kafka backup storage costs?
Yes, proportionally to the ratio achieved. Zstd typically compresses text-based Kafka payloads 3x to 5x, cutting storage cost by the same factor. Already-compressed payloads see much less benefit, so measure the ratio the backup summary reports rather than assuming one.
Are incremental Kafka backups cheaper than full backups?
Substantially. Incremental runs store only messages produced since the last checkpoint, so storage growth tracks topic churn instead of topic size. A topic with 2% daily churn stores 50x less per run than a daily full backup, and manifests merge across runs so point-in-time recovery is preserved.
Can Kafka backups be stored in Glacier or Deep Archive?
Yes, through S3 lifecycle policies that transition aged backup sets by prefix. Archive classes cost up to 23x less per gigabyte than S3 Standard, but retrievals take hours, so keep backups inside your recovery time objective in a warm class and archive only what you rarely expect to restore.
Does restoring a Kafka backup from S3 cost money?
Data into S3 is free, but restores pay GET requests, retrieval fees for archive classes, and egress if data leaves the region. Same-region restores over an S3 gateway VPC endpoint avoid most transfer charges. Budget the restore path when designing retention, not during an incident.
Wrapping up
The bill is data × time × storage class, and each lever attacks one factor: compression shrinks the data, incremental checkpoints stop time from multiplying it, and lifecycle rules cut the class rate on everything that ages. Applied together they are multiplicative — the difference between a naive and an optimized pipeline is regularly 40x, with better retention on the cheaper side. Price the restore path alongside the write path, and let regulation set retention while the levers set the cost of honoring it.