From Kafka PoC to Production: The Checklist
A Kafka PoC and a Kafka production cluster frequently run the exact same broker version, on the exact same defaults. The gap between them isn't code — it's everything nobody explicitly configured. Replication factor, backup, DR targets, monitoring, security, cluster topology, and retention all ship with defaults that are harmless for a three-week proof of concept and dangerous for a system carrying real traffic.
This is the checklist for closing that gap. It doesn't teach you anything new — it maps each silent default to the guide that already covers the fix.
Nothing about a Kafka PoC "becomes" production-ready through more traffic or more uptime. It becomes production-ready when someone replaces defaults — default.replication.factor, min.insync.replicas, backup configuration, monitoring, auth — with explicit decisions. That's a checklist, not a milestone.
The default that causes the most damage: replication factor 1
Most Kafka PoCs never explicitly create a topic. They produce to a topic name that doesn't exist yet, auto.create.topics.enable (default: true) creates it on the fly, and default.replication.factor (default: 1) decides how many copies of that data exist. One.
# What auto-creation silently gives you
kafka-topics --bootstrap-server localhost:9092 --describe --topic orders
# Topic: orders PartitionCount: 1 ReplicationFactor: 1
# What production requires — explicit, not left to defaults
kafka-topics --bootstrap-server localhost:9092 --create \
--topic orders --partitions 6 --replication-factor 3 \
--config min.insync.replicas=2
A single-replica topic loses its data permanently the moment its one broker goes down — not degraded, not slow, gone. This is the single most common PoC-to-production gap, because it requires no action to have and no error message to notice. The replication factor guide covers choosing partition count and replication factor deliberately; the in-sync replicas guide covers why min.insync.replicas — which also defaults to 1 and enforces nothing — has to move to 2 alongside replication factor 3 for the durability contract to mean anything. Since Kafka 3.0, producer clients default to acks=all with idempotence enabled, so the client side of that contract is often already right by default. The broker-side floor is the piece a PoC never sets.
The checklist
None of these gaps show up in a load test. A PoC that handles ten times its expected throughput can still have zero backup coverage, no DR runbook, and a replication factor of 1 — throughput and resilience are unrelated properties, and testing one tells you nothing about the other. Work through the table below once, item by item, rather than assuming volume testing already covered it.
| Area | PoC default | Production requirement | Owning guide |
|---|---|---|---|
| Replication factor | 1 (via default.replication.factor) | 3, explicitly set per topic | Replication factor guide |
| Minimum in-sync replicas | 1 (enforces nothing) | 2, paired with acks=all | In-sync replicas guide |
| Topic creation | Auto-created via auto.create.topics.enable | Explicit --create with partitions and replication set | Replication factor guide |
| Backup / recoverability | None configured | Scheduled backup with point-in-time recovery | Kafka backup strategies |
| DR plan and targets | None written down | RTO/RPO targets per topic tier, tested runbook | Kafka disaster recovery, RTO/RPO planning |
| Cluster topology | Single-node KRaft, no quorum | Multi-node controller quorum | KRaft Docker Compose, Controller quorum |
| Monitoring / alerting | None — failures surface when someone notices | Automated backup and cluster health alerting | Backup monitoring |
| Security | No TLS/SASL configured | mTLS or SASL_SSL on every Kafka connection | Security setup guide |
| Retention and cost | 7-day default, copied verbatim | Deliberate retention per topic tier | Backup cost guide |
Fixing the replication and ISR rows is a durability fix, not a backup — a correctly replicated topic still loses data to a bad deploy, an accidental delete, or a bug that writes garbage to every replica at once. Replication and backup answer different failure modes; production needs both, and a PoC typically has neither configured deliberately.
Backup and DR: the two things a PoC almost never has
A working PoC proves the pipeline moves data correctly. It says nothing about whether that pipeline recovers from broker loss, a bad deploy, or a deleted topic, because nobody has tried to recover it yet — there's usually nothing configured to recover from.
The Kafka backup strategies and Kafka disaster recovery pillar posts cover the mechanics; if you're starting from zero, read those before anything else on this list. The one decision worth making before either: write down RTO and RPO targets per topic tier. The RTO/RPO planning guide makes the case that these targets have to exist before an incident forces the conversation, not during one.
Cluster topology: single-node KRaft has no quorum to lose
Most Docker Compose PoCs run one combined broker-and-controller process. That's a legitimate way to run a PoC — the KRaft Docker Compose guide documents the pattern — but it has no controller quorum, so there is nothing to test for quorum loss, and no failover path if that one process dies.
Production KRaft clusters run a multi-node controller quorum specifically so the metadata layer survives losing a node. The controller quorum guide covers kafka-metadata-quorum and what actually changes between a single combined process and a real multi-node quorum. Moving from one to the other is a topology decision, not a config flag — plan it before traffic depends on it, and test what happens when a controller node goes down before you find out during an actual outage.
Monitoring, security, and cost: the three that fail silently
A PoC has no alerting, which means failures surface only when someone notices missing data — often days later. The backup monitoring guide covers the metrics worth alerting on before that happens.
A PoC frequently runs with no TLS or SASL configured at all, because nothing about a local proof of concept forces the question. The security setup guide covers configuring SASL_SSL and mTLS on every Kafka connection — treat it as a checklist item, not an optional hardening pass, once real data is involved.
PoC retention defaults — commonly seven days — get copied into production verbatim because nobody revisited them. That's fine until storage costs or a compliance requirement makes it not fine. The backup cost guide covers setting retention deliberately per topic tier instead of inheriting whatever the PoC happened to leave running.
Where to go next: the scored version
Once the gaps above are closed, the Self-Assessment Checklist gives a detailed, scored assessment (0–87 points across all six Well-Architected pillars) for tracking backup-architecture maturity on an ongoing basis. Treat this post's checklist as the one-time gap check before go-live, and the self-assessment as the recurring maturity score afterward — they answer different questions at different points in the cluster's life.
FAQ
Frequently asked questions
What is the default replication factor in Kafka, and why is it wrong for production?
The broker config default.replication.factor defaults to 1, and auto.create.topics.enable defaults to true — so any topic created implicitly, which is most topics in a PoC, has exactly one copy of its data. Losing that one broker loses the topic permanently. Production topics need an explicitly set replication factor, typically 3.
Does a Kafka PoC need backup configured before going live?
Yes. A PoC proves the pipeline moves data correctly, not that it recovers from broker loss, a bad deploy, or an accidental delete. Backup and disaster recovery are almost never configured during a PoC because nothing forces the question until real traffic and real failure modes exist.
What changes between a single-node and multi-node Kafka KRaft cluster?
A single-node KRaft setup runs one combined broker-and-controller process — common in PoC Docker Compose files — with no controller quorum and no failover if that process dies. A multi-node KRaft cluster runs a real controller quorum, so the metadata layer survives losing a node, which a single-node setup cannot do.
How do you know if a Kafka cluster is production-ready?
It's ready when the defaults a PoC leaves untouched have been replaced with explicit decisions: replication factor and minimum in-sync replicas set deliberately, backup and DR targets defined and tested, monitoring and alerting in place, TLS or SASL configured, and retention set per topic tier rather than left at whatever the PoC happened to use.
Conclusion
Nothing separates a Kafka PoC from a production cluster except explicit configuration. Replication factor, minimum in-sync replicas, backup, DR targets, monitoring, security, cluster topology, and retention all have defaults that work fine when nobody's watching and fail exactly when it matters most. None of them announce themselves — a PoC with replication factor 1 and no backup runs identically to one configured correctly, right up until a broker goes down. Work through the checklist above once, deliberately, before real traffic depends on the answers.
OSO Kafka Backup adds point-in-time recovery and built-in consumer group offset preservation on top of whatever replication factor you've set — replication and backup solve different failure modes, and production needs both. Get started or see the backup strategies guide for the full picture.