Kafka on AWS: MSK Architecture and the Data Protection Gaps
Running Kafka on AWS means choosing between three deployment models: Amazon MSK provisioned, where AWS manages brokers you size yourself; MSK Serverless, where capacity follows throughput automatically; and self-managed Kafka on EC2, where everything is yours to run. MSK removes broker patching, metadata-quorum management, and hardware failure from your plate. It does not remove responsibility for the data inside the topics.
That second point surprises teams more than it should. AWS Backup does not support MSK as a resource type, there is no snapshot API for topic data, and multi-AZ replication faithfully copies deletions and corruption along with everything else. This guide walks the MSK architecture, the failure scenarios AWS absorbs, the ones it leaves to you, and the options for closing the gap.
Under the shared responsibility model, AWS keeps MSK brokers healthy — patched, replaced on failure, replicated across availability zones. Your topic data, consumer offsets, and recovery from logical failures like deletion or bad writes remain your responsibility, and AWS ships no native backup for them.
Three ways to run Kafka on AWS
Each model trades control for operational load. The right choice depends on how much of Kafka you want to own.
| MSK provisioned | MSK Serverless | Self-managed on EC2 | |
|---|---|---|---|
| Broker operations | AWS | AWS (invisible) | You |
| Capacity model | You pick instance types and broker count | Scales with throughput | You size everything |
| Storage | EBS per broker, up to 16,384 GiB | Managed | EBS volumes you own |
| Tiered storage | Yes (provisioned only) | No | Manual, if your Kafka version supports it |
| Config control | Most broker and topic configs | Subset of topic configs | Everything |
| Native topic-data backup | None | None | None (EBS snapshots are volume-level) |
MSK provisioned is the default for most production estates. You choose broker instance types, broker count, and storage; AWS handles the metadata quorum, patching, and broker replacement. You keep enough configuration control for real tuning.
MSK Serverless removes capacity planning for spiky or unpredictable workloads. The trade is control: you manage a subset of topic configurations, and tiered storage is not available. It suits event-driven workloads that would otherwise idle provisioned brokers.
Self-managed on EC2 gives you every knob and every burden: version choice, broker tuning, quorum operations, upgrades. EBS snapshots exist here, but they capture volumes, not Kafka state — a snapshot taken mid-write is crash-consistent at best. The EBS snapshots comparison covers why that falls short for Kafka.
Inside the MSK provisioned architecture
An MSK provisioned cluster places brokers across availability zones, attached to subnets in your VPC. Each broker gets its own EBS volume, expandable to 16,384 GiB. The metadata quorum — KRaft or ZooKeeper, depending on cluster version — runs managed by AWS at no extra charge and stays invisible to you.
Storage runs in one of two modes. EBS-only keeps every log segment on broker volumes. Tiered storage, available on provisioned clusters, moves older segments to S3-backed remote storage so retention can grow without growing broker disks. That is retention economics, not data protection — tiered segments still expire with retention and still hold whatever corruption was written. The tiered storage comparison draws that line in detail.
Client authentication comes in four flavors, each on its own port:
- IAM access control (port 9098) — authorization through AWS IAM policies; works with clients that carry the AWS IAM auth library
- SASL/SCRAM (port 9096) — username and password stored in AWS Secrets Manager
- Mutual TLS (port 9094) — client certificates issued through AWS Private CA
- Plaintext (port 9092) — unauthenticated and unencrypted; avoid it outside experiments
IAM auth only works for clients built with the AWS SDK libraries. Most third-party tools — backup tools included — connect over SASL/SCRAM or mutual TLS instead, which is why those two listeners matter for the protection architecture later in this guide.
What MSK protects — and what stays your responsibility
The shared responsibility model applied to Kafka is blunt. AWS owns the infrastructure layer: broker health, patching, AZ-level failure, quorum availability. You own the data layer: what topics hold, how long retention keeps it, where consumer groups sit, and how to get any of it back after a logical failure.
| Failure scenario | Covered by MSK? | What covers it |
|---|---|---|
| Broker instance failure | Yes | Managed replacement + in-cluster replication |
| Availability zone outage | Yes | Multi-AZ broker placement |
| Region outage | No | Cross-region replication or restore from backup |
| Accidental topic deletion | No | Independent backup only |
| Application writes corrupt data | No | Point-in-time restore from backup |
| Retention expires data you still need | No | Backup or archive outside the cluster |
The bottom three rows are where teams get caught. Replication — whether the in-cluster kind or cross-region MSK Replicator — copies every write with equal loyalty. Delete a topic, and the deletion propagates. Ship a deploy that writes garbage for four hours, and every replica holds four hours of garbage.
There is no AWS-native escape hatch. AWS Backup has no MSK support, MSK exposes no topic-data snapshot, and broker EBS volumes are not accessible from your account, so volume snapshots are off the table too. Recovery from logical failure requires a copy of the data that lives outside the cluster and remembers earlier states. Replication is not backup; on MSK the distinction is enforced by what the platform does and does not offer.
Data protection options for Kafka on AWS
The realistic options split into two families: replication, which keeps a live copy for availability, and backup, which keeps historical state for recoverability.
| Option | Protects against | Recovery character | Gaps |
|---|---|---|---|
| Multi-AZ replication (built in) | Broker and AZ failure | Automatic, near-instant | Nothing logical, nothing regional |
| MSK Replicator | Region or cluster failure | Fail over to live standby | Replicates deletes and corruption too |
| MirrorMaker 2 | Region or cluster failure | Fail over, self-operated | Same class of gap, more operations |
| Tiered storage | Broker disk cost at long retention | Not a recovery tool | Segments still expire and still corrupt |
| S3 sink connector | Data loss after archiving | Re-produce from archive | One-way; no offsets, no restore tooling |
| Dedicated backup to S3 | Deletion, corruption, retention expiry, region loss | Point-in-time restore with offsets | Bounded by backup cadence |
The pairing rule follows from the table. Replication answers "how do we keep serving traffic when infrastructure fails?" Backup answers "how do we get back the state from before the incident?" A production MSK estate that matters usually needs one of each — and the failure modes in the middle rows explain why neither substitutes for the other.
A backup architecture for Amazon MSK
The working pattern is small: run the backup process on EC2 or EKS inside the brokers' VPC, authenticate over SASL/SCRAM or mutual TLS, and land compressed backups in an S3 bucket in the same region. S3 credentials come from the instance profile or IRSA service account, so no static keys exist to rotate or leak.
mode: backup
backup_id: "msk-prod-aws"
source:
bootstrap_servers:
- b-1.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
- b-2.mymsk.abcd12.c2.kafka.us-east-1.amazonaws.com:9096
security:
security_protocol: SASL_SSL
sasl_mechanism: SCRAM-SHA512
sasl_username: ${MSK_SCRAM_USER}
sasl_password: ${MSK_SCRAM_PASSWORD}
connection:
tcp_keepalive: true
topics:
include:
- "*"
exclude:
- "__consumer_offsets"
storage:
backend: s3
bucket: my-kafka-backups
region: us-east-1
prefix: msk/production
backup:
compression: zstd
include_offset_headers: true
consumer_group_snapshot: true
Consumer group offsets ride along with the topic data, so a restore returns applications to a consistent position rather than just refilling topics. For durability past the first month, S3 lifecycle rules tier old backups to cheaper storage classes, and cross-region bucket replication keeps a DR copy without a second backup system.
This is the condensed version. The Kafka backup on AWS solution page walks the full build — IAM policy, bucket layout, compute placement — the Amazon MSK integration page covers connection details, and the backup-to-S3 tutorial is the step-by-step walkthrough. Storage-side setup lives in the AWS S3 docs, and every key above is defined in the configuration reference.
Planning recovery objectives on AWS
Recovery objectives decide how much of the table above you need. Replication gives near-zero RPO against infrastructure failure, because the standby is seconds behind — but it offers no protection window at all against logical damage, since the damage arrives on the standby seconds later. Backups give a recovery window whose RPO equals the backup cadence: hourly incremental backups mean at most an hour of exposure.
Work the numbers from the business side first. The RTO/RPO planning guide covers how to set targets per topic class, and the disaster recovery use case shows the reference architectures that meet them.
AWS keeps MSK brokers healthy; it does not keep yesterday's data. OSO Kafka Backup streams MSK topics and consumer group offsets to S3 with point-in-time restore at millisecond precision. Get started.
MSK removes broker operations, not data responsibility
Amazon MSK is the right default for most teams running Kafka on AWS. It takes broker patching, quorum management, and AZ failure off your plate, and provisioned clusters add tiered storage for cheap long retention. What it does not take is responsibility for the data: no AWS Backup support, no topic snapshots, and replication that copies mistakes as faithfully as records. Know which failures multi-AZ absorbs, add cross-region replication where region loss matters, and put an independent point-in-time backup under everything replication cannot undo.
Frequently asked questions
What is MSK Kafka?
Amazon MSK (Managed Streaming for Apache Kafka) is the AWS managed service for running Apache Kafka. AWS operates the brokers and metadata quorum, handles patching and failed-broker replacement, and places brokers across availability zones. It comes in two forms: provisioned, where you size brokers and storage, and Serverless, where capacity scales with throughput.
When should I choose Amazon MSK over Confluent Kafka?
MSK fits teams committed to AWS: it runs in your VPC, integrates with IAM and Secrets Manager, and prices as AWS infrastructure. Confluent Cloud fits teams that want multi-cloud coverage or its wider managed ecosystem of connectors and stream processing. Both are solid managed Kafka services, and neither includes point-in-time backup of topic data — that layer is yours to add on either platform.
How do I find the Kafka version of an MSK cluster?
The cluster detail page in the AWS console shows the Apache Kafka version. From the CLI, run aws kafka describe-cluster with the cluster ARN; the response includes the current broker software info with the Kafka version. Knowing the version matters for client compatibility and for features like tiered storage that depend on it.
How do I back up Kafka on AWS?
Run a dedicated backup tool on EC2 or EKS in the same VPC as the brokers, connect over SASL/SCRAM or mutual TLS, and stream topic data plus consumer group offsets to S3. AWS offers no native backup for MSK topic data, and MSK Replicator is replication rather than backup. OSO Kafka Backup provides this layer with compression and millisecond point-in-time restore.