Kafka Connect S3 Sink for Archiving: Setup, Configuration, and Limits
The Kafka Connect S3 sink connector consumes Kafka topics and writes their records to Amazon S3 as partitioned object files in Avro, JSON, Parquet, or raw bytes. If you already run a Kafka Connect cluster, it is the standard way to archive topic data for analytics, long-term retention, and compliance records.
This guide covers how the connector works, a complete working configuration, the partitioners and formats that shape your archive, the exact conditions for exactly-once delivery, licensing, and the one boundary every team should plan around: an S3 sink archive has no first-class path back into Kafka.
The S3 sink connector solves the write path: topic records land in S3 as query-ready files, with exactly-once object writes when configured deterministically. It does not solve the read-back path. Restoring topics, consumer group offsets, and a precise point in time is a different job that needs a purpose-built backup tool.
What the Kafka Connect S3 sink connector does
The S3 sink is a sink connector that runs inside a Kafka Connect cluster, in either distributed or standalone mode. Connect's framework consumes the configured topics through a consumer group and hands batches of records to the connector. The connector buffers records per topic partition, then uploads completed files to S3 using multipart upload, sized by s3.part.size (25 MB parts by default).
The connector class is io.confluent.connect.s3.S3SinkConnector, from Confluent's kafka-connect-storage-cloud project. It scales the usual Connect way: raise tasks.max and the framework spreads topic partitions across tasks and workers.
Objects land under a configurable top-level directory (topics.dir, default topics), then a path chosen by the partitioner, with filenames that encode the topic, Kafka partition, and starting offset:
s3://example-kafka-archive/topics/orders/year=2026/month=08/day=01/hour=09/orders+0+0000010000.parquet
That layout is the point. It is built for query engines — Athena, Spark, Trino — to prune by prefix and scan efficiently. Typical uses are data-lake ingestion, long-term archive of event streams, compliance record retention, and feeding warehouse ETL. What the layout is not built for is replaying data back into Kafka in original order, which is where the restore gap comes from.
A working archiving configuration
Here is a complete connector config for archiving two topics to S3 as hourly-partitioned Parquet, submitted to the Connect REST API:
{
"name": "s3-archive-orders",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"tasks.max": "2",
"topics": "orders,payments",
"s3.bucket.name": "example-kafka-archive",
"s3.region": "eu-west-1",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
"flush.size": "10000",
"rotate.interval.ms": "3600000",
"partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"partition.duration.ms": "3600000",
"path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
"locale": "en-US",
"timezone": "UTC",
"timestamp.extractor": "Record"
}
}
Three settings decide when a file is closed and uploaded, and they interact:
flush.size— the maximum number of records per topic partition in one file. The first trigger to fire wins, so this also caps file size for hot partitions.rotate.interval.ms— closes a file once the span of record timestamps inside it exceeds the interval. Because it is driven by record timestamps, it is deterministic: reprocessing the same records produces the same files.rotate.schedule.interval.ms— closes files on a wall-clock schedule. Useful for flushing low-traffic topics promptly, but wall-clock time is not reproducible across restarts, so it breaks exactly-once delivery.
For credentials, give the Connect workers an IAM role (instance profile or IRSA on Kubernetes) with s3:PutObject, s3:AbortMultipartUpload, and s3:ListBucket on the archive bucket. Keep static keys out of connector configs — they end up readable in the Connect REST API.
Formats and partitioners: what your archive looks like
format.class controls the file format written to S3:
| Format class | Output | Best for |
|---|---|---|
io.confluent.connect.s3.format.parquet.ParquetFormat | Columnar Parquet | Athena, Spark, warehouse queries |
io.confluent.connect.s3.format.avro.AvroFormat | Avro container files | Schema-aware pipelines, compact storage |
io.confluent.connect.s3.format.json.JsonFormat | Newline-delimited JSON | Schemaless topics, easy inspection |
io.confluent.connect.s3.format.bytearray.ByteArrayFormat | Raw record bytes | Verbatim archive, format-agnostic |
partitioner.class controls the S3 path structure:
| Partitioner | Path shape | Deterministic? |
|---|---|---|
DefaultPartitioner | topics/<topic>/partition=<n>/ | Yes |
TimeBasedPartitioner | Custom path.format, e.g. year=/month=/day=/hour=/ | Only with timestamp.extractor=Record or RecordField |
DailyPartitioner / HourlyPartitioner | Preset daily or hourly paths | Same condition as TimeBased |
FieldPartitioner | topics/<topic>/<field>=<value>/ | Yes |
There is a tradeoff to name plainly. The more query-friendly the layout — time buckets, field values — the further your objects drift from the original topic-partition ordering. Great for WHERE day = '2026-08-01' queries; awkward for reconstructing a topic in order later.
Exactly-once delivery — and its exact conditions
The S3 sink can deliver records to S3 exactly once, even though Kafka Connect's framework only guarantees at-least-once. The trick is determinism rather than transactions. If the same records always produce the same files with the same names, a retry after failure overwrites an identical object instead of duplicating data.
That guarantee holds only when the whole write path is deterministic. Per the Confluent documentation, you need both of the following:
- A deterministic partitioner.
DefaultPartitionerandFieldPartitionerqualify as-is. The time-based partitioners qualify only whentimestamp.extractorisRecordorRecordField, so partitioning depends on the data, not on when it was processed. - Deterministic rotation. File boundaries driven by
flush.sizeandrotate.interval.msonly. Wall-clock rotation viarotate.schedule.interval.msdisqualifies the guarantee.
Label this guarantee for what it is: exactly-once into S3. It deduplicates the archive's object writes. It says nothing about reading the archive back into Kafka, where no equivalent machinery exists.
Licensing and alternatives
Confluent's S3 sink is source-available under the Confluent Community License. You can self-manage it in production for free, but it is not OSI-approved open source — the license restricts offering the connector itself as a competing managed service. For most teams archiving their own data, that restriction never bites; it matters to platform vendors.
If you want a permissively licensed S3 sink, Aiven maintains cloud-storage-connectors-for-apache-kafka under Apache 2.0, which includes an S3 sink connector.
| Confluent S3 sink | Aiven S3 sink | |
|---|---|---|
| License | Confluent Community License | Apache 2.0 |
| Maintainer | Confluent | Aiven (open source) |
| Formats | Avro, JSON, Parquet, raw bytes | Depends on connector version — check the repo |
Both run anywhere Kafka Connect runs: a self-managed Connect cluster next to any Kafka distribution, or a managed Connect platform such as MSK Connect, where you register the connector as a custom plugin. Confluent Cloud also offers its S3 sink fully managed. If your cluster runs on MSK, the Kafka on AWS guide covers where Connect fits in that architecture.
The restore gap: what an S3 sink archive cannot give back
Writing to S3 is half a backup. The read-back half is where the sink stops being the right tool.
There is a separate S3 source connector that can republish sink-written objects to a topic. It is a Confluent proprietary connector: free for a 30-day trial, then it requires a Confluent subscription. Its Backup-and-Restore mode reads only data written by the S3 sink — and, per Confluent's own documentation, it does not restore record keys or headers that the sink exported. For keyed topics, compacted downstream consumers, or header-based routing, that is not a restore; it is a lossy re-ingestion.
Some things are never in the archive at all, no matter which source connector you buy:
| Capability | S3 sink archive |
|---|---|
| Data-lake ingestion for analytics | Yes |
| Long-term record archive in open formats | Yes |
| Restore topics back into Kafka | No first-class path |
| Preserve consumer group offsets | No — never captured |
| Point-in-time recovery to a precise moment | No — files are partitioned for queries, not replay windows |
| Capture topic configuration | No |
Consumer group offsets are the sharpest edge. After any replay assembled from sink files, every consumer group has lost its position: reprocess everything or skip to latest, with duplicates or gaps either way. Offset capture and restore is precisely what a purpose-built tool does — the consumer offsets backup guide explains why that state matters as much as the records.
The pairing rule follows. Run the S3 sink for what it is good at — feeding the lake — and run an independent, restore-oriented backup for recoverability. The side-by-side comparison walks the differences line by line, the backup-to-S3 runbook shows the recoverable path into the same bucket family, and the disaster recovery use cases cover the failure scenarios only a restorable copy survives. If what you actually want from S3 is cheaper broker retention rather than an archive, that is a third tool again: Kafka tiered storage.
The S3 sink lands records in your lake. OSO Kafka Backup lands recoverable copies — topic data, topic configuration, and consumer group offsets, restorable to any cluster with millisecond point-in-time precision, from a single CLI with no Connect cluster to operate. Get started.
The right tool, used for its actual job
The Kafka Connect S3 sink is mature, widely deployed, and good at what it was built for: landing topic records in S3 as query-ready, partitioned files, exactly once when configured deterministically. Archive with it. Query what it lands. Just be honest about the day you need topics, offsets, and a precise restore point back inside Kafka — an archive is not the tool that gets you there, and the comparison of backup approaches shows what is.
Frequently asked questions
Can the Kafka Connect S3 sink connector restore data back into Kafka?
Not by itself — it only writes to S3. Confluent sells a separate proprietary S3 source connector whose Backup-and-Restore mode can republish sink-written objects to a topic, but it does not restore record keys or headers, and consumer group offsets and topic configuration were never captured in the first place.
Is the Kafka Connect S3 sink connector free to use?
The sink is source-available under the Confluent Community License, so you can self-manage it in production at no cost, though it is not OSI-approved open source. The companion S3 source connector is different: it is proprietary, with a 30-day trial followed by a paid Confluent subscription. Aiven maintains an Apache-2.0 licensed S3 sink alternative.
How does the S3 sink connector achieve exactly-once delivery?
Through determinism rather than transactions. When a deterministic partitioner is used and file rotation is driven by flush.size or rotate.interval.ms, the same records always produce identically named files, so retries overwrite identical objects instead of duplicating data. Wall-clock rotation via rotate.schedule.interval.ms breaks the guarantee.
Which formats can the Kafka Connect S3 sink write to S3?
Four formats via format.class: Avro container files, newline-delimited JSON, Parquet for columnar queries, and raw bytes via ByteArrayFormat. Parquet is the usual choice for Athena or Spark analytics, while ByteArrayFormat preserves record values verbatim for format-agnostic archives.
Is the Kafka Connect S3 sink a Kafka backup?
It is an archive, not a backup. It covers the write path well, but there is no first-class restore into Kafka, no consumer group offset preservation, no topic configuration capture, and no point-in-time recovery. Teams that need recoverability pair the sink with a purpose-built backup tool such as OSO Kafka Backup.