Kafka Replicator Guide: Architecture, Setup, and Recovery Limits
Kafka Replicator is Confluent's Kafka Connect connector for asynchronously copying selected topics from one Kafka cluster to another. It can preserve topic settings, translate consumer positions, and fit into an existing Confluent Platform operating model.
Replicator keeps a second Kafka cluster close to the source's current state. It does not create historical restore points, so it cannot recover the state before a bad write or deletion.
Use Kafka Replicator when its Connect-based flow fits an existing Confluent Platform estate. Prove topic delivery and consumer cutover, then add independent backup when recovery must reach an earlier state.
What Kafka Replicator does
The name "Kafka Replicator" usually refers to Confluent Replicator. It is different from Kafka's native replica mechanism, which copies partitions among brokers inside one cluster.
Replicator runs as a Kafka Connect connector or through Confluent's executable wrapper. An embedded consumer reads selected source topics. Connect tasks pass those records to producers that write them into the destination cluster.
The connector can create destination topics and preserve their partition count, replication factor, and configuration overrides. It can also detect new matching topics and partition increases.
Confluent documents an at-least-once delivery guarantee. A connector restart can therefore produce duplicate records. Downstream applications must handle that delivery model safely.
| Component | Usual location | Responsibility | Evidence to monitor |
|---|---|---|---|
| Source consumer | Replicator task | Reads selected source partitions | Fetch throughput and source lag |
| Connect worker | Destination site | Coordinates tasks and stores connector state | Worker and task status |
| Destination producer | Replicator task | Writes records into target topics | Produce errors and buffer pressure |
| Destination Kafka | Target cluster | Retains the replicated log | Topic health, capacity, and record arrival |
Confluent recommends placing the workers near the destination cluster. A WAN failure then interrupts remote consumption instead of remote production. Local producer writes and Connect state remain close to the destination brokers.
The current Confluent Replicator overview also says Cluster Linking and Schema Linking are preferred for most new use cases. Replicator still matters for supported existing deployments and certain migration paths.
When Kafka Replicator is still the right fit
Replicator is strongest when a team already operates it inside Confluent Platform. Existing monitoring, security, licensing, and Connect procedures can make continued use practical.
It also supports migration, aggregation, and active-passive designs. Topic filters and rename rules let operators control which logs move and how they appear at the target.
For a new Confluent topology, review Cluster Linking and Schema Linking first. That is Confluent's stated direction for most use cases. Confirm feature and version requirements against the release you run.
| Need | First tool to assess | Important boundary |
|---|---|---|
| Existing Confluent Replicator estate | Confluent Replicator | Commercial connector with operational state in Connect |
| New Confluent cross-cluster design | Cluster Linking | Check supported deployment and licensing requirements |
| Vendor-neutral cross-cluster copy | MirrorMaker 2 | You operate its workers, checkpoints, and naming policy |
| Recovery before corruption or deletion | Independent backup | Restore time differs from warm-cluster failover |
This is a scope decision, not a single-winner contest. The Kafka replication guide explains the failure boundaries covered by broker replicas and cross-cluster tools.
Plan the source-to-destination flow
Start with one documented direction. Name the source and destination clusters, list the topics, choose a destination naming rule, and define the accepted lag.
Confirm that the destination has enough brokers, storage, network capacity, and retention for copied traffic. It must also handle the full application workload during a failover.
Decide where Connect stores its config, offset, and status topics. Their replication factors must fit the destination broker count and durability policy.
Grant the Replicator principal access only to the required resources. It needs source read and describe access. It also needs destination create, write, and describe access for selected topics and connector state.
Confirm your Confluent Platform license and compatibility before deployment. Schema and offset features can carry extra version requirements. Use the current compatibility notes for the installed release.
Consumer cutover needs its own decision. Replicator's timestamp-based offset translation uses a consumer interceptor that supports Java clients. Non-Java applications need another tested position-management plan.
Configure Kafka Replicator on a Connect cluster
In the recommended destination-side pattern, the Connect worker's
bootstrap.servers points to the destination Kafka cluster. The Replicator
connector receives the source broker addresses.
The following connector copies two application topics. It adds .replica to
each destination name, making the direction visible during the first test.
{
"name": "orders-replicator",
"config": {
"connector.class": "io.confluent.connect.replicator.ReplicatorSourceConnector",
"tasks.max": "4",
"key.converter": "io.confluent.connect.replicator.util.ByteArrayConverter",
"value.converter": "io.confluent.connect.replicator.util.ByteArrayConverter",
"src.kafka.bootstrap.servers": "source-1:9092,source-2:9092,source-3:9092",
"topic.regex": "^(orders|payments)$",
"topic.rename.format": "${topic}.replica",
"offset.topic.commit": "true"
}
}
The connector class and property names come from Confluent's Replicator configuration reference. Add the required license and security properties through your approved secret and configuration process before submission.
Do not place passwords or license values in source control. Configure TLS, SASL, and trust material for both cluster connections using the method supported by your Connect deployment.
topic.regex allows future matching topics to enter the flow. An explicit
topic.whitelist is stricter, but every listed topic must exist when recent
Replicator versions start.
Review the matched topic inventory before using a broad expression. A future topic can otherwise cross a regional or data-residency boundary without a separate connector change.
Submit the reviewed JSON to the Connect REST API:
curl -X POST \
--header 'Content-Type: application/json' \
--data @orders-replicator.json \
http://connect-destination:8083/connectors
The command follows Confluent's current Replicator run guide. Treat this first POST as a controlled deployment with a rollback owner.
Verify the first replicated records
A successful REST response proves only that Connect accepted the request. Check the connector and every assigned task next.
curl -s http://connect-destination:8083/connectors/orders-replicator/status
The connector and all tasks should report RUNNING. A healthy connector with a
failed task can leave some partitions stale.
Describe a destination topic and compare its partition count with the source:
bin/kafka-topics.sh \
--bootstrap-server destination-1:9092 \
--describe \
--topic orders.replica
Check replica assignments, in-sync replicas, retention, and any required topic overrides. Topic existence does not prove that current records still flow.
Send a unique smoke record to the source:
printf 'replicator-smoke-2026-07-16\n' | bin/kafka-console-producer.sh \
--bootstrap-server source-1:9092 \
--topic orders
Consume it from the renamed destination topic:
bin/kafka-console-consumer.sh \
--bootstrap-server destination-1:9092 \
--topic orders.replica \
--from-beginning \
--max-messages 1
Use a dedicated smoke topic or uniquely keyed record for recurring checks. A shared topic can return an older record and create a false pass.
The cross-datacenter replication guide places these checks inside a wider network and capacity plan.
Handle consumer offsets and failover safely
Source and destination offsets are not interchangeable. Each topic has its own log, so the same record can have different numeric offsets across clusters.
Replicator can translate positions using timestamps. A Java consumer interceptor
records group, topic, partition, offset, and timestamp data in the source
__consumer_timestamps topic.
Replicator reads that data and maps the source position to a destination offset. It can write the translated position into the destination group's offsets.
The destination group must be inactive for that write. Active consumers commit their own positions, so Replicator will not replace them during normal use.
A practical failover drill follows this order:
- Stop source consumers and record their final committed positions.
- Wait until measured replication lag meets the accepted recovery point.
- Confirm translated destination offsets for every selected group.
- Fence source producers before enabling writes at the destination.
- Start destination consumers with the same tested group IDs.
- Check duplicate handling, missing-record checks, and downstream effects.
- Record elapsed time, owners, evidence, and rollback steps.
Confluent's Replicator failover guide documents the timestamp interceptor and translation process. Test it with your client type before making it part of an RTO claim.
Monitor Kafka Replicator in production
Process uptime is a weak signal. Monitor source consumption, task state, destination production, broker health, and a business-level smoke path together.
| Signal | What it answers | Investigate when |
|---|---|---|
| Connector and task state | Is each assignment running? | Any task is failed, paused, or restarting |
records-lag-max | Can the source consumer keep up? | The value rises across several samples |
| Source bytes consumed | Is data leaving the selected topics? | Throughput falls despite active producers |
| Producer buffer pressure | Can Connect write fast enough? | Waiting threads or buffer wait time rise |
| Destination broker health | Can the target accept and retain data? | ISR, disk, or request health degrades |
| Smoke-record age | Does the full application path still work? | The record misses the accepted delay |
Confluent recommends current JMX metrics for Replicator lag instead of the legacy consumer-group method. Keep dashboards aligned with the version that runs in production.
Historical catch-up can distort a timestamp-based latency metric. Old source records naturally have old timestamps. Pair that metric with record lag, throughput, and backlog clearance time.
Test link loss, worker restart, task rebalance, destination broker failure, and new topic discovery. Measure how long the connector takes to clear each backlog.
Common Kafka Replicator failures
Start with filters, permissions, and worker placement before changing throughput settings. Preserve the failing configuration and logs so each change has clear evidence.
| Symptom | Likely cause | Response |
|---|---|---|
| Connector runs but creates no topics | No whitelist or regex matches | Inspect the selected source inventory and pattern |
| Connector fails during startup | A whitelisted topic is missing or hidden by ACLs | Create the topic or fix source describe access |
| Records arrive under an unexpected name | Rename format differs from the runbook | Verify topic.rename.format before client cutover |
| Some partitions stop moving | One task failed while the connector stayed running | Inspect every task and its worker logs |
| Lag rises with producer pressure | Source fetch or destination produce capacity is constrained | Check JMX, quotas, buffers, and target brokers |
| Large records fail after decompression | Producer record-size limits are too small | Align limits with tested uncompressed batches |
| Offset translation does not update | Client type is unsupported or target group is active | Review the interceptor path and stop target consumers |
| Connector stops after a license event | License is missing or expired | Restore valid entitlement through the approved process |
Known issues and defaults change across releases. Check the installed version's official documentation before copying a setting from another estate.
Kafka Replicator is not backup
Replicator is good at keeping another Kafka cluster near the present. That makes it useful for migration, regional access, and warm standby designs.
It also copies the live record stream forward. Bad writes, tombstones, and application mistakes can reach the destination like valid events.
Both copies remain inside live Kafka clusters. Their broker retention and administrative controls still apply. Replicator cannot reconstruct the topic as it existed before an incident.
Independent backup creates another storage and retention boundary. OSO Kafka Backup stores topic data and consumer group offsets in S3, S3-compatible storage, Azure Blob, GCS, or a filesystem.
Run a configured backup, then validate its stored segments:
kafka-backup backup --config /etc/kafka-backup/backup.yaml
kafka-backup validate \
--path s3://kafka-dr-backups/production \
--backup-id production-daily-001 \
--deep
These commands come from the CLI basics guide. The disaster recovery documentation explains how retained restore points cover failures that replication cannot reverse.
For a direct feature boundary, see OSO Kafka Backup vs Confluent Replicator. The backup alternatives guide covers other tool categories.
Store Kafka topics and consumer offsets in independent storage. Validate the copy before an incident. Start with OSO Kafka Backup.
Finish with a tested recovery outcome
A healthy Kafka Replicator deployment has evidence at every layer. The connector runs, selected records arrive, topic settings match policy, and measured lag stays inside the accepted limit.
Recovery evidence goes further. Consumers resume at tested positions, producers cannot write to both sites by accident, and operators can execute rollback.
Keep existing Replicator deployments aligned with their installed Confluent Platform version. Assess Cluster Linking for new Confluent designs. Add backup whenever recovery must return to an earlier state.
Frequently asked questions
Does Kafka replicate messages on each cluster?
Native Kafka replication copies partition data among brokers inside one cluster. It does not automatically copy records to another cluster. Confluent Replicator creates an explicit asynchronous path from selected source topics to a destination cluster.
How does Kafka replication work?
Native replicas follow a partition leader within one cluster. Confluent Replicator covers a different scope. Its Kafka Connect tasks consume selected source records and produce them into destination topics, with optional topic configuration and consumer position handling.
What is replication in Kafka?
Replication is the copying of Kafka partition data across a defined failure boundary. Broker replication covers broker loss inside one cluster. A cross-cluster tool such as Confluent Replicator copies selected topic records into another Kafka cluster.
Why is replication required in Kafka?
Replication keeps records available when a broker, cluster, or site fails, depending on its scope. It does not preserve older clean states. Independent backup is required when recovery must return to a point before corruption, deletion, or another logical incident.