Kafka Dead Letter Queue: Patterns and Config
A Kafka dead letter queue holds records that failed processing. Kafka has no broker setting that creates one for a classic consumer group. The pattern is a separate topic.
Apache Kafka 4.4 documents that topic in Kafka Connect, in Kafka Streams, and on share groups. Each path has its own config, and each one defaults to off.
Kafka has no broker-level dead letter queue. A DLQ is a separate topic for records a consumer or connector could not process. Kafka Connect includes one for sink connectors. Set errors.tolerance to all and set errors.deadletterqueue.topic.name. Source connectors do not get that topic. Plain consumers produce to their own topic.
What is a Kafka dead letter queue?
A dead letter queue in Kafka is a topic you reserve for failed records. The original topic stays readable for records that succeeded. The DLQ keeps the ones that did not.
A classic consumer group has no broker DLQ. Your code catches the failure, produces the record to a DLQ topic, and commits the source offset. Until that produce succeeds, the source offset is the only copy.
| Path | DLQ support in Kafka 4.4 | What you set |
|---|---|---|
| Classic consumer | None built in | Produce to <topic>.dlq, then commit |
| Connect sink | Built in | errors.tolerance=all and a DLQ topic name |
| Connect source | No DLQ topic | errors.tolerance and errors.log.enable only |
| Kafka Streams | Built in since 4.2.0 | errors.dead.letter.queue.topic.name |
| Share group | Off until configured | Group DLQ name, plus the topic enable flag |
Does Kafka Streams have a DLQ?
Yes, starting with Kafka Streams 4.2.0. The 4.2.0 release lists KIP-1034, which adds dead letter queue support on exception handlers.
The config is errors.dead.letter.queue.topic.name. The default is null, so nothing is sent. When you set a topic name, the default exception handler builds a DLQ record and sends it.
A custom deserialization, production, or processing handler ignores that config. The handler has to build the DLQ record itself. The same page limits the setting to regular stream tasks. It does not apply to global state store updates.
The 4.4 upgrade guide still says DLQ support is not available for global store and KTable processing. That note is KIP-1270, from Kafka Streams 4.3.
Three handler configs decide whether the thread stops.
| Handler config | Default class |
|---|---|
deserialization.exception.handler | LogAndFailExceptionHandler |
production.exception.handler | DefaultProductionExceptionHandler |
processing.exception.handler | LogAndFailProcessingExceptionHandler |
LogAndFailExceptionHandler logs the deserialization error and stops processing more records. The Streams config guide documents LogAndContinueExceptionHandler for the skip path.
The older names default.deserialization.exception.handler and default.production.exception.handler are deprecated since Kafka Streams 4.0. KIP-1056 is that change. Use the names in the table.
A custom handler that produces with its own KafkaProducer sits outside the Streams processing guarantee. The config guide says those writes are side effects the runtime does not track. Prefer the built-in topic name, or return DLQ records from the handler response. The same guide calls that the recommended approach.
Do share groups have a DLQ?
A share group can name a DLQ topic. It does not have one until you set it. errors.deadletterqueue.topic.name on the group defaults to an empty string. Empty means the group has no DLQ topic.
The topic itself must allow the write. errors.deadletterqueue.group.enable defaults to false. False means the topic rejects share-group DLQ writes.
errors.deadletterqueue.copy.record.enable also defaults to false. False writes a record of context headers. True also copies the original record onto the DLQ topic. Turn that on when you need the payload later.
This is still a topic. It is not a broker DLQ for classic consumer groups.
How do I configure a DLQ in Kafka Connect?
Kafka Connect can write failed sink records to a DLQ topic. The Connect user guide shows the fail-fast defaults and a working override. The Connect configs page lists the DLQ keys under sink connector configs only.
Set errors.tolerance to all and set errors.deadletterqueue.topic.name. all skips problematic records. The default none fails the task on the first error.
A blank topic name records nothing. With all and a blank name, the bad record is skipped and not stored.
This example follows the user guide. The retry cap is 600000 milliseconds, which that page describes as at most 10 minutes. The max delay is 30000 milliseconds.
Replication factor 3 is the documented default. Context headers stay off until you set them true.
errors.retry.timeout=600000
errors.retry.delay.max.ms=30000
errors.log.enable=true
errors.log.include.messages=false
errors.deadletterqueue.topic.name=my-connector-errors
errors.deadletterqueue.topic.replication.factor=3
errors.deadletterqueue.context.headers.enable=true
errors.tolerance=all
| Config | Default | What it does |
|---|---|---|
errors.tolerance | none | none fails the task. all skips the record |
errors.retry.timeout | 0 | Retry window in milliseconds. -1 retries forever |
errors.retry.delay.max.ms | 60000 | Max delay between retries. Jitter is added at the cap |
errors.log.enable | false | Log each tolerated error and the record location |
errors.log.include.messages | false | Also log the key, value, and headers |
errors.deadletterqueue.topic.name | empty | DLQ topic. Empty stores nothing |
errors.deadletterqueue.topic.replication.factor | 3 | Replication factor when Connect creates the topic |
errors.deadletterqueue.context.headers.enable | false | Add __connect.errors. headers |
errors.log.include.messages can write keys, values, and headers to the Connect log. The user guide warns that this may log sensitive data. Leave it false unless you need the payload in the log.
Which context headers does Connect add?
Connect adds context headers only when errors.deadletterqueue.context.headers.enable is true. The config page says every context header key starts with __connect.errors. so the keys do not clash with the original record.
Kafka 4.4.0 names those keys in DeadLetterQueueReporter:
__connect.errors.topic__connect.errors.partition__connect.errors.offset__connect.errors.connector.name__connect.errors.task.id__connect.errors.stage__connect.errors.class.name__connect.errors.exception.class.name__connect.errors.exception.message__connect.errors.exception.stacktrace
The same class writes the original key, value, and headers. If the DLQ topic is missing, it creates the topic with one partition and the configured replication factor. An existing topic is left as it is.
Source connectors do not get these keys. The config page lists errors.tolerance, errors.log.enable, and the retry settings for sources. It does not list the dead letter queue topic settings for sources.
An S3 sink is a sink connector, so the settings above apply to it. The S3 sink guide is the archive path. A DLQ is the path for records that sink could not write.
How do I build a DLQ in a consumer?
Catch the processing error. Produce the original key and value to a topic such as orders.dlq. Add headers for the source topic, partition, offset, and exception class.
Wait until that produce finishes. Then commit the next source offset. Those header names are yours. They are not the Connect __connect.errors. keys.
byte[] errorName = ex.getClass().getName()
.getBytes(StandardCharsets.UTF_8);
ProducerRecord<byte[], byte[]> out =
new ProducerRecord<>(
record.topic() + ".dlq",
null,
record.key(),
record.value(),
record.headers());
out.headers().add("error.exception", errorName);
producer.send(out).get();
TopicPartition tp = new TopicPartition(
record.topic(), record.partition());
OffsetAndMetadata next =
new OffsetAndMetadata(record.offset() + 1);
consumer.commitSync(Map.of(tp, next));
Add error.topic, error.partition, and error.offset the same way before send. get() blocks until the broker ack arrives. Commit only after that ack. StandardCharsets.UTF_8 is the charset for the header bytes.
Commit first and a failed produce drops the record. The consumer has moved on, and the DLQ never got it. Produce first and a crash before commit can write the DLQ record again. The DLQ consumer must treat the same source topic, partition, and offset as one failure.
Should I use a retry topic or a DLQ?
Use a retry topic when the record may succeed later. Use a DLQ when retries are exhausted, or when the record cannot succeed until code or the payload changes.
| Topic | Put a record here when |
|---|---|
<topic>.retry | A later attempt may work, and the retry count is still under the cap |
<topic>.dlq | The retry cap is spent, or the record fails until something changes |
A poison pill fails on every attempt. Publishing it back to the source topic from the same consumer loops. Cap the retries, then send it to the DLQ and commit the source offset.
How do I replay messages from a DLQ?
Read the DLQ with a consumer group that does not run the failing logic. Fix the bug or the payload first. Produce to the original topic. Keep the original key so the default partitioner picks the same partition.
The replayed record gets a new offset. Records produced after the failure can already sit ahead of that replay. Order on the source topic is not restored.
The sink must tolerate a second delivery. A DLQ produce that succeeded, followed by a crash before commit, writes a duplicate on the next start. An idempotent write keys off the source topic, partition, and offset headers.
Do not replay a poison pill until the handler or the record changes. A replay that still fails, into a consumer that still routes failures to the DLQ, fills the DLQ again.
How should I set retention and backup for a DLQ topic?
A DLQ topic uses normal topic retention. There is no separate DLQ timer. The broker default is 7 days, from log.retention.hours set to 168.
retention.ms defaults to 604800000 milliseconds. retention.bytes defaults to -1, which means no size cap. The retention policy guide walks through that chain.
Set retention.ms on the DLQ on purpose. After the consumer commits, the DLQ is often the only copy of the failed record. Source-topic retention can delete the original.
A longer window on the same cluster is the live log. Replication and MirrorMaker copy that log. Point-in-time recovery needs a copy outside the cluster, restored to a millisecond inside the backup window.
Back up the DLQ topic the same way you back up any topic. The point-in-time recovery page is that restore. The feature comparison separates replication from backup.
FAQ
Frequently asked questions
Does Kafka have a built-in dead letter queue?
No. The broker does not create a DLQ for a classic consumer group. Kafka Connect writes one for sink connectors, and Kafka Streams 4.2.0 writes one when errors.dead.letter.queue.topic.name is set. A share group can name a DLQ topic only when that topic sets errors.deadletterqueue.group.enable to true.
What is errors.tolerance in Kafka Connect?
It chooses whether a connector task fails or skips a bad record. The default is none, which fails the task on the first error, and all skips problematic records. Valid values are none and all. Pair all with errors.deadletterqueue.topic.name when a sink connector should keep the failed record.
How long should DLQ topics be retained?
There is no special DLQ timer. The topic uses normal retention, and the broker default is 7 days from log.retention.hours set to 168. retention.ms defaults to 604800000 milliseconds, and retention.bytes defaults to -1. Set a longer retention.ms when the DLQ is the only copy, and back the topic up.
How do I reprocess DLQ messages?
Read the DLQ with a separate consumer group and produce to the original topic. Wait until the bug or the bad payload is fixed. Keep the original key, because the new record gets a new offset. The sink must tolerate a duplicate, and a record that still fails goes back to the DLQ.
Does a Kafka Connect source connector have a DLQ?
No. The dead letter queue configs are sink connector settings. errors.deadletterqueue.topic.name applies to a sink connector, its transforms, or its converters. A source connector can set errors.tolerance and errors.log.enable, and it does not get a DLQ topic from those sink settings.
Conclusion
A Kafka dead letter queue is a topic. Kafka Connect writes it for sink connectors when errors.tolerance is all and errors.deadletterqueue.topic.name is set. A classic consumer produces that topic itself, then commits.
Kafka Streams 4.2.0 can send to errors.dead.letter.queue.topic.name on regular tasks. Commit only after the DLQ produce succeeds. Replay only after the failure is fixed, and make the write idempotent.
Set retention on the DLQ, and keep a backup. The topic may be the only copy of the failed record.
A DLQ topic still lives on the cluster. Back up the topic, then restore it with point-in-time recovery.