Kafka RecordTooLargeException: Fix Message Too Large

Quick answer​

A Kafka message too large error means the record batch is over a cap. Raise producer max.request.size, and buffer.memory when one record exceeds it. Raise broker message.max.bytes and topic max.message.bytes to the same ceiling. On current Kafka, consumer and replica fetch sizes are not hard caps. Still set them with the batch limit. Change the produce-side caps together.

What causes RecordTooLargeException?​

org.apache.kafka.common.errors.RecordTooLargeException:
The message is 2000000 bytes when serialized which is larger than 1048576,
which is the value of the max.request.size configuration.

The RecordTooLargeException javadoc states the rule in one line. This record is larger than the maximum allowable size.

The producer can throw that before the broker sees the record. The exception names max.request.size when the serialized message is larger than that cap. The same method throws when the record is larger than buffer.memory. Both messages are in KafkaProducer on the 4.3.1 tag.

The broker throws it when the record batch is too big. Protocol error 10 is MESSAGE_TOO_LARGE. Errors.java defines that text. The request included a message larger than the max message size the server will accept. The client reports that as RecordTooLargeException.

The cap applies to the record batch. message.max.bytes is the largest record batch size allowed by Kafka, after compression if compression is enabled. A batch of many small records can fail it. One record that cannot fit inside a legal batch fails it too.

Which Kafka settings control maximum message size?​

These defaults are from the Apache Kafka 4.3 configuration reference. Set the three produce-side caps to one number you choose. The table shows the stock defaults, not a recommended production size.

max.request.size limits how many record batches go in one request. It is also a cap on the maximum uncompressed record batch size. The server has its own cap on the batch after compression. That server cap can differ from the producer cap.

buffer.memory is the total bytes the producer can use to buffer records. A single serialized record above that total throws RecordTooLargeException as well.

Topic max.message.bytes uses the same definition as the broker setting. Its server default property is message.max.bytes. A topic override wins over that default.

How do I fix a Kafka message too large error on the producer?​

Read the exception text first. It tells you which client cap failed.

1. Raise max.request.size when the exception names it. Set it above the serialized size printed in the exception. A 2 MB failure against the default means the client never called the broker.

2. Raise buffer.memory when the exception names it. The default is 33554432. A record under that size does not need a larger buffer. Keep buffer.memory at or above max.request.size.

3. Restart the producer process. Both settings live on the client. A broker change does not reload them.

max.request.size=10485760
# Raise buffer.memory only when one record exceeds 33554432.
# buffer.memory=67108864
compression.type=zstd

10485760 is an example ceiling of 10 MiB. Use the size of your largest real batch. Do not copy a round number from another cluster.

Producer compression.type defaults to none. Valid values are none, gzip, snappy, lz4, and zstd. Compression applies to full batches. More batching means better compression. It does not shrink the upper bound used for the max.request.size check.

How do I fix it on the broker and topic?​

The broker rejects the batch when the compressed record batch exceeds the effective topic cap. Raising only message.max.bytes is not enough if the topic already has a max.message.bytes override.

Kafka's Updating Broker Configs section defines the update modes. read-only needs a broker restart. cluster-wide can change on a running cluster. message.max.bytes is cluster-wide.

Updating Default Topic Configuration lists max.message.bytes as a default you can change without a restart. That default applies to topics that do not already override it.

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type brokers --entity-default \
--alter --add-config message.max.bytes=10485760

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type brokers --entity-default --describe

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders \
--alter --add-config max.message.bytes=10485760

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders --describe

mode: backup
backup_id: "large-record-snapshot"

source:
bootstrap_servers:
  - broker-1.prod.kafka:9092
topics:
  include:
    - "orders"

storage:
backend: s3
bucket: my-kafka-backups
region: us-west-2
prefix: backups/orders

backup:
compression: zstd
fetch_max_bytes: 10485760
include_offset_headers: true

Describe the topic before you assume the broker default is in force. A listed max.message.bytes is an override. It stays until you alter that topic.

On Confluent Cloud the cluster type adds its own cap. The cluster types table lists message size as 8 MB on Basic and Standard. Enterprise, Dedicated, and Freight list 20 MB. A client setting above that cloud cap still fails.

Why do consumers or replicas still fail after raising the limit?​

Usually the record was never committed. The consumer cannot read a batch the broker rejected. Check the producer exception and the topic override before you touch fetch settings.

On Kafka 4.3 the fetch settings are not hard maxima. max.partition.fetch.bytes still returns the first record batch in the first non-empty partition when that batch is larger than the limit. The consumer can make progress. fetch.max.bytes uses the same rule. The docs call it not an absolute maximum. The batch size the broker will accept is still message.max.bytes or topic max.message.bytes.

replica.fetch.max.bytes is the follower equivalent. It is not an absolute maximum. The first oversized batch is still returned so the follower can make progress. replica.fetch.response.max.bytes works the same way for the whole response. Both are read-only. Changing them takes a rolling restart.

Set the fetch values at or above your batch ceiling anyway. Older clients and some ops checks still compare them to max.message.bytes. During that restart, a broker can leave the ISR. Writes with acks=all then fail with a different error. Use the not enough replicas runbook for that case. Do not treat it as another size limit.

socket.request.max.bytes defaults to 104857600 (100 mebibytes). It is read-only. Keep each request under that cap, or change server.properties and restart.

Should I compress or split large messages instead?​

Compress when the payload is text or JSON and the broker is the side that rejects the batch. The broker measures the batch after compression. A batch that shrinks under max.message.bytes will pass that check.

Compression does not clear a producer-side failure. max.request.size caps the uncompressed record batch. The client throws before a smaller compressed batch exists.

Do not raise cluster limits for photos, files, or other opaque blobs. Put the bytes in object storage. Write a short pointer record to Kafka. Brokers, consumers, and replicas then move a small record. That claim-check pattern stays stable as the file grows.

Backup compression is a different knob. OSO Kafka Backup compresses archived segments with Zstd or LZ4. That shrinks the object you store. It does not change the Kafka batch the broker will accept. The codecs and levels are in the compression architecture.

Do oversized records affect backup and restore?​

Yes. A backup reads the topic, and a restore writes the batches to a target cluster. The target must accept the same record batch the source did.

backup.fetch_max_bytes is the max bytes requested per Fetch. The configuration reference says an unset value requests the smaller of segment_max_bytes and 16777216. The default segment size is 134217728, so the usual request is 16 MiB. Raise fetch_max_bytes when your batches are larger than that request. The sample in the tabs above is 10 MiB.

The well-architected FAQ says OSO Kafka Backup has been tested with messages up to 10 MB. If the cluster maximum is not the default, match the backup consumer configuration to it.

A restore into a tighter cluster fails the same way a producer does. The MSK migration precheck reports RecordTooLargeException when target message.max.bytes is below the source topic's effective max.message.bytes. It also warns when replica.fetch.max.bytes is below that source cap. Fix the target before replay.

Replica fetch is how brokers copy data to each other. That copy is not a backup. It lives and dies with the cluster. An independent copy is a backup in external storage. Connection and storage failures for that job are in common backup errors.