Kafka Backup to NetApp StorageGRID (S3)

Quick Answer​

NetApp StorageGRID exposes an S3-compatible API. OSO Kafka Backup writes to it through the s3 storage backend. Point endpoint at the StorageGRID load balancer, and set access_key and secret_key from a tenant account.

A custom HTTPS endpoint uses path-style requests. Restore reads the same bucket and prefix.

What is NetApp StorageGRID?​

NetApp StorageGRID is a software-defined object storage system with a native Amazon S3 API. It runs on appliances, virtual machines, or Linux containers, at one site or many. Tenant accounts separate the buckets stored for different teams.

The StorageGRID primer describes that model. Clients store and read objects through the S3 REST API. NetApp documents StorageGRID against the storage requirements of SEC Rule 17a-4(f), FINRA Rule 4511(c), and CFTC 17 CFR § 1.31(c)-(d).

Inside the grid, information lifecycle management (ILM) can keep replicated or erasure-coded object copies. Those copies protect objects already stored in StorageGRID. They are not a backup of Kafka.

Can I back up Kafka to StorageGRID?​

Yes. OSO Kafka Backup uses the s3 backend and a custom endpoint. StorageGRID accepts standard S3 bucket and object calls, so no StorageGRID adapter is required. The configuration reference documents that endpoint for S3-compatible stores.

The backup holds topic records, consumer group offsets, and topic settings outside the brokers. ILM placement and any later grid copies apply only after those objects are already in StorageGRID.

The same s3 backend is what the Amazon S3, MinIO, and Dell ECS pages use. The StorageGRID endpoint, tenant keys, and certificate differ. A managed Kafka cluster on Instaclustr is a separate integration.

How do I configure OSO Kafka Backup for a StorageGRID endpoint?​

Create an S3 tenant, a bucket, and an access key in Tenant Manager. An access key is an access key ID plus a secret access key. Then create an HTTPS load balancer endpoint and point the backup at that URL.

The client connection guide lists the values an S3 application needs. Those are the endpoint host and port, the certificate, the access key, and the bucket name. Access key management covers how tenant users create keys.

mode: backup
backup_id: "storagegrid-daily"

source:
bootstrap_servers:
  - kafka-1.internal:9092
  - kafka-2.internal:9092
topics:
  include:
    - "*"
  exclude:
    - "__consumer_offsets"

storage:
backend: s3
bucket: kafka-backup-grid
region: us-east-1
endpoint: https://s3.grid.example.com:10433
access_key: ${STORAGEGRID_ACCESS_KEY}
secret_key: ${STORAGEGRID_SECRET_KEY}
prefix: backups/prod-cluster

backup:
compression: zstd
compression_level: 5
include_offset_headers: true

mode: restore
backup_id: "storagegrid-daily"

target:
bootstrap_servers:
  - kafka-dr-1.internal:9092

storage:
backend: s3
bucket: kafka-backup-grid
region: us-east-1
endpoint: https://s3.grid.example.com:10433
access_key: ${STORAGEGRID_ACCESS_KEY}
secret_key: ${STORAGEGRID_SECRET_KEY}
prefix: backups/prod-cluster

restore:
# Restore through the moment before the incident (Unix ms)
time_window_end: 1754006400000
consumer_group_strategy: header-based

kafka-backup backup --config backup.yaml
kafka-backup validate-restore --config restore.yaml
kafka-backup restore --config restore.yaml

OSO compresses topic records with Zstd or LZ4 before upload. The storage format reference explains the objects written under the prefix.

Which load balancer endpoint should I use?​

Use an HTTPS load balancer endpoint on a Gateway Node or an Admin Node. S3 clients can also call Storage Nodes directly. The client connection steps start with that load balancer endpoint.

The first load balancer endpoint defaults to port 10433. You can choose another unused port. Ports 80 and 443 run the load balancer on Gateway Nodes only, because Admin Nodes reserve them. These rules are in the load balancing guide.

Which region and addressing style should I set?​

A custom endpoint implies path-style requests. StorageGRID accepts both path-style and virtual-hosted-style requests. Virtual-hosted-style also needs S3 endpoint domain names, DNS records, and a certificate that covers the bucket hostnames. This configuration uses path-style, so those bucket DNS names are not required.

Set region to us-east-1 when the bucket uses the StorageGRID default. CreateBucket uses us-east-1 unless the request names a region defined in Grid Manager. A name that is not defined returns an error, per bucket operations.

The region field is the value used to sign requests. On a global namespace endpoint, the SigV4 region must match the grid default, except for CreateBucket and ListBuckets. See global namespace.

How do I handle TLS and private certificates?​

Use HTTPS on the load balancer endpoint. Attach a server certificate when you create that endpoint. You can upload a certificate, generate one, or use a custom global S3 API certificate.

The load balancer endpoint guide lists those choices. The endpoint hostname must match a name on the certificate.

The storage configuration has no CA field. If the grid CA or your private CA signed the certificate, the backup process must trust that CA. NetApp notes that you may need to install the Grid CA in the S3 client, depending on the root CA. See S3 API certificates.

On Kubernetes, the operator chart documents a mounted CA bundle and SSL_CERT_FILE for a private CA. The operator configuration shows that pattern. It is not a key under storage.

How do retention and object lock work with Kafka backups?​

ILM decides where StorageGRID keeps an object and for how long. An active policy matches each object to an ordered rule. The default rule covers objects that earlier rules skip, as ILM policies describe. That policy never reads Kafka topics or consumer offsets.

S3 Object Lock is a grid setting, not a backup setting. A grid administrator enables it globally before a tenant can create a locked bucket. After that global setting is on, it stays on.

A version under a legal hold cannot be deleted. A version cannot be deleted before its retain-until date. Object retention describes that order. Object Lock requirements cover the global setting.

Those locks can block pruning of old backup objects. Align the retain-until window with the backup retention schedule before you rely on deletes.

How do I restore topics from StorageGRID?​

Keep the same bucket, endpoint, region, credentials, and prefix on the restore config. Point target.bootstrap_servers at the cluster that should receive the records. Set time_window_end to the last millisecond you want. The point-in-time restore guide walks through that window.

consumer_group_strategy: header-based uses the offset headers stored at backup time. include_offset_headers defaults to true. Run kafka-backup validate-restore before kafka-backup restore. Both commands are in the CLI reference.