Bahriya Tersane

Documentation / kafka

Kafka

Installs Apache Kafka in KRaft mode: a single node, or a cluster of nodes that are each a broker and a controller, with the cluster’s metadata kept by the controllers’ quorum and no ZooKeeper.

The chart provides:

  • a cluster of three nodes by default, with three replicas of every partition and writes acknowledged by two;
  • TLS on every connection, with a certificate authority of the cluster’s own, made with cert-manager;
  • SASL/SCRAM-SHA-512 authentication for every client and between brokers, and mutual TLS between the controllers;
  • access only where an ACL allows it;
  • users, topics and ACLs declared in values, with generated passwords;
  • rolling restarts that take one node at a time, each waiting until the node before it is back in sync;
  • the metadata version raised after each upgrade, once every node runs the new release;
  • a NetworkPolicy, and Apache’s image from the Mamluk registry, pinned by digest, with every Pod meeting the restricted Pod Security Standard.

Prometheus metrics are not yet included; they are planned for a later version of the chart.

Prerequisites

  • Kubernetes 1.33 or later.
  • cert-manager, for the cluster’s certificates. It is included with the Bahriya Kubernetes Engine; on other Kubernetes, install it first.
  • A pull secret for 1x.ax named 1x-ax in the release’s namespace, created from an image token as described in the access documentation.
  • A StorageClass for the volumes: the cluster’s default, or one named in persistence.storageClass.

Installation

helm install events tersane/kafka \
  --namespace events --create-namespace \
  --version <chart version>

A single node, with one replica of each partition:

helm install events tersane/kafka --namespace events --set replicas=1

The release’s resources are named <release>-kafka (here events-kafka), or after the release alone if its name already contains kafka. Check that every node is ready:

kubectl --namespace events get pods -l app.kubernetes.io/instance=events

Connecting

Clients bootstrap from <release>-kafka:9092 (in full, <release>-kafka.<namespace>.svc.cluster.local:9092), and Kafka then gives them each node’s address. They connect with:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="orders-service" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka/ca.crt

The certificate authority is ca.crt in the Secret <release>-kafka-certs. The administrator is admin, a super user; its password is in the Secret <release>-kafka, key admin-password.

Users, topics and ACLs

users:
  - name: orders-service
topics:
  - name: orders
    partitions: 6
    config:
      retention.ms: "604800000"
acls:
  - user: orders-service
    topic: orders
    operations: [Read, Write, Describe]
  - user: orders-service
    group: orders-
    patternType: prefixed
    operations: [Read]

After every install and upgrade, a Job creates each user, with a password generated once and kept in the Secret <release>-kafka-user-<name> (keys username and password); creates each topic that does not exist and sets its configuration; and adds each ACL. To set a user’s password yourself, create a Secret with a password key and name it in the user’s passwordSecret.

Every client needs an ACL for what it does, since access is denied where none allows it. A consumer needs Read on its topic and on its consumer group; a producer Write and Describe on its topic. The Job adds and updates, and removes nothing: a user, topic or ACL removed from the lists is removed with Kafka’s tools (kafka-configs.sh, kafka-topics.sh, kafka-acls.sh), as the administrator. A topic’s partition count can be raised by declaring more, and never lowered. Topics are not created by clients that write to a topic that does not exist (autoCreateTopics: false).

Availability

With three nodes, the controller quorum and every partition survive the loss of one node. Topics default to three replicas, and a write acknowledged by all in-sync replicas (acks=all) needs two of them (min.insync.replicas=2). A PodDisruptionBudget lets a node drain take one Kafka node at a time.

Rolling restarts

A change to the chart’s configuration restarts the nodes one at a time. A stopping node first moves the leadership of its partitions to the others. A restarted node is ready only once it is back in sync for every partition it holds a replica of, and the next node is not restarted until then, so a rolling restart never leaves a partition with fewer in-sync replicas than one node’s loss would.

Upgrades

Upgrade one minor release at a time. After an upgrade, a Job raises the cluster’s metadata version to the new release’s once every node runs it, which the next upgrade needs. Until then the cluster can still be returned to the previous release.

Adding nodes

The controllers are fixed when the cluster is created, so replicas should be decided at install. Raising it later adds brokers that hold no partitions until partitions are moved to them; Kafka does not move them by itself. Move them with kafka-reassign-partitions.sh, as the administrator:

kafka-reassign-partitions.sh --bootstrap-server events-kafka:9092 \
  --command-config admin.properties \
  --topics-to-move-json-file topics.json --broker-list 0,1,2,3 --generate

Review the proposed plan, save it, and apply it with --execute, then --verify.

External access

The chart serves clients inside the Kubernetes cluster only. A Kafka client outside it first reaches any node and is then given every node’s address, and must reach each of them, so publishing Kafka needs one address per node, not one for the cluster. Behind a Kong ingress controller, that is one TCP port per node, each routed to that node alone, and a further listener on each node that advertises the port clients outside use. This is not configured by the chart; Mamluk can set it up as a service.

Network policy

With networkPolicy.enabled (the default), the nodes admit:

  • clients on 9092 from networkPolicy.clients.from, every source when it is empty;
  • always, the nodes themselves on 9092 and 9093, and the chart’s Jobs and test on 9092.

Resources

The Java heap is half the memory limit; the rest is left to the operating system’s page cache, which Kafka relies on, and to the readiness check, which runs a Kafka tool. Give the memory limit in Mi or Gi.

Certificates

The nodes’ certificate is valid for one year, and cert-manager renews it thirty days before it expires. The nodes read it when they start; restart them after a renewal, one at a time:

kubectl --namespace events rollout restart statefulset events-kafka

Pod Security

Every Pod the chart creates meets the Kubernetes restricted Pod Security Standard: it runs as the image’s user, with a read-only root filesystem, no privilege escalation, no Linux capabilities and the container runtime’s default seccomp profile. Install it in a namespace that enforces restricted.

Testing

helm test events --namespace events

The test creates a topic as the administrator, produces a message to it, consumes it back and deletes the topic.

Uninstalling

helm uninstall events --namespace events

The volumes are kept; delete them to remove the data:

kubectl --namespace events delete pvc -l app.kubernetes.io/instance=events

Parameters

Parameter Description Default
replicas Nodes, each a broker and a controller 3
image.repository / image.tag Apache’s Kafka image, by digest 1x.ax/tersane/core/apache/kafka / the release’s
imagePullSecrets Pull secrets of every Pod [{name: 1x-ax}]
auth.existingSecret A Secret with admin-password and cluster-id generated
extraDnsNames Names the certificate carries besides the Services’ []
users Users (name, passwordSecret) []
topics Topics (name, partitions, replicationFactor, config) []
acls ACLs (user, topic or group, patternType, operations) []
config server.properties settings added to the chart’s {}
autoCreateTopics Let clients create topics by writing to them false
persistence.size / .storageClass Volume size per node; StorageClass 20Gi / the default class
resources Requests and limits of each node; the heap is half the memory limit 500m/2Gi, limits 2000m/2Gi
toolResources Requests and limits of the init container, Jobs and test 100m/384Mi, limits 1000m/384Mi
terminationGracePeriodSeconds The longest a node takes to stop 300
podDisruptionBudget.enabled / .maxUnavailable Nodes a drain may take at once true / 1
features.upgradeOnUpgrade / .timeoutSeconds Raise the metadata version after an upgrade true / 900
affinity.topologyKey / .podAntiAffinityType Spread of the nodes kubernetes.io/hostname / preferred
nodeSelector, tolerations, priorityClassName Placement of the nodes none
networkPolicy.enabled / .clients.from The NetworkPolicy; sources admitted to 9092 true / [] (every source)