Documentation / rabbitmq

RabbitMQ

Installs RabbitMQ, the message broker: a single node, or a cluster of nodes that replicate quorum queues between them, so that every queue stays available while a node is down.

The chart provides:

• a cluster of three nodes by default, formed from a fixed list of the StatefulSet’s nodes, with no access to the Kubernetes API;
• quorum queues as the default queue type;
• restarts that never take a queue below its quorum: before a node stops, it waits until its queues can lose it and moves its clients to the other nodes;
• the feature flags of a new RabbitMQ release enabled after each upgrade, once every node runs it, as RabbitMQ requires before the next upgrade;
• users, virtual hosts, permissions and policies declared in values, with generated passwords kept across upgrades;
• AMQP over TLS, with a certificate of your own or one issued by cert-manager;
• the management interface, on an optional Ingress, and Prometheus metrics;
• a NetworkPolicy, and optional access from outside the cluster through Kong;
• the image from the Mamluk registry, pinned by digest, and every Pod meeting the restricted Pod Security Standard.

Prerequisites

• Kubernetes 1.33 or later.
• A pull secret for 1x.ax named 1x-ax in the release’s namespace, created from an image token as described in the access documentation.
• A StorageClass for the volumes: the cluster’s default, or one named in persistence.storageClass.
• For TLS with a certificate issued for you, cert-manager and an issuer. It is included with the Bahriya Kubernetes Engine; on other Kubernetes, install cert-manager first, or give a certificate of your own instead.

Installation

helm install events tersane/rabbitmq \
  --namespace events --create-namespace \
  --version <chart version>

A single node, with no replication:

helm install events tersane/rabbitmq --namespace events --set replicas=1

The release’s resources are named <release>-rabbitmq (here events-rabbitmq), or after the release alone if its name already contains rabbitmq. Check that every node is ready:

kubectl --namespace events get pods -l app.kubernetes.io/instance=events

Connecting

Applications connect to the Service <release>-rabbitmq:

Port	Protocol
5672	AMQP (closed when tls.requireTls is true)
5671	AMQP over TLS (with tls.enabled)
15672	Management interface and HTTP API
15692	Prometheus metrics

The Service reaches every node; a client may connect to any of them, and RabbitMQ routes its messages to the queue wherever its leader is.

Users, virtual hosts and permissions

The administrator (auth.username, admin by default) has every permission on every virtual host. Its password is in the Secret <release>-rabbitmq, key admin-password. Further users, virtual hosts and permissions are declared:

users:
  - name: orders
    tags: []
vhosts:
  - name: orders
permissions:
  - user: orders
    vhost: orders
    configure: ".*"
    write: ".*"
    read: ".*"

Each user’s password is generated once and kept in the Secret <release>-rabbitmq-user-<name> (keys username and password); an upgrade keeps it. To set it yourself, create a Secret with a password key and name it in the user’s passwordSecret. The administrator’s password and the Erlang cookie can likewise come from auth.existingSecret, with the keys admin-username, admin-password and erlang-cookie.

The declarations are imported when the nodes start, and again whenever they change, which restarts the nodes one at a time. The import adds and updates. A user, virtual host or permission removed from the lists is left in place; remove it with the management interface or rabbitmqctl. A password changed in a Secret of your own takes effect when the nodes next restart:

kubectl --namespace events rollout restart statefulset events-rabbitmq

Policies are given in RabbitMQ’s definitions format:

policies:
  - name: orders-length
    vhost: orders
    pattern: "^orders\\."
    apply-to: queues
    definition:
      max-length: 100000
    priority: 0

Queues and availability

defaultQueueType (quorum) is the type of every queue declared without one, in every virtual host. A quorum queue has a replica on every node and stays available while a majority of them is up: with three nodes, while one is down. A classic queue lives on one node and is unavailable while that node is.

With more than one node, the cluster stops serving on the minority side of a network partition (pause_minority), so that two halves never accept writes to the same queue.

Restarts and upgrades

Any change to the chart’s configuration restarts the nodes one at a time. Before a node stops, it runs rabbitmq-upgrade await_online_quorum_plus_one, which waits until every quorum queue it hosts has enough replicas online on the other nodes to stay available without it, and then rabbitmq-upgrade drain, which moves its clients and queue leaders to the other nodes. The wait is bounded by terminationGracePeriodSeconds. The PodDisruptionBudget lets a node drain take only one RabbitMQ node at a time.

After an upgrade to a new RabbitMQ release, a Job waits until every node runs it and then enables every feature flag the release offers. RabbitMQ refuses to upgrade a node whose feature flags are not all enabled, so this keeps the next upgrade possible. Upgrade one minor release at a time, as RabbitMQ’s upgrade documentation requires.

TLS

tls:
  enabled: true
  issuerRef:
    kind: ClusterIssuer
    name: my-issuer

cert-manager then issues a certificate for the Service <release>-rabbitmq in its short and namespaced forms, and any names in tls.extraDnsNames, into the Secret <release>-rabbitmq-tls. Clients verify the server with that Secret’s ca.crt. To use a certificate of your own instead, name a kubernetes.io/tls Secret with a ca.crt key in tls.existingSecret. Set tls.requireTls to close the unencrypted port.

Management interface

The management interface and HTTP API are on port 15672. To publish them:

ingress:
  enabled: true
  host: rabbitmq.example.com
  tlsSecret: rabbitmq-example-com-tls

TLS is terminated by the ingress controller. Sign in as the administrator, or as a user tagged management, monitoring, policymaker or administrator; RabbitMQ refuses the interface and the HTTP API to a user with no tag.

The management interface and API manage users, virtual hosts, queues, exchanges and bindings, but report no message counts or rates: the chart leaves RabbitMQ’s statistics to its Prometheus endpoint, which RabbitMQ recommends in place of the interface’s own collection (see Metrics).

Network policy

With networkPolicy.enabled (the default), the nodes admit:

• AMQP on 5672 and 5671 from networkPolicy.amqp.from, the management interface on 15672 from networkPolicy.management.from, and metrics on 15692 from networkPolicy.metrics.from, every source when a list is empty;
• always, the nodes themselves and the chart’s feature flag Job on the clustering ports, 4369 and 25672, and the chart’s helm test Pod;
• with external access, the ingress controller’s namespace on 5671.

External access

To reach RabbitMQ from outside the Kubernetes cluster through a Kong ingress controller, publish AMQPS on one of Kong’s TCP ports:

tls:
  enabled: true
  issuerRef: {kind: ClusterIssuer, name: my-issuer}
  extraDnsNames:
    - rabbitmq.example.com
exposure:
  external:
    enabled: true
    port: 5671
    ipAllowlist:
      - 203.0.113.0/24

Kong must already listen on that port as a TCP stream listener. It routes the connection by port and forwards it unchanged, so the client and RabbitMQ negotiate TLS end to end. TLS and an IP allow-list are required.

Metrics

Each node serves Prometheus metrics on port 15692, and carries prometheus.io/scrape and prometheus.io/port annotations (metrics.podAnnotations):

• /metrics: the node’s totals, such as connections, channels, memory and alarms.
• /metrics/detailed?family=queue_coarse_metrics: the depth of each queue (rabbitmq_detailed_queue_messages, _ready and _unacked), reported by the node that leads the queue. Add &vhost=<name> to limit it to one virtual host. Scrape this path on every node for a complete set.

Resources

The memory limit is given to RabbitMQ as the memory it may use, and publishers are blocked when RabbitMQ reaches memoryHighWatermark of it, and when free disk on the volume falls below diskFreeLimit. A node uses about 200 MB with no messages queued, so do not set the memory limit below the default of 1Gi: at 512Mi a node reaches the watermark at rest and blocks every publisher. Requests and limits for every container are set in resources (the nodes) and jobResources (their init container, the feature flag Job and the test).

Pod Security

Every Pod the chart creates meets the Kubernetes restricted Pod Security Standard: the nodes run as the image’s rabbitmq user, with a read-only root filesystem, no privilege escalation, no Linux capabilities and the container runtime’s default seccomp profile. Install it in a namespace that enforces restricted.

Testing

helm test events --namespace events

The test declares a queue as the administrator, publishes a message to it, reads it back and deletes the queue.

Uninstalling

helm uninstall events --namespace events

The nodes all stop at once, and each waits up to terminationGracePeriodSeconds for quorum replicas that will not return, so an uninstall of a cluster takes that long. The volumes are kept; delete them to remove the data:

kubectl --namespace events delete pvc -l app.kubernetes.io/instance=events

Parameters

RabbitMQ

Parameter	Description	Default
replicas	Nodes: 1 is a single node	3
image.repository	Image	1x.ax/tersane/core/rabbitmq/rabbitmq
image.tag	Image tag and digest	Set in each release
imagePullSecrets	Pull secrets of every Pod	[{name: 1x-ax}]
defaultQueueType	Queue type of a queue declared without one	quorum
memoryHighWatermark	Fraction of the memory limit at which publishers are blocked	0.4
diskFreeLimit	Free disk below which publishers are blocked	2GB
extraConfig	rabbitmq.conf lines appended to the chart’s	""
extraPlugins	Plugins enabled besides management and Prometheus	[]
clusterDomain	The cluster’s DNS domain	cluster.local

Access

Parameter	Description	Default
auth.username	The administrator	admin
auth.existingSecret	A Secret with admin-username, admin-password and erlang-cookie	generated
users	Users (name, tags, passwordSecret)	[]
vhosts	Virtual hosts (name, defaultQueueType)	[]
permissions	Permissions (user, vhost, configure, write, read)	[]
policies	Policies, in RabbitMQ’s definitions format	[]

TLS

Parameter	Description	Default
tls.enabled	AMQPS on 5671	false
tls.requireTls	Close AMQP on 5672	false
tls.existingSecret	A kubernetes.io/tls Secret with ca.crt	""
tls.issuerRef	The cert-manager issuer of the certificate	{}
tls.extraDnsNames	Names the certificate carries besides the Service’s	[]

Storage, resources and scheduling

Parameter	Description	Default
persistence.size	Volume size per node	8Gi
persistence.storageClass	StorageClass	the default class
resources	Requests and limits of each node	250m/1Gi, limits 1000m/1Gi
jobResources	Requests and limits of the init container, Job and test	50m/128Mi, limits 500m/256Mi
terminationGracePeriodSeconds	The longest a node waits before it stops	600
podDisruptionBudget.enabled / .maxUnavailable	Nodes a drain may take at once	true / 1
featureFlags.enableAllOnUpgrade	Enable a new release’s feature flags after an upgrade	true
featureFlags.timeoutSeconds	How long the Job waits for every node to run the release	900
affinity.topologyKey	Topology the nodes are spread across	kubernetes.io/hostname
affinity.podAntiAffinityType	preferred or required	preferred
nodeSelector, tolerations, priorityClassName	Placement of the nodes	none

Network and metrics

Parameter	Description	Default
service.type / service.annotations	The client Service	ClusterIP / {}
ingress.enabled	Publish the management interface	false
ingress.host / .ingressClassName / .tlsSecret / .annotations	The Ingress	"" / kong / "" / {}
networkPolicy.enabled	Create the NetworkPolicy	true
networkPolicy.amqp.from / .management.from / .metrics.from	Sources admitted	[] (every source)
exposure.external.enabled	Publish AMQPS through Kong	false
exposure.external.port / .ipAllowlist	Kong’s TCP port; client addresses admitted	0 / []
exposure.external.ingressClassName / .ingressNamespace / .annotations	Kong’s class and namespace; extra annotations	kong / kong / {}
metrics.podAnnotations	Scrape annotations on the nodes	true