Bahriya Tersane

Documentation / logging

Logging

A log store for a Kubernetes cluster: OpenSearch and OpenSearch Dashboards, from Tersane’s opensearch chart, set up for logs.

  • Five log streams, each in its own data stream with its own retention: container logs, ingress access logs, the API server’s audit log, Kubernetes events and node services.
  • Index templates that fix the type of every field the dashboards use, and keep every other field out of the index, so one application’s JSON cannot break another’s.
  • Five dashboards with twelve saved searches: cluster logs, ingress traffic, Kubernetes audit, Kubernetes events and the log pipeline’s health.
  • Five alert monitors: a gap in the logs, ingress server errors, a Secret read by a person, a cluster administrator granted, and containers killed for memory.
  • Roles that separate Fluent Bit, which may only write, from readers and administrators, with single sign-on through OpenID Connect.
  • A Fluent Bit configuration that sends the five streams over TLS.

Prerequisites

  • Kubernetes 1.33 or later, with cert-manager, as the opensearch chart needs.
  • The kernel setting vm.max_map_count at least 262144 on every node.
  • Fluent Bit, such as the one BKE installs, in the namespace named by fluentBit.namespace (logging by default), which must exist before the chart is installed.
  • For the audit stream, Fluent Bit running on control-plane nodes, and the API server writing its audit log to /var/log/kubernetes/audit/audit.log.

Installation

kubectl create namespace logs
kubectl create secret docker-registry 1x-ax -n logs \
  --docker-server=1x.ax --docker-username=<account> --docker-password=<token>
helm install logs tersane/logging -n logs

Then give Fluent Bit the settings in examples/fluent-bit-values.yaml, with the cluster’s name in place of CLUSTER_NAME; see Fluent Bit.

After every install and upgrade, a Job sets up the streams, roles, saved objects and monitors, and writes the Secret Fluent Bit reads. Every step is safe to repeat. The last Job is kept until the next install or upgrade, and its log lists what it created, updated and left alone:

kubectl -n <namespace> logs job/<release>-logging-setup

Log streams and retention

Stream Data stream Source Retention (days)
Container logs logs-containers every container’s output, with Kubernetes metadata retention.containers, 30
Ingress access logs-ingress the ingress controller’s access log retention.ingress, 90
API server audit logs-audit the audit log on control-plane nodes retention.audit, 365
Kubernetes events logs-events the Kubernetes events API retention.events, 30
Node services logs-host kubelet’s and containerd’s journal retention.host, 30

Each stream rolls over to a new index daily, or when its index reaches rollover.maxPrimaryShardSize, and an index is deleted when its retention has passed since it rolled over. A change to a retention applies to indexes already written, as well as new ones, after the next upgrade.

With archive.enabled, an index of the streams in archive.streams is copied to the opensearch chart’s snapshot repository before it is deleted. This keeps, for example, 30 days of audit log searchable and the rest in object storage, from which a snapshot can be restored for an audit request. How long the snapshots are kept is the bucket’s own lifecycle rule.

Fields the dashboards use

fields.yaml lists every field the dashboards, saved searches and monitors read, per stream, with its type. The index templates map exactly these, and the chart’s tests fail if a shipped object names any other. If your Fluent Bit parses differently, check its output against this list: a renamed field leaves its panels empty.

Fields not in the list are kept in each document but not indexed. To search on more fields, add them per stream:

extraMappings:
  containers:
    app.order_id: {type: keyword}

Ingress controllers

The ingress stream is the same whichever controller writes it. Its pipeline reads these keys from a JSON access line, which Kong’s kong_json format writes:

Key Becomes Unit
host url.domain  
request (“METHOD path PROTOCOL”) url.path  
request_method http.request.method  
response_status http.response.status_code  
request_time http.duration_ms seconds, stored in milliseconds
upstream_latency, or upstream_response_time upstream.duration_ms milliseconds; seconds
upstream_addr upstream.address  
remote_addr client.ip  
http_user_agent user_agent.original  
body_bytes_sent http.response.bytes  

For ingress-nginx, set its log-format-upstream to write the same keys:

controller:
  config:
    log-format-escape-json: "true"
    log-format-upstream: '{"host":"$host","request":"$request","request_method":"$request_method","response_status":"$status","request_time":"$request_time","upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","remote_addr":"$remote_addr","http_user_agent":"$http_user_agent","body_bytes_sent":"$body_bytes_sent"}'

Dashboards

Dashboard The question it answers
Cluster logs How much is each workload logging, and where are the errors?
Ingress traffic Is traffic through the ingress healthy, and who is getting errors?
Kubernetes audit Who changed what in the cluster, who read Secrets, and what was refused?
Kubernetes events What is Kubernetes warning about: crashes, failed scheduling, failed probes, evictions?
Log pipeline health Are logs still arriving from every node and every stream?

The shipped objects’ IDs begin with tersane-logging-. On an upgrade, each is replaced with the new version only if it is unchanged since it was shipped. One you have edited is left as it is, and the new version is imported beside it, its title followed by “(Tersane” and the chart’s version. One you have deleted is not brought back.

Dashboards of your own are never touched: build them in Dashboards, or use “Save as” on a shipped one. To keep them in version control, export them as NDJSON and give them to the chart, which imports them the same way:

dashboards:
  extraObjects: |
    {"type":"dashboard","id":"payments","attributes":{...},"references":[...]}

Monitors

Monitor Fires when
Log gap A node that sent container logs in the past hour has sent none for monitors.logGapMinutes, or the audit log has been silent for monitors.auditGapMinutes
Ingress server errors More than monitors.ingressErrorPercent% of a host’s requests return 5xx over 5 minutes, with at least monitors.ingressMinRequests requests
Secret read by a person A user who is not a service account or a system component reads a Secret
Cluster administrator granted A binding to cluster-admin is created or changed
Containers killed for memory A container is killed for exceeding its memory limit

Alerts show in Dashboards under Alerting. To be told of them, create a channel under Notifications, and give its ID as monitors.channelId. Monitors are the chart’s: their thresholds are values, and a change made in Dashboards is replaced on the next upgrade.

Access

Role May Given to
logs-writer write the log streams, and nothing else the user fluent-bit
logs-reader read every stream, use the dashboards, see alerts the backend roles in access.readerBackendRoles
logs-admin also manage retention, templates, pipelines and monitors the backend roles in access.adminBackendRoles

Role mappings are added to, never replaced, so mappings you make in Dashboards stay. With single sign-on (opensearch.oidc, with opensearch.security.authConfig.managed: true; see the opensearch chart’s “Single sign-on”, which also says how to turn it off), the identity provider’s roles claim supplies the backend roles. The security plugin’s audit log records who searched which index.

Fluent Bit

examples/fluent-bit-values.yaml holds the settings for the fluent-bit Helm chart:

  • the five inputs, including the audit log on control-plane nodes, with a toleration for them, and the node’s journal wherever journald keeps it: /var/log/journal on disk or /run/log/journal in memory;
  • the Kubernetes metadata filter, lifting an application’s JSON line into app.*;
  • the ingress controller’s lines re-tagged into the ingress stream;
  • the cluster’s name on every record;
  • one OpenSearch output per stream over TLS, verifying OpenSearch’s certificate, as the user fluent-bit, with a buffer on the node’s disk.

The setup Job writes the Secret logging-fluent-bit into Fluent Bit’s namespace, with the user, its password, OpenSearch’s host and the CA certificate. Its account may write only that Secret, and only in that namespace. Set fluentBit.namespace to "" to leave the Secret out and supply the settings another way.

Sizing

Measured on a cluster of five nodes running about 150 containers, an ingress serving about 16,000 requests an hour, and BKE’s audit policy:

Stream Stored per day Kept On disk, with one replica
Container logs 1.2 to 3.2 GB 30 days 81 to 213 GB
Ingress access 0.4 GB 90 days 77 GB
API server audit 0.5 GB 365 days 426 GB
Kubernetes events under 0.1 GB 30 days 2 GB
Node services under 0.1 GB 30 days 3 GB

Container logs vary most with how much applications and system components log; the lower figure is with CoreDNS’s query logging off. With zstd compression, the default, the figures are about a third lower. Keeping 30 days of audit log in OpenSearch with archive.enabled brings the cluster’s total to about 135 GB, or 55 GB per node of three with headroom, plus about 145 GB in object storage.

OpenSearch stops writing to a node whose disk is 95% full, so size opensearch.persistence.size for the total above, divided by the number of nodes, with 15% to spare.

Uninstalling

helm uninstall removes OpenSearch and Dashboards, and keeps their volumes. The Secret written for Fluent Bit is owned by the release’s Role in Fluent Bit’s namespace, so Kubernetes deletes it with the release. The last setup Job is not part of the release and is left; remove it with:

kubectl -n <namespace> delete job <release>-logging-setup

Parameters

Parameter Description Default
opensearch.* Settings of the opensearch chart 3 nodes, 100Gi each, 4Gi memory, Dashboards on
retention.containers / .ingress / .audit / .events / .host Days each stream is kept 30 / 90 / 365 / 30 / 30
rollover.maxAge / .maxPrimaryShardSize When a stream’s index rolls over 1d / 10gb
codec Compression of stored logs zstd
archive.enabled / .streams Snapshot indexes before deletion false / [audit]
extraMappings Further searchable fields, per stream {}
fluentBit.namespace / .secretName Where the setup Job writes Fluent Bit’s Secret logging / logging-fluent-bit
access.readerBackendRoles / .adminBackendRoles Backend roles given reading and administration [logs-reader] / [logs-admin]
monitors.enabled / .channelId The alert monitors, and a Notifications channel true / ""
monitors.logGapMinutes / .auditGapMinutes The log gap monitor’s thresholds 10 / 15
monitors.ingressErrorPercent / .ingressMinRequests The ingress monitor’s thresholds 5 / 50
dashboards.extraObjects Saved objects of your own, as NDJSON ""
setup.resources The setup Job’s resources 50m / 128Mi requested