Documentation / logging
Logging
A log store for a Kubernetes cluster: OpenSearch and OpenSearch Dashboards, from Tersane’s opensearch chart, set up for logs.
- Five log streams, each in its own data stream with its own retention: container logs, ingress access logs, the API server’s audit log, Kubernetes events and node services.
- Index templates that fix the type of every field the dashboards use, and keep every other field out of the index, so one application’s JSON cannot break another’s.
- Five dashboards with twelve saved searches: cluster logs, ingress traffic, Kubernetes audit, Kubernetes events and the log pipeline’s health.
- Five alert monitors: a gap in the logs, ingress server errors, a Secret read by a person, a cluster administrator granted, and containers killed for memory.
- Roles that separate Fluent Bit, which may only write, from readers and administrators, with single sign-on through OpenID Connect.
- A Fluent Bit configuration that sends the five streams over TLS.
Prerequisites
- Kubernetes 1.33 or later, with cert-manager, as the opensearch chart needs.
- The kernel setting
vm.max_map_countat least 262144 on every node. - Fluent Bit, such as the one BKE installs, in the namespace named by
fluentBit.namespace(loggingby default), which must exist before the chart is installed. - For the audit stream, Fluent Bit running on control-plane nodes, and the
API server writing its audit log to
/var/log/kubernetes/audit/audit.log.
Installation
kubectl create namespace logs
kubectl create secret docker-registry 1x-ax -n logs \
--docker-server=1x.ax --docker-username=<account> --docker-password=<token>
helm install logs tersane/logging -n logs
Then give Fluent Bit the settings in examples/fluent-bit-values.yaml,
with the cluster’s name in place of CLUSTER_NAME; see
Fluent Bit.
After every install and upgrade, a Job sets up the streams, roles, saved objects and monitors, and writes the Secret Fluent Bit reads. Every step is safe to repeat. The last Job is kept until the next install or upgrade, and its log lists what it created, updated and left alone:
kubectl -n <namespace> logs job/<release>-logging-setup
Log streams and retention
| Stream | Data stream | Source | Retention (days) |
|---|---|---|---|
| Container logs | logs-containers |
every container’s output, with Kubernetes metadata | retention.containers, 30 |
| Ingress access | logs-ingress |
the ingress controller’s access log | retention.ingress, 90 |
| API server audit | logs-audit |
the audit log on control-plane nodes | retention.audit, 365 |
| Kubernetes events | logs-events |
the Kubernetes events API | retention.events, 30 |
| Node services | logs-host |
kubelet’s and containerd’s journal | retention.host, 30 |
Each stream rolls over to a new index daily, or when its index reaches
rollover.maxPrimaryShardSize, and an index is deleted when its retention
has passed since it rolled over. A change to a retention applies to indexes
already written, as well as new ones, after the next upgrade.
With archive.enabled, an index of the streams in archive.streams is
copied to the opensearch chart’s snapshot repository before it is deleted.
This keeps, for example, 30 days of audit log searchable and the rest in
object storage, from which a snapshot can be restored for an audit request.
How long the snapshots are kept is the bucket’s own lifecycle rule.
Fields the dashboards use
fields.yaml lists every field the dashboards, saved searches and monitors
read, per stream, with its type. The index templates map exactly these, and
the chart’s tests fail if a shipped object names any other. If your Fluent Bit
parses differently, check its output against this list: a renamed field
leaves its panels empty.
Fields not in the list are kept in each document but not indexed. To search on more fields, add them per stream:
extraMappings:
containers:
app.order_id: {type: keyword}
Ingress controllers
The ingress stream is the same whichever controller writes it. Its pipeline
reads these keys from a JSON access line, which Kong’s kong_json format
writes:
| Key | Becomes | Unit |
|---|---|---|
host |
url.domain |
|
request (“METHOD path PROTOCOL”) |
url.path |
|
request_method |
http.request.method |
|
response_status |
http.response.status_code |
|
request_time |
http.duration_ms |
seconds, stored in milliseconds |
upstream_latency, or upstream_response_time |
upstream.duration_ms |
milliseconds; seconds |
upstream_addr |
upstream.address |
|
remote_addr |
client.ip |
|
http_user_agent |
user_agent.original |
|
body_bytes_sent |
http.response.bytes |
For ingress-nginx, set its log-format-upstream to write the same keys:
controller:
config:
log-format-escape-json: "true"
log-format-upstream: '{"host":"$host","request":"$request","request_method":"$request_method","response_status":"$status","request_time":"$request_time","upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","remote_addr":"$remote_addr","http_user_agent":"$http_user_agent","body_bytes_sent":"$body_bytes_sent"}'
Dashboards
| Dashboard | The question it answers |
|---|---|
| Cluster logs | How much is each workload logging, and where are the errors? |
| Ingress traffic | Is traffic through the ingress healthy, and who is getting errors? |
| Kubernetes audit | Who changed what in the cluster, who read Secrets, and what was refused? |
| Kubernetes events | What is Kubernetes warning about: crashes, failed scheduling, failed probes, evictions? |
| Log pipeline health | Are logs still arriving from every node and every stream? |
The shipped objects’ IDs begin with tersane-logging-. On an upgrade, each
is replaced with the new version only if it is unchanged since it was
shipped. One you have edited is left as it is, and the new version is
imported beside it, its title followed by “(Tersane” and the chart’s version. One
you have deleted is not brought back.
Dashboards of your own are never touched: build them in Dashboards, or use “Save as” on a shipped one. To keep them in version control, export them as NDJSON and give them to the chart, which imports them the same way:
dashboards:
extraObjects: |
{"type":"dashboard","id":"payments","attributes":{...},"references":[...]}
Monitors
| Monitor | Fires when |
|---|---|
| Log gap | A node that sent container logs in the past hour has sent none for monitors.logGapMinutes, or the audit log has been silent for monitors.auditGapMinutes |
| Ingress server errors | More than monitors.ingressErrorPercent% of a host’s requests return 5xx over 5 minutes, with at least monitors.ingressMinRequests requests |
| Secret read by a person | A user who is not a service account or a system component reads a Secret |
| Cluster administrator granted | A binding to cluster-admin is created or changed |
| Containers killed for memory | A container is killed for exceeding its memory limit |
Alerts show in Dashboards under Alerting. To be told of them, create a
channel under Notifications, and give its ID as monitors.channelId.
Monitors are the chart’s: their thresholds are values, and a change made in
Dashboards is replaced on the next upgrade.
Access
| Role | May | Given to |
|---|---|---|
logs-writer |
write the log streams, and nothing else | the user fluent-bit |
logs-reader |
read every stream, use the dashboards, see alerts | the backend roles in access.readerBackendRoles |
logs-admin |
also manage retention, templates, pipelines and monitors | the backend roles in access.adminBackendRoles |
Role mappings are added to, never replaced, so mappings you make in
Dashboards stay. With single sign-on (opensearch.oidc, with
opensearch.security.authConfig.managed: true; see the opensearch chart’s
“Single sign-on”, which also says how to turn it off), the identity
provider’s roles claim supplies the backend roles. The
security plugin’s audit log records who searched which index.
Fluent Bit
examples/fluent-bit-values.yaml holds the settings for the fluent-bit Helm
chart:
- the five inputs, including the audit log on control-plane nodes, with a
toleration for them, and the node’s journal wherever journald keeps it:
/var/log/journalon disk or/run/log/journalin memory; - the Kubernetes metadata filter, lifting an application’s JSON line into
app.*; - the ingress controller’s lines re-tagged into the ingress stream;
- the cluster’s name on every record;
- one OpenSearch output per stream over TLS, verifying OpenSearch’s
certificate, as the user
fluent-bit, with a buffer on the node’s disk.
The setup Job writes the Secret logging-fluent-bit into Fluent Bit’s
namespace, with the user, its password, OpenSearch’s host and the CA
certificate. Its account may write only that Secret, and only in that
namespace. Set fluentBit.namespace to "" to leave the Secret out and
supply the settings another way.
Sizing
Measured on a cluster of five nodes running about 150 containers, an ingress serving about 16,000 requests an hour, and BKE’s audit policy:
| Stream | Stored per day | Kept | On disk, with one replica |
|---|---|---|---|
| Container logs | 1.2 to 3.2 GB | 30 days | 81 to 213 GB |
| Ingress access | 0.4 GB | 90 days | 77 GB |
| API server audit | 0.5 GB | 365 days | 426 GB |
| Kubernetes events | under 0.1 GB | 30 days | 2 GB |
| Node services | under 0.1 GB | 30 days | 3 GB |
Container logs vary most with how much applications and system components
log; the lower figure is with CoreDNS’s query logging off. With zstd
compression, the default, the figures are about a third lower. Keeping 30 days
of audit log in OpenSearch with archive.enabled brings the cluster’s total
to about 135 GB, or 55 GB per node of three with headroom, plus about 145 GB
in object storage.
OpenSearch stops writing to a node whose disk is 95% full, so size
opensearch.persistence.size for the total above, divided by the number of
nodes, with 15% to spare.
Uninstalling
helm uninstall removes OpenSearch and Dashboards, and keeps their volumes.
The Secret written for Fluent Bit is owned by the release’s Role in Fluent
Bit’s namespace, so Kubernetes deletes it with the release. The last setup
Job is not part of the release and is left; remove it with:
kubectl -n <namespace> delete job <release>-logging-setup
Parameters
| Parameter | Description | Default |
|---|---|---|
opensearch.* |
Settings of the opensearch chart | 3 nodes, 100Gi each, 4Gi memory, Dashboards on |
retention.containers / .ingress / .audit / .events / .host |
Days each stream is kept | 30 / 90 / 365 / 30 / 30 |
rollover.maxAge / .maxPrimaryShardSize |
When a stream’s index rolls over | 1d / 10gb |
codec |
Compression of stored logs | zstd |
archive.enabled / .streams |
Snapshot indexes before deletion | false / [audit] |
extraMappings |
Further searchable fields, per stream | {} |
fluentBit.namespace / .secretName |
Where the setup Job writes Fluent Bit’s Secret | logging / logging-fluent-bit |
access.readerBackendRoles / .adminBackendRoles |
Backend roles given reading and administration | [logs-reader] / [logs-admin] |
monitors.enabled / .channelId |
The alert monitors, and a Notifications channel | true / "" |
monitors.logGapMinutes / .auditGapMinutes |
The log gap monitor’s thresholds | 10 / 15 |
monitors.ingressErrorPercent / .ingressMinRequests |
The ingress monitor’s thresholds | 5 / 50 |
dashboards.extraObjects |
Saved objects of your own, as NDJSON | "" |
setup.resources |
The setup Job’s resources | 50m / 128Mi requested |