Documentation / compliance
Compliance and Assurance Convoy
Evidences a regulatory regime’s technical controls in a Kubernetes cluster. The Convoy is experimental.
No regime’s content is in the chart. Each regime’s evidence model is an image
of its own, 1x.ax/tersane/convoy/compliance/regimes/<regime>/catalogue, and
the chart holds only its reference, pinned by digest, in files/regimes.json.
The chart is granted with the Convoy and each regime is granted on its own, so
a regime the licence does not grant cannot be pulled, and stops alone.
Prerequisites
Kubernetes 1.36 or later, which the Bahriya Kubernetes Engine provides from release 2.2: each regime’s evidence model is mounted as an image volume, and the container runtime must support image volumes.
The Bahriya Tersane kyverno and trivy-operator charts, installed
separately. The chart refuses to install without them. To render it outside a
cluster, pass their APIs:
helm template compliance . \
--api-versions policies.kyverno.io/v1/ImageValidatingPolicy \
--api-versions aquasecurity.github.io/v1alpha1/ClusterComplianceReport
Regimes
| Regime | Value | Catalogue |
|---|---|---|
| NCA Essential Cybersecurity Controls, ECC-2:2024 | nca-ecc |
176 requirements |
Evidence
One CronJob per regime, <release>-collect-<regime>, runs the evidence
collector (daily at 02:17 by default, collector.schedule). It mounts the
regime’s image as a read-only image volume, pulled on every run, so the
licence is asked each time: a regime that cannot be pulled stops at the pull,
the collector never starts, and the other regimes’ runs carry on. It reads
the cluster with a ClusterRole that may only get and list. The only thing it
writes through the API is the node collector’s Jobs, in a namespace of their
own (below).
Each run is a directory on the regime’s volume, <release>-evidence-<regime>,
named by its UTC time: the report in HTML and PDF, OSCAL assessment results,
the full results, the snapshot of what was read, and a SHA-256 manifest,
signed when evidence.signingKey.existingSecret names a Secret holding a PEM
private key. Runs older than evidence.retentionDays (400) are removed. The
volume is kept when the regime or the chart is removed.
The report’s “Fix first” section, after the summary, lists every object that
fails a check, grouped by namespace, the most severe first. For each it gives
the requirements it fails, what is wrong, how to fix it, and the kubectl
command that shows the full result in trivy-operator’s or Kyverno’s report.
Fix text for trivy-operator’s checks is trivy-operator’s own, labelled so,
except where the Convoy replaces it; the rest is the Convoy’s. The same
severity, fix text and command are in results.json and in the OSCAL
assessment results.
To start a run now:
kubectl -n <namespace> create job collect-now --from=cronjob/<release>-collect-nca-ecc
The web view
The web view serves every regime’s runs from their volumes, read-only, the newest first, with each run’s report, OSCAL assessment results, results, manifest and signature:
kubectl -n <namespace> port-forward service/<release>-web-view 8080:80
then open http://localhost:8080/. It is static files served by nginx: it
writes nothing, holds no credential and no service account token, and has no
authentication of its own.
Who can read it. By default nothing reaches it over the network, so it
is read only through kubectl port-forward, by those whose Kubernetes role
allows pods/portforward in the release’s namespace. To publish it, add an
Ingress of your own and admit your ingress controller in webView.allowFrom;
the Ingress, and the authentication in front of it, are then yours.
A run waits for the web view’s node. Each evidence volume is
ReadWriteOnce, and the web view holds them mounted, so each collector run is
scheduled to the web view’s node and waits while that node is unavailable.
webView.enabled=false removes the web view and the constraint.
With email.mode: smtp, each run is sent through your own SMTP server: the
summary and “Fix first” as the message, with the PDF report attached, and the
HTML report too if email.attach names it, up to email.maxAttachmentMB.
email:
mode: smtp
smtp:
host: smtp.example.com
port: 587
tls: starttls # or implicit
existingSecret: smtp # keys username and password
egressCIDRs: [192.0.2.25/32]
from: evidence@example.com
to: [security@example.com]
TLS is required: with starttls a server that does not offer STARTTLS is
refused, and the server’s certificate is always verified, against the
system’s CAs or the CA in email.smtp.caSecret. Nothing is sent while
email.mode is none, the default.
Object storage
With s3.enabled, each run’s files are uploaded to your bucket as
<prefix><regime>/<run>/<file>, over HTTPS with the store’s certificate
verified. s3.endpoint names an S3-compatible store, such as MinIO or Ceph;
empty, it is Amazon S3 in s3.region. s3.pathStyle puts the bucket in the
path, as many S3-compatible stores expect. The keys are in the Secret
s3.existingSecret (accessKeyId, secretAccessKey).
What delivery records, and the collector’s traffic
Each delivery is recorded in the run as delivery.json. It is written after
the run’s manifest is signed, so it is not covered by the manifest. A delivery
that fails is recorded and logged, and the run still completes: the evidence
is on the volume whether or not it was sent.
While email or object storage is on, a NetworkPolicy limits the collector’s
traffic to DNS, the Kubernetes API server and the addresses you give in
email.smtp.egressCIDRs and s3.egressCIDRs; the chart refuses to install
without them. To allow any address, write 0.0.0.0/0 and ::/0.
After adding a control-plane node, upgrade the chart. The API server’s addresses are read from the
kubernetesEndpointSlices indefaultwhen the chart is installed or upgraded. Until the next upgrade, the collector cannot reach the new node’s API server. Where the chart cannot read them, for example when a GitOps tool renders it, give them inapiServer.cidrs.
Storage that binds on first use. Where the storage class binds a volume
only when a Pod first uses it (volumeBindingMode: WaitForFirstConsumer), the
evidence volumes stay Pending until each regime’s first run, and
helm install --wait waits for them until it times out. Install without
--wait, or start a first run as above.
On a Bahriya Kubernetes Engine cluster at release 2.3 or later, the collector reads
the platform’s facts from the ConfigMap bke-platform through BKE’s
ClusterRole bke-platform-reader, bound in platform.namespace when the
chart is installed or upgraded there. Elsewhere the platform checks are
reported as not evaluated, never as failed.
Node facts
What the API cannot show, the kubelet’s settings, the modes of the files
Kubernetes keeps on each node and the node’s time service, is read on the
node itself. At each run the collector starts one short Job per node in the
namespace nodeCollector.namespace (the release’s namespace with -node
added, by default). Each Pod:
- runs as user 65534, with every capability dropped and no service account token;
- mounts only named host paths, each typed as a file or a directory and read-only, so a path that is absent is never created on the node;
- reads the kubelet’s configuration file and its flag files, and records the mode and owner of the files it cannot open;
- prints the node’s facts to its log, where the collector reads them, and sends nothing over the network.
The namespace is labelled Pod Security privileged, because every other
level refuses host paths. It holds nothing but these Jobs, the Convoy’s own
policies do not apply in it, and a NetworkPolicy allows no traffic in or out.
The collector may create, read and delete Jobs there and read their Pods and
logs; it cannot create a Pod directly. Facts read within
nodeCollector.maxAgeMinutes (60) by another regime’s run are used again.
The collector’s pull secrets must also exist in this namespace, since the node Jobs run the collector’s image.
The kubelet’s settings are those of its files: a change made after the
kubelet last started, and not yet in effect, is reported as though it were.
A node whose Job does not finish within nodeCollector.timeoutSeconds is
reported as not read, with the reason, and its checks as not evaluated.
nodeCollector.enabled=false removes the namespace and leaves every node
check not evaluated.
To pull the images from a mirror of your own, set
collector.image.repository and regimeImages.repositoryPrefix; each image
is still pulled by the digest the chart pins, so copy with
skopeo copy --all --preserve-digests.
In-cluster summary
The chart also creates trivy-operator’s ClusterComplianceReport
tersane-compliance-nist: trivy-operator’s checks grouped by the NIST SP
800-53 Rev. 5 controls the Convoy maps them to, refreshed every six hours
(summary.cron), for a quick view in the cluster:
kubectl get clustercompliancereport tersane-compliance-nist -o wide
It is indicative, not evidence. trivy-operator shows a check that has not
run as passed, and the summary covers only trivy-operator’s checks, keyed on
NIST rather than on any regime. The evidence for a regime is the report the
collector writes, which computes every result from the raw reports and
records a check with no result as not evaluated, and says what to fix. It is the same whichever
regimes the licence grants. summary.enabled=false removes it.
Checks the Convoy does not use, and why
The report names, for each of the Convoy’s checks, the trivy-operator checks it is evidenced from. Where a check of trivy-operator’s would give a wrong result for the clusters the Convoy serves, the Convoy does not use it, and the check is listed here with the reason.
AVD-KSV-0125: images from trusted registries
AVD-KSV-0125 reports an image as coming from an untrusted registry unless its
registry ends with an entry of a built-in list. That list names the registries
of three providers only: Microsoft Azure (azurecr.io), Amazon Elastic
Container Registry (the ecr.<region>.amazonaws.com domains) and Google
(gcr.io, us.gcr.io, eu.gcr.io and asia.gcr.io). All three providers are
based in the United States. Every other registry is reported as untrusted: a
registry in your own data centre, a registry in a national or in-country cloud,
and 1x.ax, from which Tersane’s own images are served.
For an organisation whose regulator or policy requires its systems and data to remain in the country, or inside its own perimeter, the result therefore reflects whether an image comes from one of those three providers, not whether it comes from a source the organisation trusts. Every registry the organisation runs itself, and every in-country registry other than those three providers’, is reported as untrusted.
The list cannot be replaced without losing the other checks. The check reads a
replacement list as rule data, which the trivy-operator version the
trivy-operator chart installs accepts only as a custom rule or library in its
policies ConfigMap; with any custom rule or
library present, it reports no built-in configuration check at all. Measured on
a test cluster, each workload’s report fell from 55 checks to none, with no
error logged.
The Convoy judges the same property against the registries you name instead.
You list your authorised registries in registries.allowed, each as the
beginning of an image reference (1x.ax/ by default).
policy.registries-allowed warns about an image from any other registry, or
refuses it once policies.actions.registries-allowed is Deny, and the report
evidences the result for every workload. trivy.image-sources continues to use AVD-KSV-0034 (images
from public registries) and AVD-KSV-0013 (images referenced by a moving tag).
Admission policies
Eleven ValidatingAdmissionPolicy objects, built into Kubernetes, so they
work whatever policy engine the cluster runs, or none. Each judges an object
as it is created or changed; Kyverno reports each one’s results for the
objects already in the cluster.
| Policy | Applies to | Requires |
|---|---|---|
registries-allowed |
Workloads | Images only from registries.allowed |
no-latest-tag |
Workloads | Every image by a fixed tag or digest |
no-privileged |
Workloads | No privileged container; no capability added but NET_BIND_SERVICE |
no-host-namespaces |
Workloads | No host network, process or IPC namespace |
no-host-ports |
Workloads | No host port |
no-host-path |
Workloads | No host path volume |
no-default-token |
Workloads | automountServiceAccountToken stated, true or false |
resource-limits |
Workloads | CPU and memory limits on every container |
namespace-environment |
Namespaces | The label tersane.bahriya.cloud/environment, one of namespaces.environments |
namespace-classification |
Namespaces | The label tersane.bahriya.cloud/data-classification, one of namespaces.classifications |
ingress-tls |
Ingresses | TLS for every host |
A workload is judged at its top-level object: a Deployment, StatefulSet, DaemonSet, Job, CronJob, or a Pod or ReplicaSet that nothing owns. The ReplicaSets and Pods a controller creates are not judged again, so a warning reaches whoever applied the workload and the report holds one result for it.
Each policy is Warn by default: the object is admitted, the person who
applied it sees a warning naming what failed, and the result is recorded. Move
a policy to Deny once nothing that should be admitted fails it, or to Off
to remove it:
policies:
defaultAction: Warn
actions:
resource-limits: Deny
no-default-token: Off
No policy applies in the namespaces of policies.exemptNamespaces, nor in the
Convoy’s own.
Kyverno policies
What the API server cannot decide itself is done with Kyverno.
| Policy | Kind | Establishes |
|---|---|---|
images-signed |
ImageValidatingPolicy |
Each container’s image is signed with cosign by the key that covers it |
namespace-network-policy |
ValidatingPolicy |
Each application namespace holds at least one NetworkPolicy |
default-deny |
GeneratingPolicy |
A default-deny NetworkPolicy in each application namespace, when turned on |
Image signatures
Mamluk’s published signing keys are held by the chart, in files/signing/,
and verify every image matching signatures.mamluk.images (1x.ax/*). Each
entry of signatures.keys is a policy of its own,
tersane-compliance-images-signed-<name>, for the images matching its
patterns:
signatures:
keys:
- name: registry
images: ["registry.example.com/*"]
publicKeys:
- |
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----
An image passes a policy when it is signed with any one of that policy’s keys,
so a key and its successor are listed together while a key is replaced. An
image matching no policy’s patterns is not checked for a signature;
registries-allowed decides where images may come from. Signatures in
cosign’s bundle format and in its earlier .sig format are both verified.
The policy changes nothing in the Pod: tags are not replaced with digests.
It follows policies.actions.images-signed like the built-in policies: Warn
admits the Pod and warns, naming each unsigned image, Deny refuses it, and
Off removes the policies. Kyverno reads signatures with its own pull
secrets; for another registry, name Secrets in Kyverno’s namespace in
signatures.credentialSecrets.
When Mamluk’s key is replaced, the new key is added to files/signing/
beside the old one, from bke/signing/, and the old one removed once every
published image carries the new signature.
NetworkPolicies
namespace-network-policy is evaluated by Kyverno’s background scan only,
because a namespace holds no NetworkPolicy at the moment it is created. Its
results are in Kyverno’s cluster policy reports, renewed when a namespace
changes and at each background scan. It is never refused at admission:
policies.defaultAction does not apply to it, and Deny named for it is
refused.
generate.defaultDenyNetworkPolicy.enabled has Kyverno write the
NetworkPolicy tersane-default-deny into each application namespace as it is
created. It selects every Pod and allows no traffic, in or out, except DNS
lookups to the cluster’s DNS service while allowDNS is true, so each
workload then needs NetworkPolicies of its own allowing what it uses. Kyverno
restores it when it is deleted or changed.
What a workload loses when it is written. On a cluster whose network enforces egress, as every Bahriya Kubernetes Engine cluster’s does, every Pod in the namespace at once stops reaching:
- other Pods and Services, in its namespace and elsewhere;
- the Kubernetes API server, so an operator, a controller or any workload that calls the API stops working;
- anything outside the cluster, including registries, databases and APIs it calls by address;
and nothing reaches it, ingress controllers included. Only DNS lookups remain,
while allowDNS is true. Before turning it on, give each application
namespace NetworkPolicies that allow its workloads’ traffic: for the API
server, an egress rule to the control-plane nodes’ addresses on port 6443.
existingNamespaces also writes it into the namespaces that exist when it is
turned on, which stops their traffic at once. It is off by default. Turning it
off again removes the policy and every NetworkPolicy it wrote.
The chart adds the permissions Kyverno’s controllers need for these, through ClusterRoles that Kyverno’s own roles aggregate.
Parameters
| Parameter | Description | Default |
|---|---|---|
regimes |
Regimes evidenced | [nca-ecc] |
policies.defaultAction |
Warn or Deny for every policy |
Warn |
policies.actions |
Warn, Deny or Off for a policy, by name, including images-signed and namespace-network-policy |
{} |
policies.exemptNamespaces |
Namespaces no policy applies to | the platform’s, Kyverno’s and trivy-operator’s |
registries.allowed |
Registries images may come from, as reference prefixes | [1x.ax/] |
signatures.mamluk.images |
Images Mamluk’s keys verify, as patterns | [1x.ax/*] |
signatures.keys |
Keys of your own: name, images, publicKeys |
[] |
signatures.credentialSecrets |
Secrets in Kyverno’s namespace for reading signatures | [] |
generate.defaultDenyNetworkPolicy.enabled |
Write a default-deny NetworkPolicy into each application namespace | false |
generate.defaultDenyNetworkPolicy.allowDNS |
Allow DNS lookups in it | true |
generate.defaultDenyNetworkPolicy.existingNamespaces |
Also write it into existing namespaces | false |
namespaces.environments |
Values of the environment label | [production, test, development] |
namespaces.classifications |
Values of the data classification label | [top-secret, secret, restricted, public] |
summary.enabled |
Create the NIST-keyed in-cluster summary (indicative) | true |
summary.cron |
When trivy-operator refreshes it | 0 */6 * * * |
collector.image.repository, .tag, .digest |
The collector image, pinned by digest | 1x.ax/tersane/convoy/compliance/core/collector |
collector.imagePullSecrets |
Pull secrets for the collector and regime images | [{name: 1x-ax}] |
collector.schedule |
When each regime’s collector runs | 17 2 * * * |
collector.activeDeadlineSeconds |
A run taking longer is stopped | 1800 |
collector.scanMaxAgeHours |
Scans older than this fail scan freshness | 48 |
collector.organisation, collector.cluster |
Shown on the report’s first page | "" |
collector.resources |
The collector’s requests and limits | 100m/256Mi, 1/1Gi |
webView.enabled |
Serve the runs, read-only, in the cluster | true |
webView.allowFrom |
NetworkPolicy peers that may reach the web view | [] (port-forward only) |
webView.url |
The web view’s address, shown in each email | "" |
email.mode |
none or smtp |
none |
email.smtp.host, .port, .tls |
Your SMTP server; starttls or implicit |
"", 587, starttls |
email.smtp.existingSecret, .usernameKey, .passwordKey |
The SMTP credential | "", username, password |
email.smtp.caSecret, .caKey |
A CA for the server’s certificate | "", ca.crt |
email.smtp.egressCIDRs |
The server’s addresses; required | [] |
email.from, email.to |
Sender and recipients | "", [] |
email.attach, email.maxAttachmentMB |
report.pdf, report.html; the size limit |
[report.pdf], 10 |
s3.enabled |
Upload each run | false |
s3.endpoint, .region, .bucket, .prefix, .pathStyle |
The store and where the objects go | "" (Amazon S3), "", "", "", false |
s3.existingSecret, .accessKeyIdKey, .secretAccessKeyKey |
The access key | "", accessKeyId, secretAccessKey |
s3.caSecret, .caKey |
A CA for the store’s certificate | "", ca.crt |
s3.egressCIDRs |
The store’s addresses; required | [] |
apiServer.cidrs, .port |
The API server’s addresses, where the chart cannot read them | [], 6443 |
nodeCollector.enabled |
Read each node’s facts | true |
nodeCollector.namespace |
The node Jobs’ namespace, labelled privileged | "" (<namespace>-node) |
nodeCollector.maxAgeMinutes |
Node facts this recent are used again | 60 |
nodeCollector.timeoutSeconds |
How long a run waits for the nodes | 300 |
evidence.retentionDays |
Days of runs kept | 400 |
evidence.size, evidence.storageClass |
Each regime’s volume | 5Gi, the default class |
evidence.signingKey.existingSecret, .key |
A Secret with the PEM key that signs each manifest | "", key.pem |
regimeImages.repositoryPrefix |
A mirror holding the regime images | "" (1x.ax) |
platform.namespace |
Where BKE records its platform facts | bahriya-system |
prerequisites.check |
Refuse to install without Kyverno and trivy-operator | true |