Documentation / kyverno

Kyverno

Installs Kyverno, the Kubernetes policy engine: it validates and mutates resources as the API server admits them, verifies the signatures of the images a Pod runs, and records the result of every policy for the resources already in the cluster in policy reports.

The chart installs Kyverno’s own Helm chart, unchanged, under the name engine, with settings for a production cluster on your own premises:

every image from the Mamluk registry, pinned by digest;
the admission controller with three replicas and the other controllers with two, each with a PodDisruptionBudget;
the namespaces of the cluster’s own components left outside Kyverno’s webhooks, so that a Kyverno outage cannot stop the control plane, the network or storage from recovering;
no call to any service outside the cluster;
every Pod meeting the restricted Pod Security Standard;
an optional policy that verifies image signatures against a public key;
removal of every webhook Kyverno registered when the chart is uninstalled.

Prerequisites

Kubernetes 1.33 or later.
One Kyverno per cluster. Its webhooks and custom resources are cluster-wide; do not install a second release, in any namespace.
A namespace of its own, conventionally kyverno.
A pull secret for 1x.ax named 1x-ax in that namespace, created from an image token as described in the access documentation. Kyverno’s Pods pull their images with it, and Kyverno reads image signatures with it. To use a secret of another name, set both engine.global.imagePullSecrets and engine.existingImagePullSecrets.

Installation

helm install kyverno tersane/kyverno \
  --namespace kyverno --create-namespace \
  --version <chart version>

Check that the four controllers are ready:

kubectl --namespace kyverno get deployments

The chart installs no policy of its own unless image signature verification is configured (below). Policies are then written by you, or installed by a Convoy.

Writing policies

Write policies with Kyverno’s CEL-based kinds in the policies.kyverno.io/v1 API: ValidatingPolicy, MutatingPolicy, GeneratingPolicy, DeletingPolicy and ImageValidatingPolicy, and their namespaced equivalents. Kyverno’s earlier kinds, ClusterPolicy, Policy, CleanupPolicy and ClusterCleanupPolicy, are deprecated and are removed in Kyverno’s next minor release; a policy written in them stops working when this chart moves to that release.

Start a new policy with validationActions: [Audit], read its results in the policy reports, and change it to Deny once nothing that should be admitted fails it:

kubectl get policyreports --all-namespaces
kubectl get clusterpolicyreports

Namespaces Kyverno does not see

Kyverno’s webhooks are not called for resources in the namespaces listed in engine.config.webhooks, nor in Kyverno’s own namespace. No policy can act on them, and they keep working when Kyverno is unavailable. The default list holds the namespaces a Bahriya Kubernetes Engine cluster uses by default:

kube-system, kube-node-lease, kube-public, tigera-operator, calico-system, calico-apiserver, longhorn-system, cert-manager, kong, kuma-system, logging and bahriya-system.

If a component of your cluster is installed in another namespace, add it. The list replaces the default as a whole, so repeat the namespaces you keep:

engine:
  config:
    webhooks:
      namespaceSelector:
        matchExpressions:
          - key: kubernetes.io/metadata.name
            operator: NotIn
            values:
              - kube-system
              - kube-node-lease
              - kube-public
              - tigera-operator
              - calico-system
              - calico-apiserver
              - longhorn-system
              - cert-manager
              - kong
              - kuma-system
              - logging
              - bahriya-system
              - my-ingress

A namespace that does not exist is ignored. Workloads of your own do not belong in this list: a policy is how you choose what Kyverno does to them.

Kyverno’s webhooks fail closed: while no admission controller is answering, the API server refuses a request that a Deny policy with failurePolicy: Fail matches. The three replicas and their PodDisruptionBudget are there so that draining a node never leaves none.

Image signature verification

Set the public key with which your images are signed, and the chart creates an ImageValidatingPolicy named kyverno-verify-image-signatures:

helm upgrade kyverno tersane/kyverno --namespace kyverno --reuse-values \
  --set-file imageVerification.publicKey=cosign.pub

It checks every container image of a new or updated Pod whose reference matches one of imageVerification.images (1x.ax/* by default). An image matching none is not checked.
In Audit mode, the default, every Pod is admitted, and an image without a valid signature is recorded as a failure in the namespace’s policy report. With background: true the Pods already running are checked too.
In Deny mode, a Pod with an image without a valid signature is refused. Change the mode once the policy report shows no failure you did not expect: --set imageVerification.action=Deny.
Signatures are verified against the key alone, with no transparency log and no certificate authority, so verification works without internet access. The signature must be stored in the same registry as the image, in either of cosign’s formats: its current bundle format or its earlier .sig tag.
The Pod keeps the image references it was given. An image referenced by tag is verified by the digest the tag resolves to when the Pod is admitted, and the tag is not rewritten to that digest. Mamluk’s images in 1x.ax are safe to use by tag: a published tag is never pushed again or overwritten, so it always names the image that was verified. For images in a registry of your own, where a tag can be moved, two switches pin by digest: imageVerification.mutateDigest=true rewrites each verified tag in the Pod to its digest, so a later restart runs the image that was verified, and imageVerification.verifyDigest=true refuses any image referenced by tag alone. Both are off by default, and both apply to every image the policy checks.
Kyverno reads the signatures from the registry itself, with the pull secrets named in engine.existingImagePullSecrets, which must exist in Kyverno’s namespace. To verify images in a private registry of your own, add its pull secret to that list. If your registry’s TLS certificate is issued by a certificate authority of your own, give Kyverno that authority’s certificate: --set-file engine.global.caCertificates.data=ca.crt.

To remove the policy, set the key to an empty string; the next upgrade deletes it.

Uninstalling

helm uninstall kyverno --namespace kyverno

Kyverno registers its webhooks itself, so Helm does not remove them. On uninstall, Kyverno’s own steps scale its controllers to zero, and then this chart waits until they have stopped and removes every webhook configuration Kyverno registered. If the controllers do not stop within webhookRemoval.timeoutSeconds, the uninstall stops with Kyverno still running, which is safe; run it again. A webhook left registered with nothing to answer it would block every request it matches.

Uninstalling deletes Kyverno’s custom resource definitions, and with them every policy and policy report in the cluster. Kyverno’s kyverno ConfigMap is kept; delete it before installing Kyverno again under another release name.

Pod Security

Every Pod the chart creates meets the Kubernetes restricted Pod Security Standard: the four controllers, the jobs that run on upgrade and uninstall, and the helm test Pods. Each runs as a user other than root, cannot escalate its privileges, drops all Linux capabilities and runs under the container runtime’s default seccomp profile. Install it in a namespace that enforces restricted.

Parameters

Values under engine are passed to Kyverno’s chart. Those below are set by this chart; any other value of Kyverno’s chart may be set as well, under engine, at your own discretion.

Kyverno

Parameter	Description	Default
`engine.fullnameOverride`	The name Kyverno’s uninstall steps find its controllers by. Do not change it.	`kyverno`
`engine.admissionController.replicas`	Admission controller replicas	`3`
`engine.backgroundController.replicas`	Background controller replicas	`2`
`engine.cleanupController.replicas`	Cleanup controller replicas	`2`
`engine.reportsController.replicas`	Reports controller replicas	`2`
`engine.admissionController.container.resources.limits.cpu`	Admission controller CPU limit; the other resources are Kyverno’s defaults	`1000m`
`engine.backgroundController.resources.limits.cpu`	Background controller CPU limit	`500m`
`engine.cleanupController.resources.limits.cpu`	Cleanup controller CPU limit	`500m`
`engine.reportsController.resources.limits.cpu`	Reports controller CPU limit	`500m`
`engine.config.webhooks`	Namespaces left outside Kyverno’s webhooks	see above
`engine.global.imagePullSecrets`	Pull secrets of every Pod the chart creates	`[{name: 1x-ax}]`
`engine.existingImagePullSecrets`	Pull secrets Kyverno reads image signatures with, by name	`["1x-ax"]`
`engine.features.registryClient.credentialHelpers`	Registry credentials Kyverno uses: no cloud provider’s	`["default"]`
`engine.global.caCertificates.data`	Certificate authorities Kyverno trusts when it reads a registry, in PEM	not set

Images

Each image has a registry and a tag; the tag carries the image’s digest, which decides what runs. To pull from a mirror of your own, change each registry and keep the tags.

Parameter	Description	Default
`engine.admissionController.container.image`	Admission controller	Set in each release
`engine.admissionController.initContainer.image`	Admission controller start-up	Set in each release
`engine.backgroundController.image`	Background controller	Set in each release
`engine.cleanupController.image`	Cleanup controller	Set in each release
`engine.reportsController.image`	Reports controller	Set in each release
`engine.crds.migration.image`	Resource migration on upgrade	Set in each release
`engine.webhooksCleanup.image`	Kyverno’s uninstall steps	Set in each release
`engine.test.image`	`helm test`	Set in each release
`webhookRemoval.image.repository`	Webhook removal on uninstall, and policy removal on upgrade	`1x.ax/tersane/core/kubernetes/kubectl`

Webhook removal

Parameter	Description	Default
`webhookRemoval.enabled`	Remove Kyverno’s webhooks on uninstall	`true`
`webhookRemoval.timeoutSeconds`	Seconds to wait for the controllers to stop	`120`
`webhookRemoval.resources`	Resources of the removal Job	requests `10m`/`32Mi`, limits `100m`/`128Mi`

Image signature verification

Parameter	Description	Default
`imageVerification.publicKey`	cosign public key, PEM; the policy is created only when set	`""`
`imageVerification.action`	`Audit` or `Deny`	`Audit`
`imageVerification.images`	Image references checked, as glob patterns	`["1x.ax/*"]`
`imageVerification.background`	Also check running Pods	`true`
`imageVerification.mutateDigest`	Rewrite each verified tag in the Pod to its digest	`false`
`imageVerification.verifyDigest`	Refuse images referenced by tag alone	`false`