Documentation / concourse

Concourse

Installs Concourse, for continuous integration and delivery: pipelines of resources, jobs and tasks, run in containers on workers.

The chart provides:

Concourse 8.3, for amd64 and arm64, with its database in a PostgreSQL of your own, such as a release of the postgresql chart, reached over TLS with the server’s certificate checked;
web nodes that meet the restricted Pod Security Standard, with a read-only root filesystem;
workers in a release and a namespace of their own, privileged, as Concourse requires;
keys, local users and the database credentials taken from Secrets you create, which the chart never creates, changes or prints;
NetworkPolicies that admit workers to the TSA only from their own namespace, and admit nothing to the workers;
Concourse’s image from the Mamluk registry, pinned by digest.

Workers and Pod Security

A worker must be privileged. It runs each build step in a container it creates itself, with a container runtime inside its Pod, and that needs privilege. Upstream’s own chart sets it on every worker, with no setting to turn it off. The worker container is the only privileged container in this chart: it uses no host path, no host network and no service account token, and a clean-up container before it runs as root without privilege.

A privileged worker can do anything its node can. A pipeline step marked privileged, and any step on a compromised worker, has the access of root on the node. Install workers in a namespace that holds nothing else, on nodes you are content to give to builds, and let only those you trust change pipelines.

For that reason the chart installs in two releases: the web nodes, in the namespace of your choice under restricted, and the workers, in a namespace of their own, which must allow the privileged Pod Security Standard. The chart refuses to install both in one release, and refuses workers in default or any kube- namespace.

Why the workers have a namespace of their own

Kubernetes applies these controls to a namespace as a whole, so keeping the workers apart confines the privilege to them:

Pod Security. A namespace’s level applies to every Pod in it. In a shared namespace the web nodes would lose restricted; apart, only the workers’ namespace allows privileged Pods.
Admission policies. The Compliance and Assurance Convoy exempts whole namespaces, from every policy at once. With the workers alone in theirs, the exemption covers nothing else, and the web nodes stay under every policy.
Who can run privileged Pods. Anyone able to create Pods in the workers’ namespace can run privileged containers. A namespace holding only the workers lets you grant that to as few people as possible, apart from those who operate the web nodes.
Network. The web nodes admit connections to the TSA only from the workers’ namespace.

A namespace does not contain a privileged container: a worker can reach the node it runs on. To keep builds away from other workloads, run the workers on nodes of their own, with nodeSelector and tolerations in the workers’ release.

The cost is two releases instead of one, the keys in two Secrets, and, if you run the Convoy, the exemption described below.

Admission policies

Any admission policy forbidding privileged containers will flag the workers, including the Compliance and Assurance Convoy’s no-privileged. That is by design: Concourse cannot run builds otherwise.

The Convoy does not exempt the workers’ namespace by default, since Concourse is optional, and on a cluster without it a namespace of that name would be open to anyone able to create it. If you run the Convoy and install workers, exempt their namespace yourself. The Convoy then applies no policy at all in that namespace, not only no-privileged, so keep it to the workers and limit who can create Pods in it.

The Convoy exempts a namespace by its name, which it matches on the kubernetes.io/metadata.name label. Kubernetes sets that label to the namespace’s name and lets nobody change it, so the exemption cannot be moved to another namespace by relabelling one. Do not exempt namespaces by a label of your own: anyone able to edit a namespace could set it.

Do this before installing the workers: a policy set to Deny would otherwise refuse the install. Set this in the values of the compliance chart. The list replaces the default as a whole, so it repeats the default, which may be longer in a later version of that chart: check it against helm show values, and add any namespaces of your own.

policies:
  exemptNamespaces:
    - kube-system
    - kube-node-lease
    - kube-public
    - tigera-operator
    - calico-system
    - calico-apiserver
    - longhorn-system
    - cert-manager
    - kong
    - kuma-system
    - logging
    - bahriya-system
    - kyverno
    - trivy-system
    - ci-workers

Record the exemption, and the reason above, wherever your regime asks you to account for exceptions. An exempted namespace is not a passed control: the workers remain privileged.

Prerequisites

Kubernetes 1.33 or later.
A namespace for the web nodes, and one for the workers alone, allowed the privileged Pod Security Standard.
A pull secret for 1x.ax named 1x-ax in both, created from an image token as described in the access documentation.

A PostgreSQL database owned by the user Concourse connects as. A release of the postgresql chart in the web nodes’ namespace:

helm install pg tersane/postgresql --namespace ci --create-namespace \
  --set bootstrap.database=concourse --set bootstrap.owner=concourse

Keys, generated with Concourse’s own image, which needs nothing but a scratch directory:

mkdir keys
docker run --rm -v "$PWD/keys:/keys" --entrypoint /usr/local/concourse/bin/concourse \
  1x.ax/tersane/core/concourse/concourse:<tag> generate-key -t rsa -f /keys/session_signing_key
# the same with: -t ssh -f /keys/tsa_host_key, and -t ssh -f /keys/worker_key

Then the web nodes’ Secret, and the workers’:

kubectl --namespace ci create secret generic concourse-web-keys \
  --from-file=session_signing_key=keys/session_signing_key \
  --from-file=tsa_host_key=keys/tsa_host_key \
  --from-file=authorized_worker_keys=keys/worker_key.pub
kubectl --namespace ci-workers create secret generic concourse-worker-keys \
  --from-file=worker_key=keys/worker_key \
  --from-file=tsa_host_key.pub=keys/tsa_host_key.pub

Local users, in a Secret: a comma-separated list of name:password, where the password may be a bcrypt hash.

kubectl --namespace ci create secret generic concourse-users \
  --from-file=local_users=./local-users

Installation

The web nodes:

helm install concourse tersane/concourse --namespace ci \
  --set web.externalURL=https://ci.example.com \
  --set webKeys.existingSecret=concourse-web-keys \
  --set auth.existingSecret=concourse-users --set 'auth.mainTeam={admin}' \
  --set database.host=pg-postgresql-rw \
  --set database.existingSecret=pg-postgresql-app \
  --set database.caSecret=pg-postgresql-ca \
  --set 'networkPolicy.workerNamespaces={ci-workers}' \
  --set ingress.className=<class> --set ingress.tlsSecret=<certificate Secret> \
  --version <chart version>

The workers, after exempting their namespace if you run the Convoy:

kubectl create namespace ci-workers
kubectl label namespace ci-workers pod-security.kubernetes.io/enforce=privileged

helm install concourse-workers tersane/concourse --namespace ci-workers \
  --set web.enabled=false --set worker.enabled=true \
  --set worker.tsaHost=concourse-tsa.ci.svc.cluster.local:2222 \
  --set worker.keys.existingSecret=concourse-worker-keys \
  --set networkPolicy.webNamespace=ci \
  --version <chart version>

The web nodes migrate the database before they serve, in an init container. Sign in at the external URL, or with fly:

fly --target ci login --concourse-url https://ci.example.com --username admin

Database

database.sslmode is verify-full by default: the server’s certificate must be signed by the authority in database.caSecret and name database.host. require encrypts without checking the certificate.

Workers

Each worker keeps its containers and caches on a volume of worker.workDir.size. A worker that stops is retired: it finishes the builds it is running, within worker.terminationGracePeriodSeconds, and leaves; its volume is emptied before it registers again.

Network policy

The web nodes admit the user interface and API from networkPolicy.ingressFrom, any source when empty, and the TSA only from the namespaces in networkPolicy.workerNamespaces, matched by name.

The workers admit nothing: the web nodes reach them back through the SSH connection each worker opens to the TSA. Their egress is DNS, the TSA, and networkPolicy.workerExternalEgress, every address that is not private by default, which is what builds fetch from: registries, Git hosts and package mirrors. An air-gapped installation narrows it to its own mirrors, and adds any address inside the cluster that builds use with networkPolicy.extraEgress.

DNS is reached at the Pods labelled k8s-app: kube-dns in kube-system, on port 53, which is where most distributions run CoreDNS. Where the cluster’s DNS servers run in another namespace, carry other labels or listen on another port, set networkPolicy.dns to match them, or the workers cannot resolve names.

Uninstalling

helm uninstall concourse-workers --namespace ci-workers
helm uninstall concourse --namespace ci

The workers’ volumes are kept with the namespace; delete their claims to free them. The database is yours, and is untouched.

Parameters

Web

Parameter	Description	Default
`web.enabled`	Install the web nodes	`true`
`web.replicas`	Web nodes	`1`
`web.externalURL`	The address users and `fly` reach Concourse at; required	`""`
`web.clusterName`	Shown on the dashboard	`""`
`web.resources`	Resources of each web node	200m and 512Mi requested, 2 CPUs and 2Gi limits
`web.service.type` / `web.service.port`	The web Service	`ClusterIP` / `8080`
`web.tsa.port` / `web.tsa.service.type` / `.nodePort`	The TSA and its Service	`2222` / `ClusterIP` / `""`
`webKeys.existingSecret`	Secret with `session_signing_key`, `tsa_host_key`, `authorized_worker_keys`; required	`""`
`auth.existingSecret` / `auth.key`	Secret with the local users; required	`""` / `local_users`
`auth.mainTeam`	Local users who own the main team; required	`[]`
`database.host` / `.port` / `.name`	The PostgreSQL server and database; host required	`""` / `5432` / `concourse`
`database.existingSecret` / `.usernameKey` / `.passwordKey`	The Secret with the user and password; required	`""` / `username` / `password`
`database.sslmode`	`disable`, `require`, `verify-ca` or `verify-full`	`verify-full`
`database.caSecret` / `database.caKey`	The server’s certificate authority	`""` / `ca.crt`
`ingress.enabled` / `.className` / `.annotations` / `.tlsSecret`	The Ingress	`true` / `""` / `{}` / `""`

Workers

Parameter	Description	Default
`worker.enabled`	Install workers	`false`
`worker.replicas`	Workers	`2`
`worker.tsaHost`	host:port of the web release’s TSA; required	`""`
`worker.keys.existingSecret`	Secret with `worker_key` and `tsa_host_key.pub`; required	`""`
`worker.workDir.size` / `.storageClass`	Each worker’s volume	`50Gi` / the default class
`worker.baggageclaimDriver`	`overlay` or `naive`	`overlay`
`worker.resources`	Resources of each worker	500m and 1Gi requested, 4 CPUs and 8Gi limits
`worker.terminationGracePeriodSeconds`	Time to finish builds when stopped	`3600`

Common

Parameter	Description	Default
`image.repository` / `image.tag`	The image	`1x.ax/tersane/core/concourse/concourse` / the release’s
`imagePullPolicy` / `imagePullSecrets`	Pulling	`IfNotPresent` / `[{name: 1x-ax}]`
`logLevel`	`debug`, `info`, `error` or `fatal`	`info`
`networkPolicy.enabled`	The NetworkPolicies	`true`
`networkPolicy.ingressFrom`	Sources of the user interface and API; empty admits any	`[]`
`networkPolicy.workerNamespaces`	Namespaces whose workers may reach the TSA; required on the web release	`[]`
`networkPolicy.webNamespace`	The web release’s namespace; required on the worker release	`""`
`networkPolicy.workerExternalEgress`	Where builds may connect outside the cluster	every address that is not private
`networkPolicy.extraEgress`	Further egress rules for the workers	`[]`
`networkPolicy.dns.namespace` / `.podLabels` / `.ports`	The cluster’s DNS servers, which the workers reach	`kube-system` / `{k8s-app: kube-dns}` / `[53]`
`podSecurityContext` / `securityContext`	The web nodes’ security contexts	user 10000, `restricted`
`nodeSelector` / `tolerations`	Scheduling	`{}` / `[]`