Upgrading the bundled components

The gateway ships with third-party components around it: a directory, an identity provider, a secret store, a database, an object store, a mail trap and the monitoring pair. Every one of them runs from a container image that is pinned to a digest, and upgrading one is always the same two steps — rewrite the pin, re-run the role. What differs, component by component, is what the restart costs. This page is that list.

How a component is pinned

Each image reference in the shipped automation has the form tag@sha256:…: the tag is for the reader, the digest is the binding. A version tag alone does not bind — an upstream release can move a tag to new bytes without anything in this repository changing — so the digest is what the container engine pulls. The agreement between the places that name the same image is itself measured, by make ansible-image-pin-check.

The pin lives in the role's own defaults, one variable per component:

An upgrade is therefore: change the variable, run site.yml with the --limit that selects the hosts the component runs on, and the role's handler restarts the unit. There is no upgrade playbook for these components — the one that exists, upgrade-rolling.yml, upgrades the ISG brokers themselves (drain, replace, verify, one broker at a time), not the components around them.

Before and after: the version report

Run the report before you change a pin and again after the role has run:

ansible-playbook -i inventory/<environment>.ini -e isg_env=<environment> \
  playbooks/component-versions.yml

It only reads. Per host it prints one line per container with three columns — the pin, the image actually running (name and digest), and the version the software reports about itself — and then one summary line per component for the whole fleet. Each line carries a verdict:

A mismatch makes the run fail — after the table and the summary are printed, so one run gives you the whole picture. A not_measured does not: a host that is down is not a version discrepancy.

For a before-snapshot of an upgrade — the pin already names the new version, the host still runs the old one — mismatch is the expected state. Add -e isg_cv_fail_on_mismatch=false: the report is printed (and written) the same way, and the run does not fail.

What an upgrade costs, component by component

OpenLDAP — the directory is rebuilt from the seed

Replacing the image recreates the container, and the directory is built again from the seed. The read-only accounts the gateway binds with are put back by the role; entries added while the directory was running are gone, and the brokers' bind drops for a short while during the restart. Measured twice, on 2026-10-08.

Treat it accordingly: if anything other than the seeded content lives in this directory, export it first.

OpenBao — sealed after every restart

A restart leaves the secret store sealed. This is not a failure mode to be worked around: the role says it out loud, in red, and stops rather than pretending the store is usable. Unsealing is a separate, deliberate step, and it has its own playbook — bao-init.yml. Plan the unseal into the same maintenance window as the upgrade; until it runs, anything that needs a secret from the store will fail.

PostgreSQL — a major version is a dump and a restore

The major version is part of the pin. The automation does not move a deployment across a major version: a new data directory is needed, and the data reaches it through a dump and a restore. From 18 on, the image also keeps its data in a different layout — the store's data directory is mounted as /var/lib/postgresql, and the cluster lives in its 18/docker subdirectory.

The role refuses to start an 18 image on a data directory in the old layout (PG_VERSION at its root): started there, Postgres would initialise a new, empty database next to the old files. Measured once (2026-10-11, a lab deployment, 16.15 → 18.6): a fresh backup, the unit stopped, the old directory renamed aside, an empty one restored from the dump with the new image, then the role — the row counts matched the backup's manifest. A new deployment starts on the pinned major directly and needs none of this.

A minor-version image swap on the existing data volume is expected to be uneventful — but that is a hypothesis, not a measurement: it has not been measured in this deployment. Back up the volume before you try it, and read the upstream release notes for the minor you are moving to.

Keycloak — the built-in database is on a volume

The identity provider runs in start-dev mode with its built-in database on the keycloak/data volume. Swapping the image therefore implies a database migration of that volume, performed by the new version on first start. Measured once (2026-10-11, a lab deployment): 26.3.5 → 26.8.0, five minor versions in one jump — the migration ran on the first start, the identity provider answered after about half a minute, and the realm, the client and the LDAP federation came through unchanged. One measurement, not a guarantee for every jump.

So: back up the volume first, then change the pin. If the migration goes wrong, the backup is the way back to the state before it. The files on the volume belong to the container's user, which on the host is a subordinate uid — read and restore them inside the user namespace, for example:

podman unshare tar czf <archive> -C <keycloak-home> data

From 26.8 on, Keycloak asks for the bind password again whenever the LDAP federation's connection URL or bind DN changes; a masked password is refused. The role's own path for this (the retransport switch) already sends the real password. A manual change in the admin console needs the password typed in again.

RustFS, Prometheus, Grafana, node_exporter and Mailpit

For these five the automation has no extra step: change the pin, run the role, and the unit restarts with the new image. Whether a given version jump changes the on-disk format of a component's data is not measured here — read the upstream release notes before a jump.

One of them deserves an explicit note: Mailpit is a test SMTP trap. It is there so that a deployment can see the mail the gateway sends without sending it; it is not a mail server for anything real, and nothing is lost when its container is replaced.

What the report does not tell you

The report does not ask upstream what the newest version is, and it makes no outbound call at all: the one version command that would check online by default, Mailpit's, runs with --no-release-check. Comparing a measured version against what upstream publishes is a separate step, on a machine you choose for it.

Monthly upstream check

That separate step is playbooks/component-upstream.yml. It runs on the controller only, asks each registry for the highest stable tag of every pinned component, and gives each one a verdict: current, behind or not_measured. A behind is not a failure — it is an input to a decision.

What runs, when, and where it writes. playbooks/upstream-schedule.yml installs a systemd user timer on the controller (the isg_upstream_schedule role). It fires on the first day of every month at 07:20, with up to 30 minutes of random delay, and Persistent=true brings a missed run back after the controller comes up. Each run writes the report as a JSON and a Markdown file into the isg-component-reports directory in the controller user's home (isg_upstream_schedule_report_dir). The timer needs no root, no token and no sudo password: every play in the check runs on localhost without become, and the role measures this on the tree the timer runs from before it installs anything.

cd ansible
ansible-playbook -i inventory/<environment>.ini -e isg_env=<environment> \
  playbooks/upstream-schedule.yml
journalctl --user -u isg-upstream-<environment>.service    # the runs

When a mail goes out, and to whom. When at least one component is behind, the run mails the Markdown table to the addresses in isg_cfg_upstream_notify_to, over the deployment's existing SMTP path (isg_cfg_smtp_host, isg_cfg_smtp_port, isg_cfg_smtp_from). Nothing behind means no mail, and the log says so in one line; so does an empty recipient list. An SMTP error turns the run red, naming the host and port — a mail that silently did not go out is exactly what the monthly check exists to prevent. The sender speaks plain SMTP without TLS or authentication; a real relay is the operator's to provide.

What to do with a behind. Do not upgrade one component the day its mail arrives. Collect the behind rows into one batch, then upgrade the batch together: change each pin to the new tag's index digest, take a version report before and after (above), and read the per-component costs on this page for anything that needs more than a restart.

The firewall on the SMTP path. On the directory host the SMTP port opens only to the brokers by default (isg_directory_firewall_client_sources); the mail now also comes from the controller, so its address goes into isg_directory_firewall_smtp_extra_sources. Without that entry the firewall rejects the controller (measured: "No route to host"), and the run fails red, naming the host and port.