Useful commands

The checks an operator runs most often, as ready-to-paste one-liners: the version and the self-diagnostics on a broker, the same across the whole fleet, the resolved lockdown level of a target, which broker currently holds the shared address, the collected break-glass log and its rotation, the Ansible audit run, and the break-glass drill. After those, the operations the controller drives: the decision on an approval request, the services on the infrastructure hosts, the scheduled jobs, the rolling upgrade, a configuration rollout, the store backup with its restore drill, and the bundled component versions. Web Access has its own block: the system-check rows, the application sessions and the locks. Where it helps to know what to look for, a section shows an output sample with the same placeholders.

Every block on this page is environment-independent. <broker>, <target>, <vip name>, <admin user>, <admin key> and <break-glass key> are placeholders for this installation's own names, and <env> and <config source> name the Ansible environment and the configuration source directory on the controller. Because the one-liners carry those placeholders, they are not measured against the source tree the way the code spans in this portal's prose are: they are runs measured in a reference environment with the names taken out. What each section says a command shows comes from one of those runs.

The commands that read the collected log, the rotation configuration or an approval record need sudo on the broker. Up to the break-glass drill, nothing on this page changes state, and the drill itself is only a real login. The sections after it do change state, on purpose: an upgrade, a rollout, a backup — each says what it touches.

The version and the self-diagnostics on a broker

Run on the broker itself. isg system check prints one row per check with its status marker first, so the second line keeps only the rows that need attention, and the third only the rows about identity, configuration, the CMDB and break-glass.

isg --version | head -1
isg system check 2>&1 | grep -E '^\[(WARN|FAIL)\]'
isg system check 2>&1 | grep -E '^\[[A-Z ]+\] (ldap\.yaml|LDAP bind|Config key|cmdb\.yaml|Broker version skew|Break-glass)'

The same across every broker

One ssh per broker with the admin key, so a whole fleet answers on one screen. This is the form that showed a version skew between two brokers in the reference environment: the rows stand next to each other, and the one that differs is the one to read.

for h in <broker> <broker>; do ssh -i <admin key> <admin user>@$h 'isg --version | head -1; isg system check 2>&1 | grep -E "^\[[A-Z ]+\] (cmdb\.yaml|Broker version skew)"'; done

The lockdown level of the targets

isg servers level answers for one host at a time by design — a real CMDB connector has no host list to offer — so a loop is how a whole inventory gets read. The row names the level and where it came from, which is the part worth reading: a level served from the cache and a level served from the CMDB look the same otherwise.

for t in <target> <target>; do isg servers level $t 2>/dev/null; done

Which broker holds the shared address

On a keepalived pair exactly one broker carries the shared address at a time. It matters for reading anything else: that broker is the one operators are actually connected through, and the one a rolling upgrade takes last. The address is resolved from its name on the controller, and every broker answers with its own hostname and a plain yes or no.

vip=$(getent hosts <vip name> | awk '{print $1}')
for h in <broker> <broker>; do ssh -i <admin key> <admin user>@$h "hostname; ip -br a | grep -qw $vip && echo 'VIP (<vip name>): yes' || echo 'VIP (<vip name>): no'"; done

<broker-1>
VIP (<vip name>): yes
<broker-2>
VIP (<vip name>): no

The collected break-glass log

Every break-glass login and every forwarded command record lands in /var/log/isg-breakglass-remote.log on each broker. In order: the line count; the per-day, per-sending-host breakdown, which reads the month, the day and the host out of the syslog line; the break-glass lines themselves, counted with the same pattern the forward filter passes; the last entries; the last successful break-glass logins; and the forwarded command audit (type=EXECVE).

sudo wc -l /var/log/isg-breakglass-remote.log
sudo awk '{print $1, $2, $4}' /var/log/isg-breakglass-remote.log | sort | uniq -c
sudo grep -c -E 'isg-breakglass|isg-bg:|type=EXECVE' /var/log/isg-breakglass-remote.log
sudo tail -5 /var/log/isg-breakglass-remote.log
sudo grep -E '(Accepted|Failed) [a-z]+ for isg-breakglass' /var/log/isg-breakglass-remote.log | tail -3
sudo grep 'type=EXECVE' /var/log/isg-breakglass-remote.log | tail -5

A login line distinguishes the two break-glass layers itself: the certificate path shows up as ED25519-CERT with the certificate's own ID field, the last-resort static key without either.

The dry run of the log rotation

The rotation of the collected log is size-based, not daily, so the answer to "will it rotate tonight" is a dry run rather than a date. It parses the configuration on the way, which is the other half of what one wants to know.

sudo logrotate -d /etc/logrotate.d/isg-breakglass-remote

The Ansible audit run and its deprecation warnings

verify.yml changes nothing — it asserts, host by host, and a clean run ends with changed=0 everywhere. The DEPRECATION WARNING lines are counted separately because a deprecation is not a failure and never reaches the recap.

ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> playbooks/verify.yml 2>&1 | tee /tmp/verify.out | tail -15
grep -c 'DEPRECATION WARNING' /tmp/verify.out

The break-glass drill on one target

The drill is a real login through the emergency door, with the gateway bypassed entirely: the key is offered to the isg-breakglass account on the target, and sudo -n true shows that the account still reaches root without a password. It also leaves the trace the gateway measures — the drill trace row on isg system check returns to OK only once both break-glass layers have a recent login, so a drill on the static key alone leaves the row at WARN.

ssh -i <break-glass key> -o IdentitiesOnly=yes isg-breakglass@<target> 'id -un; sudo -n true && echo SUDO-OK'

The decision on an approval request

When a command waits for four-eyes approval, the session prints the request's ID after request=. Who decided it, when, and with what reason is stored with the request; reading it needs the service account, because the record store's credentials are readable only to it. A decision made in the web UI says so in the reason.

sudo setpriv --reuid=isg --regid=isg-operators --clear-groups /usr/local/bin/isg approvals show <approval-id>

State:             approved
Requested at:      <time>
Decided by:        <approver>
Decided at:        <time>
Decision reason:   <reason> (via admin UI)

Web Access: the rows, the application sessions and the locks

Run on a broker. The first line keeps the three Web Access rows of the system check — the file loads, the shared accounts' secrets are readable, the listener listens and its certificate covers every application. The second lists the application sessions (protocol http); reading the record store needs root, like the approval record above. The third shows the locks, users and applications alike.

isg system check 2>&1 | grep -E '^\[[A-Z ]+\] (apps\.yaml|Web Access)'
sudo env ISG_HOME=/var/lib/isg /usr/local/bin/isg sessions list | grep '(http)'
isg lock list

[OK  ] apps.yaml                    <n> app(s): <app id> <name>→<host> …
[OK  ] Web Access credentials       <n> account(s) readable
[OK  ] Web Access listener          listening on [::]:<port>; certificate covers <n> name(s) …

A running application session's transcript is on the serving broker while the session is open, one JSON line per request (command is the method and the path, exit_code the status the console answered):

sudo tail -n 5 /var/lib/isg/transcript-<session-id>.jsonl

The services on the store, directory and monitoring hosts

The shared store, the directory and the monitoring stack run as rootless containers under one unprivileged account per host, each container behind a systemd --user unit. One ssh per host lists both views: the container with its uptime, and the unit with its state.

for h in <store host> <directory host> <monitoring host>; do ssh -i <admin key> <admin user>@$h 'hostname; podman ps --format "{{.Names}}  {{.Status}}"; systemctl --user list-units --type=service --no-legend "isg-*" | awk "{print \$1, \$3, \$4}"'; done

<store host>
isg-postgres  Up <uptime>
isg-bao  Up <uptime>
isg-rustfs  Up <uptime>
isg-postgres.service active running
…

The scheduled jobs on the controller

The nightly store backup, the daily check of the newest backup's age (09:00) and the monthly upstream check run from systemd --user timers on the controller, not on the fleet. The listing shows the next and the last run; a manual run is the same unit the timer starts, so it measures exactly what the schedule will do.

systemctl --user list-timers --no-pager 'isg-*'
systemctl --user start isg-upstream-<env>.service; systemctl --user status isg-upstream-<env>.service --no-pager | tail -n 15

A one-shot unit that succeeded reads inactive (dead) with status=0/SUCCESS.

The backup-age check can also run straight from the repository on the controller. It is red when the newest archive is older than 26 hours or there is none; without -e isg_backup_age_notify=true it sends no mail.

ansible-playbook -e isg_env=<env> playbooks/backup-age.yml

The rolling upgrade, one broker at a time

On a running fleet the binary is replaced with upgrade-rolling.yml, one broker per run, with a drain before the restart and a measurement after it — not with site.yml, which would restart every broker at once. The broker that does not hold the shared address goes first. The binary is a fixed copy built from the release tag; the play stops before anything is installed if its version does not match the repository's.

ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> -e isg_binary_src=<binary copy> playbooks/upgrade-rolling.yml --limit <broker without the shared address>,<store host>
ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> -e isg_binary_src=<binary copy> playbooks/upgrade-rolling.yml --limit <broker with the shared address>,<store host>

The summary at the end of each run is the part to read:

- 'broker B1 (<broker>): <old version> → <new version> (installed before: <old version>; …
- 'fleet compatibility (B401): 1 → 1; …
- 'drain wait: 1 s, 1 poll(s); sessions at drain on: 0'
- 'post-check: isg system check rc=0 (after 1 attempt(s)); [OK  ] Broker version skew …

The audit run (verify.yml, above) after the second broker closes the upgrade.

A configuration change, rolled out

A change to the configuration source in the repository reaches the hosts in two steps: the configuration source on the controller is regenerated from the repository, then the hosts' role runs. The second run of the same command is the check that the first one finished: it ends with changed=0.

ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> playbooks/config-src.yml
ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> -l <group> playbooks/site.yml

<group> is the inventory group of the hosts the change belongs to. A change that reaches the brokers goes through the rolling upgrade instead (previous section), for the same reason the binary does.

The store backup and the restore drill

The backup run takes the record store, the secret store and the object store into one archive on the controller; the drill restores the newest archive into throwaway containers next to the live ones and measures the restore times, without touching the live store.

ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> playbooks/backup.yml
ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_config_src_dir=<config source> playbooks/backup-restore-drill.yml

The drill's summary names one restore time per store (RTO) and the row counts it compared.

The bundled component versions

The installed version of every third-party component is read from the hosts; the newest stable version upstream is asked from the controller alone — the hosts make no outbound call. A component that is behind is a finding to plan, not a failure.

ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_cv_report_dir=<report directory> playbooks/component-versions.yml
ansible-playbook -i inventory/<env>.ini -e isg_env=<env> -e isg_cv_report_dir=<report directory> playbooks/component-upstream.yml

B429 UPSTREAM — 9 komponens; current 8, behind 0, not_measured 1