Target host preparation

A target host is not ready because the gateway can reach it. It is ready when isg system check says it is — and that verdict is not a summary of this page, it is the definition this page describes. Everything below exists to make one row read OK:

Ready to broker: <host>

What "ready" is measured as

The readiness row is an aggregate of seven sub-checks. It never dials anything itself: each answer is a measurement one of the neighbouring checks in the same run already made. Any sub-check not met makes the row FAIL; otherwise any sub-check not measured makes it WARN; only all-met is OK. A not-met and a not-measured are different verdicts and are never folded together.

Sub-check 7 is not a threshold to clear. The lockdown level is a direct input to the per-command policy gate and to four-eyes approval, so the row would be hiding one of its own inputs if it left the value out.

Note what follows from the table: a plain isg system check cannot produce an OK readiness row. Sub-checks 3, 4 and 5 are gated behind probe flags, so without them the row reads WARN with each unmeasured sub-check naming the flag that would measure it. That is the intended reading — the row does not claim a host is ready on evidence it did not gather.

sshd requirements

Two directives, and they are two different directions of trust. Neither replaces the other.

TrustedUserCAKeys — the target accepts us. This is what makes the gateway's ephemeral session certificate an acceptable login. Install the CA's public key and point sshd at it:

# on the gateway
isg ca pubkey > isg_ca.pub

# on the target
sudo install -m 644 isg_ca.pub /etc/ssh/ca/isg_ca.pub
printf '%s\n' 'TrustedUserCAKeys /etc/ssh/ca/isg_ca.pub' 'LogLevel VERBOSE' \
  | sudo tee /etc/ssh/sshd_config.d/50-isg-userca.conf
sudo sshd -t && sudo systemctl reload sshd

The directive is additive: existing key-based logins keep working, which is why this step is safe to apply before anything else. Check sshd_config for an Include /etc/ssh/sshd_config.d/*.conf line first — without one, the directive belongs in the main file. Run sshd -t before the reload, every time.

LogLevel VERBOSE goes into the same drop-in for the same reason it is in the break-glass Match block below: it is what makes the target's own log attribute a certificate login to the key fingerprint, and the drop-in is the place that covers every brokered login, not only the emergency account.

Cert-only posture is a separate, opt-in step. Once certificate logins are measured working on this host, PasswordAuthentication no can be added to the same drop-in so the certificate is the only way in. Do not add it earlier: with it, a host whose TrustedUserCAKeys line is wrong has no login left. The isg_target_host Ansible role has this behind isg_target_host_cert_only, off by default, for exactly that reason.

HostCertificate — the target proves itself to us. The gateway's default host-key policy is strict, which means it accepts only a host certificate signed by its own CA: not a bare host key, and not a key it has seen before. Issue one over the target's existing host public key — the CA never sees the private half — and install it:

# on the gateway
isg ca host-cert issue --host <host> --key ./<host>_host_ed25519.pub

# on the target
sudo install -m 644 <host>-cert.pub /etc/ssh/ssh_host_ed25519_key-cert.pub
echo 'HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub' \
  | sudo tee /etc/ssh/sshd_config.d/60-isg-hostcert.conf
sudo sshd -t && sudo systemctl reload sshd

The file to copy to the target is the published mirror under pub/hosts/, not the gateway's own copy under ca/. A reload (SIGHUP) is enough: sshd picks up the HostCertificate on re-reading its config, open sessions are left alone, and the gateway's strict host-key check passes right after — no restart is needed, and the isg_target_host role reloads for the same reason.

The certificate's principals must match the name the gateway dials. --principals defaults to --host, and the strict host-key check compares the certificate's principals against the exact hostname used for the connection. A certificate issued for host.example.com is rejected when the inventory calls the machine host. List every alias: --principals host,host.example.com.

There is no principals file to maintain. The mechanism is the certificate's own principal list: the gateway mints a certificate whose single principal is the OS user of the session, and sshd admits that certificate as a login for that user because the CA is trusted. Nothing in ISG writes, requires or reads an AuthorizedPrincipalsFile.

Accounts and groups

The gateway itself needs no account on the target.

The break-glass block is a small, self-contained drop-in:

Match User isg-breakglass
    AuthorizedKeysFile /etc/ssh/breakglass_authorized_keys
    PubkeyAuthentication yes
    LogLevel VERBOSE
Match all

The closing Match all is required, not decoration: without it every global directive that follows the Include in the main sshd_config would fall under this Match block. LogLevel VERBOSE is what makes the target's own log attribute a login to a key fingerprint. The keyset file is expected to be mode 0644 owned by root:root — the account must not be able to rewrite the list of keys that admit it — and every line, including a cert-authority line, is expected to carry the restrict option.

Named local accounts, one per person

On a target host the account name is not a convention to agree on — it is the only name the login can have. On the gateway-login front the session is always opened under the name of the person who authenticated; there is no --as to redirect it, and there is no principal mapping on the target either: the role installs TrustedUserCAKeys and no AuthorizedPrincipalsFile, so sshd admits the certificate as a login for the principal the certificate carries, and that principal is the person's own user name. A shared account would have nothing to match against — the certificate names a person, the host offers one account for everybody, and the connection is simply refused.

So everyone who should reach a target host has their own local account on it. The isg_target_host Ansible role creates them from one list, isg_target_host_users, and three properties of those accounts are deliberate rather than incidental.

A shell, a home, and a locked password. The account exists in order to run commands — that is the thing ISG brokers — so it gets a login shell (isg_target_host_users_shell, /bin/bash) and a home directory. The password field is locked: the certificate is the credential, and a password must not become a second way in. One directive makes that combination work and is worth knowing by name: UsePAM yes. Without PAM, sshd reads the locked shadow field as a disabled account and refuses the login even though the certificate and the trusted CA are both in order. Both distribution families ship yes; the role does not set it, it measures it.

No sudo. The role writes no sudoers file and adds nobody to wheel or sudo. An account it manages carries exactly two groups: its own primary group and the managed group. Escalation on a target host stays what the table above says it is — the target's own sudo rules, decided per host by whoever owns the host, not a side effect of account creation. The verification measures both directions: the group list with id -Gn, and what the account may actually run with sudo -l -U.

The managed group is the scope — and an account that already exists is never adopted. Every account the role creates is a member of one group (isg_target_host_users_group, isg-testers), and that group is the role's reach: a name on the list becomes an account, a member of the group that has left the list is removed with its home, and an account outside the group is neither read nor touched. That last part is what keeps the admin account the automation itself connects with out of range.

It also fixes what happens when a listed name is already taken on the host by an account that is not in the managed group. The role refuses to run, and names the collision. Adopting the account instead would look harmless — it would simply join the group — but it would also move that account inside the role's scope, where a later, entirely innocent narrowing of the name list deletes it along with its home. The failure would arrive months after the change that caused it, which is why the answer is a stop rather than a warning: either take the name off the list, or rename the existing account by hand first.

An empty isg_target_host_users list has a stated meaning too. It does not mean "leave accounts alone"; it means the managed group is empty, so its current members are removed. There is no state in which the role silently manages nothing.

Directory-backed accounts and password sudo

A local account created with a locked password is enough to log in — the certificate is the credential — but it cannot use a password sudo: the target asks for the account's password, and there is none that anyone knows. Escalation needs an account whose password the person already has. In an enterprise that is the directory account: the target host is joined to the same directory that ISG authenticates against (SSSD, or the AD integration of the platform), sudo asks for the person's own directory password, and the target checks it — the same password the person signed in to ISG with, typed again. The gateway does not reuse the sign-in password and does not store it; with single sign-on it never sees one at all.

The isg_target_directory Ansible role sets up exactly that on a target host: SSSD over ldaps with certificate verification required, a read-only bind account of its own (isg_cfg_ldap_target_bind_dn — separate from the brokers' and the IdP's, so each can be rotated on its own), home directories created at first login, and one sudoers drop-in that names a directory group (isg_target_directory_sudo_group). Who may escalate is therefore decided in the directory, by group membership. When the role is enabled, the named local accounts of the previous section are not used on that host: a local account with the same name would shadow the directory one, and the role's verification stops on such a collision. It also stops when one of the user's group IDs has no name in the directory — typically the primary group (gidNumber) with no matching posixGroup —, because id then fails on the target.

The two gates, in order. Every command line is decided by ISG first — the policy rules, the approval requirements, the per-host level and human_access — and ISG reads through sudo to the program it runs, so a rule that matches reboot also matches sudo reboot. Only a command ISG let through reaches the target, and only then does the target's sudo apply its own rules and ask for the password. The target can narrow what ISG allowed; it cannot widen what ISG refused. The effective right is the intersection of the two.

Where the target is the only control. ISG refuses interactive root shells (sudo su, sudo -i, sudo -s, sudo bash) unconditionally. A program that can start a shell of its own (sudo vim, then :!sh) is different: inside it, ISG no longer sees the individual commands. Against that, the control is the target's sudoers — NOEXEC, or not granting such programs at all. The session recording still captures the output; preventing it is the target's job.

Network

One direction, one port.

Nothing on the target ever connects back to the gateway. There is no agent to install, no callback, and no second port.

One operational property of the probes is worth planning around: a single --probe-hosts run spends 4 of each target's 9-connection PerSourcePenalties budget, and the target scores that against the gateway's source address, not against an operator. Two runs in quick succession is what trips a target's lockout — and a lockout produces a false FAIL, which invites the retry that trips it again. The gateway enforces a minimum interval between fleet-wide probe runs for exactly this reason, and refuses the second one with the number of seconds to wait. Do not retry to confirm a failure; wait it out.

That interval is enforced by each gateway on its own, so a target facing more than one of them sees the rate multiplied by however many there are. This is deliberate — the interval bounds load rather than protecting correctness, and coordinating it across gateways would put a shared write on the probe path — but it does mean the budget above is the one to size against the whole fleet, not against a single gateway.

Verification

The static half, which needs nothing but the gateway:

isg system check

It validates every configuration file and reports the readiness row for every host in the inventory plus every host with an issued certificate — the gateways' own front certificates excepted: a gateway is not a brokered target unless the inventory lists it, and the same rule shapes the target set every --probe-hosts probe below dials. Add --verbose to see the sub-checks behind each row, and each row's caveat — what it did not measure.

The live half, which needs isg serve running:

isg system check --probe-hosts --probe-keyset --verbose

--probe-hosts refuses to run without the daemon, and says so, rather than opening connections to production hosts that would appear in each target's sshd log with no ISG audit entry to explain them. (--probe-without-audit overrides that for an urgent measurement, and states in the output what it gave up.) Every probe is audited separately: REACHABILITY_PROBE, BYPASS_PROBE, HUMAN_CERT_PROBE, BREAK_GLASS_AUTH_METHOD_PROBE, and — for the keyset read, which is a real login rather than a preauth close — BREAK_GLASS_KEYSET_READ.

The inventory's own view:

isg servers show
isg servers level <host>

What a green row does and does not prove

Ready to broker: <host> reading OK means: an access rule covers this host, a live host certificate exists for it, the target accepted a certificate from this CA, the emergency path is in place and its keyset is intact, the host answers on TCP 22 with a trusted host key, and the lockdown level resolved.

It does not prove that sudo works for isg-breakglass once logged in, and it does not prove any operator holds the private half of an accepted break-glass key — the probe only ever offers a public key and observes the answer. Confirming either needs a real break-glass login by a human holding a real key. It also does not prove that a particular person may connect: readiness is a host-level question, and whether a given principal is allowed is decided per request by the access gates.

Common drift

Everything here is a state the checks actually name. The quoted text is what the check prints.

The host certificate was never installed, or sshd started without it. The reachability check reports target offers a plain host key, not a certificate — sshd may have started without the host cert (fail-open); strict-mode connect would be rejected. This is the fail-open case: sshd came up, the port answers, and a strict connect would be refused. It is caught by a plain isg system check, without any probe flag.

The certificate expired. host cert expired on sub-check 2, and the host becomes unreachable in strict mode. The host-cert row warns for 30 days before that happens, so this state is always preceded by a warning somebody had time to act on.

The certificate was revoked. host cert revoked. Revocation is gateway-side: the target's configuration is untouched, and the serial is consulted on every dial with no cache to go stale.

TrustedUserCAKeys was removed or points at the wrong file. Sub-check 3 reports certificate rejected (cert_rejected). This one is only visible under --probe-hosts — nothing on the gateway can see the target's sshd config, so it takes an actual offer to the target to find out.

The host is not covered by any access rule. no human_access rule names this host. A host can be reachable, certified and perfectly healthy, and still be one nobody can connect to, because no allow rule mentions it. There is no host wildcard to catch it — use a host_labels selector.

Two hosts share one host key. The host-cert row warns duplicate host key shared with <other host>. The normal cause is a golden image or a cloned VM snapshot that shipped one /etc/ssh/ssh_host_*_key to every machine, and the consequence is that either host can present the other's identity to the strict check. Issuance refuses a second certificate over an already-certified key for this reason; the fix is to regenerate the clone's host keys (ssh-keygen -A) and re-issue.

The break-glass keyset drifted. The keyset row reports each finding rather than stopping at the first: unexpected keyset mode/owner: … (want 0644 root:root), N of M key(s) missing the restrict option, no cert-authority line in the keyset — the break-glass CA is not trusted on this host, or — the one that fails rather than warns — keyset trusts an unexpected break-glass CA. A host whose emergency door is not ours is not ready to broker.

The break-glass account is missing. isg-breakglass account does not exist on this host, or break-glass keyset is missing or unreadable (/etc/ssh/breakglass_authorized_keys).

The Match block is no longer in effect. break-glass Match block NOT in effect — no offered key was accepted by isg-breakglass: the keyset file was read and every key in it was offered, and the target accepted none. Either the Match block is not routing that account there, or sshd is not reading this file for it.

For anything not on this list, Troubleshooting catalogues the messages by symptom.