The previous gate did not detect whether the restarted minion was back:
systemctl is-active --quiet salt-minion \
&& salt-call --local --timeout=5 --out=quiet test.ping
Both halves are near-vacuous. `--local` sets file_client=local, so test.ping
runs in a throwaway minion that never contacts the master and never inspects
the running daemon; it only proves python and the module loader work. And the
shipped unit is Type=notify with notify_systemd() called before the daemon
imports salt.cli.daemons, so is-active goes true at process launch, not at
connection. The script could return ready while the minion was still
authenticating, which is the race it exists to prevent.
Gate instead on the condition salt itself uses to log "Minion is ready to
receive requests!", requiring both signals of the current daemon instance:
1. the pid-tagged ready line in the minion log. tune_in() emits it only
after sync_connect_master() returns, i.e. the pub channel authenticated,
the req channel connected, and _post_master_init() finished loading
modules and compiling pillar.
2. that same pid holding an ESTABLISHED req connection to a master on 4506
plus a second (publish) connection to the same master IP. The publish
port is absent from minion config -- the minion learns it from the
master's auth reply -- so it is derived from the connection.
Resolve the daemon pid from systemd (MainPID -> pgrep -P), never from
/var/run/salt-minion.pid. salt_minion() runs the minion in a multiprocessing
child; that child writes the pidfile, owns the sockets and logs the ready
line, while MainPID is the parent. During a restart the pidfile still names
the old child, whose own ready line is already in the log, so keying off it
reports ready instantly. Children of the current MainPID exclude the old
instance structurally, with no timing assumptions.
Degrade deterministically rather than spinning to the timeout: if
log_level_logfile does not emit INFO records the ready line can never appear,
so detect that up front from the merged config and fall back to the socket
check. log_level_logfile defaults to None (inherit log_level), so resolve the
inheritance before deciding. If ss is unavailable, fall back to the log gate.
If neither signal is usable, fail immediately with a clear message.
Requiring master connectivity adds no new dependency: every path that applies
salt.minion or a highstate does so without --local, so file_client=remote
already required a reachable master to fetch salt:// files. No salt-call
master round-trip is added; the daemon's own successful auth already proves
the key is accepted.
Also fix the comment above wait_for_salt_minion_ready, which attributed the
script to common_sbin/common/tools/sbin (it is deployed by salt_sbin from
salt/tools/sbin) and asserted a --no-block restart that appears nowhere in
the repo. No state logic changed.
Only the RHCK->UEK flavor cross needs grubby --set-default; a UEK7->UEK8
update stays in the kernel-uek lineage and auto-promotes on its own. Detect
the running kernel and act accordingly:
- UEK8: already on target, no-op.
- UEK7: populate the repo and install UEK8, then verify it auto-promoted
(warn with the manual grubby command if it did not) -- no grubby change.
- RHCK: install UEK8 and set the boot default explicitly, as before.
Also make an already-installed UEK8 skip the repo entirely so a disabled or
empty kernel repo can't block flipping the default, and correct the header
comment that claimed every transition needs grubby.
Three stages of the UEK8 path fail silently, and the script only handled
the last one:
1. Populate. so-repo-sync runs before the highstate deploys the
[securityonionkernel] section into repodownload.conf, so the first
kernel-aware soup skips the kernel sync. kernelrepo_init_empty then
seeds valid-but-empty repodata, leaving an enabled repo with zero
packages. dnf resolves it happily and installs nothing, no error.
2. Install. `dnf install kernel-uek` on a UEK7 node sees kernel-uek 5.15
already installed, prints "Nothing to do" and exits 0 -- so the script
sailed past the install and died later with a misleading grubby error.
3. Boot. Already handled: grubby only auto-promotes within the running
kernel's flavor lineage, so 5.x -> 6.x UEK never promotes on its own.
Add ensure_kernel_repo(), which verifies the repo is enabled (necessary
because skip_if_unavailable=1 hides a broken repo) and that it can serve a
6.x kernel-uek. When it cannot, a manager runs so-repo-sync to populate
/nsm/kernelrepo and re-checks; a minion cannot fix it and exits non-zero
pointing the admin at the manager. Airgap managers bail, since their repo
comes from the ISO rather than a sync.
Install the explicit UEK8 NEVRA instead of the bare package name so the
"Nothing to do" exit-0 case cannot mask a no-op, and pin the repoquery to
securityonionkernel so a UEK7 kernel-uek in the main repo is never picked.
Still idempotent and still never reboots.
The script assumed the UEK8 kernel was already installed and only switched
the boot default to it. On a node running the EL9 stock kernel (RHCK 5.14)
there is no kernel-uek* package at all, so `dnf update` has nothing to
upgrade and UEK8 never lands -- the script just logged "nothing to do" and
exited 0.
When no 6.x UEK boot entry exists, install the kernel-uek metapackage (it
pulls kernel-uek-core plus the module subpackages, including
kernel-uek-modules-extra-netfilter) and then proceed with the grubby
switch. Fail loudly if securityonionkernel is not an enabled repo, since
that assignment is gated on the NIC-pin marker and the salt version match
and a silent no-op there is hard to diagnose.
Also point DEFAULTKERNEL at kernel-uek-core so later kernel updates stay on
the UEK line rather than falling back to RHCK.
Still idempotent and still never reboots.
On a fresh install the surirulesync file.recurse creates .gitkeep before
SOC has generated all-rulesets.rules. That change satisfied the
surirulereload onchanges requisite, so the reload ran with no ruleset
present, failed to stat the file, and reported the state (and install)
as failed.
Add an onlyif guard so the reload only runs when all-rulesets.rules
exists. A .gitkeep-only sync now leaves the state a clean success
(onlyif condition false); once SOC writes the ruleset, the reload fires
normally.
Route the reload/verify output (ours plus so-common's retry/fail lines)
through a synchronous timestamping pipeline so every line in reload.log
is prefixed with a date/time, and preserve the real exit code via
PIPESTATUS.
Josh Patterson
Jorge Reyes
Mike Reeves
Jason Ertel
Jason Ertel
coreyogburn
Josh Brower
Mike Reeves
Corey Ogburn
Matthew Wright