With two independent checks now writing to the same log, messages like
"system uptime only N seconds does not meet 1800 second requirement" were
ambiguous about which check they came from. Prefix every line with a
[minion-restart-check] or [boot-highstate-check] tag and reword the uptime,
threshold, and healthy messages to say what was evaluated and why it was
skipped.
Restructure the boot-highstate check from a nested if into an if/elif chain so
each outcome (restart already queued, uptime too low, healthy, already running,
forcing) logs its own reason instead of silently doing nothing.
Add a second, independent trigger to the every-5-minute health check: if the
host has been up >= 15 minutes (HIGHSTATE_UPTIME_REQ) and no highstate has
completed since this boot (lasthighstate mtime older than boot time), run
salt-call state.highstate. This recovers a host whose boot highstate
(so-boot-highstate.service) failed or was skipped, even while the minion is
otherwise healthy and touching state-apply-test.
The new path deliberately does not enable highstate, so a soup-disabled
highstate is respected and never forced mid-upgrade. A saltutil.running guard
plus queue=True prevents stacking across successive cron runs, and a RESTARTED
flag suppresses the new block when the existing minion-restart path already
queued a highstate.
Live testing on a standalone node found the previous commit still reported
ready on the OUTGOING daemon. Reproduced by running the production sequence:
systemctl restart --no-block salt-minion # what service.restart issues
/usr/sbin/so-salt-minion-wait # what cmd.run then runs
so-salt-minion-wait: gating on pid-tagged ready line ... plus master sockets
salt-minion (pid 2750297) ready after 3s # 2750297 is the OLD child
salt restarts this unit with --no-block -- _no_block_default() in
salt/modules/systemd_service.py returns True when the unit is the salt-minion
service -- so service.restart returns as soon as the job is enqueued. Measured
on the host, systemd does not swap MainPID until ~7.3s later. Throughout that
window the old daemon is still running, still holds its master sockets, and
its own ready line is already in the log, so every gate passes on the instance
that is about to be torn down. INITIAL_SLEEP=3 expired inside that window.
Wait for systemd's job queue for the unit to drain before resolving MainPID.
That is deterministic rather than a timing guess: the job exists from the
moment --no-block returns until the new instance signals READY, and MainPID is
new by the time it clears. Measured transition:
t=0.0s job pending, child=OLD, sockets up, ready line present
t=7.9s job drained, child=NEW, sockets up, ready line ABSENT
t=10.7s ready line for NEW child appears <- script returns here
The same run also confirms empirically why the log line is required in
addition to the sockets: for 2.8s the new child has both master connections
while _post_master_init() is still loading modules and compiling pillar, so a
socket-only gate would return that much too early.
Correct the comment claim from the previous commit. The --no-block restart is
real; it lives in salt's systemd_service module, not in this repo, which is
why searching the repo for it turned up nothing.
The reposync section in repodownload.conf and the client repo assigned in
repo/client/oracle.sls both used the bare name securityonionkernel, colliding
across the two roles. Rename the reposync-side section (and its --repoid, the
so-repo-sync guard, and the so-kernel-upgrade presence check) to
securityonionkernelsync, mirroring the existing securityonion/securityonionsync
split for the main repo. The client repo stays securityonionkernel. Also give
the section its own name=Security Onion Kernel Repo repo.
When the minion is deemed hung and restarted, wait for it to become
ready via so-salt-minion-wait, then kick off salt-call state.highstate
(queued, backgrounded) so the box re-applies its states and recovers on
its own rather than waiting for the next scheduled highstate.
The previous gate did not detect whether the restarted minion was back:
systemctl is-active --quiet salt-minion \
&& salt-call --local --timeout=5 --out=quiet test.ping
Both halves are near-vacuous. `--local` sets file_client=local, so test.ping
runs in a throwaway minion that never contacts the master and never inspects
the running daemon; it only proves python and the module loader work. And the
shipped unit is Type=notify with notify_systemd() called before the daemon
imports salt.cli.daemons, so is-active goes true at process launch, not at
connection. The script could return ready while the minion was still
authenticating, which is the race it exists to prevent.
Gate instead on the condition salt itself uses to log "Minion is ready to
receive requests!", requiring both signals of the current daemon instance:
1. the pid-tagged ready line in the minion log. tune_in() emits it only
after sync_connect_master() returns, i.e. the pub channel authenticated,
the req channel connected, and _post_master_init() finished loading
modules and compiling pillar.
2. that same pid holding an ESTABLISHED req connection to a master on 4506
plus a second (publish) connection to the same master IP. The publish
port is absent from minion config -- the minion learns it from the
master's auth reply -- so it is derived from the connection.
Resolve the daemon pid from systemd (MainPID -> pgrep -P), never from
/var/run/salt-minion.pid. salt_minion() runs the minion in a multiprocessing
child; that child writes the pidfile, owns the sockets and logs the ready
line, while MainPID is the parent. During a restart the pidfile still names
the old child, whose own ready line is already in the log, so keying off it
reports ready instantly. Children of the current MainPID exclude the old
instance structurally, with no timing assumptions.
Degrade deterministically rather than spinning to the timeout: if
log_level_logfile does not emit INFO records the ready line can never appear,
so detect that up front from the merged config and fall back to the socket
check. log_level_logfile defaults to None (inherit log_level), so resolve the
inheritance before deciding. If ss is unavailable, fall back to the log gate.
If neither signal is usable, fail immediately with a clear message.
Requiring master connectivity adds no new dependency: every path that applies
salt.minion or a highstate does so without --local, so file_client=remote
already required a reachable master to fetch salt:// files. No salt-call
master round-trip is added; the daemon's own successful auth already proves
the key is accepted.
Also fix the comment above wait_for_salt_minion_ready, which attributed the
script to common_sbin/common/tools/sbin (it is deployed by salt_sbin from
salt/tools/sbin) and asserted a --no-block restart that appears nowhere in
the repo. No state logic changed.
Only the RHCK->UEK flavor cross needs grubby --set-default; a UEK7->UEK8
update stays in the kernel-uek lineage and auto-promotes on its own. Detect
the running kernel and act accordingly:
- UEK8: already on target, no-op.
- UEK7: populate the repo and install UEK8, then verify it auto-promoted
(warn with the manual grubby command if it did not) -- no grubby change.
- RHCK: install UEK8 and set the boot default explicitly, as before.
Also make an already-installed UEK8 skip the repo entirely so a disabled or
empty kernel repo can't block flipping the default, and correct the header
comment that claimed every transition needs grubby.
Three stages of the UEK8 path fail silently, and the script only handled
the last one:
1. Populate. so-repo-sync runs before the highstate deploys the
[securityonionkernel] section into repodownload.conf, so the first
kernel-aware soup skips the kernel sync. kernelrepo_init_empty then
seeds valid-but-empty repodata, leaving an enabled repo with zero
packages. dnf resolves it happily and installs nothing, no error.
2. Install. `dnf install kernel-uek` on a UEK7 node sees kernel-uek 5.15
already installed, prints "Nothing to do" and exits 0 -- so the script
sailed past the install and died later with a misleading grubby error.
3. Boot. Already handled: grubby only auto-promotes within the running
kernel's flavor lineage, so 5.x -> 6.x UEK never promotes on its own.
Add ensure_kernel_repo(), which verifies the repo is enabled (necessary
because skip_if_unavailable=1 hides a broken repo) and that it can serve a
6.x kernel-uek. When it cannot, a manager runs so-repo-sync to populate
/nsm/kernelrepo and re-checks; a minion cannot fix it and exits non-zero
pointing the admin at the manager. Airgap managers bail, since their repo
comes from the ISO rather than a sync.
Install the explicit UEK8 NEVRA instead of the bare package name so the
"Nothing to do" exit-0 case cannot mask a no-op, and pin the repoquery to
securityonionkernel so a UEK7 kernel-uek in the main repo is never picked.
Still idempotent and still never reboots.
The script assumed the UEK8 kernel was already installed and only switched
the boot default to it. On a node running the EL9 stock kernel (RHCK 5.14)
there is no kernel-uek* package at all, so `dnf update` has nothing to
upgrade and UEK8 never lands -- the script just logged "nothing to do" and
exited 0.
When no 6.x UEK boot entry exists, install the kernel-uek metapackage (it
pulls kernel-uek-core plus the module subpackages, including
kernel-uek-modules-extra-netfilter) and then proceed with the grubby
switch. Fail loudly if securityonionkernel is not an enabled repo, since
that assignment is gated on the NIC-pin marker and the salt version match
and a silent no-op there is hard to diagnose.
Also point DEFAULTKERNEL at kernel-uek-core so later kernel updates stay on
the UEK line rather than falling back to RHCK.
Still idempotent and still never reboots.
Josh Patterson
Jorge Reyes
Jason Ertel
Jason Ertel
Mike Reeves
coreyogburn
Josh Brower
Mike Reeves