Before making any changes, verify the grid is in a good state:
- check_cluster_health: waits for Elasticsearch to reach at least 'yellow'
(blocks only on red/unreachable, since yellow is normal), modeled on the
wait in so-elasticsearch-roles-load.
- check_fleet_server: confirms the Fleet Server status API returns HTTP 200,
modeled on the wait_for_so-elastic-fleet state in elasticfleet/enabled.sls.
Both run alongside the existing check_pillar_items (manager pillar render) and
verify_es_version_compatibility, before soup modifies anything, so a failure
exits cleanly with an actionable message and no partial changes. Valid on all
manager roles soup runs on (eval/standalone/manager/managerhype/managersearch/
import), which all run Elasticsearch and the Fleet Server.
After a partial upgrade, /etc/soversion already reads the target version, so
soup's startup line "Found that Security Onion X is currently installed" made
it look finished even as soup resumed. When a resume marker is present and
differs from the installed version, print an explicit NOTE that the grid is only
partially upgraded and this run will resume and complete it.
Also clear any stale resume marker in the already-latest path so a successfully
completed upgrade is never mistaken for a partial one and re-run on a later
invocation (the marker is normally removed at the end of postupgrade_changes;
this is a belt-and-suspenders guard).
When soup fails via the EXIT trap after it has begun modifying the system, print
a prominent UPGRADE INCOMPLETE banner instructing the user to run soup again to
resume and complete the update. Gated on a new SOUP_UPGRADE_STARTED flag set at
the start of the hotfix and upgrade branches, so pre-flight gate failures (ES
compatibility, disk, network) that abort before any changes are made do not show
it.
A failed highstate mid-upgrade left /etc/soversion already advanced to the
target version (the highstate stamps it from the pillar via the soversionfile
state), so a re-run of soup saw INSTALLEDVERSION == NEWVERSION and reported
"already running the latest version", stranding the box with post-upgrade
steps never run.
Introduce /etc/sopostversion, a soup-owned marker (no salt state manages it)
that records post-upgrade walk progress. It is seeded from the pre-upgrade
version before the highstate, advanced after each post_to_* step, and removed
on successful completion. upgrade_check treats a leftover marker as "upgrade
not finished" and resumes the remaining post steps instead of bailing.
Also fix the hotfix path: /etc/sohotfix was written before the hotfix
highstate, so a failed hotfix highstate looked already-applied on re-run.
Since no salt state manages /etc/sohotfix, defer its write (update_version)
until after the highstate succeeds so it is an honest completion marker.
The reposync section in repodownload.conf and the client repo assigned in
repo/client/oracle.sls both used the bare name securityonionkernel, colliding
across the two roles. Rename the reposync-side section (and its --repoid, the
so-repo-sync guard, and the so-kernel-upgrade presence check) to
securityonionkernelsync, mirroring the existing securityonion/securityonionsync
split for the main repo. The client repo stays securityonionkernel. Also give
the section its own name=Security Onion Kernel Repo repo.
Only the RHCK->UEK flavor cross needs grubby --set-default; a UEK7->UEK8
update stays in the kernel-uek lineage and auto-promotes on its own. Detect
the running kernel and act accordingly:
- UEK8: already on target, no-op.
- UEK7: populate the repo and install UEK8, then verify it auto-promoted
(warn with the manual grubby command if it did not) -- no grubby change.
- RHCK: install UEK8 and set the boot default explicitly, as before.
Also make an already-installed UEK8 skip the repo entirely so a disabled or
empty kernel repo can't block flipping the default, and correct the header
comment that claimed every transition needs grubby.
Three stages of the UEK8 path fail silently, and the script only handled
the last one:
1. Populate. so-repo-sync runs before the highstate deploys the
[securityonionkernel] section into repodownload.conf, so the first
kernel-aware soup skips the kernel sync. kernelrepo_init_empty then
seeds valid-but-empty repodata, leaving an enabled repo with zero
packages. dnf resolves it happily and installs nothing, no error.
2. Install. `dnf install kernel-uek` on a UEK7 node sees kernel-uek 5.15
already installed, prints "Nothing to do" and exits 0 -- so the script
sailed past the install and died later with a misleading grubby error.
3. Boot. Already handled: grubby only auto-promotes within the running
kernel's flavor lineage, so 5.x -> 6.x UEK never promotes on its own.
Add ensure_kernel_repo(), which verifies the repo is enabled (necessary
because skip_if_unavailable=1 hides a broken repo) and that it can serve a
6.x kernel-uek. When it cannot, a manager runs so-repo-sync to populate
/nsm/kernelrepo and re-checks; a minion cannot fix it and exits non-zero
pointing the admin at the manager. Airgap managers bail, since their repo
comes from the ISO rather than a sync.
Install the explicit UEK8 NEVRA instead of the bare package name so the
"Nothing to do" exit-0 case cannot mask a no-op, and pin the repoquery to
securityonionkernel so a UEK7 kernel-uek in the main repo is never picked.
Still idempotent and still never reboots.
The script assumed the UEK8 kernel was already installed and only switched
the boot default to it. On a node running the EL9 stock kernel (RHCK 5.14)
there is no kernel-uek* package at all, so `dnf update` has nothing to
upgrade and UEK8 never lands -- the script just logged "nothing to do" and
exited 0.
When no 6.x UEK boot entry exists, install the kernel-uek metapackage (it
pulls kernel-uek-core plus the module subpackages, including
kernel-uek-modules-extra-netfilter) and then proceed with the grubby
switch. Fail loudly if securityonionkernel is not an enabled repo, since
that assignment is gated on the NIC-pin marker and the salt version match
and a silent no-op there is hard to diagnose.
Also point DEFAULTKERNEL at kernel-uek-core so later kernel updates stay on
the UEK line rather than falling back to RHCK.
Still idempotent and still never reboots.
Mike ReevesandGitHub
Mike Reeves
Jason ErtelandGitHub
Jason Ertel
Jorge ReyesandGitHub
coreyogburnandGitHub
Josh Brower
Corey Ogburn