Live testing on a standalone node found the previous commit still reported
ready on the OUTGOING daemon. Reproduced by running the production sequence:
systemctl restart --no-block salt-minion # what service.restart issues
/usr/sbin/so-salt-minion-wait # what cmd.run then runs
so-salt-minion-wait: gating on pid-tagged ready line ... plus master sockets
salt-minion (pid 2750297) ready after 3s # 2750297 is the OLD child
salt restarts this unit with --no-block -- _no_block_default() in
salt/modules/systemd_service.py returns True when the unit is the salt-minion
service -- so service.restart returns as soon as the job is enqueued. Measured
on the host, systemd does not swap MainPID until ~7.3s later. Throughout that
window the old daemon is still running, still holds its master sockets, and
its own ready line is already in the log, so every gate passes on the instance
that is about to be torn down. INITIAL_SLEEP=3 expired inside that window.
Wait for systemd's job queue for the unit to drain before resolving MainPID.
That is deterministic rather than a timing guess: the job exists from the
moment --no-block returns until the new instance signals READY, and MainPID is
new by the time it clears. Measured transition:
t=0.0s job pending, child=OLD, sockets up, ready line present
t=7.9s job drained, child=NEW, sockets up, ready line ABSENT
t=10.7s ready line for NEW child appears <- script returns here
The same run also confirms empirically why the log line is required in
addition to the sockets: for 2.8s the new child has both master connections
while _post_master_init() is still loading modules and compiling pillar, so a
socket-only gate would return that much too early.
Correct the comment claim from the previous commit. The --no-block restart is
real; it lives in salt's systemd_service module, not in this repo, which is
why searching the repo for it turned up nothing.
When the minion is deemed hung and restarted, wait for it to become
ready via so-salt-minion-wait, then kick off salt-call state.highstate
(queued, backgrounded) so the box re-applies its states and recovers on
its own rather than waiting for the next scheduled highstate.
The previous gate did not detect whether the restarted minion was back:
systemctl is-active --quiet salt-minion \
&& salt-call --local --timeout=5 --out=quiet test.ping
Both halves are near-vacuous. `--local` sets file_client=local, so test.ping
runs in a throwaway minion that never contacts the master and never inspects
the running daemon; it only proves python and the module loader work. And the
shipped unit is Type=notify with notify_systemd() called before the daemon
imports salt.cli.daemons, so is-active goes true at process launch, not at
connection. The script could return ready while the minion was still
authenticating, which is the race it exists to prevent.
Gate instead on the condition salt itself uses to log "Minion is ready to
receive requests!", requiring both signals of the current daemon instance:
1. the pid-tagged ready line in the minion log. tune_in() emits it only
after sync_connect_master() returns, i.e. the pub channel authenticated,
the req channel connected, and _post_master_init() finished loading
modules and compiling pillar.
2. that same pid holding an ESTABLISHED req connection to a master on 4506
plus a second (publish) connection to the same master IP. The publish
port is absent from minion config -- the minion learns it from the
master's auth reply -- so it is derived from the connection.
Resolve the daemon pid from systemd (MainPID -> pgrep -P), never from
/var/run/salt-minion.pid. salt_minion() runs the minion in a multiprocessing
child; that child writes the pidfile, owns the sockets and logs the ready
line, while MainPID is the parent. During a restart the pidfile still names
the old child, whose own ready line is already in the log, so keying off it
reports ready instantly. Children of the current MainPID exclude the old
instance structurally, with no timing assumptions.
Degrade deterministically rather than spinning to the timeout: if
log_level_logfile does not emit INFO records the ready line can never appear,
so detect that up front from the merged config and fall back to the socket
check. log_level_logfile defaults to None (inherit log_level), so resolve the
inheritance before deciding. If ss is unavailable, fall back to the log gate.
If neither signal is usable, fail immediately with a clear message.
Requiring master connectivity adds no new dependency: every path that applies
salt.minion or a highstate does so without --local, so file_client=remote
already required a reachable master to fetch salt:// files. No salt-call
master round-trip is added; the daemon's own successful auth already proves
the key is accepted.
Also fix the comment above wait_for_salt_minion_ready, which attributed the
script to common_sbin/common/tools/sbin (it is deployed by salt_sbin from
salt/tools/sbin) and asserted a --no-block restart that appears nowhere in
the repo. No state logic changed.
On a fresh install the surirulesync file.recurse creates .gitkeep before
SOC has generated all-rulesets.rules. That change satisfied the
surirulereload onchanges requisite, so the reload ran with no ruleset
present, failed to stat the file, and reported the state (and install)
as failed.
Add an onlyif guard so the reload only runs when all-rulesets.rules
exists. A .gitkeep-only sync now leaves the state a clean success
(onlyif condition false); once SOC writes the ruleset, the reload fires
normally.
Route the reload/verify output (ours plus so-common's retry/fail lines)
through a synchronous timestamping pipeline so every line in reload.log
is prefixed with a date/time, and preserve the real exit code via
PIPESTATUS.
Treating an in-progress reload as instant success could report success
while Suricata was still running a stale ruleset (the in-flight reload
may have started before the new all-rulesets.rules was written).
Make success conditional on Suricata actually having loaded the current
ruleset: capture the rules-file mtime up front, trigger a blocking
reload-rules, then query ruleset-reload-time and only succeed when
last_reload >= mtime. An in-progress reload now retries (waits for it to
clear so our own fresh reload runs) instead of short-circuiting, and a
ruleset that never catches up within the retry window fails via fail().
Also drop the redundant ruleset-reload-nonblocking call (the verified
blocking reload is authoritative and the async call was what left a
reload running) and log human-readable timestamps.
so-suricata-reload-rules failed the surirulereload state when a rule
reload was already running: suricatasc returns
{"message":"Reload already in progress","return":"NOK"}, which never
matched the expected output, so retry looped all 60 attempts (~3 min)
and called fail.
Wrap the suricatasc calls so an in-progress reload is treated as
success (the in-flight reload picks up the new rules) while genuine
container-not-ready conditions still retry and ultimately fail.
Rename the two custom push-detection beacons for clarity:
- pillar_db -> postgres_pillar_beacon
- rules_db -> rules_beacon
Salt resolves a beacon by its config-key name to a _beacons/ module of the
same filename and tags its events salt/beacon/<minion>/<name>/<tag>, so each
rename touches the module file, the beacon config key in
beacons_pushstate.conf.jinja, and the reactor tag patterns in
reactor_pushstate.conf together. Watermark filenames and log prefixes are
updated to match; reactor run() logic is unchanged.
Salt's stock inotify beacon leaks one kernel inotify instance every time
the minion rebuilds the beacon loader's __context__ (the orphaned
pyinotify.Notifier is never stopped), accumulating against
fs.inotify.max_user_instances=128 until inotify_init() fails with EMFILE
and rule-change push detection silently stops. This is independent of
disable_during_state_run.
Add a custom poll-based beacon (salt/_beacons/rules_db.py) modeled on
pillar_db.py: it fingerprints the suricata/strelka rule dirs each interval
(relpath + mtime_ns + size, temp files excluded) against a per-dir
watermark, emitting an event only on change. It holds zero inotify
instances, so the leak is impossible, and it keeps firing during state
runs. Swap the inotify beacon config and reactor tag mappings accordingly;
the push_suricata/push_strelka reactors are unchanged (they read only
data['path']).
The /nsm/kernelrepo bind mount exposed the files, but without a matching
location block external requests to /kernelrepo/ fell through to the SOC
app and returned HTML, so minions hit 'repomd.xml parser error'. Add a
/kernelrepo/ location mirroring /repo/.
highstate_interval_hours describes the per-minion highstate schedule, not the
active-push pipeline, so relocate it from salt.auto_apply to a new salt.schedule
settings subtree. Repoint so-salt-minion-check at the new pillar path (it had
been left on the stale global:push path) so its restart grace period tracks the
schedule again.
- Add salt.schedule.highstate_interval_hours to defaults.yaml/soc_salt.yaml and a
side-effect-free salt/salt/schedule.map.jinja (SCHEDULEMERGED), matching the
*MERGED map convention. Consumers read SCHEDULEMERGED.highstate_interval_hours.
- Split salt/schedule.sls into salt/salt/highstate_schedule.sls (every minion) and
salt/salt/push_drain_schedule.sls (managers); update top.sls to apply the
highstate schedule via '*' and the drainer schedule via the configured-manager
block. Remove the now-empty schedule.sls aggregator.
- pillar_push_map.yaml and so-push-drainer: comment/doc updates only.
During soup the grid is mid-salt-upgrade. Only assign the UEK8 kernel
repo once the node's grains.saltversion matches salt.minion.version from
minion.defaults.yaml, so the kernel repo and the update it enables don't
activate until the node is fully on the target salt.
Installing kernel-uek-core adds a UEK8 (6.x) boot entry but doesn't make
it the default, because grubby only auto-promotes within the running
kernel's flavor lineage and we cross from a 5.x kernel to the new UEK8
flavor. so-kernel-upgrade finds the newest installed 6.x UEK kernel and
grubby --set-default's it (idempotent, verifies the change, no reboot).
Josh Patterson
Josh Brower
Mike Reeves
Jorge Reyes
Jason Ertel
Jason Ertel
Mike Reeves