mirror of
https://github.com/Security-Onion-Solutions/securityonion.git
synced 2026-07-12 20:11:51 +02:00
so-salt-minion-wait: wait for the restart job before reading MainPID
Live testing on a standalone node found the previous commit still reported ready on the OUTGOING daemon. Reproduced by running the production sequence: systemctl restart --no-block salt-minion # what service.restart issues /usr/sbin/so-salt-minion-wait # what cmd.run then runs so-salt-minion-wait: gating on pid-tagged ready line ... plus master sockets salt-minion (pid 2750297) ready after 3s # 2750297 is the OLD child salt restarts this unit with --no-block -- _no_block_default() in salt/modules/systemd_service.py returns True when the unit is the salt-minion service -- so service.restart returns as soon as the job is enqueued. Measured on the host, systemd does not swap MainPID until ~7.3s later. Throughout that window the old daemon is still running, still holds its master sockets, and its own ready line is already in the log, so every gate passes on the instance that is about to be torn down. INITIAL_SLEEP=3 expired inside that window. Wait for systemd's job queue for the unit to drain before resolving MainPID. That is deterministic rather than a timing guess: the job exists from the moment --no-block returns until the new instance signals READY, and MainPID is new by the time it clears. Measured transition: t=0.0s job pending, child=OLD, sockets up, ready line present t=7.9s job drained, child=NEW, sockets up, ready line ABSENT t=10.7s ready line for NEW child appears <- script returns here The same run also confirms empirically why the log line is required in addition to the sockets: for 2.8s the new child has both master connections while _post_master_init() is still loading modules and compiling pillar, so a socket-only gate would return that much too early. Correct the comment claim from the previous commit. The --no-block restart is real; it lives in salt's systemd_service module, not in this repo, which is why searching the repo for it turned up nothing.
This commit is contained in:
@@ -138,7 +138,8 @@ salt_minion_service:
|
||||
# /usr/sbin/so-salt-minion-wait (deployed by salt_sbin from salt/tools/sbin/); it keys the ready
|
||||
# line to the current daemon pid (resolved via systemd, not the pidfile) and corroborates with the
|
||||
# master req/publish sockets. set_log_levels above enforces the log_level_logfile: info that the
|
||||
# ready line depends on.
|
||||
# ready line depends on. salt restarts this unit with --no-block, so mod_watch returns while the old
|
||||
# daemon is still up; the script waits for systemd's restart job to drain before it reads MainPID.
|
||||
wait_for_salt_minion_ready:
|
||||
cmd.run:
|
||||
- name: /usr/sbin/so-salt-minion-wait
|
||||
|
||||
@@ -26,6 +26,14 @@
|
||||
# logs the ready line, while systemd's MainPID is the parent. During a restart the pidfile can still
|
||||
# name the OLD child, whose own ready line is already in the log -- matching it would report ready
|
||||
# instantly. Children of the current MainPID structurally exclude the old instance.
|
||||
#
|
||||
# That is only true once systemd has actually swapped MainPID. Salt restarts this unit with
|
||||
# --no-block (salt/modules/systemd_service.py:_no_block_default returns True for salt-minion), so
|
||||
# service.restart returns as soon as the job is enqueued and the state proceeds to run this script
|
||||
# while the OLD daemon is still up -- observed at ~7s before MainPID flips. Reading MainPID in that
|
||||
# window names the outgoing instance, which is still fully connected and has its own ready line, so
|
||||
# every gate below would pass on the daemon that is about to die. Wait for systemd's job queue for
|
||||
# the unit to drain first; that is the deterministic "the swap has happened" signal.
|
||||
|
||||
. /usr/sbin/so-common
|
||||
|
||||
@@ -76,6 +84,13 @@ else
|
||||
echo "so-salt-minion-wait: INFO file logging unavailable (log_level_logfile='${LOG_LEVEL_LOGFILE:-unset}'); gating on master sockets only"
|
||||
fi
|
||||
|
||||
# True while systemd still has a queued or running job for the unit, i.e. an in-flight --no-block
|
||||
# restart. MainPID still names the outgoing daemon until this drains. The unit name is passed as a
|
||||
# filter and grepped as well, so this stays correct if an older systemctl ignores the filter.
|
||||
restart_pending() {
|
||||
systemctl list-jobs --no-legend salt-minion.service 2>/dev/null | grep -q 'salt-minion\.service'
|
||||
}
|
||||
|
||||
# Emit the pid(s) of the current daemon instance. systemd's MainPID is the parent keepalive process;
|
||||
# its child runs tune_in. Fall back to MainPID when there is no child (--disable-keepalive path).
|
||||
resolve_daemon_pids() {
|
||||
@@ -134,8 +149,16 @@ sleep "$INITIAL_SLEEP"
|
||||
|
||||
elapsed="$INITIAL_SLEEP"
|
||||
pids=""
|
||||
announced_pending=0
|
||||
while [ "$elapsed" -lt "$TIMEOUT" ]; do
|
||||
if pids=$(resolve_daemon_pids); then
|
||||
if restart_pending; then
|
||||
# An in-flight --no-block restart: MainPID still names the outgoing daemon. Evaluating now
|
||||
# would bless the instance that is about to be torn down.
|
||||
if [ "$announced_pending" -eq 0 ]; then
|
||||
echo "so-salt-minion-wait: systemd restart job in flight; waiting for it to drain"
|
||||
announced_pending=1
|
||||
fi
|
||||
elif pids=$(resolve_daemon_pids); then
|
||||
# shellcheck disable=SC2086
|
||||
for pid in $pids; do
|
||||
if instance_ready "$pid"; then
|
||||
@@ -149,5 +172,6 @@ while [ "$elapsed" -lt "$TIMEOUT" ]; do
|
||||
done
|
||||
|
||||
mainpid=$(systemctl show -p MainPID --value salt-minion 2>/dev/null)
|
||||
echo "salt-minion did not become ready within ${TIMEOUT}s (MainPID=${mainpid:-unknown}, candidate pids='${pids:-none}', log_gate=${USE_LOG_GATE}, socket_gate=${USE_SOCKET_GATE}, log_file=${LOG_FILE})" >&2
|
||||
restart_pending && pending=yes || pending=no
|
||||
echo "salt-minion did not become ready within ${TIMEOUT}s (MainPID=${mainpid:-unknown}, candidate pids='${pids:-none}', restart_job_pending=${pending}, log_gate=${USE_LOG_GATE}, socket_gate=${USE_SOCKET_GATE}, log_file=${LOG_FILE})" >&2
|
||||
exit 1
|
||||
|
||||
Reference in New Issue
Block a user