so-salt-minion-wait: wait for the restart job before reading MainPID

Live testing on a standalone node found the previous commit still reported
ready on the OUTGOING daemon. Reproduced by running the production sequence:

  systemctl restart --no-block salt-minion   # what service.restart issues
  /usr/sbin/so-salt-minion-wait              # what cmd.run then runs

  so-salt-minion-wait: gating on pid-tagged ready line ... plus master sockets
  salt-minion (pid 2750297) ready after 3s   # 2750297 is the OLD child

salt restarts this unit with --no-block -- _no_block_default() in
salt/modules/systemd_service.py returns True when the unit is the salt-minion
service -- so service.restart returns as soon as the job is enqueued. Measured
on the host, systemd does not swap MainPID until ~7.3s later. Throughout that
window the old daemon is still running, still holds its master sockets, and
its own ready line is already in the log, so every gate passes on the instance
that is about to be torn down. INITIAL_SLEEP=3 expired inside that window.

Wait for systemd's job queue for the unit to drain before resolving MainPID.
That is deterministic rather than a timing guess: the job exists from the
moment --no-block returns until the new instance signals READY, and MainPID is
new by the time it clears. Measured transition:

  t=0.0s   job pending, child=OLD, sockets up, ready line present
  t=7.9s   job drained, child=NEW, sockets up, ready line ABSENT
  t=10.7s  ready line for NEW child appears   <- script returns here

The same run also confirms empirically why the log line is required in
addition to the sockets: for 2.8s the new child has both master connections
while _post_master_init() is still loading modules and compiling pillar, so a
socket-only gate would return that much too early.

Correct the comment claim from the previous commit. The --no-block restart is
real; it lives in salt's systemd_service module, not in this repo, which is
why searching the repo for it turned up nothing.
This commit is contained in:
Josh Patterson
2026-07-10 09:35:21 -04:00
parent fbeac25ee9
commit 89e6a746c8
2 changed files with 28 additions and 3 deletions
+2 -1
View File
@@ -138,7 +138,8 @@ salt_minion_service:
# /usr/sbin/so-salt-minion-wait (deployed by salt_sbin from salt/tools/sbin/); it keys the ready
# line to the current daemon pid (resolved via systemd, not the pidfile) and corroborates with the
# master req/publish sockets. set_log_levels above enforces the log_level_logfile: info that the
# ready line depends on.
# ready line depends on. salt restarts this unit with --no-block, so mod_watch returns while the old
# daemon is still up; the script waits for systemd's restart job to drain before it reads MainPID.
wait_for_salt_minion_ready:
cmd.run:
- name: /usr/sbin/so-salt-minion-wait
+26 -2
View File
@@ -26,6 +26,14 @@
# logs the ready line, while systemd's MainPID is the parent. During a restart the pidfile can still
# name the OLD child, whose own ready line is already in the log -- matching it would report ready
# instantly. Children of the current MainPID structurally exclude the old instance.
#
# That is only true once systemd has actually swapped MainPID. Salt restarts this unit with
# --no-block (salt/modules/systemd_service.py:_no_block_default returns True for salt-minion), so
# service.restart returns as soon as the job is enqueued and the state proceeds to run this script
# while the OLD daemon is still up -- observed at ~7s before MainPID flips. Reading MainPID in that
# window names the outgoing instance, which is still fully connected and has its own ready line, so
# every gate below would pass on the daemon that is about to die. Wait for systemd's job queue for
# the unit to drain first; that is the deterministic "the swap has happened" signal.
. /usr/sbin/so-common
@@ -76,6 +84,13 @@ else
echo "so-salt-minion-wait: INFO file logging unavailable (log_level_logfile='${LOG_LEVEL_LOGFILE:-unset}'); gating on master sockets only"
fi
# True while systemd still has a queued or running job for the unit, i.e. an in-flight --no-block
# restart. MainPID still names the outgoing daemon until this drains. The unit name is passed as a
# filter and grepped as well, so this stays correct if an older systemctl ignores the filter.
restart_pending() {
systemctl list-jobs --no-legend salt-minion.service 2>/dev/null | grep -q 'salt-minion\.service'
}
# Emit the pid(s) of the current daemon instance. systemd's MainPID is the parent keepalive process;
# its child runs tune_in. Fall back to MainPID when there is no child (--disable-keepalive path).
resolve_daemon_pids() {
@@ -134,8 +149,16 @@ sleep "$INITIAL_SLEEP"
elapsed="$INITIAL_SLEEP"
pids=""
announced_pending=0
while [ "$elapsed" -lt "$TIMEOUT" ]; do
if pids=$(resolve_daemon_pids); then
if restart_pending; then
# An in-flight --no-block restart: MainPID still names the outgoing daemon. Evaluating now
# would bless the instance that is about to be torn down.
if [ "$announced_pending" -eq 0 ]; then
echo "so-salt-minion-wait: systemd restart job in flight; waiting for it to drain"
announced_pending=1
fi
elif pids=$(resolve_daemon_pids); then
# shellcheck disable=SC2086
for pid in $pids; do
if instance_ready "$pid"; then
@@ -149,5 +172,6 @@ while [ "$elapsed" -lt "$TIMEOUT" ]; do
done
mainpid=$(systemctl show -p MainPID --value salt-minion 2>/dev/null)
echo "salt-minion did not become ready within ${TIMEOUT}s (MainPID=${mainpid:-unknown}, candidate pids='${pids:-none}', log_gate=${USE_LOG_GATE}, socket_gate=${USE_SOCKET_GATE}, log_file=${LOG_FILE})" >&2
restart_pending && pending=yes || pending=no
echo "salt-minion did not become ready within ${TIMEOUT}s (MainPID=${mainpid:-unknown}, candidate pids='${pids:-none}', restart_job_pending=${pending}, log_gate=${USE_LOG_GATE}, socket_gate=${USE_SOCKET_GATE}, log_file=${LOG_FILE})" >&2
exit 1