From d3da6b393971455a8b91ebce3a52e34acf299563 Mon Sep 17 00:00:00 2001 From: Josh Patterson Date: Wed, 5 Aug 2026 14:06:21 -0400 Subject: [PATCH 1/6] soup: actively push a grid-wide highstate to remote minions after upgrade The per-minion highstate schedule moved from 15 minutes to 120 minutes (salt:schedule:highstate_interval_minutes), so after soup upgrades the manager, non-manager minions could otherwise sit on the old version for up to ~2.5 hours (interval + splay) before their scheduled highstate applies the new code. Add so-grid-highstate, a detached best-effort driver soup fires at the end of an upgrade. It uses the existing orch.push_batch runner to highstate the grid in role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node grids, and when Salt itself was upgraded first runs an untiered pass and waits for minions to reconnect on the new salt-minion before the tiered pass. soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so an SSH drop can't kill it) called on both the hotfix and full-upgrade paths, wires the previously-dead -b flag through as --batch (now accepting N or N%), and updates the distributed-deployment message to reflect the active push. --- salt/manager/tools/sbin/so-grid-highstate | 164 ++++++++++++++++++++++ salt/manager/tools/sbin/soup | 48 ++++++- 2 files changed, 207 insertions(+), 5 deletions(-) create mode 100644 salt/manager/tools/sbin/so-grid-highstate diff --git a/salt/manager/tools/sbin/so-grid-highstate b/salt/manager/tools/sbin/so-grid-highstate new file mode 100644 index 000000000..050ddf35f --- /dev/null +++ b/salt/manager/tools/sbin/so-grid-highstate @@ -0,0 +1,164 @@ +#!/bin/bash +# +# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one +# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at +# https://securityonion.net/license; you may not use this file except in compliance with the +# Elastic License 2.0. + +# so-grid-highstate +# ================= +# Drives a batched, role-tiered highstate across every non-manager minion in the +# grid. soup fires this (detached) after it finishes upgrading the manager so the +# rest of the grid converges immediately instead of waiting for its own scheduled +# highstate -- which, since the schedule moved from 15 minutes to 120 minutes +# (salt:schedule:highstate_interval_minutes), could otherwise leave nodes on the +# old version for up to ~2.5 hours (interval + splay) while the manager runs new code. +# +# Work is done by the existing orch.push_batch orchestration (salt/orch/push_batch.sls), +# the same runner the active-push drainer uses, so batching/queueing behavior matches. +# Tiers are dispatched in declaration order: searchnodes/heavynodes (Elasticsearch data +# nodes) first, then receivers, then everything else -- so the data tier converges before +# the ingest tier before sensors/fleet/idh/etc. +# +# When soup also upgraded Salt itself, remote minions must first highstate onto the new +# salt-minion package (top.sls gates every real state on G@saltversion, so a stale-version +# minion only gets salt.minion until it upgrades and reconnects). --salt-upgraded runs that +# preliminary pass and waits for the fleet to settle before the tiered pass. +# +# This is best-effort: soup has already completed by the time this runs, and the 120-minute +# scheduled highstate remains the backstop for any node that is offline or missed a batch. + +LOG_FILE=/opt/so/log/salt/so-grid-highstate.log +LOCK_FILE=/opt/so/state/so-grid-highstate.lock +SETTLE_MAX_WAIT=${GRID_HIGHSTATE_SETTLE_WAIT:-900} # backstop for the post-salt-upgrade settle loop +SETTLE_INTERVAL=15 +SETTLE_STABLE_CHECKS=3 + +BATCH="" +BATCH_WAIT="" +SALT_UPGRADED=false +REASON="manual" + +log() { + echo "$(date '+%Y-%m-%d %H:%M:%S') | $*" | tee -a "$LOG_FILE" +} + +usage() { + echo "Usage: so-grid-highstate [--batch ] [--batch-wait ] [--salt-upgraded] [--reason ]" + exit 1 +} + +while [ $# -gt 0 ]; do + case "$1" in + --batch) BATCH="$2"; shift 2 ;; + --batch-wait) BATCH_WAIT="$2"; shift 2 ;; + --salt-upgraded) SALT_UPGRADED=true; shift ;; + --reason) REASON="$2"; shift 2 ;; + -h|--help) usage ;; + *) echo "Unknown option: $1"; usage ;; + esac +done + +mkdir -p "$(dirname "$LOG_FILE")" "$(dirname "$LOCK_FILE")" + +# Serialize: a second invocation (e.g. two soups, or a manual run overlapping soup's) +# should not dispatch a competing set of batches. +exec 9>"$LOCK_FILE" +if ! flock -n 9; then + log "another so-grid-highstate is already running (lock $LOCK_FILE held); exiting" + exit 0 +fi + +# Resolve batch settings from the salt:auto_apply pillar when not overridden on the +# command line, falling back to the same defaults orch.push_batch/salt.defaults use. +if [ -z "$BATCH" ]; then + BATCH=$(salt-call --out=newline_values_only pillar.get salt:auto_apply:batch 2>/dev/null) + [ -z "$BATCH" ] && BATCH='25%' +fi +if [ -z "$BATCH_WAIT" ]; then + BATCH_WAIT=$(salt-call --out=newline_values_only pillar.get salt:auto_apply:batch_wait 2>/dev/null) + [ -z "$BATCH_WAIT" ] && BATCH_WAIT=15 +fi + +MINIONID=$(salt-call --local --out=newline_values_only grains.get id 2>/dev/null) +[ -z "$MINIONID" ] && MINIONID=$(cat /etc/salt/minion_id 2>/dev/null) +if [ -z "$MINIONID" ]; then + log "could not determine this minion's id; aborting" + exit 1 +fi + +# Single-node grids (eval/standalone/import with no other accepted keys) have nothing +# remote to push -- the manager already highstated during soup. +NUM_ACCEPTED=$(salt-key --out=json --list=accepted 2>/dev/null | jq -r '.minions | length' 2>/dev/null) +NUM_ACCEPTED=${NUM_ACCEPTED:-0} +if [ "$NUM_ACCEPTED" -le 1 ]; then + log "single node grid ($NUM_ACCEPTED accepted minion(s)); nothing to push (reason=$REASON)" + exit 0 +fi + +log "starting grid highstate: reason=$REASON minion=$MINIONID accepted=$NUM_ACCEPTED batch=$BATCH batch_wait=$BATCH_WAIT salt_upgraded=$SALT_UPGRADED" + +# Dispatch a single synchronous orch.push_batch run for the given actions JSON. +# Synchronous is fine: soup launched us detached, so blocking here does not hold soup up. +dispatch() { + local desc="$1" + local actions="$2" + log "dispatching $desc" + if salt-run state.orchestrate orch.push_batch pillar="{\"actions\": $actions}" >>"$LOG_FILE" 2>&1; then + log "$desc dispatch completed (rc=0)" + else + log "WARNING: $desc dispatch returned rc=$?; nodes it missed will converge on the scheduled highstate" + fi +} + +# Wait for the responsive minion set to stop growing. Used after the salt-upgrade pass, +# where minions restart salt-minion (~30s delayed, see salt/salt/minion/init.sls) and drop +# off the bus before reconnecting on the new version. Mirrors so-boot-mine-update's settle +# loop: settle on the reachable set rather than requiring up==accepted, since an operator may +# have intentionally powered a node off. Bounded by SETTLE_MAX_WAIT so a down node never stalls us. +wait_for_settle() { + local elapsed=0 prev=-1 stable=0 up=0 + while [ "$elapsed" -lt "$SETTLE_MAX_WAIT" ]; do + up=$(salt-run manage.up --out=json 2>/dev/null \ + | python3 -c 'import sys,json; print(len(json.load(sys.stdin)))' 2>/dev/null) + up=${up:-0} + if [ "$up" -gt 0 ] && [ "$up" -eq "$prev" ]; then + stable=$((stable + 1)) + [ "$stable" -ge "$SETTLE_STABLE_CHECKS" ] && break + else + stable=0 + fi + prev=$up + sleep "$SETTLE_INTERVAL" + elapsed=$((elapsed + SETTLE_INTERVAL)) + done + log "fleet settled at ${up} minions up after ${elapsed}s" + [ "$elapsed" -ge "$SETTLE_MAX_WAIT" ] && log "WARNING: ${SETTLE_MAX_WAIT}s settle backstop hit; proceeding with whoever is up" +} + +# Pass 0: when Salt itself was upgraded, remote minions still on the old version only match +# top.sls's 'not G@saltversion' block (salt.minion, which performs the package upgrade). Push +# an untiered highstate so they upgrade+reconnect, then wait for them to come back before the +# real tiered pass applies the new version's states. +if [ "$SALT_UPGRADED" = "true" ]; then + dispatch "salt-upgrade pass (all remote minions)" \ + "[{\"highstate\": true, \"tgt\": \"not $MINIONID\", \"tgt_type\": \"compound\", \"batch\": \"$BATCH\", \"batch_wait\": $BATCH_WAIT}]" + log "waiting for minions to reconnect on the new salt version" + wait_for_settle +fi + +# Tiered pass: Elasticsearch data nodes first, then receivers, then the remainder. The last +# tier is defined as the complement of the earlier tiers (and of this manager) so coverage is +# exhaustive -- sensors, fleet, idh, desktop, hypervisor, and any future role are all included. +TIERED_ACTIONS=$(cat < receivers -> remainder)" "$TIERED_ACTIONS" + +log "grid highstate complete (reason=$REASON)" +exit 0 diff --git a/salt/manager/tools/sbin/soup b/salt/manager/tools/sbin/soup index 30d26124d..23da8b87b 100755 --- a/salt/manager/tools/sbin/soup +++ b/salt/manager/tools/sbin/soup @@ -24,7 +24,10 @@ else POSTVERSION=$INSTALLEDVERSION fi INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk '{print $2}') -BATCHSIZE=5 +# Optional -b override for the grid highstate batch size (a count like "5" or a +# percentage like "25%"). Empty means so-grid-highstate uses the salt:auto_apply:batch +# pillar default. +BATCHSIZE= SOUP_LOG=/root/soup.log SOUP_DEBUG_LOG=/root/soup-debug.log WHATWOULDYOUSAYYAHDOHERE=soup @@ -452,6 +455,31 @@ highstate() { salt-call state.highstate -l info queue=True } +push_grid_highstate() { + # Drive a batched, role-tiered highstate across the rest of the grid so remote minions + # pick up this upgrade now instead of waiting up to ~2.5 hours for their own scheduled + # highstate (the schedule moved from 15 to 120 minutes). so-grid-highstate does the work + # via orch.push_batch; it only exists once the manager highstate has deployed this + # version's sbin files, so guard on it. Launch fully detached (setsid) so it survives an + # SSH drop, and never let it affect soup's exit status -- it is best-effort with the + # scheduled highstate as backstop. + if [[ ! -x /usr/sbin/so-grid-highstate ]]; then + echo "so-grid-highstate not present; remote nodes will converge on their scheduled highstate." + return 0 + fi + + local extra_args=() + if [[ $SALTUPGRADED == true || $UPGRADESALT -eq 1 ]]; then + extra_args+=(--salt-upgraded) + fi + if [[ -n "$BATCHSIZE" ]]; then + extra_args+=(--batch "$BATCHSIZE") + fi + + echo "Dispatching a grid-wide highstate to remote nodes. Progress: /opt/so/log/salt/so-grid-highstate.log" + setsid nohup /usr/sbin/so-grid-highstate --reason soup "${extra_args[@]}" >/dev/null 2>&1 & +} + masterlock() { echo "Locking Salt Master" mv -v $TOPFILE $BACKUPTOPFILE @@ -1936,6 +1964,9 @@ main() { # rather than reporting "already latest". The soversion/pillar writes in # update_version are no-ops here since the version is unchanged for a hotfix. update_version + # Push the hotfix out to the rest of the grid rather than waiting for the scheduled + # highstate. Hotfixes never upgrade Salt, so no --salt-upgraded pass is needed. + push_grid_highstate else SOUP_UPGRADE_STARTED=true echo "" @@ -2102,13 +2133,18 @@ main() { if [[ $NUM_MINIONS -gt 1 ]]; then + # Actively drive the rest of the grid to this version now. The scheduled highstate + # runs only every 120 minutes (salt:schedule:highstate_interval_minutes), so without + # this remote nodes could sit on the old version for a couple of hours after soup finishes. + push_grid_highstate + cat << EOF -This appears to be a distributed deployment. Other nodes should update themselves at the next Salt highstate (typically within 15 minutes). Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch. +This appears to be a distributed deployment. soup has dispatched a batched, grid-wide highstate to update the other nodes now: Elasticsearch data nodes (search/heavy nodes) first, then receivers, then sensors and the remaining nodes. Progress is logged to /opt/so/log/salt/so-grid-highstate.log, and you can watch nodes update from the Grid section of SOC. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch. -Each minion is on a random 15 minute check-in period and things like network bandwidth can be a factor in how long the actual upgrade takes. If you have a heavy node on a slow link, it is going to take a while to get the containers to it. Depending on what changes happened between the versions, Elasticsearch might not be able to talk to said heavy node until the update is complete. +Nodes are updated in batches, and things like network bandwidth can be a factor in how long the actual upgrade takes. If you have a heavy node on a slow link, it is going to take a while to get the containers to it. Depending on what changes happened between the versions, Elasticsearch might not be able to talk to said heavy node until the update is complete. Any node that is offline or missed a batch will converge on its own scheduled highstate (every 120 minutes by default). If it looks like you’re missing data after the upgrade, please avoid restarting services and instead make sure at least one search node has completed its upgrade. The best way to do this is to run 'sudo salt-call state.highstate' from a search node and make sure there are no errors. Typically if it works on one node it will work on the rest. Sensor nodes are less complex and will update as they check in so you can monitor those from the Grid section of SOC. @@ -2148,8 +2184,10 @@ while getopts ":b:f:y" opt; do case ${opt} in b ) BATCHSIZE="$OPTARG" - if ! [[ "$BATCHSIZE" =~ ^[1-9][0-9]*$ ]]; then - echo "Batch size must be a number greater than 0." + # Accept either a plain count (e.g. 5) or a percentage (e.g. 25%); passed through + # to so-grid-highstate --batch, which salt's batch/batch_wait accepts in both forms. + if ! [[ "$BATCHSIZE" =~ ^[1-9][0-9]*%?$ ]]; then + echo "Batch size must be a number greater than 0, optionally with a trailing % (e.g. 5 or 25%)." exit 1 fi ;; From 36833fdad1bb4873cc32e6b8fc9b3737dcd7c48a Mon Sep 17 00:00:00 2001 From: Josh Patterson Date: Wed, 5 Aug 2026 15:05:24 -0400 Subject: [PATCH 2/6] so-grid-highstate: emit single-line pillar JSON and skip empty tiers Two defects surfaced testing BRANCH=asasoup soup on a manager+heavynode grid: 1. The tiered-pass actions JSON was built from a multi-line heredoc. salt parses 'pillar=' kwargs with a non-DOTALL regex, so the embedded newlines made salt-run treat the whole token as a positional saltenv -- 'No matching salt environment for environment pillar={...}' -- and the highstate never ran, leaving the heavynode on the old version. Emit the actions JSON on a single line (matching how so-push-drainer's json.dumps payload already works). 2. orch.push_batch's salt.state step reports 'No minions returned' (a failure) for a tier whose compound target matches nothing, so any grid lacking a role (no receiver, small grids) always logged a warning and returned rc=1 even when every present node converged. Pre-check each tier with 'salt -C --preview-target' and include only tiers that match >=1 minion; exit cleanly if none match. Verified live: heavynode highstated 3.2.0 -> 3.3.0 (427 states, 0 failed) and a follow-up run skips the empty receiver/remainder tiers with rc=0. --- salt/manager/tools/sbin/so-grid-highstate | 41 ++++++++++++++++++----- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/salt/manager/tools/sbin/so-grid-highstate b/salt/manager/tools/sbin/so-grid-highstate index 050ddf35f..ef65039ba 100644 --- a/salt/manager/tools/sbin/so-grid-highstate +++ b/salt/manager/tools/sbin/so-grid-highstate @@ -150,15 +150,40 @@ fi # Tiered pass: Elasticsearch data nodes first, then receivers, then the remainder. The last # tier is defined as the complement of the earlier tiers (and of this manager) so coverage is # exhaustive -- sensors, fleet, idh, desktop, hypervisor, and any future role are all included. -TIERED_ACTIONS=$(cat < receivers -> remainder)" "$TIERED_ACTIONS" + +# Count minions a compound target matches, using the master's key/cache data (no execution). +tier_count() { + salt --out=json -C "$1" --preview-target 2>/dev/null | jq 'length' 2>/dev/null +} + +# Build the actions JSON, including only tiers that actually match minions. An empty target +# would make orch.push_batch's salt.state step return "No minions returned" -- a failure -- +# even though nothing needed to run, and grids commonly lack a tier (no receiver, etc.). +# Keep the JSON on a single line: salt parses `pillar=` kwargs with a non-DOTALL +# regex, so an embedded newline makes it treat the whole token as a positional saltenv +# instead ("No matching salt environment for environment 'pillar=...'"). +actions="" +for tgt in "${TIER_TGTS[@]}"; do + n=$(tier_count "$tgt"); n=${n:-0} + if [ "$n" -ge 1 ]; then + [ -n "$actions" ] && actions="$actions, " + actions="$actions{\"highstate\": true, \"tgt\": \"$tgt\", \"tgt_type\": \"compound\", \"batch\": \"$BATCH\", \"batch_wait\": $BATCH_WAIT}" + log "tier matched $n minion(s): $tgt" + else + log "tier matched 0 minions, skipping: $tgt" + fi +done + +if [ -z "$actions" ]; then + log "no remote minions matched any tier; nothing to push (reason=$REASON)" + exit 0 +fi +dispatch "tiered pass (searchnodes/heavynodes -> receivers -> remainder)" "[$actions]" log "grid highstate complete (reason=$REASON)" exit 0 From 6abf382ea87d7317dd3dc321c5ea5197d7fd45f7 Mon Sep 17 00:00:00 2001 From: Josh Patterson Date: Thu, 6 Aug 2026 09:46:12 -0400 Subject: [PATCH 3/6] so-grid-highstate: wait for fleet recovery and quiet the expected salt-upgrade warning Two refinements to the --salt-upgraded path, surfaced testing a salt downgrade+soup on a manager+heavynode grid: - The post-upgrade settle loop settled on any stable reachable count >0, so when a target was briefly down for its salt-minion restart it could settle on the not-yet-restarted subset (observed: 'settled at 1' with 2 accepted) and release the tiered pass before nodes reconnected. Capture the reachable count just before the pass and wait for it to recover to that count (up >= pre-upgrade target) and hold steady, with an initial grace so the delayed restart dip is observed rather than skipped. Still compares against the pre-upgrade reachable set, not accepted keys, so an intentionally powered-off node never stalls past the backstop. - The salt-upgrade pass returns non-zero by design (targets restart salt-minion mid-run), but it logged the generic 'nodes it missed will converge on the scheduled highstate' warning, which reads like a real failure. Mark that dispatch as expect_restart so it logs a benign, explanatory line instead. Verified live: with the heavynode's salt-minion bounced during the settle window, the loop logged 'fleet recovered to 2 minions up (>= pre-upgrade 2)' and only then ran the tiered pass (heavynode highstate 427 succeeded, 0 failed). --- salt/manager/tools/sbin/so-grid-highstate | 60 +++++++++++++++++------ 1 file changed, 45 insertions(+), 15 deletions(-) diff --git a/salt/manager/tools/sbin/so-grid-highstate b/salt/manager/tools/sbin/so-grid-highstate index ef65039ba..6abe43279 100644 --- a/salt/manager/tools/sbin/so-grid-highstate +++ b/salt/manager/tools/sbin/so-grid-highstate @@ -33,6 +33,10 @@ LOCK_FILE=/opt/so/state/so-grid-highstate.lock SETTLE_MAX_WAIT=${GRID_HIGHSTATE_SETTLE_WAIT:-900} # backstop for the post-salt-upgrade settle loop SETTLE_INTERVAL=15 SETTLE_STABLE_CHECKS=3 +# salt-minion on an upgraded node restarts ~30s after the upgrade state runs +# (salt/salt/minion/init.sls start_minion_post_upgrade); wait past that before sampling +# so the settle loop sees the drop-off instead of settling on the pre-restart set. +SETTLE_INITIAL_WAIT=${GRID_HIGHSTATE_SETTLE_INITIAL_WAIT:-45} BATCH="" BATCH_WAIT="" @@ -98,31 +102,51 @@ fi log "starting grid highstate: reason=$REASON minion=$MINIONID accepted=$NUM_ACCEPTED batch=$BATCH batch_wait=$BATCH_WAIT salt_upgraded=$SALT_UPGRADED" +# Count minions currently responsive on the bus (includes this manager). +count_up() { + salt-run manage.up --out=json 2>/dev/null \ + | python3 -c 'import sys,json; print(len(json.load(sys.stdin)))' 2>/dev/null +} + # Dispatch a single synchronous orch.push_batch run for the given actions JSON. # Synchronous is fine: soup launched us detached, so blocking here does not hold soup up. +# expect_restart=true marks a dispatch (the salt-upgrade pass) where a non-zero rc is normal +# because targets restart salt-minion mid-run -- so we don't log a misleading failure warning. dispatch() { local desc="$1" local actions="$2" + local expect_restart="${3:-false}" + local rc log "dispatching $desc" - if salt-run state.orchestrate orch.push_batch pillar="{\"actions\": $actions}" >>"$LOG_FILE" 2>&1; then + salt-run state.orchestrate orch.push_batch pillar="{\"actions\": $actions}" >>"$LOG_FILE" 2>&1 + rc=$? + if [ "$rc" -eq 0 ]; then log "$desc dispatch completed (rc=0)" + elif [ "$expect_restart" = "true" ]; then + log "$desc returned rc=$rc; this is expected during a salt upgrade (targets restart salt-minion mid-run). Waiting for them to reconnect before the tiered pass." else - log "WARNING: $desc dispatch returned rc=$?; nodes it missed will converge on the scheduled highstate" + log "WARNING: $desc dispatch returned rc=$rc; nodes it missed will converge on the scheduled highstate" fi } -# Wait for the responsive minion set to stop growing. Used after the salt-upgrade pass, -# where minions restart salt-minion (~30s delayed, see salt/salt/minion/init.sls) and drop -# off the bus before reconnecting on the new version. Mirrors so-boot-mine-update's settle -# loop: settle on the reachable set rather than requiring up==accepted, since an operator may -# have intentionally powered a node off. Bounded by SETTLE_MAX_WAIT so a down node never stalls us. +# Wait for the reachable minion set to recover to its pre-upgrade size and hold steady. +# Used after the salt-upgrade pass, where targets restart salt-minion (~30s delayed, see +# salt/salt/minion/init.sls) and drop off the bus before reconnecting on the new version. +# target = how many minions were reachable just before the pass; requiring up >= target keeps +# us from releasing the tiered pass while nodes are still down for their restart (settling on +# the not-yet-restarted subset). We deliberately compare against the pre-upgrade reachable +# count, not accepted keys, so a node an operator intentionally powered off never stalls us. +# Bounded by SETTLE_MAX_WAIT. wait_for_settle() { + local target="$1" local elapsed=0 prev=-1 stable=0 up=0 + # Let the delayed salt-minion restart begin before we start counting stability, otherwise + # we could see the pre-restart set as "stable" and settle before the drop-off even happens. + sleep "$SETTLE_INITIAL_WAIT" + elapsed=$SETTLE_INITIAL_WAIT while [ "$elapsed" -lt "$SETTLE_MAX_WAIT" ]; do - up=$(salt-run manage.up --out=json 2>/dev/null \ - | python3 -c 'import sys,json; print(len(json.load(sys.stdin)))' 2>/dev/null) - up=${up:-0} - if [ "$up" -gt 0 ] && [ "$up" -eq "$prev" ]; then + up=$(count_up); up=${up:-0} + if [ "$up" -ge "$target" ] && [ "$up" -eq "$prev" ]; then stable=$((stable + 1)) [ "$stable" -ge "$SETTLE_STABLE_CHECKS" ] && break else @@ -132,8 +156,11 @@ wait_for_settle() { sleep "$SETTLE_INTERVAL" elapsed=$((elapsed + SETTLE_INTERVAL)) done - log "fleet settled at ${up} minions up after ${elapsed}s" - [ "$elapsed" -ge "$SETTLE_MAX_WAIT" ] && log "WARNING: ${SETTLE_MAX_WAIT}s settle backstop hit; proceeding with whoever is up" + if [ "$up" -ge "$target" ]; then + log "fleet recovered to ${up} minions up (>= pre-upgrade ${target}) after ${elapsed}s" + else + log "WARNING: ${SETTLE_MAX_WAIT}s settle backstop hit; only ${up}/${target} pre-upgrade minions back up; proceeding (stragglers converge on the scheduled highstate)" + fi } # Pass 0: when Salt itself was upgraded, remote minions still on the old version only match @@ -141,10 +168,13 @@ wait_for_settle() { # an untiered highstate so they upgrade+reconnect, then wait for them to come back before the # real tiered pass applies the new version's states. if [ "$SALT_UPGRADED" = "true" ]; then + PRE_UP=$(count_up); PRE_UP=${PRE_UP:-1} + log "pre-upgrade reachable minions (incl. this manager): $PRE_UP" dispatch "salt-upgrade pass (all remote minions)" \ - "[{\"highstate\": true, \"tgt\": \"not $MINIONID\", \"tgt_type\": \"compound\", \"batch\": \"$BATCH\", \"batch_wait\": $BATCH_WAIT}]" + "[{\"highstate\": true, \"tgt\": \"not $MINIONID\", \"tgt_type\": \"compound\", \"batch\": \"$BATCH\", \"batch_wait\": $BATCH_WAIT}]" \ + true log "waiting for minions to reconnect on the new salt version" - wait_for_settle + wait_for_settle "$PRE_UP" fi # Tiered pass: Elasticsearch data nodes first, then receivers, then the remainder. The last From 2d0ea48c39a323b1018528fce52d46401dc410d8 Mon Sep 17 00:00:00 2001 From: Josh Patterson Date: Mon, 10 Aug 2026 08:27:40 -0400 Subject: [PATCH 4/6] so-grid-highstate: drop the .log extension from the log path Rename the grid-highstate log from /opt/so/log/salt/so-grid-highstate.log to /opt/so/log/salt/so-grid-highstate. Updates the LOG_FILE var in so-grid-highstate and the two /opt/so/log/salt/so-grid-highstate.log references in soup (the progress echo and the distributed-deployment message). --- salt/manager/tools/sbin/so-grid-highstate | 2 +- salt/manager/tools/sbin/soup | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/salt/manager/tools/sbin/so-grid-highstate b/salt/manager/tools/sbin/so-grid-highstate index 6abe43279..e0153b2ee 100644 --- a/salt/manager/tools/sbin/so-grid-highstate +++ b/salt/manager/tools/sbin/so-grid-highstate @@ -28,7 +28,7 @@ # This is best-effort: soup has already completed by the time this runs, and the 120-minute # scheduled highstate remains the backstop for any node that is offline or missed a batch. -LOG_FILE=/opt/so/log/salt/so-grid-highstate.log +LOG_FILE=/opt/so/log/salt/so-grid-highstate LOCK_FILE=/opt/so/state/so-grid-highstate.lock SETTLE_MAX_WAIT=${GRID_HIGHSTATE_SETTLE_WAIT:-900} # backstop for the post-salt-upgrade settle loop SETTLE_INTERVAL=15 diff --git a/salt/manager/tools/sbin/soup b/salt/manager/tools/sbin/soup index bc7ac559c..3644b9d83 100755 --- a/salt/manager/tools/sbin/soup +++ b/salt/manager/tools/sbin/soup @@ -476,7 +476,7 @@ push_grid_highstate() { extra_args+=(--batch "$BATCHSIZE") fi - echo "Dispatching a grid-wide highstate to remote nodes. Progress: /opt/so/log/salt/so-grid-highstate.log" + echo "Dispatching a grid-wide highstate to remote nodes. Progress: /opt/so/log/salt/so-grid-highstate" setsid nohup /usr/sbin/so-grid-highstate --reason soup "${extra_args[@]}" >/dev/null 2>&1 & } @@ -2154,7 +2154,7 @@ main() { -This appears to be a distributed deployment. soup has dispatched a batched, grid-wide highstate to update the other nodes now: Elasticsearch data nodes (search/heavy nodes) first, then receivers, then sensors and the remaining nodes. Progress is logged to /opt/so/log/salt/so-grid-highstate.log, and you can watch nodes update from the Grid section of SOC. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch. +This appears to be a distributed deployment. soup has dispatched a batched, grid-wide highstate to update the other nodes now: Elasticsearch data nodes (search/heavy nodes) first, then receivers, then sensors and the remaining nodes. Progress is logged to /opt/so/log/salt/so-grid-highstate, and you can watch nodes update from the Grid section of SOC. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch. Nodes are updated in batches, and things like network bandwidth can be a factor in how long the actual upgrade takes. If you have a heavy node on a slow link, it is going to take a while to get the containers to it. Depending on what changes happened between the versions, Elasticsearch might not be able to talk to said heavy node until the update is complete. Any node that is offline or missed a batch will converge on its own scheduled highstate (every 120 minutes by default). From ee1d2167e8bedc507232438c3106322b0e39340d Mon Sep 17 00:00:00 2001 From: Josh Patterson Date: Mon, 10 Aug 2026 08:30:40 -0400 Subject: [PATCH 5/6] logrotate: rotate /opt/so/log/salt/so-grid-highstate Add a logrotate entry for the grid-highstate driver's log, matching the existing /opt/so/log/salt/{minion,master,so-salt-minion-check} entries (daily, rotate 14, copytruncate, compress). Registered in both logrotate/defaults.yaml and the SOC config schema logrotate/soc_logrotate.yaml. --- salt/logrotate/defaults.yaml | 10 ++++++++++ salt/logrotate/soc_logrotate.yaml | 7 +++++++ 2 files changed, 17 insertions(+) diff --git a/salt/logrotate/defaults.yaml b/salt/logrotate/defaults.yaml index e37193d0d..d48665f9c 100644 --- a/salt/logrotate/defaults.yaml +++ b/salt/logrotate/defaults.yaml @@ -220,6 +220,16 @@ logrotate: - extension .log - dateext - dateyesterday + /opt/so/log/salt/so-grid-highstate: + - daily + - rotate 14 + - missingok + - copytruncate + - compress + - create + - extension .log + - dateext + - dateyesterday /nsm/idh/*_x_log: - daily - rotate 14 diff --git a/salt/logrotate/soc_logrotate.yaml b/salt/logrotate/soc_logrotate.yaml index f329ed623..089772c0f 100644 --- a/salt/logrotate/soc_logrotate.yaml +++ b/salt/logrotate/soc_logrotate.yaml @@ -140,6 +140,13 @@ logrotate: multiline: True global: True forcedType: "[]string" + "/opt/so/log/salt/so-grid-highstate": + description: List of logrotate options for this file. + title: /opt/so/log/salt/so-grid-highstate + advanced: True + multiline: True + global: True + forcedType: "[]string" "/nsm/idh/*_x_log": description: List of logrotate options for this file. title: /nsm/idh/*.log From 7400e3dffa0d65db4a6010158b2601ed8dc4cbe7 Mon Sep 17 00:00:00 2001 From: Josh Patterson Date: Tue, 11 Aug 2026 10:12:40 -0400 Subject: [PATCH 6/6] Rename so-grid-highstate to so-soup-grid-highstate --- salt/logrotate/defaults.yaml | 2 +- salt/logrotate/soc_logrotate.yaml | 4 ++-- ...{so-grid-highstate => so-soup-grid-highstate} | 12 ++++++------ salt/manager/tools/sbin/soup | 16 ++++++++-------- 4 files changed, 17 insertions(+), 17 deletions(-) rename salt/manager/tools/sbin/{so-grid-highstate => so-soup-grid-highstate} (96%) diff --git a/salt/logrotate/defaults.yaml b/salt/logrotate/defaults.yaml index d48665f9c..90d01cac9 100644 --- a/salt/logrotate/defaults.yaml +++ b/salt/logrotate/defaults.yaml @@ -220,7 +220,7 @@ logrotate: - extension .log - dateext - dateyesterday - /opt/so/log/salt/so-grid-highstate: + /opt/so/log/salt/so-soup-grid-highstate: - daily - rotate 14 - missingok diff --git a/salt/logrotate/soc_logrotate.yaml b/salt/logrotate/soc_logrotate.yaml index 089772c0f..42b1f9694 100644 --- a/salt/logrotate/soc_logrotate.yaml +++ b/salt/logrotate/soc_logrotate.yaml @@ -140,9 +140,9 @@ logrotate: multiline: True global: True forcedType: "[]string" - "/opt/so/log/salt/so-grid-highstate": + "/opt/so/log/salt/so-soup-grid-highstate": description: List of logrotate options for this file. - title: /opt/so/log/salt/so-grid-highstate + title: /opt/so/log/salt/so-soup-grid-highstate advanced: True multiline: True global: True diff --git a/salt/manager/tools/sbin/so-grid-highstate b/salt/manager/tools/sbin/so-soup-grid-highstate similarity index 96% rename from salt/manager/tools/sbin/so-grid-highstate rename to salt/manager/tools/sbin/so-soup-grid-highstate index e0153b2ee..b4f82f82b 100644 --- a/salt/manager/tools/sbin/so-grid-highstate +++ b/salt/manager/tools/sbin/so-soup-grid-highstate @@ -5,8 +5,8 @@ # https://securityonion.net/license; you may not use this file except in compliance with the # Elastic License 2.0. -# so-grid-highstate -# ================= +# so-soup-grid-highstate +# ====================== # Drives a batched, role-tiered highstate across every non-manager minion in the # grid. soup fires this (detached) after it finishes upgrading the manager so the # rest of the grid converges immediately instead of waiting for its own scheduled @@ -28,8 +28,8 @@ # This is best-effort: soup has already completed by the time this runs, and the 120-minute # scheduled highstate remains the backstop for any node that is offline or missed a batch. -LOG_FILE=/opt/so/log/salt/so-grid-highstate -LOCK_FILE=/opt/so/state/so-grid-highstate.lock +LOG_FILE=/opt/so/log/salt/so-soup-grid-highstate +LOCK_FILE=/opt/so/state/so-soup-grid-highstate.lock SETTLE_MAX_WAIT=${GRID_HIGHSTATE_SETTLE_WAIT:-900} # backstop for the post-salt-upgrade settle loop SETTLE_INTERVAL=15 SETTLE_STABLE_CHECKS=3 @@ -48,7 +48,7 @@ log() { } usage() { - echo "Usage: so-grid-highstate [--batch ] [--batch-wait ] [--salt-upgraded] [--reason ]" + echo "Usage: so-soup-grid-highstate [--batch ] [--batch-wait ] [--salt-upgraded] [--reason ]" exit 1 } @@ -69,7 +69,7 @@ mkdir -p "$(dirname "$LOG_FILE")" "$(dirname "$LOCK_FILE")" # should not dispatch a competing set of batches. exec 9>"$LOCK_FILE" if ! flock -n 9; then - log "another so-grid-highstate is already running (lock $LOCK_FILE held); exiting" + log "another so-soup-grid-highstate is already running (lock $LOCK_FILE held); exiting" exit 0 fi diff --git a/salt/manager/tools/sbin/soup b/salt/manager/tools/sbin/soup index 3644b9d83..7fe03da85 100755 --- a/salt/manager/tools/sbin/soup +++ b/salt/manager/tools/sbin/soup @@ -25,7 +25,7 @@ else fi INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk '{print $2}') # Optional -b override for the grid highstate batch size (a count like "5" or a -# percentage like "25%"). Empty means so-grid-highstate uses the salt:auto_apply:batch +# percentage like "25%"). Empty means so-soup-grid-highstate uses the salt:auto_apply:batch # pillar default. BATCHSIZE= SOUP_LOG=/root/soup.log @@ -458,13 +458,13 @@ highstate() { push_grid_highstate() { # Drive a batched, role-tiered highstate across the rest of the grid so remote minions # pick up this upgrade now instead of waiting up to ~2.5 hours for their own scheduled - # highstate (the schedule moved from 15 to 120 minutes). so-grid-highstate does the work + # highstate (the schedule moved from 15 to 120 minutes). so-soup-grid-highstate does the work # via orch.push_batch; it only exists once the manager highstate has deployed this # version's sbin files, so guard on it. Launch fully detached (setsid) so it survives an # SSH drop, and never let it affect soup's exit status -- it is best-effort with the # scheduled highstate as backstop. - if [[ ! -x /usr/sbin/so-grid-highstate ]]; then - echo "so-grid-highstate not present; remote nodes will converge on their scheduled highstate." + if [[ ! -x /usr/sbin/so-soup-grid-highstate ]]; then + echo "so-soup-grid-highstate not present; remote nodes will converge on their scheduled highstate." return 0 fi @@ -476,8 +476,8 @@ push_grid_highstate() { extra_args+=(--batch "$BATCHSIZE") fi - echo "Dispatching a grid-wide highstate to remote nodes. Progress: /opt/so/log/salt/so-grid-highstate" - setsid nohup /usr/sbin/so-grid-highstate --reason soup "${extra_args[@]}" >/dev/null 2>&1 & + echo "Dispatching a grid-wide highstate to remote nodes. Progress: /opt/so/log/salt/so-soup-grid-highstate" + setsid nohup /usr/sbin/so-soup-grid-highstate --reason soup "${extra_args[@]}" >/dev/null 2>&1 & } masterlock() { @@ -2154,7 +2154,7 @@ main() { -This appears to be a distributed deployment. soup has dispatched a batched, grid-wide highstate to update the other nodes now: Elasticsearch data nodes (search/heavy nodes) first, then receivers, then sensors and the remaining nodes. Progress is logged to /opt/so/log/salt/so-grid-highstate, and you can watch nodes update from the Grid section of SOC. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch. +This appears to be a distributed deployment. soup has dispatched a batched, grid-wide highstate to update the other nodes now: Elasticsearch data nodes (search/heavy nodes) first, then receivers, then sensors and the remaining nodes. Progress is logged to /opt/so/log/salt/so-soup-grid-highstate, and you can watch nodes update from the Grid section of SOC. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch. Nodes are updated in batches, and things like network bandwidth can be a factor in how long the actual upgrade takes. If you have a heavy node on a slow link, it is going to take a while to get the containers to it. Depending on what changes happened between the versions, Elasticsearch might not be able to talk to said heavy node until the update is complete. Any node that is offline or missed a batch will converge on its own scheduled highstate (every 120 minutes by default). @@ -2197,7 +2197,7 @@ while getopts ":b:f:y" opt; do b ) BATCHSIZE="$OPTARG" # Accept either a plain count (e.g. 5) or a percentage (e.g. 25%); passed through - # to so-grid-highstate --batch, which salt's batch/batch_wait accepts in both forms. + # to so-soup-grid-highstate --batch, which salt's batch/batch_wait accepts in both forms. if ! [[ "$BATCHSIZE" =~ ^[1-9][0-9]*%?$ ]]; then echo "Batch size must be a number greater than 0, optionally with a trailing % (e.g. 5 or 25%)." exit 1