Two refinements to the --salt-upgraded path, surfaced testing a salt downgrade+soup
on a manager+heavynode grid:
- The post-upgrade settle loop settled on any stable reachable count >0, so when a
target was briefly down for its salt-minion restart it could settle on the
not-yet-restarted subset (observed: 'settled at 1' with 2 accepted) and release the
tiered pass before nodes reconnected. Capture the reachable count just before the
pass and wait for it to recover to that count (up >= pre-upgrade target) and hold
steady, with an initial grace so the delayed restart dip is observed rather than
skipped. Still compares against the pre-upgrade reachable set, not accepted keys, so
an intentionally powered-off node never stalls past the backstop.
- The salt-upgrade pass returns non-zero by design (targets restart salt-minion
mid-run), but it logged the generic 'nodes it missed will converge on the scheduled
highstate' warning, which reads like a real failure. Mark that dispatch as
expect_restart so it logs a benign, explanatory line instead.
Verified live: with the heavynode's salt-minion bounced during the settle window, the
loop logged 'fleet recovered to 2 minions up (>= pre-upgrade 2)' and only then ran the
tiered pass (heavynode highstate 427 succeeded, 0 failed).
Two defects surfaced testing BRANCH=asasoup soup on a manager+heavynode grid:
1. The tiered-pass actions JSON was built from a multi-line heredoc. salt parses
'pillar=<value>' kwargs with a non-DOTALL regex, so the embedded newlines made
salt-run treat the whole token as a positional saltenv -- 'No matching salt
environment for environment pillar={...}' -- and the highstate never ran, leaving
the heavynode on the old version. Emit the actions JSON on a single line (matching
how so-push-drainer's json.dumps payload already works).
2. orch.push_batch's salt.state step reports 'No minions returned' (a failure) for a
tier whose compound target matches nothing, so any grid lacking a role (no receiver,
small grids) always logged a warning and returned rc=1 even when every present node
converged. Pre-check each tier with 'salt -C <tgt> --preview-target' and include only
tiers that match >=1 minion; exit cleanly if none match.
Verified live: heavynode highstated 3.2.0 -> 3.3.0 (427 states, 0 failed) and a
follow-up run skips the empty receiver/remainder tiers with rc=0.
The per-minion highstate schedule moved from 15 minutes to 120 minutes
(salt:schedule:highstate_interval_minutes), so after soup upgrades the manager,
non-manager minions could otherwise sit on the old version for up to ~2.5 hours
(interval + splay) before their scheduled highstate applies the new code.
Add so-grid-highstate, a detached best-effort driver soup fires at the end of an
upgrade. It uses the existing orch.push_batch runner to highstate the grid in
role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node
grids, and when Salt itself was upgraded first runs an untiered pass and waits
for minions to reconnect on the new salt-minion before the tiered pass.
soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so
an SSH drop can't kill it) called on both the hotfix and full-upgrade paths,
wires the previously-dead -b flag through as --batch (now accepting N or N%),
and updates the distributed-deployment message to reflect the active push.
salt.schedule.highstate_interval_hours could not express a sub-hour
cadence, so an operator who disables salt.auto_apply had no way back to
the legacy 15-minute highstate. Rename the setting to
highstate_interval_minutes (default 120, behavior unchanged) and enforce
a 15-minute floor in SOC.
Non-manager splay is now a quarter of the interval clamped to [5, 30]
minutes, so a short interval no longer gets jitter larger than itself; at
the 120-minute default it stays 1800s. The so-salt-minion-check restart
threshold keeps its interval-plus-one-hour grace, now in minute math.
Previously salt/vars/elasticsearch.map.jinja read the elasticsearch heap
size via a raw pillar access (INIT.PILLAR.elasticsearch.esheap). If the
esheap key was missing from a minion pillar, building GLOBALS raised
'dict object has no attribute esheap', cascading into every state that
imports GLOBALS (elasticsearch.enabled, logstash.config, telegraf.config,
etc.) and blocking highstate entirely.
Add an esheap default ('600m', matching the es_heapsize() floor) to
elasticsearch/defaults.yaml and fall back to it via dict .get() so a
missing pillar value degrades gracefully. The per-node pillar value still
wins when present.
The agent name is also the agentMapping config key and a SOC setting id
segment, so the space made it awkward to target. Matches the rename in
securityonion-soc.
Josh Patterson
Jason ErtelandGitHub
Jason Ertel
Mike ReevesandGitHub
Mike Reeves
coreyogburnandGitHub
Corey Ogburn
Josh BrowerandGitHub
Jorge ReyesandGitHub