Two defects surfaced testing BRANCH=asasoup soup on a manager+heavynode grid:
1. The tiered-pass actions JSON was built from a multi-line heredoc. salt parses
'pillar=<value>' kwargs with a non-DOTALL regex, so the embedded newlines made
salt-run treat the whole token as a positional saltenv -- 'No matching salt
environment for environment pillar={...}' -- and the highstate never ran, leaving
the heavynode on the old version. Emit the actions JSON on a single line (matching
how so-push-drainer's json.dumps payload already works).
2. orch.push_batch's salt.state step reports 'No minions returned' (a failure) for a
tier whose compound target matches nothing, so any grid lacking a role (no receiver,
small grids) always logged a warning and returned rc=1 even when every present node
converged. Pre-check each tier with 'salt -C <tgt> --preview-target' and include only
tiers that match >=1 minion; exit cleanly if none match.
Verified live: heavynode highstated 3.2.0 -> 3.3.0 (427 states, 0 failed) and a
follow-up run skips the empty receiver/remainder tiers with rc=0.
The per-minion highstate schedule moved from 15 minutes to 120 minutes
(salt:schedule:highstate_interval_minutes), so after soup upgrades the manager,
non-manager minions could otherwise sit on the old version for up to ~2.5 hours
(interval + splay) before their scheduled highstate applies the new code.
Add so-grid-highstate, a detached best-effort driver soup fires at the end of an
upgrade. It uses the existing orch.push_batch runner to highstate the grid in
role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node
grids, and when Salt itself was upgraded first runs an untiered pass and waits
for minions to reconnect on the new salt-minion before the tiered pass.
soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so
an SSH drop can't kill it) called on both the hotfix and full-upgrade paths,
wires the previously-dead -b flag through as --batch (now accepting N or N%),
and updates the distributed-deployment message to reflect the active push.
salt.schedule.highstate_interval_hours could not express a sub-hour
cadence, so an operator who disables salt.auto_apply had no way back to
the legacy 15-minute highstate. Rename the setting to
highstate_interval_minutes (default 120, behavior unchanged) and enforce
a 15-minute floor in SOC.
Non-manager splay is now a quarter of the interval clamped to [5, 30]
minutes, so a short interval no longer gets jitter larger than itself; at
the 120-minute default it stays 1800s. The so-salt-minion-check restart
threshold keeps its interval-plus-one-hour grace, now in minute math.
Previously salt/vars/elasticsearch.map.jinja read the elasticsearch heap
size via a raw pillar access (INIT.PILLAR.elasticsearch.esheap). If the
esheap key was missing from a minion pillar, building GLOBALS raised
'dict object has no attribute esheap', cascading into every state that
imports GLOBALS (elasticsearch.enabled, logstash.config, telegraf.config,
etc.) and blocking highstate entirely.
Add an esheap default ('600m', matching the es_heapsize() floor) to
elasticsearch/defaults.yaml and fall back to it via dict .get() so a
missing pillar value degrades gracefully. The per-node pillar value still
wins when present.
The agent name is also the agentMapping config key and a SOC setting id
segment, so the space made it awkward to target. Matches the rename in
securityonion-soc.
Josh Patterson
Jason Ertel
Jason Ertel
Mike Reeves
Mike Reeves
coreyogburn
Corey Ogburn
Josh Brower
Jorge Reyes