Holding dispatch in the drainer cannot help an orchestration that is
already running when salt-master restarts: its first action raises, and
the next action's refresh_pillar is published seconds after the restart,
before minions have rejoined the publisher, so it fails with "No minions
responded".
refresh_pillar now retries (3 attempts, 10s apart; a failed attempt takes
about 15s to time out, so the later attempts land after the reconnect
window). It also runs ahead of highstate actions, which now require it,
so they get the same reachability check. Retrying the highstate step
itself is not an option because it would re-run a fleetwide highstate
whenever any state failed.
Run straight after a salt-master restart on a standalone 3.5.0 install,
the orchestration previously failed with "No minions responded"; it now
succeeds on the second attempt. A target that is genuinely offline takes
about 65s to fail instead of 15s.
A push dispatched seconds after a salt-master restart was lost: minions
rejoin the new publisher on a randomized ZeroMQ reconnect delay (up to
~22s with the default recon_default/recon_max), and a job published
before then reaches nobody. The orchestration's refresh_pillar step timed
out with "No minions responded", the drainer logged "push failed" as an
ERROR, and so-log-check failed the run.
Seen on a standalone 3.5.0 install: a highstate restarted salt-master
(new reactor/engine config), the drainer dispatched a soc push 8s later,
and the job cache shows the refresh_pillar job matched the minion but got
no return. Measured on the same host over five restarts, jobs published
in the first 4-9s were lost every time.
The drainer now leaves ready intents in place for a later pass while
/var/run/salt-master.pid is less than 45s old. On a multi-node grid this
also avoids a batch state run silently skipping the minions that have
not reconnected yet, since batch only sends the job to minions that
answer its ping.
getKeyValue and removeKey now return early when a parent of the key is
null or not a mapping, so get, remove and purgelistitem report the key as
missing instead of crashing on a hand-edited pillar such as
`firewall: null`.
so-push-drainer computed a record's age before the per-record
try/except. A record whose dispatched_at or checked_at was not a number
raised TypeError there, so every pass exited before the drain and
auto-apply stopped. The traceback went to the stderr of a schedule with
return_job: False, so nothing reached so-push-drainer.log.
Only the drainer writes these records, so this takes a hand-edited file,
but result tracking is not supposed to be able to block the drain. Log a
warning and remove the record, as is already done for unreadable JSON
and a missing jid.
Also reword the so-log-check exclusion comment, which said the queued
state run "still completes" where the drainer only claims it may have,
and note at the drainer's warning that so-log-check matches its text.
so-push-drainer logs "push result unknown" as a WARNING when salt-master
restarts while a pushed state run is queued: the orchestration's next
publish fails with AuthenticationError, but the minion still runs the
queued state. so-log-check matches any line containing "error", so the
quoted exception still failed the check.
Reproduced on a distributed 3.5.0 grid: a sensoroni push queued behind a
150s state run on the manager, then salt-master was restarted. The
orchestration (which runs under salt-minion.service, so it survives the
restart) raised AuthenticationError 9s later, the drainer logged the
WARNING, and the queued state run completed with 0 failed states.
so-log-check --unknown flagged the line.
The pattern requires AuthenticationError so an unknown result from any
other raised exception, e.g. a bug in a state module, is still reported.
For a setting saved for a single node, push_pillar replaced the app's
target from pillar_push_map.yaml with the node id. The app's state was
then applied to that node even if the node does not run the app.
Seen in the lab: the SOC config test saves
elasticsearch.config.cluster.routing.allocation.disk.threshold_enabled
for the first node SOC lists, and SOC lists nodes in random order. When
that node was a sensor, the elasticsearch state was applied to it and
failed to render (so_roles has no so-sensor entry), and so-push-drainer
logged "push failed".
Narrow the app's target to the node instead of replacing it:
"L@<node> and ( <app target> )". A node outside the app's target then
matches nothing, and so-push-drainer skips the action.
so-push-drainer dispatched every queued action, even when its target
matched no accepted minion. The orchestration's saltutil.refresh_pillar
step then failed with "No minions responded", and the drainer logged a
push failure that so-log-check reports as an ERROR.
Seen on a fresh distributed manager: the first drain after setup sent
strelka and suricata rule pushes targeting sensor roles before any
sensor had joined the grid. Nothing had failed; no node held the role.
Before dispatch, resolve each action's target with the master's
CkMinions (accepted keys and cached grains) and drop the ones that
match nothing, logging each at INFO. If none remain, clear the intents
without dispatching. Uncached minions still count as a match, and a
lookup error keeps the action, so pushes are never dropped when the
answer is uncertain.
Fail 4xx ILM PUTs immediately (retry 429/5xx/connect), print the
Elasticsearch reason in the salt summary, and retry the load on the
next highstate via /opt/so/state/esilm.txt when the last run failed.
Fixes https://github.com/Security-Onion-Solutions/sos/issues/277
Mount the host trust store into zeek, suricata, tcpreplay, the strelka
containers and idh, and restart them when it changes. Point the Python
requests clients in strelka-backend and idh at it instead of certifi.
Remove the frozen /opt/so/conf/ca bundles from every node in ca.trust
rather than only on the manager in soup.
Anchor the grid CA and blocklist the eMudhra roots in the host trust
store (new ca.trust state, also used by the desktop role), then mount the
extracted store into each container instead of the frozen copies made by
so-catrust. Containers restart when the host trust changes.
Every container in docker/defaults.yaml exposes custom_bind_mounts via
soc_docker.yaml, but kafka and soc enabled.sls never appended them to
binds, so the setting was silently ignored for those two containers.
Salt 3006.29 (saltstack/salt#70466) answered the eMudhra/emSign incident by
bumping certifi to 2026.10.19, which drops the "emSign ECC Root CA - G3" root
whose intermediates had recoverable private keys, and by shipping pip 26.2.2,
whose vendored certifi gets the same fix. We stay on 3006.19, so apply the
same two upgrades inside /opt/saltstack/salt from vendored wheels.
relenv is upgraded to 0.22.27 in the same pip invocation. relenv 0.22.2's
runtime wrapper hardcodes pre-26.2 pip install() signatures, so a lone pip
26.2.2 upgrade breaks every later pip run with an unexpected
'script_executable' argument. 0.22.27 is what 3006.29 builds against and its
runtime diff is otherwise cosmetic. Installing all three at once under the
stock pip means no host ever sits in the broken intermediate state.
The state lives in salt.python_modules so new installs get it from
salt_install_module_deps before salt-master starts, and existing grids get it
from the soup highstate plus grid push. It is gated on saltversion 3006.19 so
a future Salt bump disables it, and ordered before the docker module install
so docker is always installed with the fixed pip. The docker wheel set now
carries certifi 2026.10.19 instead of 2024.7.4.
Verified on a fresh 3.4.0 manager + sensor grid: state applies on both,
reruns are no-ops, full highstate passes on both nodes.
pillar.get replaces lists, so once alerts queries are saved to pillar
from the SOC config UI, queries added to defaults later (e.g. the
licensed Group By Assessment, Name) never reach soc.json. Merge them by
name the same way availableModels are merged by id: pillar order and
edits are kept, and default queries missing from pillar are appended.