Holding dispatch in the drainer cannot help an orchestration that is
already running when salt-master restarts: its first action raises, and
the next action's refresh_pillar is published seconds after the restart,
before minions have rejoined the publisher, so it fails with "No minions
responded".
refresh_pillar now retries (3 attempts, 10s apart; a failed attempt takes
about 15s to time out, so the later attempts land after the reconnect
window). It also runs ahead of highstate actions, which now require it,
so they get the same reachability check. Retrying the highstate step
itself is not an option because it would re-run a fleetwide highstate
whenever any state failed.
Run straight after a salt-master restart on a standalone 3.5.0 install,
the orchestration previously failed with "No minions responded"; it now
succeeds on the second attempt. A target that is genuinely offline takes
about 65s to fail instead of 15s.
A push dispatched seconds after a salt-master restart was lost: minions
rejoin the new publisher on a randomized ZeroMQ reconnect delay (up to
~22s with the default recon_default/recon_max), and a job published
before then reaches nobody. The orchestration's refresh_pillar step timed
out with "No minions responded", the drainer logged "push failed" as an
ERROR, and so-log-check failed the run.
Seen on a standalone 3.5.0 install: a highstate restarted salt-master
(new reactor/engine config), the drainer dispatched a soc push 8s later,
and the job cache shows the refresh_pillar job matched the minion but got
no return. Measured on the same host over five restarts, jobs published
in the first 4-9s were lost every time.
The drainer now leaves ready intents in place for a later pass while
/var/run/salt-master.pid is less than 45s old. On a multi-node grid this
also avoids a batch state run silently skipping the minions that have
not reconnected yet, since batch only sends the job to minions that
answer its ping.
getKeyValue and removeKey now return early when a parent of the key is
null or not a mapping, so get, remove and purgelistitem report the key as
missing instead of crashing on a hand-edited pillar such as
`firewall: null`.
so-push-drainer computed a record's age before the per-record
try/except. A record whose dispatched_at or checked_at was not a number
raised TypeError there, so every pass exited before the drain and
auto-apply stopped. The traceback went to the stderr of a schedule with
return_job: False, so nothing reached so-push-drainer.log.
Only the drainer writes these records, so this takes a hand-edited file,
but result tracking is not supposed to be able to block the drain. Log a
warning and remove the record, as is already done for unreadable JSON
and a missing jid.
Also reword the so-log-check exclusion comment, which said the queued
state run "still completes" where the drainer only claims it may have,
and note at the drainer's warning that so-log-check matches its text.
so-push-drainer logs "push result unknown" as a WARNING when salt-master
restarts while a pushed state run is queued: the orchestration's next
publish fails with AuthenticationError, but the minion still runs the
queued state. so-log-check matches any line containing "error", so the
quoted exception still failed the check.
Reproduced on a distributed 3.5.0 grid: a sensoroni push queued behind a
150s state run on the manager, then salt-master was restarted. The
orchestration (which runs under salt-minion.service, so it survives the
restart) raised AuthenticationError 9s later, the drainer logged the
WARNING, and the queued state run completed with 0 failed states.
so-log-check --unknown flagged the line.
The pattern requires AuthenticationError so an unknown result from any
other raised exception, e.g. a bug in a state module, is still reported.
For a setting saved for a single node, push_pillar replaced the app's
target from pillar_push_map.yaml with the node id. The app's state was
then applied to that node even if the node does not run the app.
Seen in the lab: the SOC config test saves
elasticsearch.config.cluster.routing.allocation.disk.threshold_enabled
for the first node SOC lists, and SOC lists nodes in random order. When
that node was a sensor, the elasticsearch state was applied to it and
failed to render (so_roles has no so-sensor entry), and so-push-drainer
logged "push failed".
Narrow the app's target to the node instead of replacing it:
"L@<node> and ( <app target> )". A node outside the app's target then
matches nothing, and so-push-drainer skips the action.
so-push-drainer dispatched every queued action, even when its target
matched no accepted minion. The orchestration's saltutil.refresh_pillar
step then failed with "No minions responded", and the drainer logged a
push failure that so-log-check reports as an ERROR.
Seen on a fresh distributed manager: the first drain after setup sent
strelka and suricata rule pushes targeting sensor roles before any
sensor had joined the grid. Nothing had failed; no node held the role.
Before dispatch, resolve each action's target with the master's
CkMinions (accepted keys and cached grains) and drop the ones that
match nothing, logging each at INFO. If none remain, clear the intents
without dispatching. Uncached minions still count as a match, and a
lookup error keeps the action, so pushes are never dropped when the
answer is uncertain.
Fail 4xx ILM PUTs immediately (retry 429/5xx/connect), print the
Elasticsearch reason in the salt summary, and retry the load on the
next highstate via /opt/so/state/esilm.txt when the last run failed.
Fixes https://github.com/Security-Onion-Solutions/sos/issues/277