getKeyValue and removeKey now return early when a parent of the key is
null or not a mapping, so get, remove and purgelistitem report the key as
missing instead of crashing on a hand-edited pillar such as
`firewall: null`.
so-push-drainer computed a record's age before the per-record
try/except. A record whose dispatched_at or checked_at was not a number
raised TypeError there, so every pass exited before the drain and
auto-apply stopped. The traceback went to the stderr of a schedule with
return_job: False, so nothing reached so-push-drainer.log.
Only the drainer writes these records, so this takes a hand-edited file,
but result tracking is not supposed to be able to block the drain. Log a
warning and remove the record, as is already done for unreadable JSON
and a missing jid.
Also reword the so-log-check exclusion comment, which said the queued
state run "still completes" where the drainer only claims it may have,
and note at the drainer's warning that so-log-check matches its text.
so-push-drainer logs "push result unknown" as a WARNING when salt-master
restarts while a pushed state run is queued: the orchestration's next
publish fails with AuthenticationError, but the minion still runs the
queued state. so-log-check matches any line containing "error", so the
quoted exception still failed the check.
Reproduced on a distributed 3.5.0 grid: a sensoroni push queued behind a
150s state run on the manager, then salt-master was restarted. The
orchestration (which runs under salt-minion.service, so it survives the
restart) raised AuthenticationError 9s later, the drainer logged the
WARNING, and the queued state run completed with 0 failed states.
so-log-check --unknown flagged the line.
The pattern requires AuthenticationError so an unknown result from any
other raised exception, e.g. a bug in a state module, is still reported.
For a setting saved for a single node, push_pillar replaced the app's
target from pillar_push_map.yaml with the node id. The app's state was
then applied to that node even if the node does not run the app.
Seen in the lab: the SOC config test saves
elasticsearch.config.cluster.routing.allocation.disk.threshold_enabled
for the first node SOC lists, and SOC lists nodes in random order. When
that node was a sensor, the elasticsearch state was applied to it and
failed to render (so_roles has no so-sensor entry), and so-push-drainer
logged "push failed".
Narrow the app's target to the node instead of replacing it:
"L@<node> and ( <app target> )". A node outside the app's target then
matches nothing, and so-push-drainer skips the action.
so-push-drainer dispatched every queued action, even when its target
matched no accepted minion. The orchestration's saltutil.refresh_pillar
step then failed with "No minions responded", and the drainer logged a
push failure that so-log-check reports as an ERROR.
Seen on a fresh distributed manager: the first drain after setup sent
strelka and suricata rule pushes targeting sensor roles before any
sensor had joined the grid. Nothing had failed; no node held the role.
Before dispatch, resolve each action's target with the master's
CkMinions (accepted keys and cached grains) and drop the ones that
match nothing, logging each at INFO. If none remain, clear the intents
without dispatching. Uncached minions still count as a match, and a
lookup error keeps the action, so pushes are never dropped when the
answer is uncertain.
Fail 4xx ILM PUTs immediately (retry 429/5xx/connect), print the
Elasticsearch reason in the salt summary, and retry the load on the
next highstate via /opt/so/state/esilm.txt when the last run failed.
Fixes https://github.com/Security-Onion-Solutions/sos/issues/277