Auto State Apply fires on SOC config saves and on suricata/strelka rule
updates. Files a user creates or edits by hand under
/opt/so/saltstack/local/salt/ change no pillar, so nothing fired and the
change waited for the next scheduled highstate, now 120 minutes by default.
That gap is the 3.2 Known Issue in the docs.
Watch the directories the docs tell users to edit, and route them through
the push pipeline that already exists:
zeek/policy -> zeek (covers intel/ and custom/)
zeek/zkg -> zeek
elasticsearch/files/ingest -> elasticsearch
elasticsearch/roles -> elasticsearch
logstash/pipelines/config/custom -> logstash
Tags are pillar_push_map.yaml app names, so the existing entries already
carry the right state and compound target, and no map entry changes.
Rename the beacon rules_beacon -> local_files_beacon. Rules are now one of
five kinds of file it watches, and the new name matches how its sibling
postgres_pillar_beacon is named: source, then what it watches.
Replace push_suricata.sls and push_strelka.sls with one push_files.sls bound
to salt/beacon/*/local_files_beacon/*, which looks the tag up in
pillar_push_map.yaml the same way push_pillar.sls does. The map's suricata
and strelka targets match the compounds those two reactors hardcoded, so
rule pushes are unchanged. The app comes from the event tag rather than the
payload because salt's beacon loop pops the beacon's tag key off the data.
Key watermarks by watched directory instead of by tag. zeek/policy and
zeek/zkg both emit the tag zeek, and a shared watermark would make them
overwrite each other's digest and emit on every poll.
Prune .git from the fingerprint walk. zkg packages must be git clones with a
clean working tree, so the watched tree carries full git metadata; walking it
every 15s is wasted work and git's own index and ref mtime churn would fire a
grid-wide zeek apply on its own. Placing or updating a package always touches
working-tree files too, so detection is unaffected.
The watch is an allowlist rather than the whole local salt tree because salt
writes into that tree itself: hypervisor/hosts/ is rewritten continuously by
virtual_node_manager.py and virtual_power_manager.py, libvirt/images/ holds
multi-GB qcow2 files, and elasticfleet/files/so_agent-installers/,
elasticsearch/files/users, ca/files/ and filebeat/files/ are all state-written.
Watching any of them would either self-retrigger or make the 15s poll walk
gigabytes.
Add 100%-coverage unit tests for the three custom salt beacons
(postgres_pillar_beacon, rules_beacon, zeek) and add salt/_beacons to
the python-test workflow's paths trigger and matrix.
To pass the workflow's flake8 lint over the whole directory:
- zeek.py: reindent to 4 spaces, drop trailing blank line, noqa the
Salt-injected __salt__ references (F821); no logic change.
- postgres_pillar_beacon.py: noqa C901 on beacon() (complexity 13 > 12).
Rename the two custom push-detection beacons for clarity:
- pillar_db -> postgres_pillar_beacon
- rules_db -> rules_beacon
Salt resolves a beacon by its config-key name to a _beacons/ module of the
same filename and tags its events salt/beacon/<minion>/<name>/<tag>, so each
rename touches the module file, the beacon config key in
beacons_pushstate.conf.jinja, and the reactor tag patterns in
reactor_pushstate.conf together. Watermark filenames and log prefixes are
updated to match; reactor run() logic is unchanged.
Salt's stock inotify beacon leaks one kernel inotify instance every time
the minion rebuilds the beacon loader's __context__ (the orphaned
pyinotify.Notifier is never stopped), accumulating against
fs.inotify.max_user_instances=128 until inotify_init() fails with EMFILE
and rule-change push detection silently stops. This is independent of
disable_during_state_run.
Add a custom poll-based beacon (salt/_beacons/rules_db.py) modeled on
pillar_db.py: it fingerprints the suricata/strelka rule dirs each interval
(relpath + mtime_ns + size, temp files excluded) against a per-dir
watermark, emitting an event only on change. It holds zero inotify
instances, so the leak is impossible, and it keeps firing during state
runs. Swap the inotify beacon config and reactor tag mappings accordingly;
the push_suricata/push_strelka reactors are unchanged (they read only
data['path']).
The SOC postgres database was renamed so_soc -> securityonion (see
POSTGRES_DB in salt/postgres/enabled.sls and the SOC postgres config in
salt/soc/defaults.yaml). The pillar_db beacon still hardcoded so_soc, so
every poll failed with 'database "so_soc" does not exist' (rc=2),
silently disabling active-push detection of audit_settings changes.
Update DATABASE to 'securityonion' and refresh the now-stale so_soc
references in the beacon and push_pillar reactor comments.
The active-push feature detected pillar/settings changes via an inotify
beacon on the manager watching /opt/so/saltstack/local/pillar. Replace
that pillar watch with a custom salt beacon (pillar_db) that polls the
SOC so_soc.audit_settings table on a monotonic id watermark, so changes
made through SOC drive immediate pushes from the database instead of the
files. The suricata/strelka rule inotify watches (and pyinotify) are kept
unchanged, since rule-file edits are not recorded in audit_settings.
- salt/_beacons/pillar_db.py: new beacon. Polls audit_settings via
`docker exec so-postgres psql` (unix-socket trust auth), tracks the last
processed id in /opt/so/state/pillar_db_watch.id, seeds to MAX(id) on
first run (no history replay), and emits one event per new row.
- salt/reactor/push_pillar.sls: consume setting_id/node_id from the beacon
event instead of a file path. App = first dotted segment of setting_id,
looked up in pillar_push_map.yaml. Empty node_id -> grid-wide actions as
is; populated node_id -> the app's state(s) retargeted to that one node.
- salt/manager/files/beacons_pushstate.conf.jinja: drop the pillar inotify
block, add the pillar_db beacon (interval = push.drain_interval); keep
the suricata/strelka inotify watches.
- salt/salt/files/reactor_pushstate.conf: map salt/beacon/*/pillar_db/
audit_settings to push_pillar.sls; remove the pillar inotify reactor
lines; keep suricata/strelka.
The intent -> so-push-drainer -> orch.push_batch pipeline is unchanged.
Verified end-to-end on a standalone: a grid-wide telegraf.output change
re-applied telegraf fleetwide (container replaced), and a per-host
ntp.config.servers change applied ntp to only that node.
Josh Patterson