On a fresh install the surirulesync file.recurse creates .gitkeep before
SOC has generated all-rulesets.rules. That change satisfied the
surirulereload onchanges requisite, so the reload ran with no ruleset
present, failed to stat the file, and reported the state (and install)
as failed.
Add an onlyif guard so the reload only runs when all-rulesets.rules
exists. A .gitkeep-only sync now leaves the state a clean success
(onlyif condition false); once SOC writes the ruleset, the reload fires
normally.
Route the reload/verify output (ours plus so-common's retry/fail lines)
through a synchronous timestamping pipeline so every line in reload.log
is prefixed with a date/time, and preserve the real exit code via
PIPESTATUS.
Treating an in-progress reload as instant success could report success
while Suricata was still running a stale ruleset (the in-flight reload
may have started before the new all-rulesets.rules was written).
Make success conditional on Suricata actually having loaded the current
ruleset: capture the rules-file mtime up front, trigger a blocking
reload-rules, then query ruleset-reload-time and only succeed when
last_reload >= mtime. An in-progress reload now retries (waits for it to
clear so our own fresh reload runs) instead of short-circuiting, and a
ruleset that never catches up within the retry window fails via fail().
Also drop the redundant ruleset-reload-nonblocking call (the verified
blocking reload is authoritative and the async call was what left a
reload running) and log human-readable timestamps.
so-suricata-reload-rules failed the surirulereload state when a rule
reload was already running: suricatasc returns
{"message":"Reload already in progress","return":"NOK"}, which never
matched the expected output, so retry looped all 60 attempts (~3 min)
and called fail.
Wrap the suricatasc calls so an in-progress reload is treated as
success (the in-flight reload picks up the new rules) while genuine
container-not-ready conditions still retry and ultimately fail.
Rename the two custom push-detection beacons for clarity:
- pillar_db -> postgres_pillar_beacon
- rules_db -> rules_beacon
Salt resolves a beacon by its config-key name to a _beacons/ module of the
same filename and tags its events salt/beacon/<minion>/<name>/<tag>, so each
rename touches the module file, the beacon config key in
beacons_pushstate.conf.jinja, and the reactor tag patterns in
reactor_pushstate.conf together. Watermark filenames and log prefixes are
updated to match; reactor run() logic is unchanged.
Salt's stock inotify beacon leaks one kernel inotify instance every time
the minion rebuilds the beacon loader's __context__ (the orphaned
pyinotify.Notifier is never stopped), accumulating against
fs.inotify.max_user_instances=128 until inotify_init() fails with EMFILE
and rule-change push detection silently stops. This is independent of
disable_during_state_run.
Add a custom poll-based beacon (salt/_beacons/rules_db.py) modeled on
pillar_db.py: it fingerprints the suricata/strelka rule dirs each interval
(relpath + mtime_ns + size, temp files excluded) against a per-dir
watermark, emitting an event only on change. It holds zero inotify
instances, so the leak is impossible, and it keeps firing during state
runs. Swap the inotify beacon config and reactor tag mappings accordingly;
the push_suricata/push_strelka reactors are unchanged (they read only
data['path']).
Jason Ertel
reyesj2
coreyogburnandGitHub
Josh Patterson
Josh BrowerandGitHub
Mike ReevesandGitHub
Corey Ogburn
Jason ErtelandGitHub
Matthew WrightandCorey Ogburn