salt.schedule.highstate_interval_hours could not express a sub-hour
cadence, so an operator who disables salt.auto_apply had no way back to
the legacy 15-minute highstate. Rename the setting to
highstate_interval_minutes (default 120, behavior unchanged) and enforce
a 15-minute floor in SOC.
Non-manager splay is now a quarter of the interval clamped to [5, 30]
minutes, so a short interval no longer gets jitter larger than itself; at
the 120-minute default it stays 1800s. The so-salt-minion-check restart
threshold keeps its interval-plus-one-hour grace, now in minute math.
Previously salt/vars/elasticsearch.map.jinja read the elasticsearch heap
size via a raw pillar access (INIT.PILLAR.elasticsearch.esheap). If the
esheap key was missing from a minion pillar, building GLOBALS raised
'dict object has no attribute esheap', cascading into every state that
imports GLOBALS (elasticsearch.enabled, logstash.config, telegraf.config,
etc.) and blocking highstate entirely.
Add an esheap default ('600m', matching the es_heapsize() floor) to
elasticsearch/defaults.yaml and fall back to it via dict .get() so a
missing pillar value degrades gracefully. The per-node pillar value still
wins when present.
The agent name is also the agentMapping config key and a SOC setting id
segment, so the space made it awkward to target. Matches the rename in
securityonion-soc.
agentMapping values are model displayNames (the canonical selector); the
stock config used the model id, which only resolved via the legacy
id@adapter fallback. Use the Claude Sonnet displayName so agent-to-model
resolution matches the documented contract.
init-db.sh only runs on a fresh PGDATA (docker-entrypoint-initdb.d), so a
cluster left partially initialized -- e.g. the container restarted mid first-init
by a watch trigger -- never recovered: the so_postgres role existed but its
schema grants and the so_telegraf database were missing, breaking SOC with
"permission denied for schema public".
- init-db.sh: make the role upsert race-safe. Try CREATE ROLE and fall back to
ALTER on duplicate_object/unique_violation instead of IF NOT EXISTS, so a
concurrent creator no longer aborts the script (set -e + ON_ERROR_STOP=1)
before the grants and so_telegraf creation run. The whole script is now
idempotent and safe to re-run.
- postgres/enabled.sls: move postgres_wait_ready here (was telegraf-gated in
telegraf_users.sls) and add postgres_bootstrap_soc_db, which re-runs init-db.sh
every highstate so a partially-initialized cluster self-heals.
- telegraf_users.sls: drop its now-duplicate postgres_wait_ready definition; it
resolves from postgres.enabled via the existing include.
- soup: remove bootstrap_so_soc_database and its post_to_3.2.0 call. The
highstates soup runs before postupgrade now reconcile the SOC DB, making the
one-shot bootstrap redundant.
PostgreSQL's log_min_error_statement defaults to 'error', so whenever a
CREATE/ALTER ROLE ... PASSWORD statement errored, the full statement text --
including the plaintext password -- was written to /opt/so/log/postgres/postgres.log.
The role-provisioning paths (init-db.sh for so_postgres, so-telegraf-postgres
for per-minion telegraf roles) both dispatch such DDL, the latter on every
state.apply.
- init-db.sh / so-telegraf-postgres: SET log_min_error_statement = panic before
the password-bearing DDL so an error no longer emits the STATEMENT line. The
ERROR message itself (no password) still logs, preserving debuggability.
- logrotate: add a postgres stanza (daily, keep 14, copytruncate, compress) so
postgres.log is rotated like every other service and leaked content can't
persist indefinitely. copytruncate is required because the container holds the
log open via redirected stderr.
- soup: scrub any already-logged PASSWORD lines from postgres.log during
post_to_3.2.0, rewriting in place to preserve the inode postgres is writing to.
Running the wait on a healthy, steady-state minion always reported readiness
after a 3s floor:
salt-minion (pid 4114640) ready after 3s
That floor came entirely from the unconditional sleep "$INITIAL_SLEEP" (3s)
before the poll loop. The sleep is vestigial: it predates restart_pending and
never even covered the restart race (see 89e6a746c -- "INITIAL_SLEEP=3 expired
inside that window"). Salt restarts the unit with --no-block and the restart
job is enqueued before service.restart returns, so an in-flight restart is
visible to restart_pending on the first loop iteration; the sleep protects
nothing now.
Drop the INITIAL_SLEEP constant and the pre-loop sleep and start elapsed at 0.
The loop already sleeps at the bottom of each iteration, so the first readiness
check now runs immediately: an already-ready minion returns "ready after 0s",
while the restart path (guarded by restart_pending) and the mid-startup log
gate are unchanged.
The wait required both a socket gate and a log gate to pass. The log gate
greps the minion log for salt's one-time startup line "Minion is ready to
receive requests!", which scrolls out of the log tail on a minion that has
not restarted recently. On such a minion the log gate could never pass, so
the script ran to its full 120s timeout and exited 1 even though the minion
was healthy and connected. This also false-timed-out when salt_minion_service
reported a non-restart change (e.g. an enable toggle).
The log gate's only remaining purpose is closing the ~2.8s post-connect window
where the master sockets are up but _post_master_init() is still loading. Gate
it on the current pid's uptime: enforce the ready line only within
READY_LINE_WINDOW (90s) of (re)start, and let the already restart-independent
socket gate be the steady-state authority past that. The fresh-restart path is
unchanged, and if uptime can't be read the strict behavior is kept.
Mike ReevesandGitHub
Josh Brower
Josh PattersonandGitHub
Jorge ReyesandGitHub
Mike Reeves
Jason Ertel
Jason ErtelandGitHub