agentMapping values are model displayNames (the canonical selector); the
stock config used the model id, which only resolved via the legacy
id@adapter fallback. Use the Claude Sonnet displayName so agent-to-model
resolution matches the documented contract.
init-db.sh only runs on a fresh PGDATA (docker-entrypoint-initdb.d), so a
cluster left partially initialized -- e.g. the container restarted mid first-init
by a watch trigger -- never recovered: the so_postgres role existed but its
schema grants and the so_telegraf database were missing, breaking SOC with
"permission denied for schema public".
- init-db.sh: make the role upsert race-safe. Try CREATE ROLE and fall back to
ALTER on duplicate_object/unique_violation instead of IF NOT EXISTS, so a
concurrent creator no longer aborts the script (set -e + ON_ERROR_STOP=1)
before the grants and so_telegraf creation run. The whole script is now
idempotent and safe to re-run.
- postgres/enabled.sls: move postgres_wait_ready here (was telegraf-gated in
telegraf_users.sls) and add postgres_bootstrap_soc_db, which re-runs init-db.sh
every highstate so a partially-initialized cluster self-heals.
- telegraf_users.sls: drop its now-duplicate postgres_wait_ready definition; it
resolves from postgres.enabled via the existing include.
- soup: remove bootstrap_so_soc_database and its post_to_3.2.0 call. The
highstates soup runs before postupgrade now reconcile the SOC DB, making the
one-shot bootstrap redundant.
PostgreSQL's log_min_error_statement defaults to 'error', so whenever a
CREATE/ALTER ROLE ... PASSWORD statement errored, the full statement text --
including the plaintext password -- was written to /opt/so/log/postgres/postgres.log.
The role-provisioning paths (init-db.sh for so_postgres, so-telegraf-postgres
for per-minion telegraf roles) both dispatch such DDL, the latter on every
state.apply.
- init-db.sh / so-telegraf-postgres: SET log_min_error_statement = panic before
the password-bearing DDL so an error no longer emits the STATEMENT line. The
ERROR message itself (no password) still logs, preserving debuggability.
- logrotate: add a postgres stanza (daily, keep 14, copytruncate, compress) so
postgres.log is rotated like every other service and leaked content can't
persist indefinitely. copytruncate is required because the container holds the
log open via redirected stderr.
- soup: scrub any already-logged PASSWORD lines from postgres.log during
post_to_3.2.0, rewriting in place to preserve the inode postgres is writing to.
Running the wait on a healthy, steady-state minion always reported readiness
after a 3s floor:
salt-minion (pid 4114640) ready after 3s
That floor came entirely from the unconditional sleep "$INITIAL_SLEEP" (3s)
before the poll loop. The sleep is vestigial: it predates restart_pending and
never even covered the restart race (see 89e6a746c -- "INITIAL_SLEEP=3 expired
inside that window"). Salt restarts the unit with --no-block and the restart
job is enqueued before service.restart returns, so an in-flight restart is
visible to restart_pending on the first loop iteration; the sleep protects
nothing now.
Drop the INITIAL_SLEEP constant and the pre-loop sleep and start elapsed at 0.
The loop already sleeps at the bottom of each iteration, so the first readiness
check now runs immediately: an already-ready minion returns "ready after 0s",
while the restart path (guarded by restart_pending) and the mid-startup log
gate are unchanged.
The wait required both a socket gate and a log gate to pass. The log gate
greps the minion log for salt's one-time startup line "Minion is ready to
receive requests!", which scrolls out of the log tail on a minion that has
not restarted recently. On such a minion the log gate could never pass, so
the script ran to its full 120s timeout and exited 1 even though the minion
was healthy and connected. This also false-timed-out when salt_minion_service
reported a non-restart change (e.g. an enable toggle).
The log gate's only remaining purpose is closing the ~2.8s post-connect window
where the master sockets are up but _post_master_init() is still loading. Gate
it on the current pid's uptime: enforce the ready line only within
READY_LINE_WINDOW (90s) of (re)start, and let the already restart-independent
socket gate be the steady-state authority past that. The fresh-restart path is
unchanged, and if uptime can't be read the strict behavior is kept.
Add 100%-coverage unit tests for the three custom salt beacons
(postgres_pillar_beacon, rules_beacon, zeek) and add salt/_beacons to
the python-test workflow's paths trigger and matrix.
To pass the workflow's flake8 lint over the whole directory:
- zeek.py: reindent to 4 spaces, drop trailing blank line, noqa the
Salt-injected __salt__ references (F821); no logic change.
- postgres_pillar_beacon.py: noqa C901 on beacon() (complexity 13 > 12).
Containers now start on boot via restart_policy unless-stopped, so a
highstate is no longer required to bring them up. Gather and display the
container table even when no highstate has completed since reboot, while
still warning the user. The exit code / JSON status_code stays 2 in that
state so SOC's Grid continues to show the restarting message unchanged.
Before making any changes, verify the grid is in a good state:
- check_cluster_health: waits for Elasticsearch to reach at least 'yellow'
(blocks only on red/unreachable, since yellow is normal), modeled on the
wait in so-elasticsearch-roles-load.
- check_fleet_server: confirms the Fleet Server status API returns HTTP 200,
modeled on the wait_for_so-elastic-fleet state in elasticfleet/enabled.sls.
Both run alongside the existing check_pillar_items (manager pillar render) and
verify_es_version_compatibility, before soup modifies anything, so a failure
exits cleanly with an actionable message and no partial changes. Valid on all
manager roles soup runs on (eval/standalone/manager/managerhype/managersearch/
import), which all run Elasticsearch and the Fleet Server.
After a partial upgrade, /etc/soversion already reads the target version, so
soup's startup line "Found that Security Onion X is currently installed" made
it look finished even as soup resumed. When a resume marker is present and
differs from the installed version, print an explicit NOTE that the grid is only
partially upgraded and this run will resume and complete it.
Also clear any stale resume marker in the already-latest path so a successfully
completed upgrade is never mistaken for a partial one and re-run on a later
invocation (the marker is normally removed at the end of postupgrade_changes;
this is a belt-and-suspenders guard).
When soup fails via the EXIT trap after it has begun modifying the system, print
a prominent UPGRADE INCOMPLETE banner instructing the user to run soup again to
resume and complete the update. Gated on a new SOUP_UPGRADE_STARTED flag set at
the start of the hotfix and upgrade branches, so pre-flight gate failures (ES
compatibility, disk, network) that abort before any changes are made do not show
it.
Josh PattersonandGitHub
Josh BrowerandGitHub
Mike ReevesandGitHub
Mike Reeves
Jason ErtelandGitHub
Jason Ertel
Doug BurksandGitHub
Jorge ReyesandGitHub