mirror of
https://github.com/Security-Onion-Solutions/securityonion.git
synced 2026-05-11 05:40:32 +02:00
make so-yaml PG-canonical and add pillar-change reactor stack
Two coupled changes that together let so_pillar.* be the canonical config store, with config edits driving service reloads automatically: so-yaml PG-canonical mode - Adds /opt/so/conf/so-yaml/mode (and SO_YAML_BACKEND env override) with three values: dual (legacy), postgres (PG-only for managed paths), disk (emergency rollback). Bootstrap files (secrets.sls, ca/init.sls, *.nodes.sls, top.sls, ...) stay disk-only regardless via the existing SkipPath allowlist in so_yaml_postgres.locate. - loadYaml/writeYaml/purgeFile now route to so_pillar.* in postgres mode: replace/add/get all read+write the database with no disk file ever appearing. PG failure is fatal in postgres mode (no silent fallback); dual mode preserves the prior best-effort mirror. - so_yaml_postgres gains read_yaml(path), is_pg_managed(path), and is_enabled() so so-yaml can answer "is this path PG-managed and is PG up" without reaching into private helpers. - schema_pillar.sls writes /opt/so/conf/so-yaml/mode = postgres after the importer succeeds, so flipping postgres:so_pillar:enabled flips so-yaml's behavior in lockstep with the schema being live. pg_notify-driven change fan-out - 008_change_notify.sql adds so_pillar.change_queue + an AFTER trigger on pillar_entry that enqueues the locator and pg_notifies 'so_pillar_change'. Queue is drained at-least-once so engine restarts don't lose events; pg_notify is just the wakeup signal. - New salt-master engine pg_notify_pillar.py LISTENs on the channel, drains the queue with FOR UPDATE SKIP LOCKED, debounces bursts, and fires 'so/pillar/changed' events grouped by (scope, role, minion). - Reactor so_pillar_changed.sls catches the tag and dispatches to orch.so_pillar_reload, which carries a DISPATCH map of pillar-path prefix -> (state sls, role grain set) so adding a new service to the auto-reload list is a one-line edit instead of a new reactor. - Engine + reactor wiring is gated on the same postgres:so_pillar:enabled flag as the schema and ext_pillar config so the whole stack flips on/off together. Tests: 21 new cases (112 total, all passing) covering mode resolution, PG-managed detection, and PG-canonical read/write/purge routing with the PG client stubbed.
This commit is contained in:
@@ -69,6 +69,12 @@ class SkipPath(Exception):
|
||||
"""Raised when a file path is intentionally not mirrored to PG."""
|
||||
|
||||
|
||||
def is_enabled():
|
||||
"""Public alias for callers that want to probe PG reachability without
|
||||
relying on a leading-underscore private name."""
|
||||
return _is_enabled()
|
||||
|
||||
|
||||
def _is_enabled():
|
||||
"""PG dual-write only fires if so-postgres is reachable. Cheap probe.
|
||||
Returns True when docker exec succeeds, False otherwise. We never
|
||||
@@ -149,6 +155,55 @@ def _conflict_target(scope):
|
||||
raise ValueError(f"unknown scope {scope!r}")
|
||||
|
||||
|
||||
def is_pg_managed(path):
|
||||
"""True if this path maps to a so_pillar.* row (locate() succeeds).
|
||||
Bootstrap and mine-driven files return False — they always live on
|
||||
disk regardless of so-yaml's backend mode."""
|
||||
try:
|
||||
locate(path)
|
||||
return True
|
||||
except SkipPath:
|
||||
return False
|
||||
|
||||
|
||||
def read_yaml(path):
|
||||
"""Return the content dict stored in so_pillar.pillar_entry for `path`,
|
||||
or None when no row exists. Raises SkipPath when `path` is not part of
|
||||
the PG-managed surface (caller should read disk in that case).
|
||||
|
||||
Used by so-yaml.py PG-canonical mode so `replace`, `get`, etc. resolve
|
||||
against the database rather than a stale (or absent) disk file."""
|
||||
if not _is_enabled():
|
||||
return None
|
||||
scope, role, minion_id, pillar_path = locate(path)
|
||||
|
||||
if scope == "minion":
|
||||
sql = ("SELECT data FROM so_pillar.pillar_entry "
|
||||
"WHERE scope='minion' "
|
||||
f"AND minion_id={_pg_str(minion_id)} "
|
||||
f"AND pillar_path={_pg_str(pillar_path)}")
|
||||
elif scope == "role":
|
||||
sql = ("SELECT data FROM so_pillar.pillar_entry "
|
||||
"WHERE scope='role' "
|
||||
f"AND role_name={_pg_str(role)} "
|
||||
f"AND pillar_path={_pg_str(pillar_path)}")
|
||||
else:
|
||||
sql = ("SELECT data FROM so_pillar.pillar_entry "
|
||||
"WHERE scope='global' "
|
||||
f"AND pillar_path={_pg_str(pillar_path)}")
|
||||
|
||||
try:
|
||||
out = _docker_psql(sql).strip()
|
||||
except Exception:
|
||||
return None
|
||||
if not out:
|
||||
return None
|
||||
try:
|
||||
return json.loads(out)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def write_yaml(path, content_dict, *, reason="so-yaml dual-write"):
|
||||
"""Mirror the disk write at `path` (whose content was just rendered as
|
||||
`content_dict`) into so_pillar.pillar_entry. Best-effort: any failure
|
||||
|
||||
Reference in New Issue
Block a user