so-telegraf-partition-repair cleared the backlog but left the cause in place:
pg_cron's launcher is still dead, so the grid re-stalls as soon as it walks off
the premade window. Operators on the preview release need something they can run
once, before they soup, that leaves Telegraf collecting again.
Replace it with so-telegraf-repair, which fixes both halves. The running release
already creates the pg_cron extension and registers telegraf-partman-maintenance
in so_telegraf; only the launcher is missing, because so_telegraf did not exist
when the postmaster started. Restarting so-postgres is therefore enough to get
the existing job firing, so this touches no configuration and duplicates none of
the postgres state's SQL -- group_role still migrates the job to the postgres
database on the next soup. It also reconciles premake to 7 and prefers
so_admin.telegraf_maintenance() when that state has already landed.
Exit status separates healthy (0) from needs-repair (1) from does-not-apply (2),
which is what soup now gates on. postupgrade_changes runs after the highstate,
so the database is already converted by then and the backlog is the only thing
left to detect. Truncating is destructive and most grids were never affected --
fresh installs in particular, since they have no Telegraf history at all -- so
soup asks first and skips silently rather than clearing defaults on every host.
Rename the grid-highstate log from /opt/so/log/salt/so-grid-highstate.log to
/opt/so/log/salt/so-grid-highstate. Updates the LOG_FILE var in so-grid-highstate
and the two /opt/so/log/salt/so-grid-highstate.log references in soup (the progress
echo and the distributed-deployment message).
Truncating default partitions is destructive and premaking them is pg_cron's
job, so neither belongs in a state that runs on every checkin. Replace the two
telegraf_users states with so-telegraf-partition-repair, a standalone tool that
reports partition health and clears the backlog, and call it once from soup.
The script depends only on pg_partman, so it also runs against a grid that has
not yet picked up the new postgres state. It no-ops when nothing is stranded,
refuses to discard rows non-interactively without --yes, and reports when the
pg_cron job has never fired, which is the underlying cause rather than a
symptom the truncate addresses.
Hourly self-healing stays with so_admin.telegraf_maintenance() via pg_cron, so
a grid that never soups still recovers, just gradually and without discarding
in-retention metrics.
Two refinements to the --salt-upgraded path, surfaced testing a salt downgrade+soup
on a manager+heavynode grid:
- The post-upgrade settle loop settled on any stable reachable count >0, so when a
target was briefly down for its salt-minion restart it could settle on the
not-yet-restarted subset (observed: 'settled at 1' with 2 accepted) and release the
tiered pass before nodes reconnected. Capture the reachable count just before the
pass and wait for it to recover to that count (up >= pre-upgrade target) and hold
steady, with an initial grace so the delayed restart dip is observed rather than
skipped. Still compares against the pre-upgrade reachable set, not accepted keys, so
an intentionally powered-off node never stalls past the backstop.
- The salt-upgrade pass returns non-zero by design (targets restart salt-minion
mid-run), but it logged the generic 'nodes it missed will converge on the scheduled
highstate' warning, which reads like a real failure. Mark that dispatch as
expect_restart so it logs a benign, explanatory line instead.
Verified live: with the heavynode's salt-minion bounced during the settle window, the
loop logged 'fleet recovered to 2 minions up (>= pre-upgrade 2)' and only then ran the
tiered pass (heavynode highstate 427 succeeded, 0 failed).
Two defects surfaced testing BRANCH=asasoup soup on a manager+heavynode grid:
1. The tiered-pass actions JSON was built from a multi-line heredoc. salt parses
'pillar=<value>' kwargs with a non-DOTALL regex, so the embedded newlines made
salt-run treat the whole token as a positional saltenv -- 'No matching salt
environment for environment pillar={...}' -- and the highstate never ran, leaving
the heavynode on the old version. Emit the actions JSON on a single line (matching
how so-push-drainer's json.dumps payload already works).
2. orch.push_batch's salt.state step reports 'No minions returned' (a failure) for a
tier whose compound target matches nothing, so any grid lacking a role (no receiver,
small grids) always logged a warning and returned rc=1 even when every present node
converged. Pre-check each tier with 'salt -C <tgt> --preview-target' and include only
tiers that match >=1 minion; exit cleanly if none match.
Verified live: heavynode highstated 3.2.0 -> 3.3.0 (427 states, 0 failed) and a
follow-up run skips the empty receiver/remainder tiers with rc=0.
The per-minion highstate schedule moved from 15 minutes to 120 minutes
(salt:schedule:highstate_interval_minutes), so after soup upgrades the manager,
non-manager minions could otherwise sit on the old version for up to ~2.5 hours
(interval + splay) before their scheduled highstate applies the new code.
Add so-grid-highstate, a detached best-effort driver soup fires at the end of an
upgrade. It uses the existing orch.push_batch runner to highstate the grid in
role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node
grids, and when Salt itself was upgraded first runs an untiered pass and waits
for minions to reconnect on the new salt-minion before the tiered pass.
soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so
an SSH drop can't kill it) called on both the hotfix and full-upgrade paths,
wires the previously-dead -b flag through as --batch (now accepting N or N%),
and updates the distributed-deployment message to reflect the active push.
init-db.sh only runs on a fresh PGDATA (docker-entrypoint-initdb.d), so a
cluster left partially initialized -- e.g. the container restarted mid first-init
by a watch trigger -- never recovered: the so_postgres role existed but its
schema grants and the so_telegraf database were missing, breaking SOC with
"permission denied for schema public".
- init-db.sh: make the role upsert race-safe. Try CREATE ROLE and fall back to
ALTER on duplicate_object/unique_violation instead of IF NOT EXISTS, so a
concurrent creator no longer aborts the script (set -e + ON_ERROR_STOP=1)
before the grants and so_telegraf creation run. The whole script is now
idempotent and safe to re-run.
- postgres/enabled.sls: move postgres_wait_ready here (was telegraf-gated in
telegraf_users.sls) and add postgres_bootstrap_soc_db, which re-runs init-db.sh
every highstate so a partially-initialized cluster self-heals.
- telegraf_users.sls: drop its now-duplicate postgres_wait_ready definition; it
resolves from postgres.enabled via the existing include.
- soup: remove bootstrap_so_soc_database and its post_to_3.2.0 call. The
highstates soup runs before postupgrade now reconcile the SOC DB, making the
one-shot bootstrap redundant.
PostgreSQL's log_min_error_statement defaults to 'error', so whenever a
CREATE/ALTER ROLE ... PASSWORD statement errored, the full statement text --
including the plaintext password -- was written to /opt/so/log/postgres/postgres.log.
The role-provisioning paths (init-db.sh for so_postgres, so-telegraf-postgres
for per-minion telegraf roles) both dispatch such DDL, the latter on every
state.apply.
- init-db.sh / so-telegraf-postgres: SET log_min_error_statement = panic before
the password-bearing DDL so an error no longer emits the STATEMENT line. The
ERROR message itself (no password) still logs, preserving debuggability.
- logrotate: add a postgres stanza (daily, keep 14, copytruncate, compress) so
postgres.log is rotated like every other service and leaked content can't
persist indefinitely. copytruncate is required because the container holds the
log open via redirected stderr.
- soup: scrub any already-logged PASSWORD lines from postgres.log during
post_to_3.2.0, rewriting in place to preserve the inode postgres is writing to.
Before making any changes, verify the grid is in a good state:
- check_cluster_health: waits for Elasticsearch to reach at least 'yellow'
(blocks only on red/unreachable, since yellow is normal), modeled on the
wait in so-elasticsearch-roles-load.
- check_fleet_server: confirms the Fleet Server status API returns HTTP 200,
modeled on the wait_for_so-elastic-fleet state in elasticfleet/enabled.sls.
Both run alongside the existing check_pillar_items (manager pillar render) and
verify_es_version_compatibility, before soup modifies anything, so a failure
exits cleanly with an actionable message and no partial changes. Valid on all
manager roles soup runs on (eval/standalone/manager/managerhype/managersearch/
import), which all run Elasticsearch and the Fleet Server.
After a partial upgrade, /etc/soversion already reads the target version, so
soup's startup line "Found that Security Onion X is currently installed" made
it look finished even as soup resumed. When a resume marker is present and
differs from the installed version, print an explicit NOTE that the grid is only
partially upgraded and this run will resume and complete it.
Also clear any stale resume marker in the already-latest path so a successfully
completed upgrade is never mistaken for a partial one and re-run on a later
invocation (the marker is normally removed at the end of postupgrade_changes;
this is a belt-and-suspenders guard).
When soup fails via the EXIT trap after it has begun modifying the system, print
a prominent UPGRADE INCOMPLETE banner instructing the user to run soup again to
resume and complete the update. Gated on a new SOUP_UPGRADE_STARTED flag set at
the start of the hotfix and upgrade branches, so pre-flight gate failures (ES
compatibility, disk, network) that abort before any changes are made do not show
it.
A failed highstate mid-upgrade left /etc/soversion already advanced to the
target version (the highstate stamps it from the pillar via the soversionfile
state), so a re-run of soup saw INSTALLEDVERSION == NEWVERSION and reported
"already running the latest version", stranding the box with post-upgrade
steps never run.
Introduce /etc/sopostversion, a soup-owned marker (no salt state manages it)
that records post-upgrade walk progress. It is seeded from the pre-upgrade
version before the highstate, advanced after each post_to_* step, and removed
on successful completion. upgrade_check treats a leftover marker as "upgrade
not finished" and resumes the remaining post steps instead of bailing.
Also fix the hotfix path: /etc/sohotfix was written before the hotfix
highstate, so a failed hotfix highstate looked already-applied on re-run.
Since no salt state manages /etc/sohotfix, defer its write (update_version)
until after the highstate succeeds so it is an honest completion marker.
Josh Patterson
Mike Reeves
Mike Reeves
reyesj2
Jason Ertel