The Hydra admin API creates OAuth clients and introspects tokens without
authentication, relying on network isolation the same way Kratos does, but
so-hydra published 0.0.0.0:4445:4445 and sat on sobridge. With userland-proxy
left at its default of true, Docker ran a proxy listener on the host, so any
container reached the admin API through the manager IP or the bridge gateway
regardless of its docker network.
so-hydra now sits alone on soauth and no longer publishes 4445. 4444 is still
published, so the nginx proxy for /oauth2/token and the well-known endpoints is
unchanged. so-soc is already dual homed from the Kratos change and reaches the
admin API over soauth, so only its hostUrl moves. wait_for_hydra polls the
container address instead of the host port.
so-client reaches the admin API through docker exec, mirroring so-user. Exit
codes still propagate, so the --fail-with-body error handling is unchanged.
so-hydra is removed in up_to_3.4.0 alongside so-kratos and so-soc so the
highstate recreates it with the correct network membership. The auth network
range is already handled by set_soauth_range.
Upstream Elastic now ships binaries built for the x86-64-v3
micro-architecture level. This is not a Security Onion choice: nodes whose
CPUs predate x86-64-v3 can no longer run Elastic's own builds, so those
nodes break once they are upgraded.
Check for support before soup modifies anything, and require the operator
to type "override" to proceed when a node is unsupported or offline.
Runs after upgrade_check so a grid that is already current exits without
prompting. Targets only the roles that run a container built from the
so-elastic-agent image, using the role lists already maintained in
salt/reactor/pillar_push_map.yaml. Adds --skip-cpu-check to bypass the gate
for automation, and exit code 162 when the operator declines to override.
The Kratos admin API is unauthenticated by design and relies on network
isolation, but so-kratos published 0.0.0.0:4434:4434 and daemon.json leaves
userland-proxy at its default of true. Docker therefore ran a proxy listener
on the host, and any container reached the admin API through the manager IP
or the bridge gateway regardless of its docker network.
so-kratos now sits alone on a new soauth network and no longer publishes
4434. 4433 is still published, so nginx is unchanged. so-soc is dual homed
and reaches the admin API over soauth, with publicHostUrl set explicitly.
so-user reaches it through docker exec, and wait_for_kratos polls the
container address instead of the host port.
firewall/iptables.jinja hardcoded sobridge in every generated DNAT, ACCEPT,
masquerade and isolation rule, so it is now driven by a docker:networks map
and a per-container networks list. Containers that do not declare one
default to sobridge, and .ip still resolves to the primary network's
address. soauth is only created on the roles that run so-kratos.
Setup already allows a custom docker range, so it now also prompts for the
auth network range and writes it to the docker pillar. Grids on the default
range need no decision during soup, since they get the same 172.17.2.0/24 a
fresh install would. Grids that set a custom range are prompted, defaulting
to the adjacent /24, and unattended upgrades take that default with a notice
rather than blocking.
so-kratos and so-soc are removed in up_to_3.4.0 so the highstate recreates
them with the correct network membership.
so-telegraf-partition-repair cleared the backlog but left the cause in place:
pg_cron's launcher is still dead, so the grid re-stalls as soon as it walks off
the premade window. Operators on the preview release need something they can run
once, before they soup, that leaves Telegraf collecting again.
Replace it with so-telegraf-repair, which fixes both halves. The running release
already creates the pg_cron extension and registers telegraf-partman-maintenance
in so_telegraf; only the launcher is missing, because so_telegraf did not exist
when the postmaster started. Restarting so-postgres is therefore enough to get
the existing job firing, so this touches no configuration and duplicates none of
the postgres state's SQL -- group_role still migrates the job to the postgres
database on the next soup. It also reconciles premake to 7 and prefers
so_admin.telegraf_maintenance() when that state has already landed.
Exit status separates healthy (0) from needs-repair (1) from does-not-apply (2),
which is what soup now gates on. postupgrade_changes runs after the highstate,
so the database is already converted by then and the backlog is the only thing
left to detect. Truncating is destructive and most grids were never affected --
fresh installs in particular, since they have no Telegraf history at all -- so
soup asks first and skips silently rather than clearing defaults on every host.
Rename the grid-highstate log from /opt/so/log/salt/so-grid-highstate.log to
/opt/so/log/salt/so-grid-highstate. Updates the LOG_FILE var in so-grid-highstate
and the two /opt/so/log/salt/so-grid-highstate.log references in soup (the progress
echo and the distributed-deployment message).
Truncating default partitions is destructive and premaking them is pg_cron's
job, so neither belongs in a state that runs on every checkin. Replace the two
telegraf_users states with so-telegraf-partition-repair, a standalone tool that
reports partition health and clears the backlog, and call it once from soup.
The script depends only on pg_partman, so it also runs against a grid that has
not yet picked up the new postgres state. It no-ops when nothing is stranded,
refuses to discard rows non-interactively without --yes, and reports when the
pg_cron job has never fired, which is the underlying cause rather than a
symptom the truncate addresses.
Hourly self-healing stays with so_admin.telegraf_maintenance() via pg_cron, so
a grid that never soups still recovers, just gradually and without discarding
in-retention metrics.
The per-minion highstate schedule moved from 15 minutes to 120 minutes
(salt:schedule:highstate_interval_minutes), so after soup upgrades the manager,
non-manager minions could otherwise sit on the old version for up to ~2.5 hours
(interval + splay) before their scheduled highstate applies the new code.
Add so-grid-highstate, a detached best-effort driver soup fires at the end of an
upgrade. It uses the existing orch.push_batch runner to highstate the grid in
role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node
grids, and when Salt itself was upgraded first runs an untiered pass and waits
for minions to reconnect on the new salt-minion before the tiered pass.
soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so
an SSH drop can't kill it) called on both the hotfix and full-upgrade paths,
wires the previously-dead -b flag through as --batch (now accepting N or N%),
and updates the distributed-deployment message to reflect the active push.
init-db.sh only runs on a fresh PGDATA (docker-entrypoint-initdb.d), so a
cluster left partially initialized -- e.g. the container restarted mid first-init
by a watch trigger -- never recovered: the so_postgres role existed but its
schema grants and the so_telegraf database were missing, breaking SOC with
"permission denied for schema public".
- init-db.sh: make the role upsert race-safe. Try CREATE ROLE and fall back to
ALTER on duplicate_object/unique_violation instead of IF NOT EXISTS, so a
concurrent creator no longer aborts the script (set -e + ON_ERROR_STOP=1)
before the grants and so_telegraf creation run. The whole script is now
idempotent and safe to re-run.
- postgres/enabled.sls: move postgres_wait_ready here (was telegraf-gated in
telegraf_users.sls) and add postgres_bootstrap_soc_db, which re-runs init-db.sh
every highstate so a partially-initialized cluster self-heals.
- telegraf_users.sls: drop its now-duplicate postgres_wait_ready definition; it
resolves from postgres.enabled via the existing include.
- soup: remove bootstrap_so_soc_database and its post_to_3.2.0 call. The
highstates soup runs before postupgrade now reconcile the SOC DB, making the
one-shot bootstrap redundant.
PostgreSQL's log_min_error_statement defaults to 'error', so whenever a
CREATE/ALTER ROLE ... PASSWORD statement errored, the full statement text --
including the plaintext password -- was written to /opt/so/log/postgres/postgres.log.
The role-provisioning paths (init-db.sh for so_postgres, so-telegraf-postgres
for per-minion telegraf roles) both dispatch such DDL, the latter on every
state.apply.
- init-db.sh / so-telegraf-postgres: SET log_min_error_statement = panic before
the password-bearing DDL so an error no longer emits the STATEMENT line. The
ERROR message itself (no password) still logs, preserving debuggability.
- logrotate: add a postgres stanza (daily, keep 14, copytruncate, compress) so
postgres.log is rotated like every other service and leaked content can't
persist indefinitely. copytruncate is required because the container holds the
log open via redirected stderr.
- soup: scrub any already-logged PASSWORD lines from postgres.log during
post_to_3.2.0, rewriting in place to preserve the inode postgres is writing to.
Before making any changes, verify the grid is in a good state:
- check_cluster_health: waits for Elasticsearch to reach at least 'yellow'
(blocks only on red/unreachable, since yellow is normal), modeled on the
wait in so-elasticsearch-roles-load.
- check_fleet_server: confirms the Fleet Server status API returns HTTP 200,
modeled on the wait_for_so-elastic-fleet state in elasticfleet/enabled.sls.
Both run alongside the existing check_pillar_items (manager pillar render) and
verify_es_version_compatibility, before soup modifies anything, so a failure
exits cleanly with an actionable message and no partial changes. Valid on all
manager roles soup runs on (eval/standalone/manager/managerhype/managersearch/
import), which all run Elasticsearch and the Fleet Server.
After a partial upgrade, /etc/soversion already reads the target version, so
soup's startup line "Found that Security Onion X is currently installed" made
it look finished even as soup resumed. When a resume marker is present and
differs from the installed version, print an explicit NOTE that the grid is only
partially upgraded and this run will resume and complete it.
Also clear any stale resume marker in the already-latest path so a successfully
completed upgrade is never mistaken for a partial one and re-run on a later
invocation (the marker is normally removed at the end of postupgrade_changes;
this is a belt-and-suspenders guard).
When soup fails via the EXIT trap after it has begun modifying the system, print
a prominent UPGRADE INCOMPLETE banner instructing the user to run soup again to
resume and complete the update. Gated on a new SOUP_UPGRADE_STARTED flag set at
the start of the hotfix and upgrade branches, so pre-flight gate failures (ES
compatibility, disk, network) that abort before any changes are made do not show
it.
A failed highstate mid-upgrade left /etc/soversion already advanced to the
target version (the highstate stamps it from the pillar via the soversionfile
state), so a re-run of soup saw INSTALLEDVERSION == NEWVERSION and reported
"already running the latest version", stranding the box with post-upgrade
steps never run.
Introduce /etc/sopostversion, a soup-owned marker (no salt state manages it)
that records post-upgrade walk progress. It is seeded from the pre-upgrade
version before the highstate, advanced after each post_to_* step, and removed
on successful completion. upgrade_check treats a leftover marker as "upgrade
not finished" and resumes the remaining post steps instead of bailing.
Also fix the hotfix path: /etc/sohotfix was written before the hotfix
highstate, so a failed hotfix highstate looked already-applied on re-run.
Since no salt state manages /etc/sohotfix, defer its write (update_version)
until after the highstate succeeds so it is an honest completion marker.
reyesj2
Mike Reeves
Mike Reeves
Josh Brower
Josh Patterson
Jason Ertel