"zeekctl cron" writes node statistics to /nsm/zeek/logs/stats. The CPU and memory
half comes from a helper that shells out to top, which the Zeek container does not
include. The helper's "command not found" output is then parsed as process data, so
every cron run appended a line per node reading "bad output from top", which
so-log-check reports.
Nothing wrote that file before, since log_stats and update_http_stats only run from
"zeekctl cron". Set StatsLogEnable to 0 so neither runs, and mark it read only since
the CPU and memory statistics cannot work with this image. The interface counters it
also collects are not used anywhere in Security Onion, which tracks Zeek packet loss
separately through packetloss.log and Telegraf, so nothing is lost by turning this
off. Note in StatsLogExpireInterval that it does nothing while the stats log is off.
MailTo, MailConnectionSummary and MailHostUpDown do nothing but send mail, and
the Zeek container has no mail program, so nothing they control can happen. Mark
them read only rather than offering knobs in SOC that cannot take effect.
MailConnectionSummary only gates the emailed copy; the connection summary is
generated and archived either way. MailTo also feeds Notice::mail_dest, but
Security Onion never enables the notice email action, so that half is inert too.
MailHostUpDown gates only the notification text - host status detection, the
plugin hook and the stored state all run regardless.
MinDiskSpace stays editable. It is not mail only: setting it to 0 skips the disk
space check entirely, and the warning it produces is not emailed but does appear
in the output of "zeekctl cron". Correct its description, and MailHostUpDown's,
which both claimed these settings have no visible effect.
LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval are only acted
on by "zeekctl cron", which nothing in the grid ran, so setting them in SOC did
nothing. Add so-zeek-cron and run it every 5 minutes, the interval upstream
recommends.
This also restarts a node that died unexpectedly and marks it crashed so a crash
report is written, which is what CrashExpireInterval then reaps.
The crontab runs as root because the script needs the docker socket; it drops to
the zeek user inside the container so the stats logs and zeekctl-config.sh it
writes stay owned by uid 937.
Annotate the five zeekctl settings that were previously undocumented. The regex
on LogExpireInterval matters: a bare number means days, and a value shorter than
LogRotationInterval raises ConfigurationError, which fails the zeekctl deploy in
the container entrypoint. Zeek then never starts while Salt still reports success
and the container still reports healthy. Excluding the min unit keeps that
unreachable at the default 3600 second rotation interval. MinDiskSpace and
MailHostUpDown only send mail and the image has no sendmail, so their
descriptions say they currently have no effect.
Telegraf never stops writing. Clearing 50 defaults with separate
TRUNCATEs left the earliest ones refilled by the time maintenance tried
to attach today's child, which then failed on the default's constraint
and aborted the whole run. Doing both under one transaction makes the
concurrent inserts wait and land in the new partition.
Retention drops every child once they all age out, partman refuses to
drop the last one, and with infinite_time_partitions off it will not
premake forward from a child that far in the past. The set is left with
one stale partition and no current one, so metrics land right back in
the default.
Set infinite_time_partitions on telegraf parents, in both the repair
script and the retention subcommand, and stop blaming the launcher when
pg_cron is not loaded at all.
so-telegraf-partition-repair cleared the backlog but left the cause in place:
pg_cron's launcher is still dead, so the grid re-stalls as soon as it walks off
the premade window. Operators on the preview release need something they can run
once, before they soup, that leaves Telegraf collecting again.
Replace it with so-telegraf-repair, which fixes both halves. The running release
already creates the pg_cron extension and registers telegraf-partman-maintenance
in so_telegraf; only the launcher is missing, because so_telegraf did not exist
when the postmaster started. Restarting so-postgres is therefore enough to get
the existing job firing, so this touches no configuration and duplicates none of
the postgres state's SQL -- group_role still migrates the job to the postgres
database on the next soup. It also reconciles premake to 7 and prefers
so_admin.telegraf_maintenance() when that state has already landed.
Exit status separates healthy (0) from needs-repair (1) from does-not-apply (2),
which is what soup now gates on. postupgrade_changes runs after the highstate,
so the database is already converted by then and the backlog is the only thing
left to detect. Truncating is destructive and most grids were never affected --
fresh installs in particular, since they have no Telegraf history at all -- so
soup asks first and skips silently rather than clearing defaults on every host.
Add a logrotate entry for the grid-highstate driver's log, matching the existing
/opt/so/log/salt/{minion,master,so-salt-minion-check} entries (daily, rotate 14,
copytruncate, compress). Registered in both logrotate/defaults.yaml and the SOC
config schema logrotate/soc_logrotate.yaml.
Rename the grid-highstate log from /opt/so/log/salt/so-grid-highstate.log to
/opt/so/log/salt/so-grid-highstate. Updates the LOG_FILE var in so-grid-highstate
and the two /opt/so/log/salt/so-grid-highstate.log references in soup (the progress
echo and the distributed-deployment message).
pg_dumpall dumped every database, and so_telegraf dominated the result: it
is the only database that grows with grid size and metric volume, while
everything else in the cluster is small and mostly static.
Nothing is lost by skipping it. The data is transient metrics on a 14-day
retention window, roles are globals so the per-minion telegraf logins are
still dumped, and the database itself is rebuilt after a restore without
operator action -- init-db.sh recreates it (run on every highstate by
postgres_bootstrap_soc_db, not just on a fresh volume), telegraf_users.sls
re-provisions the roles and schema, and Telegraf recreates its tables on
first write.
Truncating default partitions is destructive and premaking them is pg_cron's
job, so neither belongs in a state that runs on every checkin. Replace the two
telegraf_users states with so-telegraf-partition-repair, a standalone tool that
reports partition health and clears the backlog, and call it once from soup.
The script depends only on pg_partman, so it also runs against a grid that has
not yet picked up the new postgres state. It no-ops when nothing is stranded,
refuses to discard rows non-interactively without --yes, and reports when the
pg_cron job has never fired, which is the underlying cause rather than a
symptom the truncate addresses.
Hourly self-healing stays with so_admin.telegraf_maintenance() via pg_cron, so
a grid that never soups still recovers, just gradually and without discarding
in-retention metrics.
Two refinements to the --salt-upgraded path, surfaced testing a salt downgrade+soup
on a manager+heavynode grid:
- The post-upgrade settle loop settled on any stable reachable count >0, so when a
target was briefly down for its salt-minion restart it could settle on the
not-yet-restarted subset (observed: 'settled at 1' with 2 accepted) and release the
tiered pass before nodes reconnected. Capture the reachable count just before the
pass and wait for it to recover to that count (up >= pre-upgrade target) and hold
steady, with an initial grace so the delayed restart dip is observed rather than
skipped. Still compares against the pre-upgrade reachable set, not accepted keys, so
an intentionally powered-off node never stalls past the backstop.
- The salt-upgrade pass returns non-zero by design (targets restart salt-minion
mid-run), but it logged the generic 'nodes it missed will converge on the scheduled
highstate' warning, which reads like a real failure. Mark that dispatch as
expect_restart so it logs a benign, explanatory line instead.
Verified live: with the heavynode's salt-minion bounced during the settle window, the
loop logged 'fleet recovered to 2 minions up (>= pre-upgrade 2)' and only then ran the
tiered pass (heavynode highstate 427 succeeded, 0 failed).
pg_cron's launcher connects to cron.database_name at postmaster start and is
registered BGW_NEVER_RESTART. On a host upgraded onto an existing /nsm/postgres
volume, init-db.sh never runs, so so_telegraf does not exist when PostgreSQL
starts -- the launcher dies and never retries. Salt then creates the database,
the extension, and the schedule, all of which succeed, but no worker is left to
fire the job. partman.run_maintenance_proc() therefore never runs: partitions
stop being premade after create_parent's initial window and every metric lands
in <parent>_default. Retention never fires either.
That state is self-perpetuating. Once the default partition holds rows for a day
with no child, PostgreSQL cannot create that child at all -- attaching it would
violate the default partition's constraint -- so maintenance aborts on the first
parent it reaches. Fixing the scheduler alone does not recover a stalled grid.
Point cron.database_name at the always-present postgres database and register the
job with cron.schedule_in_database targeting so_telegraf, so the launcher no
longer depends on database creation order. group_role drops any registration left
behind in so_telegraf, and both halves are guarded on the live GUC so applying
postgres.telegraf_users before the postgresql.conf change has restarted the
container skips instead of failing.
Maintenance now runs so_admin.telegraf_maintenance(), which drains stranded rows
out of any default partition before calling partman: expired rows are deleted,
the rest are repartitioned. It runs from the state on every highstate as well as
hourly from pg_cron, so a grid whose worker is dead still recovers on its own.
The routines live in a postgres-owned schema so_telegraf has no rights on, since
pg_cron executes them as postgres.
Existing grids are recovered by a marker-guarded repair state that truncates the
non-empty defaults once per host. The backlog is mostly past retention already
and moving tens of GB just to delete most of it is not worth the WAL.
Also raise premake from 3 to 7, reconciled onto existing parents in the retention
subcommand, so an outage has a week of headroom before anything reaches a
default partition, and add a check subcommand reporting partition age, default
occupancy and last job status.
Two defects surfaced testing BRANCH=asasoup soup on a manager+heavynode grid:
1. The tiered-pass actions JSON was built from a multi-line heredoc. salt parses
'pillar=<value>' kwargs with a non-DOTALL regex, so the embedded newlines made
salt-run treat the whole token as a positional saltenv -- 'No matching salt
environment for environment pillar={...}' -- and the highstate never ran, leaving
the heavynode on the old version. Emit the actions JSON on a single line (matching
how so-push-drainer's json.dumps payload already works).
2. orch.push_batch's salt.state step reports 'No minions returned' (a failure) for a
tier whose compound target matches nothing, so any grid lacking a role (no receiver,
small grids) always logged a warning and returned rc=1 even when every present node
converged. Pre-check each tier with 'salt -C <tgt> --preview-target' and include only
tiers that match >=1 minion; exit cleanly if none match.
Verified live: heavynode highstated 3.2.0 -> 3.3.0 (427 states, 0 failed) and a
follow-up run skips the empty receiver/remainder tiers with rc=0.
The per-minion highstate schedule moved from 15 minutes to 120 minutes
(salt:schedule:highstate_interval_minutes), so after soup upgrades the manager,
non-manager minions could otherwise sit on the old version for up to ~2.5 hours
(interval + splay) before their scheduled highstate applies the new code.
Add so-grid-highstate, a detached best-effort driver soup fires at the end of an
upgrade. It uses the existing orch.push_batch runner to highstate the grid in
role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node
grids, and when Salt itself was upgraded first runs an untiered pass and waits
for minions to reconnect on the new salt-minion before the tiered pass.
soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so
an SSH drop can't kill it) called on both the hotfix and full-upgrade paths,
wires the previously-dead -b flag through as --batch (now accepting N or N%),
and updates the distributed-deployment message to reflect the active push.
Josh Patterson
Josh Brower
Jorge Reyes
Jason Ertel
Mike Reeves
Jason Ertel
Mike Reeves