Salt installed the so-* scripts into /usr/sbin owned by unprivileged service
UIDs (939/socore, plus 930-960 per service) at mode 755, while root executes
those same files from cron, systemd and state cmd.run. Any file-write
primitive as one of those UIDs was therefore root.
Two mechanisms behind this are not visible in the diff:
file.recurse also manages the destination directory, so /usr/sbin itself was
chowned to whichever service UID ran last. A directory's owner may always
chmod it, so that UID could replace even the scripts already declared
user: root -- so-config-backup, so-suricata-eve-clean, so-nsm-mount-nvme.
usr_sbin_perms now pins the directory to root:root 555, the mode the
filesystem RPM ships.
Omitting user:/group: is a no-op on files that already exist, because
check_perms only chowns when a user is named. Explicit user: root is what
lets upgraded grids self-heal on the next highstate, and what makes a
revert chown back rather than silently do nothing.
zeekctl's post-terminate archives the final logs in the background and returns
immediately unless StopWait is set, so the container exits and takes the archiving
with it, stranding unarchived logs in /nsm/zeek/spool/tmp on every restart. Docker's
default 10s grace is also too tight for the entrypoint's SIGTERM trap; overrunning it
means SIGKILL and crash directories on the next start.
Both are needed. StopWait alone gives the stop more work to do inside the same 10s
window, which was measured ending in SIGKILL with logs stranded in the spool.
disabled.sls used docker rm -f, which never delivers SIGTERM, so stop the container
before removing it.
Reported in discussion #16174.
"zeekctl cron" writes node statistics to /nsm/zeek/logs/stats. The CPU and memory
half comes from a helper that shells out to top, which the Zeek container does not
include. The helper's "command not found" output is then parsed as process data, so
every cron run appended a line per node reading "bad output from top", which
so-log-check reports.
Nothing wrote that file before, since log_stats and update_http_stats only run from
"zeekctl cron". Set StatsLogEnable to 0 so neither runs, and mark it read only since
the CPU and memory statistics cannot work with this image. The interface counters it
also collects are not used anywhere in Security Onion, which tracks Zeek packet loss
separately through packetloss.log and Telegraf, so nothing is lost by turning this
off. Note in StatsLogExpireInterval that it does nothing while the stats log is off.
MailTo, MailConnectionSummary and MailHostUpDown do nothing but send mail, and
the Zeek container has no mail program, so nothing they control can happen. Mark
them read only rather than offering knobs in SOC that cannot take effect.
MailConnectionSummary only gates the emailed copy; the connection summary is
generated and archived either way. MailTo also feeds Notice::mail_dest, but
Security Onion never enables the notice email action, so that half is inert too.
MailHostUpDown gates only the notification text - host status detection, the
plugin hook and the stored state all run regardless.
MinDiskSpace stays editable. It is not mail only: setting it to 0 skips the disk
space check entirely, and the warning it produces is not emailed but does appear
in the output of "zeekctl cron". Correct its description, and MailHostUpDown's,
which both claimed these settings have no visible effect.
LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval are only acted
on by "zeekctl cron", which nothing in the grid ran, so setting them in SOC did
nothing. Add so-zeek-cron and run it every 5 minutes, the interval upstream
recommends.
This also restarts a node that died unexpectedly and marks it crashed so a crash
report is written, which is what CrashExpireInterval then reaps.
The crontab runs as root because the script needs the docker socket; it drops to
the zeek user inside the container so the stats logs and zeekctl-config.sh it
writes stay owned by uid 937.
Annotate the five zeekctl settings that were previously undocumented. The regex
on LogExpireInterval matters: a bare number means days, and a value shorter than
LogRotationInterval raises ConfigurationError, which fails the zeekctl deploy in
the container entrypoint. Zeek then never starts while Salt still reports success
and the container still reports healthy. Excluding the min unit keeps that
unreachable at the default 3600 second rotation interval. MinDiskSpace and
MailHostUpDown only send mail and the image has no sendmail, so their
descriptions say they currently have no effect.
- schedule highstate every 2 hours (was 15 minutes); interval lives in
global:push:highstate_interval_hours so the SOC admin UI can tune it and
so-salt-minion-check derives its threshold as (interval + 1) * 3600
- add inotify beacon on the manager + master reactor + orch.push_batch that
writes per-app intent files, with a so-push-drainer schedule on the manager
that debounces, dedupes, and dispatches a single orchestration
- pillar_push_map.yaml allowlists the apps whose pillar changes trigger an
immediate targeted state.apply (targets verified against salt/top.sls);
edits under pillar/minions/ trigger a state.highstate on that one minion
- host-batch every push orchestration (batch: 25%, batch_wait: 15) so rule
changes don't thundering-herd large fleets
- new global:push:enabled kill-switch tears down the beacon, reactor config,
and drainer schedule on the next highstate for operators who want to keep
highstate-only behavior
- set restart_policy: unless-stopped on 23 container states so docker
recovers crashes without waiting for the next highstate; leave registry
(always), strelka/backend (on-failure), kratos, and hydra alone with
inline comments explaining why
Add ulimits as a configurable advanced setting for every container,
allowing customization through the web UI. Move hardcoded ulimits
from elasticsearch and zeek into defaults.yaml and fix elasticsearch
ulimits that were incorrectly nested under the environment key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create /opt/so/conf/zeek/zkg directory and sync custom packages
from the manager via file.recurse. Bind mount the directory into
the so-zeek container so the entrypoint can install packages on
startup.
JA4 (BSD licensed) remains always enabled, but JA4+ variants (JA4S,
JA4D, JA4H, JA4L, JA4SSH, JA4T, JA4TS, JA4X) require a FoxIO license
and are now toggleable via the SOC UI. The toggle includes a license
agreement warning and defaults to disabled.
Josh Patterson
Mike Reeves
Mike Reeves
Doug Burks
reyesj2
Josh Brower
Jason Ertel