/opt/so/saltstack/default holds the source for every root-executed script --
/usr/sbin, the reactors, _runners/_modules/_beacons, the master engines and
salt-relay.sh -- plus every state the root master renders. SOC mounts
/opt/so/saltstack rw as uid 939, so root-owning /usr/sbin alone was not
enough: the next highstate would copy attacker-controlled bytes out of the
tree into the root-owned destination and run them.
SOC never writes under default/, it only reads it. Every SOC write targets
local/, which stays socore-owned, as does /opt/so/state. No mode is enforced
on default/ -- SOC reads that tree, and 750/640 would break its config load.
Also stops copy_new_files(), so-saltstack-update and setup from chowning the
tree back to socore, and replaces preserve: True in soup_scripts.sls, which
carried uid/gid in from the /tmp staging tree and would have undone the
ownership before the first post-soup highstate.
Salt installed the so-* scripts into /usr/sbin owned by unprivileged service
UIDs (939/socore, plus 930-960 per service) at mode 755, while root executes
those same files from cron, systemd and state cmd.run. Any file-write
primitive as one of those UIDs was therefore root.
Two mechanisms behind this are not visible in the diff:
file.recurse also manages the destination directory, so /usr/sbin itself was
chowned to whichever service UID ran last. A directory's owner may always
chmod it, so that UID could replace even the scripts already declared
user: root -- so-config-backup, so-suricata-eve-clean, so-nsm-mount-nvme.
usr_sbin_perms now pins the directory to root:root 555, the mode the
filesystem RPM ships.
Omitting user:/group: is a no-op on files that already exist, because
check_perms only chowns when a user is named. Explicit user: root is what
lets upgraded grids self-heal on the next highstate, and what makes a
revert chown back rather than silently do nothing.
setup runs so-minion -o=setup before /usr/sbin/so-common is installed, so
valid_ip4 was undefined, every MAINIP was rejected, and no minion pillar
was written. Pillar compile then failed for the new manager and setup gave
up waiting for the salt master. Use an inline IPv4 regex instead.
so-minion exported every line of the minion-controlled /opt/so/install.txt
into its root shell and wrote the values unescaped into a Jinja-rendered
pillar, allowing a rogue node to redefine PILLARFILE or run code on the
master at the next pillar compile. pcapspace also fed minion-returned
disk.usage output into bash arithmetic, which evaluates array subscripts.
- Parse install.txt against an allowlist of known keys; never export
- Validate MINION_ID before building pillar paths; make them readonly
- Validate node type, IP, interface, hostname, heap and core values
before any pillar is written; strip braces and control chars from
the free-text node description
- Require a numeric disk size before pcapspace arithmetic
- Refuse manager node types on add/addVM so a remote node cannot
rewrite the CA pillar; only setup may create them
The cleanup loop had no progress check or pass limit, so once /nsm was over
threshold with nothing left to reclaim it spun at full speed, writing 1.1 GB /
12.5M lines to sensor_clean.log in five hours. Stop when a pass removes
nothing, when a pass frees no space, or at MAX_PASSES, and drop the per-pass
"no old files" logging in favor of one actionable line.
Replace the pgrep guard with flock -n. A find|while read subshell inherits the
parent's argv, so pgrep -cf counted one instance as hundreds; it was also
check-then-act, which let cron stack up overlapping runs.
Paths now derive from SENSOR_DIR with env-overridable LOG/LOCK so the
over-threshold path can be tested against a scratch filesystem.
Upstream Elastic now ships binaries built for the x86-64-v3
micro-architecture level. This is not a Security Onion choice: nodes whose
CPUs predate x86-64-v3 can no longer run Elastic's own builds, so those
nodes break once they are upgraded.
Check for support before soup modifies anything, and require the operator
to type "override" to proceed when a node is unsupported or offline.
Runs after upgrade_check so a grid that is already current exits without
prompting. Targets only the roles that run a container built from the
so-elastic-agent image, using the role lists already maintained in
salt/reactor/pillar_push_map.yaml. Adds --skip-cpu-check to bypass the gate
for automation, and exit code 162 when the operator declines to override.
zeekctl's post-terminate archives the final logs in the background and returns
immediately unless StopWait is set, so the container exits and takes the archiving
with it, stranding unarchived logs in /nsm/zeek/spool/tmp on every restart. Docker's
default 10s grace is also too tight for the entrypoint's SIGTERM trap; overrunning it
means SIGKILL and crash directories on the next start.
Both are needed. StopWait alone gives the stop more work to do inside the same 10s
window, which was measured ending in SIGKILL with logs stranded in the spool.
disabled.sls used docker rm -f, which never delivers SIGTERM, so stop the container
before removing it.
Reported in discussion #16174.
2 new config fields. Batch size is used to limit how many message turns we put in the transcript when we ask the memory agent to extract facts. The retries helps limit how many times we ask the memory agent to process a problematic session.
When auto approving tools, we might approve a tool_request before it's been saved to ES. These vars describe some leniency in retrying when the message can't be found before giving up.
Double quoted stings in yaml allow for escape sequences like `\n` and `\t` but when used around a regex, salt will hang up on `\d` not being a valid escape sequence. Switching to single quotes so escapes aren't processed.
New field that'll stop the memory scanner from scanning before an indicated date. Leaving it empty lets the scanner scan everything.
The regex for it allows YYYY-MM-DD and ensures months only allow the max number of days (no June the 43rd).
Auto State Apply fires on SOC config saves and on suricata/strelka rule
updates. Files a user creates or edits by hand under
/opt/so/saltstack/local/salt/ change no pillar, so nothing fired and the
change waited for the next scheduled highstate, now 120 minutes by default.
That gap is the 3.2 Known Issue in the docs.
Watch the directories the docs tell users to edit, and route them through
the push pipeline that already exists:
zeek/policy -> zeek (covers intel/ and custom/)
zeek/zkg -> zeek
elasticsearch/files/ingest -> elasticsearch
elasticsearch/roles -> elasticsearch
logstash/pipelines/config/custom -> logstash
Tags are pillar_push_map.yaml app names, so the existing entries already
carry the right state and compound target, and no map entry changes.
Rename the beacon rules_beacon -> local_files_beacon. Rules are now one of
five kinds of file it watches, and the new name matches how its sibling
postgres_pillar_beacon is named: source, then what it watches.
Replace push_suricata.sls and push_strelka.sls with one push_files.sls bound
to salt/beacon/*/local_files_beacon/*, which looks the tag up in
pillar_push_map.yaml the same way push_pillar.sls does. The map's suricata
and strelka targets match the compounds those two reactors hardcoded, so
rule pushes are unchanged. The app comes from the event tag rather than the
payload because salt's beacon loop pops the beacon's tag key off the data.
Key watermarks by watched directory instead of by tag. zeek/policy and
zeek/zkg both emit the tag zeek, and a shared watermark would make them
overwrite each other's digest and emit on every poll.
Prune .git from the fingerprint walk. zkg packages must be git clones with a
clean working tree, so the watched tree carries full git metadata; walking it
every 15s is wasted work and git's own index and ref mtime churn would fire a
grid-wide zeek apply on its own. Placing or updating a package always touches
working-tree files too, so detection is unaffected.
The watch is an allowlist rather than the whole local salt tree because salt
writes into that tree itself: hypervisor/hosts/ is rewritten continuously by
virtual_node_manager.py and virtual_power_manager.py, libvirt/images/ holds
multi-GB qcow2 files, and elasticfleet/files/so_agent-installers/,
elasticsearch/files/users, ca/files/ and filebeat/files/ are all state-written.
Watching any of them would either self-retrigger or make the 15s poll walk
gigabytes.
Josh Brower
Josh Patterson
Doug Burks
Mike Reeves
Corey Ogburn
Mike Reeves
Jason Ertel
reyesj2
Jason Ertel
Matthew Wright