so-telegraf mounted /var/run/docker.sock and joined the host docker group on
every node type. The :ro flag blocks write() to the inode, not connect() plus
HTTP over the socket, so any code execution inside the container could reach
POST /containers/create with Privileged:true and become root on the host.
No telegraf script used the socket; the only consumer was the native
[[inputs.docker]] plugin, and group_add 920 existed solely to feed it.
Container metrics now come from so-container-stats, a collector that runs on
the host from cron and writes influx line protocol to a file telegraf already
had mounted. This is the pattern so-status, so-raid-status and
so-elasticagent-status already use, so the privileged docker access stays on
the host side where root cron already ran it.
The collector runs as somon, a service account in the docker group with no
login shell and a locked password, rather than root. Docker group membership
is still root-equivalent on the host, so this is defense in depth rather than
a privilege boundary.
The telegraf scripts were root:939 mode 770, letting socore rewrite them for
code execution inside the container; they are now 750, which still allows the
read and execute telegraf needs. The container also ran with no group, giving
it gid 0, and now runs as 939:939. That alone would have broken
lasthighstate.sh, which reached /opt/so/log/salt only via the root group and
could not tell an unreadable file from a missing one, so it silently reported
a 56 year highstate age. Only the lasthighstate file is bind mounted now, and
the script tests readability instead of existence.
Everything inputs.docker collected beyond the five fields the shipped
dashboards query is available per stat under telegraf:container_stats,
annotated for SOC so an operator can enable it without editing files. Defaults
reproduce the previous output exactly. With every stat enabled the emitted
field set matches what inputs.docker wrote, verified by running the plugin
against the live socket and diffing: 53 fields, no type mismatches, no field
present on one side only.
Two deliberate differences: max_usage carries the real cgroup peak where the
daemon reports 0 on cgroup v2, and host-network containers emit no
docker_container_net row, matching inputs.docker. Docker label tags are not
restored, since nothing queries them and they cost significant cardinality.
so-status, the influxdb size cron, so-elasticagent-status, so-raid-status and
so-common-status-check truncated their output in place while telegraf read it,
so telegraf periodically saw an empty file and logged a parse error or emitted
empty values. They now write aside and rename. Measured on a live manager, the
old so-status cron left status.log empty for 215 of 10997 reads.
Tested on a fresh install, a converted grid and a 3.0 upgrade.
Adding a user failed with "Password does not meet the minimum requirements"
for passwords that were well over the eight character minimum.
so-user reads the password from stdin in updatePassword, but verifyEnvironment
runs first and calls kratosCurl, which ran docker exec with -i. That attaches
stdin and drains the pipe the caller sent the password on, so the later
read -rs saw EOF, password was empty, and expr length "" tripped the minimum
length check.
No kratosCurl or hydraCurl call sends a body on stdin; every one passes it as
a -d argument, so -i was never needed. Dropping it leaves the admin API
responses unchanged.
so-client had the same wrapper for the Hydra admin API. It does not currently
read stdin, but the flag drains its caller's pipe just the same, so it is
dropped there too.
sobridge carries most containers but held no annotation, so it was absent from
the config tree entirely. Its pillar entry is an empty map, so annotating it
adds a setting with no children for docker.networks.soauth.range to collide
with, and the tree still builds.
docker.map.jinja fills in sobridge's range and gateway at render time from
docker.range and docker.gateway, and resets the entry when it is not a mapping,
so nothing writes keys underneath it.
The sentence about sobridge is dropped from the soauth range description now
that sobridge documents itself.
SOC reported "Malformed config setting (docker.networks.soauth.range): Setting
name 'networks' conflicts with another similarly named setting" and refused to
render the config tree.
soc_docker.yaml annotated docker.networks itself, and every key in that block
was a scalar, so FlattenAnnotations registered docker.networks as a setting.
defaults.yaml holds a nested map there, so FlattenPillar separately registered
docker.networks.soauth.range, .gateway and .manager_only, and
HydrateAnnotations unions the two sets. The config tree gives each id segment
either a value or children, never both, so addToNode pushed docker.networks as
a childless leaf and then threw when docker.networks.soauth.range tried to
descend through it.
The annotations move down to the three leaves that actually carry values, so
docker.networks is a plain branch. This matches docker.containers, which is
nested the same way and has never carried annotations of its own.
docker.ulimits and the per-container networks list are unaffected: their pillar
values are lists rather than maps, so they stay single settings.
The description was written alongside the Kratos change and named only the
Kratos admin API, but so-hydra now sits on soauth too.
The trailing note about rebuilding the firewall is dropped, since the setting
is readonly and cannot be changed from SOC in the first place.
The Hydra admin API creates OAuth clients and introspects tokens without
authentication, relying on network isolation the same way Kratos does, but
so-hydra published 0.0.0.0:4445:4445 and sat on sobridge. With userland-proxy
left at its default of true, Docker ran a proxy listener on the host, so any
container reached the admin API through the manager IP or the bridge gateway
regardless of its docker network.
so-hydra now sits alone on soauth and no longer publishes 4445. 4444 is still
published, so the nginx proxy for /oauth2/token and the well-known endpoints is
unchanged. so-soc is already dual homed from the Kratos change and reaches the
admin API over soauth, so only its hostUrl moves. wait_for_hydra polls the
container address instead of the host port.
so-client reaches the admin API through docker exec, mirroring so-user. Exit
codes still propagate, so the --fail-with-body error handling is unchanged.
so-hydra is removed in up_to_3.4.0 alongside so-kratos and so-soc so the
highstate recreates it with the correct network membership. The auth network
range is already handled by set_soauth_range.
/opt/so/saltstack/default holds the source for every root-executed script --
/usr/sbin, the reactors, _runners/_modules/_beacons, the master engines and
salt-relay.sh -- plus every state the root master renders. SOC mounts
/opt/so/saltstack rw as uid 939, so root-owning /usr/sbin alone was not
enough: the next highstate would copy attacker-controlled bytes out of the
tree into the root-owned destination and run them.
SOC never writes under default/, it only reads it. Every SOC write targets
local/, which stays socore-owned, as does /opt/so/state. No mode is enforced
on default/ -- SOC reads that tree, and 750/640 would break its config load.
Also stops copy_new_files(), so-saltstack-update and setup from chowning the
tree back to socore, and replaces preserve: True in soup_scripts.sls, which
carried uid/gid in from the /tmp staging tree and would have undone the
ownership before the first post-soup highstate.
Salt installed the so-* scripts into /usr/sbin owned by unprivileged service
UIDs (939/socore, plus 930-960 per service) at mode 755, while root executes
those same files from cron, systemd and state cmd.run. Any file-write
primitive as one of those UIDs was therefore root.
Two mechanisms behind this are not visible in the diff:
file.recurse also manages the destination directory, so /usr/sbin itself was
chowned to whichever service UID ran last. A directory's owner may always
chmod it, so that UID could replace even the scripts already declared
user: root -- so-config-backup, so-suricata-eve-clean, so-nsm-mount-nvme.
usr_sbin_perms now pins the directory to root:root 555, the mode the
filesystem RPM ships.
Omitting user:/group: is a no-op on files that already exist, because
check_perms only chowns when a user is named. Explicit user: root is what
lets upgraded grids self-heal on the next highstate, and what makes a
revert chown back rather than silently do nothing.
setup runs so-minion -o=setup before /usr/sbin/so-common is installed, so
valid_ip4 was undefined, every MAINIP was rejected, and no minion pillar
was written. Pillar compile then failed for the new manager and setup gave
up waiting for the salt master. Use an inline IPv4 regex instead.
so-minion exported every line of the minion-controlled /opt/so/install.txt
into its root shell and wrote the values unescaped into a Jinja-rendered
pillar, allowing a rogue node to redefine PILLARFILE or run code on the
master at the next pillar compile. pcapspace also fed minion-returned
disk.usage output into bash arithmetic, which evaluates array subscripts.
- Parse install.txt against an allowlist of known keys; never export
- Validate MINION_ID before building pillar paths; make them readonly
- Validate node type, IP, interface, hostname, heap and core values
before any pillar is written; strip braces and control chars from
the free-text node description
- Require a numeric disk size before pcapspace arithmetic
- Refuse manager node types on add/addVM so a remote node cannot
rewrite the CA pillar; only setup may create them
The cleanup loop had no progress check or pass limit, so once /nsm was over
threshold with nothing left to reclaim it spun at full speed, writing 1.1 GB /
12.5M lines to sensor_clean.log in five hours. Stop when a pass removes
nothing, when a pass frees no space, or at MAX_PASSES, and drop the per-pass
"no old files" logging in favor of one actionable line.
Replace the pgrep guard with flock -n. A find|while read subshell inherits the
parent's argv, so pgrep -cf counted one instance as hundreds; it was also
check-then-act, which let cron stack up overlapping runs.
Paths now derive from SENSOR_DIR with env-overridable LOG/LOCK so the
over-threshold path can be tested against a scratch filesystem.
Josh Patterson
Jorge Reyes
Jason Ertel
Jason Ertel
Mike Reeves
Josh Brower
Doug Burks
Mike Reeves
coreyogburn
Corey Ogburn