Auto State Apply fires on SOC config saves and on suricata/strelka rule
updates. Files a user creates or edits by hand under
/opt/so/saltstack/local/salt/ change no pillar, so nothing fired and the
change waited for the next scheduled highstate, now 120 minutes by default.
That gap is the 3.2 Known Issue in the docs.
Watch the directories the docs tell users to edit, and route them through
the push pipeline that already exists:
zeek/policy -> zeek (covers intel/ and custom/)
zeek/zkg -> zeek
elasticsearch/files/ingest -> elasticsearch
elasticsearch/roles -> elasticsearch
logstash/pipelines/config/custom -> logstash
Tags are pillar_push_map.yaml app names, so the existing entries already
carry the right state and compound target, and no map entry changes.
Rename the beacon rules_beacon -> local_files_beacon. Rules are now one of
five kinds of file it watches, and the new name matches how its sibling
postgres_pillar_beacon is named: source, then what it watches.
Replace push_suricata.sls and push_strelka.sls with one push_files.sls bound
to salt/beacon/*/local_files_beacon/*, which looks the tag up in
pillar_push_map.yaml the same way push_pillar.sls does. The map's suricata
and strelka targets match the compounds those two reactors hardcoded, so
rule pushes are unchanged. The app comes from the event tag rather than the
payload because salt's beacon loop pops the beacon's tag key off the data.
Key watermarks by watched directory instead of by tag. zeek/policy and
zeek/zkg both emit the tag zeek, and a shared watermark would make them
overwrite each other's digest and emit on every poll.
Prune .git from the fingerprint walk. zkg packages must be git clones with a
clean working tree, so the watched tree carries full git metadata; walking it
every 15s is wasted work and git's own index and ref mtime churn would fire a
grid-wide zeek apply on its own. Placing or updating a package always touches
working-tree files too, so detection is unaffected.
The watch is an allowlist rather than the whole local salt tree because salt
writes into that tree itself: hypervisor/hosts/ is rewritten continuously by
virtual_node_manager.py and virtual_power_manager.py, libvirt/images/ holds
multi-GB qcow2 files, and elasticfleet/files/so_agent-installers/,
elasticsearch/files/users, ca/files/ and filebeat/files/ are all state-written.
Watching any of them would either self-retrigger or make the 15s poll walk
gigabytes.
Telegraf 1.39 deprecated bare string entries in inputs.exec commands and
will drop them in 1.45, warning on every start:
W! DeprecationWarning: Value "/scripts/esindexsize.sh" for option
"command" of plugin "inputs.exec" deprecated since version 1.39.0
Each entry is now a single-element argv array. Array form skips shell
parsing, which is fine here: every command is a bare script path from
telegraf's scripts list, no args or shell metacharacters.
The UEK8 rollout installs the new kernel and flips the boot default, but
leaves the stock EL9 (RHCK) packages behind: disk in /boot and a stale
GRUB entry on every upgraded node.
They cannot be removed in the same pass that installs UEK8. dnf's
protect_running_kernel refuses to erase the booted kernel-core, so the
removal has to wait until the node has rebooted onto 6.x. Waiting is the
safer sequencing anyway -- the node proves it comes up on UEK8 before its
fallback is deleted -- so this does not remove RHCK from the uek7 branch
either, where dnf would allow it.
so-kernel-upgrade grows a --cleanup mode that does only the removal and
no-ops (exit 0, with a log line) on a node not yet running UEK8. Its uek8
branch, which previously reported "nothing to do", now runs that cleanup
along with set_default_kernel_conf -- which also closes a gap where a node
that came up on UEK8 straight from a fresh install never had
DEFAULTKERNEL=kernel-uek-core written.
The common highstate calls --cleanup gated on the running kernel, so the
cleanup lands grid-wide as each node reboots: fresh installs reboot at the
end of setup, upgraded nodes whenever the admin schedules it. The rpm
check inside the script is the idempotency guard, so subsequent highstates
cost an rpm query rather than a dnf transaction, and the package list is
not duplicated into the state where it could drift.
Added `memoryScanIntervalSeconds` with a default of 5 mins.
Opted to set `useMemoryScanner` to false so by default our user's sessions are not sent to the cloud before they have a chance to configure the new setting.
The Logstash log level was hardcoded to info in log4j2.properties, and
logstash.yml carried no log.level key, so the only way to raise verbosity
for troubleshooting was to hand-edit a file that the next highstate
overwrites. Add log_x_level and log_x_format to logstash:config so both
render into logstash.yml, annotated as advanced per-node settings with
the value sets Logstash 9.3.7 accepts.
log4j2.properties gains jinja, so it moves to log4j2.properties.jinja and
is rendered by a discrete lslog4j2 state rather than the lsetcsync
recurse, which cannot rename. The recurse exclude_pat now matches both
names so it neither copies the template verbatim nor lets clean: True
delete the rendered file, matching how pipelines.yml is already handled.
The appender layout is selected at render time so log.format actually
changes the log output instead of being a dead setting, keeping the
existing file name so nothing downstream moves. rootLogger.level now
follows ls.log.level rather than claiming info regardless of the
configured level.
Grid testing every permitted value on the manager pipeline surfaced two
combinations the UI allows that take the pipeline down, neither of which
the descriptions mentioned.
pipeline.ordered: true requires pipeline.workers: 1; with more workers the
pipeline fails to start with "enabling the 'pipeline.ordered' setting
requires the use of a single pipeline worker". Also correct the auto
wording: it only engages when workers is explicitly set to 1.
queue.max_bytes larger than the free space on /nsm/logstash fails queue
creation with "Unable to allocate N more bytes", rather than merely being
inadvisable.
Widen the byte-size regex, which rejected values Logstash accepts and so
blocked the save in SOC: bare-letter units (1g, 512m, 64k), decimals
(1.5gb), whitespace before the unit, and a bare integer. Allow whitespace
in dead_letter_queue.retain.age (5 d). Both stay lowercase-only, matching
byte_value.rb and AbstractPipelineExt.parseToDuration.
Fix description gaps: queue.checkpoint.retry is a Windows/SAN workaround
Elastic does not otherwise recommend, batch metrics sampling is technical
preview, queue.checkpoint.interval is deprecated in 9.1, compression makes
a queue unreadable by Logstash before 9.2, flush_check_interval has a
1000ms floor, max_events counts unread events, and the path settings are
created by Logstash but reject symlinks. Note which settings apply only to
persisted queues or an enabled DLQ.
Drop the undocumented 'disabled' value from queue.compression.
Numeric fields stay stricter than NumericSetting, which has no validator
and would accept negatives, floats and NaN in event counts and intervals.
A non-mapping value under logstash:pipeline_settings:<pipeline> made
config.sls raise "'str object' has no attribute 'get'", which failed the
whole logstash.config render rather than just skipping the bad value.
pipelines.yml.jinja already guarded this; config.sls now does too, and
logs which pipeline was ignored.
Add logstash:pipeline_settings carrying the 27 pipeline-scoped settings
Logstash 9.3.7 accepts, annotated individually per pipeline and rendered
into pipelines.yml. A blank setting inherits from logstash.yml. Restart
logstash when pipelines.yml changes, and add the missing managerhype
annotation.
Fixes#15090
so-boot-highstate.service was never enabled outside managers: only the
manager branch of so-setup called mark_setup_complete, so the marker its
service.enabled gates on never existed on sensors, search nodes, receivers,
etc.
Move the marker state into salt.minion.boot_highstate as the sole owner
within a highstate. Non-managers never apply salt.minion during setup, so
reaching it means setup is done and the marker is unconditional -- this also
heals already-installed nodes. Managers keep the legacy startup_states gate,
since they do highstate mid-setup.
Also add the marker to setup.virt for salt-cloud guests (replacing the
startup_states line removed in fabecb82) and to so-setup's non-manager branch.
"zeekctl cron" writes node statistics to /nsm/zeek/logs/stats. The CPU and memory
half comes from a helper that shells out to top, which the Zeek container does not
include. The helper's "command not found" output is then parsed as process data, so
every cron run appended a line per node reading "bad output from top", which
so-log-check reports.
Nothing wrote that file before, since log_stats and update_http_stats only run from
"zeekctl cron". Set StatsLogEnable to 0 so neither runs, and mark it read only since
the CPU and memory statistics cannot work with this image. The interface counters it
also collects are not used anywhere in Security Onion, which tracks Zeek packet loss
separately through packetloss.log and Telegraf, so nothing is lost by turning this
off. Note in StatsLogExpireInterval that it does nothing while the stats log is off.
MailTo, MailConnectionSummary and MailHostUpDown do nothing but send mail, and
the Zeek container has no mail program, so nothing they control can happen. Mark
them read only rather than offering knobs in SOC that cannot take effect.
MailConnectionSummary only gates the emailed copy; the connection summary is
generated and archived either way. MailTo also feeds Notice::mail_dest, but
Security Onion never enables the notice email action, so that half is inert too.
MailHostUpDown gates only the notification text - host status detection, the
plugin hook and the stored state all run regardless.
MinDiskSpace stays editable. It is not mail only: setting it to 0 skips the disk
space check entirely, and the warning it produces is not emailed but does appear
in the output of "zeekctl cron". Correct its description, and MailHostUpDown's,
which both claimed these settings have no visible effect.
LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval are only acted
on by "zeekctl cron", which nothing in the grid ran, so setting them in SOC did
nothing. Add so-zeek-cron and run it every 5 minutes, the interval upstream
recommends.
This also restarts a node that died unexpectedly and marks it crashed so a crash
report is written, which is what CrashExpireInterval then reaps.
The crontab runs as root because the script needs the docker socket; it drops to
the zeek user inside the container so the stats logs and zeekctl-config.sh it
writes stay owned by uid 937.
Annotate the five zeekctl settings that were previously undocumented. The regex
on LogExpireInterval matters: a bare number means days, and a value shorter than
LogRotationInterval raises ConfigurationError, which fails the zeekctl deploy in
the container entrypoint. Zeek then never starts while Salt still reports success
and the container still reports healthy. Excluding the min unit keeps that
unreachable at the default 3600 second rotation interval. MinDiskSpace and
MailHostUpDown only send mail and the image has no sendmail, so their
descriptions say they currently have no effect.
Josh Patterson
Jason Ertel
Jason Ertel
Jorge Reyes
Matthew Wright
Matthew Wright
Mike Reeves
Mike Reeves
coreyogburn
Corey Ogburn
Josh Brower