Auto State Apply fires on SOC config saves and on suricata/strelka rule
updates. Files a user creates or edits by hand under
/opt/so/saltstack/local/salt/ change no pillar, so nothing fired and the
change waited for the next scheduled highstate, now 120 minutes by default.
That gap is the 3.2 Known Issue in the docs.
Watch the directories the docs tell users to edit, and route them through
the push pipeline that already exists:
zeek/policy -> zeek (covers intel/ and custom/)
zeek/zkg -> zeek
elasticsearch/files/ingest -> elasticsearch
elasticsearch/roles -> elasticsearch
logstash/pipelines/config/custom -> logstash
Tags are pillar_push_map.yaml app names, so the existing entries already
carry the right state and compound target, and no map entry changes.
Rename the beacon rules_beacon -> local_files_beacon. Rules are now one of
five kinds of file it watches, and the new name matches how its sibling
postgres_pillar_beacon is named: source, then what it watches.
Replace push_suricata.sls and push_strelka.sls with one push_files.sls bound
to salt/beacon/*/local_files_beacon/*, which looks the tag up in
pillar_push_map.yaml the same way push_pillar.sls does. The map's suricata
and strelka targets match the compounds those two reactors hardcoded, so
rule pushes are unchanged. The app comes from the event tag rather than the
payload because salt's beacon loop pops the beacon's tag key off the data.
Key watermarks by watched directory instead of by tag. zeek/policy and
zeek/zkg both emit the tag zeek, and a shared watermark would make them
overwrite each other's digest and emit on every poll.
Prune .git from the fingerprint walk. zkg packages must be git clones with a
clean working tree, so the watched tree carries full git metadata; walking it
every 15s is wasted work and git's own index and ref mtime churn would fire a
grid-wide zeek apply on its own. Placing or updating a package always touches
working-tree files too, so detection is unaffected.
The watch is an allowlist rather than the whole local salt tree because salt
writes into that tree itself: hypervisor/hosts/ is rewritten continuously by
virtual_node_manager.py and virtual_power_manager.py, libvirt/images/ holds
multi-GB qcow2 files, and elasticfleet/files/so_agent-installers/,
elasticsearch/files/users, ca/files/ and filebeat/files/ are all state-written.
Watching any of them would either self-retrigger or make the 15s poll walk
gigabytes.
Telegraf 1.39 deprecated bare string entries in inputs.exec commands and
will drop them in 1.45, warning on every start:
W! DeprecationWarning: Value "/scripts/esindexsize.sh" for option
"command" of plugin "inputs.exec" deprecated since version 1.39.0
Each entry is now a single-element argv array. Array form skips shell
parsing, which is fine here: every command is a bare script path from
telegraf's scripts list, no args or shell metacharacters.
The UEK8 rollout installs the new kernel and flips the boot default, but
leaves the stock EL9 (RHCK) packages behind: disk in /boot and a stale
GRUB entry on every upgraded node.
They cannot be removed in the same pass that installs UEK8. dnf's
protect_running_kernel refuses to erase the booted kernel-core, so the
removal has to wait until the node has rebooted onto 6.x. Waiting is the
safer sequencing anyway -- the node proves it comes up on UEK8 before its
fallback is deleted -- so this does not remove RHCK from the uek7 branch
either, where dnf would allow it.
so-kernel-upgrade grows a --cleanup mode that does only the removal and
no-ops (exit 0, with a log line) on a node not yet running UEK8. Its uek8
branch, which previously reported "nothing to do", now runs that cleanup
along with set_default_kernel_conf -- which also closes a gap where a node
that came up on UEK8 straight from a fresh install never had
DEFAULTKERNEL=kernel-uek-core written.
The common highstate calls --cleanup gated on the running kernel, so the
cleanup lands grid-wide as each node reboots: fresh installs reboot at the
end of setup, upgraded nodes whenever the admin schedules it. The rpm
check inside the script is the idempotency guard, so subsequent highstates
cost an rpm query rather than a dnf transaction, and the package list is
not duplicated into the state where it could drift.
Added `memoryScanIntervalSeconds` with a default of 5 mins.
Opted to set `useMemoryScanner` to false so by default our user's sessions are not sent to the cloud before they have a chance to configure the new setting.
The Logstash log level was hardcoded to info in log4j2.properties, and
logstash.yml carried no log.level key, so the only way to raise verbosity
for troubleshooting was to hand-edit a file that the next highstate
overwrites. Add log_x_level and log_x_format to logstash:config so both
render into logstash.yml, annotated as advanced per-node settings with
the value sets Logstash 9.3.7 accepts.
log4j2.properties gains jinja, so it moves to log4j2.properties.jinja and
is rendered by a discrete lslog4j2 state rather than the lsetcsync
recurse, which cannot rename. The recurse exclude_pat now matches both
names so it neither copies the template verbatim nor lets clean: True
delete the rendered file, matching how pipelines.yml is already handled.
The appender layout is selected at render time so log.format actually
changes the log output instead of being a dead setting, keeping the
existing file name so nothing downstream moves. rootLogger.level now
follows ls.log.level rather than claiming info regardless of the
configured level.
Grid testing every permitted value on the manager pipeline surfaced two
combinations the UI allows that take the pipeline down, neither of which
the descriptions mentioned.
pipeline.ordered: true requires pipeline.workers: 1; with more workers the
pipeline fails to start with "enabling the 'pipeline.ordered' setting
requires the use of a single pipeline worker". Also correct the auto
wording: it only engages when workers is explicitly set to 1.
queue.max_bytes larger than the free space on /nsm/logstash fails queue
creation with "Unable to allocate N more bytes", rather than merely being
inadvisable.
Widen the byte-size regex, which rejected values Logstash accepts and so
blocked the save in SOC: bare-letter units (1g, 512m, 64k), decimals
(1.5gb), whitespace before the unit, and a bare integer. Allow whitespace
in dead_letter_queue.retain.age (5 d). Both stay lowercase-only, matching
byte_value.rb and AbstractPipelineExt.parseToDuration.
Fix description gaps: queue.checkpoint.retry is a Windows/SAN workaround
Elastic does not otherwise recommend, batch metrics sampling is technical
preview, queue.checkpoint.interval is deprecated in 9.1, compression makes
a queue unreadable by Logstash before 9.2, flush_check_interval has a
1000ms floor, max_events counts unread events, and the path settings are
created by Logstash but reject symlinks. Note which settings apply only to
persisted queues or an enabled DLQ.
Drop the undocumented 'disabled' value from queue.compression.
Numeric fields stay stricter than NumericSetting, which has no validator
and would accept negatives, floats and NaN in event counts and intervals.
Jorge Reyes
Josh Patterson
Jason Ertel
Jason Ertel
Matthew Wright
Matthew Wright
Mike Reeves
Mike Reeves
coreyogburn
Corey Ogburn