The Logstash log level was hardcoded to info in log4j2.properties, and
logstash.yml carried no log.level key, so the only way to raise verbosity
for troubleshooting was to hand-edit a file that the next highstate
overwrites. Add log_x_level and log_x_format to logstash:config so both
render into logstash.yml, annotated as advanced per-node settings with
the value sets Logstash 9.3.7 accepts.
log4j2.properties gains jinja, so it moves to log4j2.properties.jinja and
is rendered by a discrete lslog4j2 state rather than the lsetcsync
recurse, which cannot rename. The recurse exclude_pat now matches both
names so it neither copies the template verbatim nor lets clean: True
delete the rendered file, matching how pipelines.yml is already handled.
The appender layout is selected at render time so log.format actually
changes the log output instead of being a dead setting, keeping the
existing file name so nothing downstream moves. rootLogger.level now
follows ls.log.level rather than claiming info regardless of the
configured level.
Grid testing every permitted value on the manager pipeline surfaced two
combinations the UI allows that take the pipeline down, neither of which
the descriptions mentioned.
pipeline.ordered: true requires pipeline.workers: 1; with more workers the
pipeline fails to start with "enabling the 'pipeline.ordered' setting
requires the use of a single pipeline worker". Also correct the auto
wording: it only engages when workers is explicitly set to 1.
queue.max_bytes larger than the free space on /nsm/logstash fails queue
creation with "Unable to allocate N more bytes", rather than merely being
inadvisable.
Widen the byte-size regex, which rejected values Logstash accepts and so
blocked the save in SOC: bare-letter units (1g, 512m, 64k), decimals
(1.5gb), whitespace before the unit, and a bare integer. Allow whitespace
in dead_letter_queue.retain.age (5 d). Both stay lowercase-only, matching
byte_value.rb and AbstractPipelineExt.parseToDuration.
Fix description gaps: queue.checkpoint.retry is a Windows/SAN workaround
Elastic does not otherwise recommend, batch metrics sampling is technical
preview, queue.checkpoint.interval is deprecated in 9.1, compression makes
a queue unreadable by Logstash before 9.2, flush_check_interval has a
1000ms floor, max_events counts unread events, and the path settings are
created by Logstash but reject symlinks. Note which settings apply only to
persisted queues or an enabled DLQ.
Drop the undocumented 'disabled' value from queue.compression.
Numeric fields stay stricter than NumericSetting, which has no validator
and would accept negatives, floats and NaN in event counts and intervals.
A non-mapping value under logstash:pipeline_settings:<pipeline> made
config.sls raise "'str object' has no attribute 'get'", which failed the
whole logstash.config render rather than just skipping the bad value.
pipelines.yml.jinja already guarded this; config.sls now does too, and
logs which pipeline was ignored.
Add logstash:pipeline_settings carrying the 27 pipeline-scoped settings
Logstash 9.3.7 accepts, annotated individually per pipeline and rendered
into pipelines.yml. A blank setting inherits from logstash.yml. Restart
logstash when pipelines.yml changes, and add the missing managerhype
annotation.
Fixes#15090
- schedule highstate every 2 hours (was 15 minutes); interval lives in
global:push:highstate_interval_hours so the SOC admin UI can tune it and
so-salt-minion-check derives its threshold as (interval + 1) * 3600
- add inotify beacon on the manager + master reactor + orch.push_batch that
writes per-app intent files, with a so-push-drainer schedule on the manager
that debounces, dedupes, and dispatches a single orchestration
- pillar_push_map.yaml allowlists the apps whose pillar changes trigger an
immediate targeted state.apply (targets verified against salt/top.sls);
edits under pillar/minions/ trigger a state.highstate on that one minion
- host-batch every push orchestration (batch: 25%, batch_wait: 15) so rule
changes don't thundering-herd large fleets
- new global:push:enabled kill-switch tears down the beacon, reactor config,
and drainer schedule on the next highstate for operators who want to keep
highstate-only behavior
- set restart_policy: unless-stopped on 23 container states so docker
recovers crashes without waiting for the next highstate; leave registry
(always), strelka/backend (on-failure), kratos, and hydra alone with
inline comments explaining why
Add ulimits as a configurable advanced setting for every container,
allowing customization through the web UI. Move hardcoded ulimits
from elasticsearch and zeek into defaults.yaml and fix elasticsearch
ulimits that were incorrectly nested under the environment key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Josh Patterson
Josh Brower
reyesj2
Mike Reeves
Doug Burks
Mike ReevesandClaude Opus 4.6
Jason Ertel