Commit Graph
18934 Commits
Author SHA1 Message Date
Mike Reeves 4dbea0e28d Merge pull request #16403 from Security-Onion-Solutions/TOoSmOotH/568
INTERNAL: Trim Telegraf collection for Postgres
2026-11-17 16:58:20 -05:00
Mike Reeves 6d7be47a69 Trim Telegraf collection on Postgres-only grids and add Postgres support to Elasticsearch growth scripts 2026-11-17 14:17:27 -05:00
Josh Patterson e51706acad Merge pull request #16399 from Security-Onion-Solutions/fix/auto-apply-push-result-unknown
FIX: queue auto state apply pushes
2026-11-17 10:53:39 -05:00
Jorge Reyes 9612b15fca Merge pull request #16400 from Security-Onion-Solutions/reyesj2-patch-2
remove unused localrules config
2026-11-16 14:58:56 -06:00
reyesj2 0f5170e17e soup missing closing } post merge resolution; 2026-11-16 14:19:22 -06:00
reyesj2 cac027d438 fix merge 2026-11-16 13:52:05 -06:00
Jorge Reyes dea6932342 Merge branch '3/dev' into reyesj2-patch-2 2026-11-16 13:50:14 -06:00
Mike Reeves 043b2185d6 Merge pull request #16398 from Security-Onion-Solutions/foundry/telegraf-partition-11tqi
Postgres: Telegraf partition maintenance over the socket, with grid-test fixes
2026-11-16 14:47:37 -05:00
reyesj2 e7bb00d19d move /nsm/rules ownership into soc/config.sls. nginx no longer serves /nsm/rules, but so-soc still mounts it read-write 2026-11-16 13:47:25 -06:00
reyesj2 8c3611ac72 only rewrite pillar if it has localrules reference(s) 2026-11-16 13:46:10 -06:00
reyesj2 5558c54632 so-yaml: treat non-mapping parents as missing keys
getKeyValue and removeKey now return early when a parent of the key is
null or not a mapping, so get, remove and purgelistitem report the key as
missing instead of crashing on a hand-edited pillar such as
`firewall: null`.
2026-11-16 13:44:58 -06:00
Josh Patterson 9045aff88d FIX: correct the batch default in the pillar_push_map.yaml comment
The comment gave the orch default batch as 25%. salt/salt/defaults.yaml
sets salt:auto_apply:batch to 10%.
2026-11-16 14:19:51 -05:00
Josh Patterson cc9dd0a101 FIX: drop malformed push_dispatched records instead of failing the pass
so-push-drainer computed a record's age before the per-record
try/except. A record whose dispatched_at or checked_at was not a number
raised TypeError there, so every pass exited before the drain and
auto-apply stopped. The traceback went to the stderr of a schedule with
return_job: False, so nothing reached so-push-drainer.log.

Only the drainer writes these records, so this takes a hand-edited file,
but result tracking is not supposed to be able to block the drain. Log a
warning and remove the record, as is already done for unreadable JSON
and a missing jid.

Also reword the so-log-check exclusion comment, which said the queued
state run "still completes" where the drainer only claims it may have,
and note at the drainer's warning that so-log-check matches its text.
2026-11-16 14:19:51 -05:00
Mike Reeves 52f9fe3092 Remove blank line in soup script 2026-11-16 13:52:14 -05:00
Mike Reeves f492e87f20 Add check for post-upgrade version file 2026-11-16 12:12:11 -05:00
Mike Reeves 01518ec25e Fix formatting issues in soup script 2026-11-16 11:14:06 -05:00
Jorge Reyes 5f3c1c84a4 Merge pull request #16396 from Security-Onion-Solutions/reyesj2-patch-3
elastic artifact repo only needs /beats/ dir
2026-11-16 09:22:23 -06:00
Josh Patterson 70053e362c FIX: exclude the drainer's push-result-unknown warning from so-log-check
so-push-drainer logs "push result unknown" as a WARNING when salt-master
restarts while a pushed state run is queued: the orchestration's next
publish fails with AuthenticationError, but the minion still runs the
queued state. so-log-check matches any line containing "error", so the
quoted exception still failed the check.

Reproduced on a distributed 3.5.0 grid: a sensoroni push queued behind a
150s state run on the manager, then salt-master was restarted. The
orchestration (which runs under salt-minion.service, so it survives the
restart) raised AuthenticationError 9s later, the drainer logged the
WARNING, and the queued state run completed with 0 failed states.
so-log-check --unknown flagged the line.

The pattern requires AuthenticationError so an unknown result from any
other raised exception, e.g. a bug in a state module, is still reported.
2026-11-16 10:20:05 -05:00
Mike Reeves b01f01da82 Fix formatting issues in soup script 2026-11-13 16:07:55 -05:00
Mike Reeves 03317ba5e4 Postgres: Telegraf partition maintenance over the socket, with grid-test fixes 2026-11-12 20:42:29 -05:00
reyesj2 9bc3ae6b26 further tighten elastic artifact registry 2026-11-12 15:49:34 -06:00
Jorge Reyes 40a30f6908 Merge pull request #16389 from Security-Onion-Solutions/reyesj2-patch-1
update elastic-agent artifact repo directory
2026-11-11 14:32:59 -06:00
Josh Patterson cc970892ed FIX: keep the app's target when pushing a per-node setting
For a setting saved for a single node, push_pillar replaced the app's
target from pillar_push_map.yaml with the node id. The app's state was
then applied to that node even if the node does not run the app.

Seen in the lab: the SOC config test saves
elasticsearch.config.cluster.routing.allocation.disk.threshold_enabled
for the first node SOC lists, and SOC lists nodes in random order. When
that node was a sensor, the elasticsearch state was applied to it and
failed to render (so_roles has no so-sensor entry), and so-push-drainer
logged "push failed".

Narrow the app's target to the node instead of replacing it:
"L@<node> and ( <app target> )". A node outside the app's target then
matches nothing, and so-push-drainer skips the action.
2026-11-11 10:07:08 -05:00
reyesj2 4e0558d2d3 remove unused localrules config 2026-11-10 17:30:01 -06:00
Josh Patterson 22a4acbb5a FIX: skip push actions whose target matches no minions
so-push-drainer dispatched every queued action, even when its target
matched no accepted minion. The orchestration's saltutil.refresh_pillar
step then failed with "No minions responded", and the drainer logged a
push failure that so-log-check reports as an ERROR.

Seen on a fresh distributed manager: the first drain after setup sent
strelka and suricata rule pushes targeting sensor roles before any
sensor had joined the grid. Nothing had failed; no node held the role.

Before dispatch, resolve each action's target with the master's
CkMinions (accepted keys and cached grains) and drop the ones that
match nothing, logging each at INFO. If none remain, clear the intents
without dispatching. Uncached minions still count as a match, and a
lookup error keeps the action, so pushes are never dropped when the
answer is uncertain.
2026-11-10 15:26:37 -05:00
Mike Reeves 1fd37251c9 Merge pull request #16390 from Security-Onion-Solutions/foundry/mwaq014injcipm
Telegraf: fix EPS double counting and duplicate metric rows
2026-11-10 14:53:30 -05:00
Mike Reeves 4a5253f688 Telegraf: fix EPS double counting and duplicate metric rows 2026-11-10 08:26:19 -05:00
reyesj2 ec220572f0 dedup artifact registry config & split ea artifacts out of /opt/socore/html/ into dedicated /opt/socore/elastic-artifacts/artifacts/ 2026-11-09 12:32:06 -06:00
Josh Patterson 9e8592e220 Merge remote-tracking branch 'origin/3/dev' into fix/auto-apply-push-result-unknown
# Conflicts:
#	salt/manager/tools/sbin/so-push-drainer
#	salt/manager/tools/sbin/so-push-drainer_test.py
2026-11-06 09:50:25 -05:00
Jorge Reyes b7b00f8233 Merge pull request #16378 from Security-Onion-Solutions/reyesj2/fix-16369
surface ilm policy errors in so-elasticsearch-ilm-policy-load output
2026-11-05 14:05:29 -06:00
Jorge Reyes 45d2b29a42 Merge pull request #16374 from Security-Onion-Solutions/fix/skip-mismatched-iso-registry
Skip leftover ISO registry.tar when tags do not match SOVERSION
2026-11-05 11:11:45 -06:00
Doug Burks 7b494439e7 Merge pull request #16377 from Security-Onion-Solutions/feature/desktop-so-crates
Install securityonion-so-crates on Security Onion Desktop
2026-11-04 15:56:12 -05:00
Doug Burks 9a8dc097b2 Install securityonion-so-crates on Security Onion Desktop 2026-11-04 15:51:16 -05:00
reyesj2 274da18929 Surface ILM policy validation errors in highstate
Fail 4xx ILM PUTs immediately (retry 429/5xx/connect), print the
Elasticsearch reason in the salt summary, and retry the load on the
next highstate via /opt/so/state/esilm.txt when the last run failed.

Fixes https://github.com/Security-Onion-Solutions/sos/issues/277
2026-11-04 13:18:31 -06:00
reyesj2 99a8ad2d4b Wait for Elasticsearch and retry ILM policy loads
Require wait_for_so-elasticsearch, check ES is responsive before
dispatch, and retry each policy PUT so soup no longer fails with
empty output when ES is not ready yet.

Fixes https://github.com/Security-Onion-Solutions/securityonion/issues/16369
2026-11-04 11:14:12 -06:00
Mike Reeves bd9cdeadef Merge pull request #16373 from Security-Onion-Solutions/foundry/mw1jl4boa6hshh
Remove x86-64-v3 CPU pre-flight check from soup
2026-11-03 22:50:54 -05:00
Mike Reeves bdf01a5a5a Remove x86-64-v3 CPU check from soup 2026-11-03 22:13:58 -05:00
reyesj2 1849c74cea Skip leftover ISO registry.tar when tags do not match SOVERSION
Fixes #16372
2026-11-03 21:07:29 -06:00
Mike Reeves 021b069706 Merge pull request #16367 from Security-Onion-Solutions/TOoSmOotH-patch-1
Bump version from 3.4.0 to 3.5.0
2026-11-02 11:52:06 -05:00
Mike Reeves 824d674eec Add version 3.5.0 to discussion template 2026-11-02 11:50:49 -05:00
Mike Reeves d43791c63e Bump version from 3.4.0 to 3.5.0 2026-11-02 11:50:08 -05:00
Mike Reeves 4dc7f2d97c Merge pull request #16365 from Security-Onion-Solutions/3.4
3.4.0
2026-11-02 10:52:49 -05:00
Mike Reeves 092be7f222 3.4.0 2026-11-02 10:46:19 -05:00
Mike Reeves 340e7928f6 3.4.0 2026-10-30 11:36:47 -04:00
Jason Ertel f5ff40bd20 Merge pull request #16361 from Security-Onion-Solutions/jertel/wip
switch analyzer deps to py 314
2026-10-29 16:15:29 -04:00
Jason Ertel 7c271ea9c1 switch analyzer deps to py 314 2026-10-29 16:06:43 -04:00
coreyogburn 6794af6f68 Merge pull request #16358 from Security-Onion-Solutions/cogburn/sync-timeouts
New Configurable Timeouts for Detection Engine Syncs
2026-10-28 11:34:02 -06:00
Corey Ogburn 691bef2cf4 New Configurable Timeouts for Detection Engine Syncs
Each engine now has timeouts for how long to allow a sync to go on after detections are modified.
2026-10-28 10:48:03 -06:00
Mike Reeves 1bbaa213cb Merge pull request #16354 from Security-Onion-Solutions/mreeves/remove-es-sokeys 2026-10-27 17:36:48 -04:00
Mike Reeves 5342765ca8 Merge pull request #16357 from Security-Onion-Solutions/mreeves/host-ca-trust
Use the host CA trust store, with the grid CA, for all containers
2026-10-27 16:40:58 -04:00