1457 Commits
Author SHA1 Message Date
Josh Patterson e5969a12aa Merge pull request #16109 from Security-Onion-Solutions/fixESHEAP
Add default esheap value so nodes highstate without the pillar value
2026-07-24 09:04:58 -04:00
Josh Patterson 6bd3c414bb Add default esheap value so nodes highstate without the pillar value
Previously salt/vars/elasticsearch.map.jinja read the elasticsearch heap
size via a raw pillar access (INIT.PILLAR.elasticsearch.esheap). If the
esheap key was missing from a minion pillar, building GLOBALS raised
'dict object has no attribute esheap', cascading into every state that
imports GLOBALS (elasticsearch.enabled, logstash.config, telegraf.config,
etc.) and blocking highstate entirely.

Add an esheap default ('600m', matching the es_heapsize() floor) to
elasticsearch/defaults.yaml and fall back to it via dict .get() so a
missing pillar value degrades gracefully. The per-node pillar value still
wins when present.
2026-07-23 15:16:26 -04:00
reyesj2 e4c14a9294 add gc.log to log4j2 delete policy 2026-07-22 16:12:24 -05:00
reyesj2 445ae58919 add optional dlm configs 2026-07-22 15:34:12 -05:00
reyesj2 382dee1d06 include global_override data_retention default for addon integrations that don't yet have a default value defined 2026-07-22 12:19:36 -05:00
reyesj2 f4aa9932ff add default templates for system integration metrics indices 2026-07-22 12:13:45 -05:00
reyesj2 8d168e9661 remove rollover config from ilm warm phase 2026-07-20 14:57:08 -05:00
reyesj2 b104f0955a es annotations 2026-07-20 14:33:25 -05:00
Josh Patterson f958212bea Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-07-15 15:07:09 -04:00
reyesj2 87a5639643 pipeline updates 2026-07-13 13:05:25 -05:00
reyesj2 99e9fc1c3b ES 9.3.7 2026-07-10 13:50:55 -05:00
reyesj2 5a57bbe4de add missing so-logs-soc annotation 2026-07-08 14:03:31 -05:00
Josh Patterson 33c24cd136 Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-06-26 15:42:56 -04:00
Josh Brower d0edfd2131 set transport for ssl.established:false logs 2026-06-25 14:18:43 -04:00
Josh Patterson d71e80cf66 Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-06-23 10:32:32 -04:00
reyesj2 96fcc0ec38 wip 2026-06-22 14:25:46 -05:00
Josh Patterson 8e33d0e1e9 Merge remote-tracking branch 'origin/3/dev' into soupmod2 2026-06-16 12:54:18 -04:00
reyesj2 a769d4c680 another unneeded default 2026-06-16 09:32:37 -05:00
reyesj2 f68e3e47a1 remove pillar merge 2026-06-16 09:19:10 -05:00
reyesj2 95cae4c734 remove so-elasticsearch-indices-delete cron when using DLM 2026-06-15 13:32:45 -05:00
reyesj2 596471e140 using new annotation config 2026-06-15 13:31:53 -05:00
reyesj2 d10f21399c remove comments 2026-06-15 13:31:23 -05:00
Josh Patterson 43f72c1f9f Parallelize so-elasticsearch-templates-load template PUTs
Load component and index templates as throttled background jobs (max 10
concurrent) instead of sequential curl PUTs, matching the bounded-concurrency
+ flock-serialized-output pattern used by the fleet/ILM load scripts. Keeps a
wait barrier between the component phase and the index phase so index
templates never load before their referenced component templates. Failures are
tracked via per-job marker files since counter increments can't escape
background subshells.
2026-06-12 15:11:34 -04:00
reyesj2 c505160480 set default DLM retention 90d 2026-06-11 15:13:28 -05:00
reyesj2 d9f6cde4e1 remove global setting from data_retention annotation 2026-06-11 15:11:29 -05:00
Josh Patterson 6c42c419e2 Serialize ILM policy-load output with flock to stop interleaving
A single printf per block was not actually one write() call, so
concurrent jobs still occasionally interleaved their label and response
lines. Hold an flock around just the printf (curl still runs in
parallel) so each policy's block prints intact, keeping live
completion-order streaming.
2026-06-11 15:42:41 -04:00
Josh Patterson 07d3b148b5 fix output 2026-06-11 13:37:26 -04:00
Josh Patterson 780d9faf0d Parallelize so-elasticsearch-ilm-policy-load PUTs
Run the ~300 ILM policy PUTs concurrently (bounded to 10 in flight via a
throttle gate) instead of one serial curl per policy. Adds a put_policy
helper and waits for all background jobs before exiting. Preserves policy
parity; only the scheduling changes. Drops the dead empty sid cookie arg
(falls back to basic auth from curl.config as before).
2026-06-11 12:08:32 -04:00
Josh Patterson 33a116357d Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-06-10 08:56:17 -04:00
reyesj2 cf456dc58c reuse existing index templates 2026-06-09 23:21:43 -05:00
reyesj2 9aa9ea3255 Iniitial DLM support 2026-06-09 23:19:26 -05:00
reyesj2 ac907ba45f fix elasticsearch template generation issue 2026-06-05 16:42:08 -05:00
Josh Patterson b4e5171415 Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-05-14 08:03:45 -04:00
Jorge Reyes 638aca97c8 Merge pull request #15877 from Security-Onion-Solutions/reyesj2-patch-1
update redis index template
2026-05-13 13:44:04 -05:00
Josh Patterson 84decc1db6 Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-05-13 14:09:15 -04:00
reyesj2 d56bf01823 add zeek.ja4d ingest pipeline 2026-05-13 12:32:54 -05:00
reyesj2 492ae80da7 add ingest latency metrics 2026-05-11 16:51:38 -05:00
Jorge Reyes 4a2177c827 update redis index template
missing redis integration component templates
2026-05-11 16:15:56 -05:00
Josh Brower 499f7102bd cleanup status code 2026-05-07 11:27:49 -04:00
Josh Patterson f774334b6c Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-05-06 08:16:41 -04:00
reyesj2 dceed421ae update grok type conversion to convert processor 2026-05-05 13:41:00 -05:00
Josh Patterson 034711d148 Merge remote-tracking branch 'origin/3/dev' into saltthangs 2026-04-28 10:47:29 -04:00
reyesj2 b6acf3b522 typo 2026-04-24 09:24:58 -05:00
reyesj2 fdfca469cc prevent non-manager nodes from running elasticsearch.cluster state manually 2026-04-23 09:53:07 -05:00
reyesj2 22f869734e add check for files before attempting to use file pattern to load templates 2026-04-22 23:11:31 -05:00
reyesj2 72dbb69a1c fix searchnodes running elasticsearch/cluster state 2026-04-22 20:37:48 -05:00
reyesj2 ebb93b4fa7 add wait_for_so-elasticsearch state and split elasticsearch cluster configuration out of enabled.sls 2026-04-17 14:43:07 -05:00
reyesj2 ba00ae8a7b supress noisy warning from ES 9.3.3 2026-04-16 14:41:25 -05:00
reyesj2 16a4a42faf check for addon-index templates dir before attempting to load addon index templates 2026-04-14 19:26:37 -05:00
reyesj2 a232cd89cc ES 9.3.3 2026-04-13 13:36:51 -05:00