Annotate the soauth network leaves instead of the networks map

SOC reported "Malformed config setting (docker.networks.soauth.range): Setting name 'networks' conflicts with another similarly named setting" and refused to render the config tree. soc_docker.yaml annotated docker.networks itself, and every key in that block was a scalar, so FlattenAnnotations registered docker.networks as a setting. defaults.yaml holds a nested map there, so FlattenPillar separately registered docker.networks.soauth.range, .gateway and .manager_only, and HydrateAnnotations unions the two sets. The config tree gives each id segment either a value or children, never both, so addToNode pushed docker.networks as a childless leaf and then threw when docker.networks.soauth.range tried to descend through it. The annotations move down to the three leaves that actually carry values, so docker.networks is a plain branch. This matches docker.containers, which is nested the same way and has never carried annotations of its own. docker.ulimits and the per-container networks list are unaffected: their pillar values are lists rather than maps, so they stay single settings.
Mention Hydra in the soauth network description
2026-09-30 19:47:17 +02:00 · 2026-09-18 13:40:05 -04:00 · 2026-09-18 10:17:20 -04:00 · 2026-09-17 16:58:49 -04:00 · 2026-09-09 16:09:00 -04:00 · 2026-09-08 15:15:12 -04:00
51 changed files with 1119 additions and 532 deletions
@@ -13,6 +13,7 @@ body:
        - 3.1.0
        - 3.2.0
        - 3.3.0
+        - 3.4.0
        - Other (please provide detail below)
    validations:
      required: true
@@ -1,17 +1,17 @@
-### 3.2.0-20260729 ISO image released on 2026/07/29
+### 3.3.0-20260908 ISO image released on 2026/09/08


 ### Download and Verify

-3.2.0-20260729 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-3.2.0-20260729.iso
+3.3.0-20260908 ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260908.iso
 
-MD5: B1E10F46DF872B655C29325DF965A4DB  
-SHA1: 0F3C7ED80F6D326B7A993C2F899B986320C01BF9  
-SHA256: 7465163C1D1ADFCDC3935530EAFB312E987C016941ADC11841B214553314D1FF  
+MD5: 5A2C42D0083F2D7B4DC2178C30EBC05F  
+SHA1: 505220A8A3315AFEE601C13772996018425CCD29  
+SHA256: 6EB8401296A1D051FEC558C520D2D4AB1912A72D351A2426FBF8C87FEE2BA844  

 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.2.0-20260729.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260908.iso.sig

 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/main/KEYS  
@@ -25,22 +25,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/

 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.2.0-20260729.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260908.iso.sig
 ```

 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-3.2.0-20260729.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260908.iso
 ```

 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-3.2.0-20260729.iso.sig securityonion-3.2.0-20260729.iso
+gpg --verify securityonion-3.3.0-20260908.iso.sig securityonion-3.3.0-20260908.iso
 ```

 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Tue 28 Jul 2026 06:17:34 PM EDT using RSA key ID FE507013
+gpg: Signature made Tue 08 Sep 2026 10:07:12 AM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
@@ -1 +1 @@
-3.3.0
+3.4.0
@@ -3,13 +3,14 @@
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.

-# Custom salt beacon that watches the suricata/strelka rule directories for changes
-# and emits a beacon event per changed directory. This replaces the stock salt
-# `inotify` beacon, which leaks a kernel inotify instance every time the minion
-# rebuilds the beacon loader's __context__ (orphaning the old pyinotify.Notifier
-# without closing it) until fs.inotify.max_user_instances is exhausted and the
-# beacon dies with EMFILE. Polling holds zero inotify instances, so the leak is
-# impossible, and it keeps firing during state runs (no blackout).
+# Custom salt beacon that watches hand-edited directories under
+# /opt/so/saltstack/local/salt/ for changes and emits a beacon event per changed
+# directory. This replaces the stock salt `inotify` beacon, which leaks a kernel
+# inotify instance every time the minion rebuilds the beacon loader's __context__
+# (orphaning the old pyinotify.Notifier without closing it) until
+# fs.inotify.max_user_instances is exhausted and the beacon dies with EMFILE.
+# Polling holds zero inotify instances, so the leak is impossible, and it keeps
+# firing during state runs (no blackout).
 #
 # Detection is poll-based with a per-directory fingerprint persisted to
 # WATERMARK_DIR: each pass walks the directory and hashes every file's
@@ -19,9 +20,10 @@
 # up on the next one).
 #
 # Each emitted event carries the watched directory path under the configured tag
-# (e.g. salt/beacon/<minion>/rules_beacon/suricata); the push_suricata / push_strelka
-# reactors write a push intent, after which the existing so-push-drainer /
-# orch.push_batch pipeline takes over unchanged.
+# (e.g. salt/beacon/<minion>/local_files_beacon/zeek); the push_files reactor
+# looks the tag up in salt/reactor/pillar_push_map.yaml and writes a push intent,
+# after which the existing so-push-drainer / orch.push_batch pipeline takes over
+# unchanged.

 import hashlib
 import logging
@@ -77,7 +79,9 @@ def _fingerprint(directory):
    h = hashlib.sha1()
    if os.path.isdir(directory):
        entries = []
-        for root, _dirs, files in os.walk(directory):
+        for root, dirs, files in os.walk(directory):
+            # zkg packages are git clones; .git churn would fire a state apply on its own.
+            dirs[:] = [d for d in dirs if d != '.git']
            for name in files:
                full = os.path.join(root, name)
                if _excluded(full):
@@ -94,20 +98,22 @@ def _fingerprint(directory):
    return h.hexdigest()


-def _watermark_file(tag):
-    return os.path.join(WATERMARK_DIR, 'rules_beacon_%s.hash' % tag)
+def _watermark_file(tag, directory):
+    # Keyed by directory: zeek/policy and zeek/zkg share the tag `zeek`.
+    scope = hashlib.sha1(directory.encode('utf-8', 'surrogateescape')).hexdigest()[:12]
+    return os.path.join(WATERMARK_DIR, 'local_files_beacon_%s_%s.hash' % (tag, scope))


-def _read_watermark(tag):
+def _read_watermark(tag, directory):
    try:
-        with open(_watermark_file(tag), 'r') as f:
+        with open(_watermark_file(tag, directory), 'r') as f:
            return (f.read() or '').strip() or None
    except IOError:
        return None


-def _write_watermark(tag, digest):
-    path = _watermark_file(tag)
+def _write_watermark(tag, directory, digest):
+    path = _watermark_file(tag, directory)
    try:
        os.makedirs(WATERMARK_DIR, exist_ok=True)
        tmp = path + '.tmp'
@@ -115,7 +121,7 @@ def _write_watermark(tag, digest):
            f.write(digest)
        os.rename(tmp, path)
    except OSError:
-        log.exception('rules_beacon: failed to persist watermark to %s', path)
+        log.exception('local_files_beacon: failed to persist watermark to %s', path)


 def beacon(config):
@@ -123,17 +129,17 @@ def beacon(config):

    for directory, tag in _paths_from_config(config).items():
        digest = _fingerprint(directory)
-        previous = _read_watermark(tag)
+        previous = _read_watermark(tag, directory)

        # First run / missing watermark: seed the digest and emit nothing so a
        # fresh host does not fire a spurious fleetwide push.
        if previous is None:
-            _write_watermark(tag, digest)
+            _write_watermark(tag, directory, digest)
            continue

        if digest != previous:
-            _write_watermark(tag, digest)
+            _write_watermark(tag, directory, digest)
            retval.append({'tag': tag, 'path': directory})
-            log.info('rules_beacon: change detected in %s, emitting %s', directory, tag)
+            log.info('local_files_beacon: change detected in %s, emitting %s', directory, tag)

    return retval
@@ -0,0 +1,221 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+import hashlib
+import os
+import shutil
+import tempfile
+import unittest
+from unittest.mock import patch
+
+import local_files_beacon
+
+
+class TestRulesBeacon(unittest.TestCase):
+
+    def setUp(self):
+        # Isolate all on-disk state (watermarks and the dirs we fingerprint) in a
+        # throwaway tree, and point WATERMARK_DIR at it so the real read/write
+        # helpers run against actual files.
+        self.tmpdir = tempfile.mkdtemp()
+        self.state = os.path.join(self.tmpdir, 'state')
+        patcher = patch.object(local_files_beacon, 'WATERMARK_DIR', self.state)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+
+    def tearDown(self):
+        shutil.rmtree(self.tmpdir, ignore_errors=True)
+
+    def _make_dir(self, name, files=None):
+        path = os.path.join(self.tmpdir, name)
+        os.makedirs(path, exist_ok=True)
+        for fname, content in (files or {}).items():
+            with open(os.path.join(path, fname), 'w') as f:
+                f.write(content)
+        return path
+
+    # -- trivial contract -------------------------------------------------
+
+    def test_virtual_returns_true(self):
+        self.assertTrue(local_files_beacon.__virtual__())
+
+    def test_validate_returns_valid(self):
+        self.assertEqual(local_files_beacon.validate({}), (True, 'valid'))
+
+    # -- _paths_from_config -----------------------------------------------
+
+    def test_paths_from_config_list_of_dicts(self):
+        config = [{'interval': 10}, {'paths': {'/a': 'suricata', '/b': 'strelka'}}]
+        self.assertEqual(
+            local_files_beacon._paths_from_config(config),
+            {'/a': 'suricata', '/b': 'strelka'},
+        )
+
+    def test_paths_from_config_plain_dict(self):
+        self.assertEqual(
+            local_files_beacon._paths_from_config({'paths': {'/a': 'suricata'}}),
+            {'/a': 'suricata'},
+        )
+
+    def test_paths_from_config_skips_non_dict_items(self):
+        self.assertEqual(local_files_beacon._paths_from_config(['bogus', 42]), {})
+
+    def test_paths_from_config_paths_not_a_dict(self):
+        self.assertEqual(local_files_beacon._paths_from_config({'paths': 'nope'}), {})
+
+    def test_paths_from_config_unexpected_type(self):
+        self.assertEqual(local_files_beacon._paths_from_config('nonsense'), {})
+
+    # -- _excluded --------------------------------------------------------
+
+    def test_excluded_matches_temp_and_editor_files(self):
+        for pathname in ('/rules/foo.swp', '/rules/foo~', '/rules/4913', '/rules/.#foo'):
+            self.assertTrue(local_files_beacon._excluded(pathname), pathname)
+
+    def test_excluded_allows_real_rule_files(self):
+        self.assertFalse(local_files_beacon._excluded('/rules/suricata.rules'))
+
+    # -- _fingerprint -----------------------------------------------------
+
+    def test_fingerprint_missing_dir_is_empty_tree_digest(self):
+        missing = os.path.join(self.tmpdir, 'does-not-exist')
+        self.assertEqual(local_files_beacon._fingerprint(missing), hashlib.sha1().hexdigest())
+
+    def test_fingerprint_changes_when_content_changes(self):
+        d = self._make_dir('rules', {'a.rules': 'alert'})
+        before = local_files_beacon._fingerprint(d)
+        with open(os.path.join(d, 'a.rules'), 'w') as f:
+            f.write('alert tcp any any -> any any')  # different size
+        self.assertNotEqual(local_files_beacon._fingerprint(d), before)
+
+    def test_fingerprint_ignores_excluded_files(self):
+        d = self._make_dir('rules', {'a.rules': 'alert'})
+        before = local_files_beacon._fingerprint(d)
+        with open(os.path.join(d, 'a.rules.swp'), 'w') as f:
+            f.write('editor swap')
+        self.assertEqual(local_files_beacon._fingerprint(d), before)
+
+    def test_fingerprint_skips_unstatable_entries(self):
+        # A dangling symlink appears in os.walk's file list but os.stat raises
+        # OSError, exercising the except-continue path.
+        d = self._make_dir('rules', {'a.rules': 'alert'})
+        good = local_files_beacon._fingerprint(d)
+        os.symlink(os.path.join(d, 'missing-target'), os.path.join(d, 'broken.link'))
+        self.assertEqual(local_files_beacon._fingerprint(d), good)
+
+    def test_fingerprint_prunes_git_metadata(self):
+        # zkg packages are git clones, so the watched tree carries .git.
+        d = self._make_dir('zkg', {'pkg.zeek': 'print 1;'})
+        before = local_files_beacon._fingerprint(d)
+        git_dir = os.path.join(d, 'pkg', '.git', 'refs', 'heads')
+        os.makedirs(git_dir)
+        with open(os.path.join(git_dir, 'main'), 'w') as f:
+            f.write('0' * 40)
+        self.assertEqual(local_files_beacon._fingerprint(d), before)
+
+    def test_fingerprint_still_sees_worktree_next_to_git(self):
+        d = self._make_dir('zkg', {'pkg.zeek': 'print 1;'})
+        os.makedirs(os.path.join(d, 'pkg', '.git'))
+        before = local_files_beacon._fingerprint(d)
+        with open(os.path.join(d, 'pkg', 'scripts.zeek'), 'w') as f:
+            f.write('print 2;')
+        self.assertNotEqual(local_files_beacon._fingerprint(d), before)
+
+    # -- _read_watermark / _write_watermark -------------------------------
+
+    def test_watermark_round_trip(self):
+        local_files_beacon._write_watermark('suricata', '/rules/suricata', 'deadbeef')
+        self.assertEqual(
+            local_files_beacon._read_watermark('suricata', '/rules/suricata'), 'deadbeef')
+
+    def test_read_watermark_missing_returns_none(self):
+        self.assertIsNone(local_files_beacon._read_watermark('suricata', '/rules/suricata'))
+
+    def test_read_watermark_empty_file_returns_none(self):
+        os.makedirs(self.state, exist_ok=True)
+        with open(local_files_beacon._watermark_file('suricata', '/rules/suricata'), 'w') as f:
+            f.write('')
+        self.assertIsNone(local_files_beacon._read_watermark('suricata', '/rules/suricata'))
+
+    def test_write_watermark_swallows_oserror(self):
+        with patch.object(local_files_beacon.os, 'makedirs', side_effect=OSError):
+            local_files_beacon._write_watermark('suricata', '/rules/suricata', 'deadbeef')
+        self.assertIsNone(local_files_beacon._read_watermark('suricata', '/rules/suricata'))
+
+    def test_watermark_file_differs_per_directory_within_one_tag(self):
+        # zeek/policy and zeek/zkg share the tag 'zeek'.
+        self.assertNotEqual(
+            local_files_beacon._watermark_file('zeek', '/local/zeek/policy'),
+            local_files_beacon._watermark_file('zeek', '/local/zeek/zkg'),
+        )
+
+    def test_watermarks_are_independent_within_one_tag(self):
+        local_files_beacon._write_watermark('zeek', '/local/zeek/policy', 'policyhash')
+        local_files_beacon._write_watermark('zeek', '/local/zeek/zkg', 'zkghash')
+        self.assertEqual(
+            local_files_beacon._read_watermark('zeek', '/local/zeek/policy'), 'policyhash')
+        self.assertEqual(
+            local_files_beacon._read_watermark('zeek', '/local/zeek/zkg'), 'zkghash')
+
+    # -- beacon -----------------------------------------------------------
+
+    def _config(self, mapping):
+        return [{'paths': mapping}]
+
+    def test_beacon_seeds_first_run_and_emits_nothing(self):
+        with patch.object(local_files_beacon, '_fingerprint', return_value='hash1'), \
+             patch.object(local_files_beacon, '_read_watermark', return_value=None), \
+             patch.object(local_files_beacon, '_write_watermark') as mock_write:
+            result = local_files_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
+        self.assertEqual(result, [])
+        mock_write.assert_called_once_with('suricata', '/rules/suricata', 'hash1')
+
+    def test_beacon_emits_on_change(self):
+        with patch.object(local_files_beacon, '_fingerprint', return_value='newhash'), \
+             patch.object(local_files_beacon, '_read_watermark', return_value='oldhash'), \
+             patch.object(local_files_beacon, '_write_watermark') as mock_write:
+            result = local_files_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
+        self.assertEqual(result, [{'tag': 'suricata', 'path': '/rules/suricata'}])
+        mock_write.assert_called_once_with('suricata', '/rules/suricata', 'newhash')
+
+    def test_beacon_no_change_emits_nothing(self):
+        with patch.object(local_files_beacon, '_fingerprint', return_value='samehash'), \
+             patch.object(local_files_beacon, '_read_watermark', return_value='samehash'), \
+             patch.object(local_files_beacon, '_write_watermark') as mock_write:
+            result = local_files_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
+        self.assertEqual(result, [])
+        mock_write.assert_not_called()
+
+    def test_beacon_end_to_end_with_real_files(self):
+        # Exercise the full stack (real fingerprint + real watermark files) across
+        # two poll passes: first seeds silently, second fires after a write.
+        d = self._make_dir('rules', {'a.rules': 'alert'})
+        config = self._config({d: 'suricata'})
+
+        self.assertEqual(local_files_beacon.beacon(config), [])  # seed pass
+        self.assertEqual(local_files_beacon.beacon(config), [])  # unchanged pass
+
+        with open(os.path.join(d, 'b.rules'), 'w') as f:
+            f.write('alert tcp any any -> any any')
+        self.assertEqual(local_files_beacon.beacon(config), [{'tag': 'suricata', 'path': d}])
+
+    def test_beacon_two_dirs_one_tag_do_not_flap(self):
+        # Tag-keyed watermarks would clobber each other and emit on every pass.
+        policy = self._make_dir('zeek/policy', {'intel.dat': '#fields\tindicator'})
+        zkg = self._make_dir('zeek/zkg', {'README': 'place packages here'})
+        config = self._config({policy: 'zeek', zkg: 'zeek'})
+
+        self.assertEqual(local_files_beacon.beacon(config), [])  # seed pass
+        self.assertEqual(local_files_beacon.beacon(config), [])  # idle
+        self.assertEqual(local_files_beacon.beacon(config), [])  # still idle
+
+        with open(os.path.join(policy, 'intel.dat'), 'a') as f:
+            f.write('\nevil.com\tIntel::DOMAIN\tsource\n')
+        self.assertEqual(local_files_beacon.beacon(config), [{'tag': 'zeek', 'path': policy}])
+        self.assertEqual(local_files_beacon.beacon(config), [])  # quiet again
+
+
+if __name__ == '__main__':
+    unittest.main()
@@ -1,172 +0,0 @@
-# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
-# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
-# https://securityonion.net/license; you may not use this file except in compliance with the
-# Elastic License 2.0.
-
-import hashlib
-import os
-import shutil
-import tempfile
-import unittest
-from unittest.mock import patch
-
-import rules_beacon
-
-
-class TestRulesBeacon(unittest.TestCase):
-
-    def setUp(self):
-        # Isolate all on-disk state (watermarks and the dirs we fingerprint) in a
-        # throwaway tree, and point WATERMARK_DIR at it so the real read/write
-        # helpers run against actual files.
-        self.tmpdir = tempfile.mkdtemp()
-        self.state = os.path.join(self.tmpdir, 'state')
-        patcher = patch.object(rules_beacon, 'WATERMARK_DIR', self.state)
-        patcher.start()
-        self.addCleanup(patcher.stop)
-
-    def tearDown(self):
-        shutil.rmtree(self.tmpdir, ignore_errors=True)
-
-    def _make_dir(self, name, files=None):
-        path = os.path.join(self.tmpdir, name)
-        os.makedirs(path, exist_ok=True)
-        for fname, content in (files or {}).items():
-            with open(os.path.join(path, fname), 'w') as f:
-                f.write(content)
-        return path
-
-    # -- trivial contract -------------------------------------------------
-
-    def test_virtual_returns_true(self):
-        self.assertTrue(rules_beacon.__virtual__())
-
-    def test_validate_returns_valid(self):
-        self.assertEqual(rules_beacon.validate({}), (True, 'valid'))
-
-    # -- _paths_from_config -----------------------------------------------
-
-    def test_paths_from_config_list_of_dicts(self):
-        config = [{'interval': 10}, {'paths': {'/a': 'suricata', '/b': 'strelka'}}]
-        self.assertEqual(
-            rules_beacon._paths_from_config(config),
-            {'/a': 'suricata', '/b': 'strelka'},
-        )
-
-    def test_paths_from_config_plain_dict(self):
-        self.assertEqual(
-            rules_beacon._paths_from_config({'paths': {'/a': 'suricata'}}),
-            {'/a': 'suricata'},
-        )
-
-    def test_paths_from_config_skips_non_dict_items(self):
-        self.assertEqual(rules_beacon._paths_from_config(['bogus', 42]), {})
-
-    def test_paths_from_config_paths_not_a_dict(self):
-        self.assertEqual(rules_beacon._paths_from_config({'paths': 'nope'}), {})
-
-    def test_paths_from_config_unexpected_type(self):
-        self.assertEqual(rules_beacon._paths_from_config('nonsense'), {})
-
-    # -- _excluded --------------------------------------------------------
-
-    def test_excluded_matches_temp_and_editor_files(self):
-        for pathname in ('/rules/foo.swp', '/rules/foo~', '/rules/4913', '/rules/.#foo'):
-            self.assertTrue(rules_beacon._excluded(pathname), pathname)
-
-    def test_excluded_allows_real_rule_files(self):
-        self.assertFalse(rules_beacon._excluded('/rules/suricata.rules'))
-
-    # -- _fingerprint -----------------------------------------------------
-
-    def test_fingerprint_missing_dir_is_empty_tree_digest(self):
-        missing = os.path.join(self.tmpdir, 'does-not-exist')
-        self.assertEqual(rules_beacon._fingerprint(missing), hashlib.sha1().hexdigest())
-
-    def test_fingerprint_changes_when_content_changes(self):
-        d = self._make_dir('rules', {'a.rules': 'alert'})
-        before = rules_beacon._fingerprint(d)
-        with open(os.path.join(d, 'a.rules'), 'w') as f:
-            f.write('alert tcp any any -> any any')  # different size
-        self.assertNotEqual(rules_beacon._fingerprint(d), before)
-
-    def test_fingerprint_ignores_excluded_files(self):
-        d = self._make_dir('rules', {'a.rules': 'alert'})
-        before = rules_beacon._fingerprint(d)
-        with open(os.path.join(d, 'a.rules.swp'), 'w') as f:
-            f.write('editor swap')
-        self.assertEqual(rules_beacon._fingerprint(d), before)
-
-    def test_fingerprint_skips_unstatable_entries(self):
-        # A dangling symlink appears in os.walk's file list but os.stat raises
-        # OSError, exercising the except-continue path.
-        d = self._make_dir('rules', {'a.rules': 'alert'})
-        good = rules_beacon._fingerprint(d)
-        os.symlink(os.path.join(d, 'missing-target'), os.path.join(d, 'broken.link'))
-        self.assertEqual(rules_beacon._fingerprint(d), good)
-
-    # -- _read_watermark / _write_watermark -------------------------------
-
-    def test_watermark_round_trip(self):
-        rules_beacon._write_watermark('suricata', 'deadbeef')
-        self.assertEqual(rules_beacon._read_watermark('suricata'), 'deadbeef')
-
-    def test_read_watermark_missing_returns_none(self):
-        self.assertIsNone(rules_beacon._read_watermark('suricata'))
-
-    def test_read_watermark_empty_file_returns_none(self):
-        os.makedirs(self.state, exist_ok=True)
-        with open(rules_beacon._watermark_file('suricata'), 'w') as f:
-            f.write('')
-        self.assertIsNone(rules_beacon._read_watermark('suricata'))
-
-    def test_write_watermark_swallows_oserror(self):
-        with patch.object(rules_beacon.os, 'makedirs', side_effect=OSError):
-            rules_beacon._write_watermark('suricata', 'deadbeef')
-        self.assertIsNone(rules_beacon._read_watermark('suricata'))
-
-    # -- beacon -----------------------------------------------------------
-
-    def _config(self, mapping):
-        return [{'paths': mapping}]
-
-    def test_beacon_seeds_first_run_and_emits_nothing(self):
-        with patch.object(rules_beacon, '_fingerprint', return_value='hash1'), \
-             patch.object(rules_beacon, '_read_watermark', return_value=None), \
-             patch.object(rules_beacon, '_write_watermark') as mock_write:
-            result = rules_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
-        self.assertEqual(result, [])
-        mock_write.assert_called_once_with('suricata', 'hash1')
-
-    def test_beacon_emits_on_change(self):
-        with patch.object(rules_beacon, '_fingerprint', return_value='newhash'), \
-             patch.object(rules_beacon, '_read_watermark', return_value='oldhash'), \
-             patch.object(rules_beacon, '_write_watermark') as mock_write:
-            result = rules_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
-        self.assertEqual(result, [{'tag': 'suricata', 'path': '/rules/suricata'}])
-        mock_write.assert_called_once_with('suricata', 'newhash')
-
-    def test_beacon_no_change_emits_nothing(self):
-        with patch.object(rules_beacon, '_fingerprint', return_value='samehash'), \
-             patch.object(rules_beacon, '_read_watermark', return_value='samehash'), \
-             patch.object(rules_beacon, '_write_watermark') as mock_write:
-            result = rules_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
-        self.assertEqual(result, [])
-        mock_write.assert_not_called()
-
-    def test_beacon_end_to_end_with_real_files(self):
-        # Exercise the full stack (real fingerprint + real watermark files) across
-        # two poll passes: first seeds silently, second fires after a write.
-        d = self._make_dir('rules', {'a.rules': 'alert'})
-        config = self._config({d: 'suricata'})
-
-        self.assertEqual(rules_beacon.beacon(config), [])  # seed pass
-        self.assertEqual(rules_beacon.beacon(config), [])  # unchanged pass
-
-        with open(os.path.join(d, 'b.rules'), 'w') as f:
-            f.write('alert tcp any any -> any any')
-        self.assertEqual(rules_beacon.beacon(config), [{'tag': 'suricata', 'path': d}])
-
-
-if __name__ == '__main__':
-    unittest.main()
@@ -418,6 +418,19 @@ is_single_node_grid() {
 	grep "role: so-" /etc/salt/grains | grep -E "eval|standalone|import" &> /dev/null
 }

+remove_elasticsearch_index_template() {
+  local template_name=$1
+  local reason=${2:-"Removing index template for upgrade"}
+
+  # check if template exists
+  if so-elasticsearch-query "_index_template/$template_name" --fail --retry 3 --retry-delay 5 >/dev/null 2>&1; then
+    echo "Removing Elasticsearch index template: $template_name ($reason)"
+    if ! so-elasticsearch-query "_index_template/$template_name" -XDELETE --fail --retry 3 --retry-delay 5; then
+      return 1
+    fi
+  fi
+}
+
 initialize_elasticsearch_indices() {
    local index_names=$1
    local default_entry=${2:-'{"@timestamp":"0"}'}
@@ -134,6 +134,7 @@ if [[ $EXCLUDE_STARTUP_ERRORS == 'Y' ]]; then
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|Redis may have been restarted" # Redis likely restarted by salt
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|file already closed"          # Go logging race condition during container restart
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|relation \"audit_settings\" does not exist" # salt checking for changes before SOC starts
+    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|Error in plugin: elasticsearch: Unable to retrieve master node information" # expected error while ES is upgrading/electing a master 
 fi

 if [[ $EXCLUDE_FALSE_POSITIVE_ERRORS == 'Y' ]]; then
@@ -240,6 +241,7 @@ if [[ $EXCLUDE_KNOWN_ERRORS == 'Y' ]]; then
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|tcp 127.0.0.1:6791: bind: address already in use" # so-elastic-fleet agent restarting. Seen starting w/ 8.18.8 https://github.com/elastic/kibana/issues/201459
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|TransformTask\] \[logs-.*user so_kibana lacks the required permissions" # Known issue with integrations starting transform jobs that are explicitly not allowed to start as a system user
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|manifest unknown"             # appears in so-dockerregistry log for so-tcpreplay following docker upgrade to 29.2.1-1
+    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|Could not index event to Elasticsearch.*\"version\" => \"9.0.8\"" # Expected during Elastic upgrade temporarily, as policies referencing older pipelines are updated
 fi

 RESULT=0
@@ -304,4 +306,4 @@ else
    echo -e "\nResult: One or more errors found"
 fi

-exit $RESULT
+exit $RESULT
@@ -1,6 +1,12 @@
 docker:
  range: '172.17.1.0/24'
  gateway: '172.17.1.1'
+  networks:
+    sobridge: {}
+    soauth:
+      range: '172.17.2.0/24'
+      gateway: '172.17.2.1'
+      manager_only: True
  ulimits:
    - name: nofile
      soft: 1048576
@@ -58,18 +64,18 @@ docker:
      ulimits: []
    'so-kratos':
      final_octet: 28
+      networks: ['soauth']
      port_bindings:
        - 0.0.0.0:4433:4433
-        - 0.0.0.0:4434:4434
      custom_bind_mounts: []
      extra_hosts: []
      extra_env: []
      ulimits: []
    'so-hydra':
      final_octet: 30
+      networks: ['soauth']
      port_bindings:
        - 0.0.0.0:4444:4444
-        - 0.0.0.0:4445:4445
      custom_bind_mounts: []
      extra_hosts: []
      extra_env: []
@@ -128,6 +134,7 @@ docker:
      ulimits: []
    'so-soc':
      final_octet: 34
+      networks: ['sobridge', 'soauth']
      port_bindings:
        - 0.0.0.0:9822:9822
      custom_bind_mounts: []
@@ -1,8 +1,26 @@
 {% import_yaml 'docker/defaults.yaml' as DOCKERDEFAULTS %}
 {% set DOCKERMERGED = salt['pillar.get']('docker', DOCKERDEFAULTS.docker, merge=True) %}
-{% set RANGESPLIT = DOCKERMERGED.range.split('.') %}
-{% set FIRSTTHREE = RANGESPLIT[0] ~ '.' ~ RANGESPLIT[1] ~ '.' ~ RANGESPLIT[2] ~ '.' %}
+
+{% if DOCKERMERGED.networks.sobridge is not mapping %}
+{%   do DOCKERMERGED.networks.update({'sobridge': {}}) %}
+{% endif %}
+{% do DOCKERMERGED.networks['sobridge'].update({'range': DOCKERMERGED.range, 'gateway': DOCKERMERGED.gateway}) %}
+
+{% for netname, net in DOCKERMERGED.networks.items() %}
+{%   set RANGESPLIT = net.range.split('.') %}
+{%   do net.update({'prefix': RANGESPLIT[0] ~ '.' ~ RANGESPLIT[1] ~ '.' ~ RANGESPLIT[2] ~ '.'}) %}
+{% endfor %}

 {% for container, vals in DOCKERMERGED.containers.items() %}
-{%   do DOCKERMERGED.containers[container].update({'ip': FIRSTTHREE ~ DOCKERMERGED.containers[container].final_octet}) %}
+{%   set CONTAINER_NETS = vals.get('networks', ['sobridge']) %}
+{%   set IPS = {} %}
+{%   for netname in CONTAINER_NETS %}
+{%     do IPS.update({netname: DOCKERMERGED.networks[netname].prefix ~ vals.final_octet}) %}
+{%   endfor %}
+{%   do DOCKERMERGED.containers[container].update({
+         'networks': CONTAINER_NETS,
+         'ips': IPS,
+         'network': CONTAINER_NETS[0],
+         'ip': IPS[CONTAINER_NETS[0]]
+       }) %}
 {% endfor %}
@@ -71,15 +71,19 @@ dockerreserveports:
    - source: salt://common/files/99-reserved-ports.conf
    - name: /etc/sysctl.d/99-reserved-ports.conf

-sos_docker_net:
+{% for NETNAME, NETWORK in DOCKERMERGED.networks.items() %}
+{% if not NETWORK.get('manager_only') or GLOBALS.get('is_manager', False) %}
+sos_docker_net_{{ NETNAME }}:
  docker_network.present:
-    - name: sobridge
-    - subnet: {{ DOCKERMERGED.range }}
-    - gateway: {{ DOCKERMERGED.gateway }}
+    - name: {{ NETNAME }}
+    - subnet: {{ NETWORK.range }}
+    - gateway: {{ NETWORK.gateway }}
    - options:
-        com.docker.network.bridge.name: 'sobridge'
+        com.docker.network.bridge.name: '{{ NETNAME }}'
        com.docker.network.driver.mtu: '1500'
        com.docker.network.bridge.enable_ip_masquerade: 'true'
        com.docker.network.bridge.enable_icc: 'true'
        com.docker.network.bridge.host_binding_ipv4: '0.0.0.0'
-    - unless: ip l | grep sobridge
+    - unless: ip l | grep {{ NETNAME }}
+{% endif %}
+{% endfor %}
@@ -7,6 +7,33 @@ docker:
    description: Default docker IP range for containers.
    helpLink: docker
    advanced: True
+  networks:
+    soauth:
+      range:
+        description: |
+          IP range for the soauth docker network. soauth is an isolated network for the
+          authentication services, so that the Kratos and Hydra admin APIs are only reachable
+          from the containers placed on it. Most containers instead use sobridge, which takes
+          its range and gateway from the docker.range and docker.gateway settings above.
+        helpLink: docker
+        readonly: True
+        advanced: True
+        global: True
+      gateway:
+        description: Gateway for the soauth docker network.
+        helpLink: docker
+        readonly: True
+        advanced: True
+        global: True
+      manager_only:
+        description: |
+          Limits the soauth network to grid members running the authentication containers,
+          instead of creating it on every node.
+        helpLink: docker
+        readonly: True
+        advanced: True
+        global: True
+        forcedType: bool
  ulimits:
    description: |
      Default ulimit settings applied to all containers via the Docker daemon. Each entry specifies a resource name (e.g. nofile, memlock, core, nproc) with soft and hard limits. Individual container ulimits override these defaults. Valid resource names include: cpu, fsize, data, stack, core, rss, nproc, nofile, memlock, as, locks, sigpending, msgqueue, nice, rtprio, rttime.
@@ -34,6 +61,16 @@ docker:
        readonly: True
        advanced: True
        global: True
+      networks:
+        description: |
+          Docker networks this container is attached to. The first entry is the container's
+          primary network and determines the address its published ports are forwarded to.
+          Defaults to sobridge when unset.
+        helpLink: docker
+        readonly: True
+        advanced: True
+        global: True
+        forcedType: "[]string"
      port_bindings:
        description: List of port bindings for the container.
        helpLink: docker
@@ -5,7 +5,7 @@
  "package": {
    "name": "endpoint",
    "title": "Elastic Defend",
-    "version": "9.3.1",
+    "version": "9.4.1",
    "requires_root": true
  },
  "enabled": true,
@@ -29,7 +29,7 @@
              "\\.gz$"
            ],
            "include_files": [],
-            "processors": "- dissect:\n    tokenizer: \"/nsm/import/%{import.id}/evtx/%{import.file}\"\n    field: \"log.file.path\"\n    target_prefix: \"\"\n- decode_json_fields:\n    fields: [\"message\"]\n    target: \"\"\n- drop_fields:\n    fields: [\"host\"]\n    ignore_missing: true\n- add_fields:\n    target: data_stream\n    fields:\n      type: logs\n      dataset: system.security\n- add_fields:\n    target: event\n    fields:\n      dataset: system.security\n      module: system\n      imported: true\n- add_fields:\n    target: \"@metadata\"\n    fields:\n        pipeline: logs-system.security-2.20.0\n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-Sysmon/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.sysmon_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.sysmon_operational\n          module: windows\n          imported: true\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.sysmon_operational-3.8.3\n- if:\n    equals:\n      winlog.channel: 'Application'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.application-2.20.0\n- if:\n    equals:\n      winlog.channel: 'System'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.system-2.20.0\n    \n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-PowerShell/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.powershell_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.powershell_operational\n          module: windows\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.powershell_operational-3.8.3\n- add_fields:\n    target: data_stream\n    fields:\n        dataset: import",
+            "processors": "- dissect:\n    tokenizer: \"/nsm/import/%{import.id}/evtx/%{import.file}\"\n    field: \"log.file.path\"\n    target_prefix: \"\"\n- decode_json_fields:\n    fields: [\"message\"]\n    target: \"\"\n- drop_fields:\n    fields: [\"host\"]\n    ignore_missing: true\n- add_fields:\n    target: data_stream\n    fields:\n      type: logs\n      dataset: system.security\n- add_fields:\n    target: event\n    fields:\n      dataset: system.security\n      module: system\n      imported: true\n- add_fields:\n    target: \"@metadata\"\n    fields:\n        pipeline: logs-system.security-2.22.3\n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-Sysmon/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.sysmon_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.sysmon_operational\n          module: windows\n          imported: true\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.sysmon_operational-3.9.0\n- if:\n    equals:\n      winlog.channel: 'Application'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.application-2.22.3\n- if:\n    equals:\n      winlog.channel: 'System'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.system-2.22.3\n    \n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-PowerShell/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.powershell_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.powershell_operational\n          module: windows\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.powershell_operational-3.9.0\n- add_fields:\n    target: data_stream\n    fields:\n        dataset: import",
            "tags": [
              "import"
            ],
@@ -16,7 +16,6 @@
    'awsfirehose.metrics': 'aws.cloudwatch',
    'cribl.logs': 'cribl',
    'cribl.metrics': 'cribl',
-    'sentinel_one_cloud_funnel.logins': 'sentinel_one_cloud_funnel.login',
    'azure_application_insights.app_insights': 'azure.app_insights',
    'azure_application_insights.app_state': 'azure.app_state',
    'azure_billing.billing': 'azure.billing',
@@ -68,6 +68,24 @@ so-elastic-fleet-package-upgrade:
    - require:
      - http: wait_for_so-kibana

+# initial so-elasticsearch-templates run is earlier, but it can skip over templates that have component templates not yet installed to avoid elasticsearch rejecting the template.
+so-elasticsearch-templates-after-fleet-packages:
+  cmd.run:
+    - name: /usr/sbin/so-elasticsearch-templates-load
+    - cwd: /opt/so
+    - unless: test -f /opt/so/state/estemplates.txt
+    - require:
+      - cmd: so-elastic-fleet-package-upgrade
+
+so-elastic-fleet-integration-upgrade:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-integration-upgrade
+    - retry:
+        attempts: 3
+        interval: 10
+    - require:
+      - cmd: so-elastic-fleet-package-upgrade
+
 so-elastic-fleet-integrations:
  cmd.run:
    - name: /usr/sbin/so-elastic-fleet-integration-policy-load
@@ -86,21 +104,13 @@ so-elastic-agent-grid-upgrade:
    - require:
      - http: wait_for_so-kibana

-so-elastic-fleet-integration-upgrade:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-integration-upgrade
-    - retry:
-        attempts: 3
-        interval: 10
-    - require:
-      - http: wait_for_so-kibana
-
 {# Optional integrations script doesn't need the retries like so-elastic-fleet-integration-upgrade which loads the default integrations #}
 so-elastic-fleet-addon-integrations:
  cmd.run:
    - name: /usr/sbin/so-elastic-fleet-optional-integrations-load
    - require:
      - http: wait_for_so-kibana
+      - cmd: so-elasticsearch-templates-after-fleet-packages

 {% if ELASTICFLEETMERGED.config.defend_filters.enable_auto_configuration %}
 so-elastic-defend-manage-filters-file-watch:
@@ -10,6 +10,25 @@

 PKG_LOAD_FAILURES=0
 PKG_LOAD_FAILURES_NAMES=()
+PKG_UPGRADED=0
+
+cleanup_elasticsearch_fleet_transforms() {
+    local transforms transform_id attempt
+    
+    if ! transforms=$(so-elasticsearch-query "_transform/logs-elasticsearch.index_pivot-default-*" --retry 1 --retry-delay 5); then
+        return 0
+    fi
+
+    while IFS= read -r transform_id; do
+        [ -n "$transform_id" ] || continue
+        for attempt in {1..3}; do
+            if so-elasticsearch-query "_transform/$transform_id?force=true" -XDELETE --fail --retry 1 --retry-delay 5; then
+                break
+            fi
+            sleep 5
+        done
+    done < <(jq -r '.transforms[]?.id' <<< "$transforms")
+}

 {%- for PACKAGE in SUPPORTED_PACKAGES %}
 if INSTALLED_VERSION=$(elastic_fleet_package_version_check "{{ PACKAGE }}") && LATEST_VERSION=$(elastic_fleet_package_latest_version_check "{{ PACKAGE }}"); then
@@ -17,10 +36,25 @@ if INSTALLED_VERSION=$(elastic_fleet_package_version_check "{{ PACKAGE }}") && L
    if [ "$INSTALLED_VERSION" == "$LATEST_VERSION" ]; then
        echo "{{ PACKAGE }} integration version $INSTALLED_VERSION is already at the reported latest version $LATEST_VERSION, skipping upgrade."
    else
+{%- if PACKAGE == 'elasticsearch' %}
+        cleanup_elasticsearch_fleet_transforms
+{%- endif %}
        echo "Upgrading {{ PACKAGE }} package from $INSTALLED_VERSION to version $LATEST_VERSION..."
        if ! elastic_fleet_package_install "{{ PACKAGE }}" "$LATEST_VERSION"; then
            PKG_LOAD_FAILURES=$((PKG_LOAD_FAILURES + 1))
            PKG_LOAD_FAILURES_NAMES+=("{{ PACKAGE }}")
+        # check that package has upgraded to the expected version after install command
+        elif ! LATEST_VERSION=$(elastic_fleet_package_latest_version_check "{{ PACKAGE }}"); then
+            echo "ERROR: Failed to get latest version information for integration {{ PACKAGE }} after upgrade attempt"
+            PKG_LOAD_FAILURES=$((PKG_LOAD_FAILURES + 1))
+            PKG_LOAD_FAILURES_NAMES+=("{{ PACKAGE }}")
+        elif INSTALLED_VERSION=$(elastic_fleet_package_version_check "{{ PACKAGE }}") && [ "$INSTALLED_VERSION" == "$LATEST_VERSION" ]; then
+            echo "{{ PACKAGE }} integration upgraded to version $LATEST_VERSION."
+            PKG_UPGRADED=$((PKG_UPGRADED + 1))
+        else
+            echo "ERROR: {{ PACKAGE }} integration still at ${INSTALLED_VERSION:-unknown}; expected $LATEST_VERSION"
+            PKG_LOAD_FAILURES=$((PKG_LOAD_FAILURES + 1))
+            PKG_LOAD_FAILURES_NAMES+=("{{ PACKAGE }}")
        fi
    fi
 else
@@ -30,6 +64,11 @@ else
 fi
 {%- endfor %}

+if [ $PKG_UPGRADED -gt 0 ]; then
+    echo "Elasticsearch template statefiles cleared after $PKG_UPGRADED package upgrade(s), so templates can reload."
+    rm -f /opt/so/state/estemplates.txt /opt/so/state/addon_estemplates.txt
+fi
+
 if [ $PKG_LOAD_FAILURES -gt 0 ]; then
    echo "ERROR: Failed to upgrade $PKG_LOAD_FAILURES package(s):"
    for PKG in "${PKG_LOAD_FAILURES_NAMES[@]}"; do
@@ -98,6 +98,13 @@ so-es-cluster-settings:
      - docker_container: so-elasticsearch
      - file: elasticsearch_sbin_jinja
      - http: wait_for_so-elasticsearch
+
+so-elasticsearch-system-indices-patch:
+  cmd.run:
+    - name: /usr/sbin/so-elasticsearch-system-indices-patch
+    - require:
+      - http: wait_for_so-elasticsearch
+      - file: so-elasticsearch-system-indices-patch-script
 {%  endif %}

 # heavynodes will only load ILM policies for SO managed indices. (Indicies defined in elasticsearch/defaults.yaml)
@@ -42,6 +42,16 @@ elasticsearch_sbin:
    - file_mode: 755
    - exclude_pat:
      - so-elasticsearch-pipelines # exclude this because we need to watch it for changes, we sync it in another state
+      - so-elasticsearch-system-indices-patch
+    - show_changes: False
+
+so-elasticsearch-system-indices-patch-script:
+  file.managed:
+    - name: /usr/sbin/so-elasticsearch-system-indices-patch
+    - source: salt://elasticsearch/tools/sbin/so-elasticsearch-system-indices-patch
+    - user: 930
+    - group: 939
+    - mode: 755
    - show_changes: False

 elasticsearch_sbin_jinja:
@@ -1,7 +1,7 @@
 elasticsearch:
  enabled: false
  esheap: '600m'
-  version: 9.3.7
+  version: 9.4.5
  index_clean: true
  data_retention_method: DLM
  vm:
@@ -0,0 +1,203 @@
+#!/bin/bash
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+set -eo pipefail
+
+SETTINGS='{"index":{"auto_expand_replicas":"0-1"}}'
+KIBANA_PASSWORD=
+INDEX_PATTERNS=(
+  '.entity_analytics.risk_score.lookup-*'
+  '.entity_analytics.watchlists.*'
+  '.entity_analytics.monitoring.users-*'
+  '.entity_analytics.entity-leads-*'
+  '.asset-criticality.asset-criticality-*'
+  '.workflows-executions'
+  '.workflows-step-executions'
+  '.entities.v2.latest.security_*'
+  '.entities.v2.history.security_*'
+  'risk-score.risk-score-latest-*'
+  '.metrics-endpoint.metadata_united_*'
+  '.metrics-endpoint.metadata_current_*'
+)
+DATA_STREAM_PATTERNS=(
+  '.entities.v2.updates.security_*'
+  'risk-score.risk-score-*'
+  '.rule-events'
+  '.alert-actions'
+)
+TEMPLATE_PATTERNS=(
+  'entities_v2_latest_security_default_index_template'
+  'entities_v2_history_security_default_index_template'
+  '.entities_v2_updates_security_default_index_template'
+  '.risk-score.risk-score-default-index-template'
+  '.rule-events'
+  '.alert-actions'
+  '.metrics-endpoint.metadata_united_default-template'
+  '.metrics-endpoint.metadata_current_default-template'
+)
+
+query_es() {
+  if so-elasticsearch-query "$@" --fail --retry 3 --retry-delay 5; then
+    return 0
+  fi
+
+  # retry failed attempts with so_kibana user (system managed indices reject so_elastic user)
+  local query_path="$1"
+  shift
+
+  if [[ -z "$KIBANA_PASSWORD" ]]; then
+    KIBANA_PASSWORD=$(salt-call pillar.get elasticsearch:auth:users:so_kibana_user:pass --out=newline_values_only)
+  fi
+  [[ -n "$KIBANA_PASSWORD" ]] || return 1
+
+  echo "Retrying ${query_path} as so_kibana." >&2
+  curl -K /opt/so/conf/elasticsearch/curl.config --user "so_kibana:${KIBANA_PASSWORD}" \
+    -s -k -L --fail --retry 3 --retry-delay 5 -H 'Content-Type: application/json' "https://localhost:9200/${query_path}" "$@"
+}
+
+# add auto_expand_replicas=0-1 to given index
+set_auto_expand_replicas() {
+  local index="$1"
+
+  echo "Setting auto_expand_replicas to 0-1 on ${index}."
+  query_es "${index}/_settings" -XPUT -d "$SETTINGS" >/dev/null
+}
+
+# resolve index patterns and find each index with an unassigned replica
+unassigned_replicas() {
+  local pattern="$1"
+  local resolved_indices response index
+
+  if ! resolved_indices=$(query_es "_resolve/index/${pattern}?expand_wildcards=all" 2>/dev/null); then
+    return 0
+  fi
+
+  while read -r index; do
+    if ! response=$(query_es "_cat/shards/${index}?format=json&h=index,prirep,state" 2>/dev/null); then
+      continue
+    fi
+    jq -r '.[]? | objects | select(.prirep == "r" and .state == "UNASSIGNED") | .index' <<<"$response"
+  done < <(jq -r '.indices[]?.name' <<<"$resolved_indices")
+}
+
+data_stream_indices() {
+  local pattern="$1"
+  local response
+
+  if ! response=$(query_es "_data_stream/${pattern}?expand_wildcards=all" 2>/dev/null); then
+    return 0
+  fi
+  jq -r '.data_streams[]?.indices[]?.index_name' <<<"$response"
+}
+
+update_system_indices() {
+  local pattern="$1"
+  local index
+
+  while read -r index; do
+    [[ -n "$index" ]] && set_auto_expand_replicas "$index"
+  done < <(unassigned_replicas "$pattern")
+}
+
+# update data stream backing indices with unassigned replicas
+update_system_ds() {
+  local pattern="$1"
+  local index
+
+  while read -r index; do
+    while read -r unassigned_index; do
+      [[ -n "$unassigned_index" ]] && set_auto_expand_replicas "$unassigned_index"
+    done < <(unassigned_replicas "$index")
+  done < <(data_stream_indices "$pattern")
+}
+
+has_unassigned_replicas() {
+  local pattern="$1"
+  local index
+
+  index=$(unassigned_replicas "$pattern" | sed -n '1p')
+  [[ -n "$index" ]]
+}
+
+data_stream_has_unassigned_replicas() {
+  local pattern="$1"
+  local index
+  while read -r index; do
+    has_unassigned_replicas "$index" && return 0
+  done < <(data_stream_indices "$pattern")
+
+  return 1
+}
+
+needs_patch() {
+  local pattern
+  for pattern in "${INDEX_PATTERNS[@]}"; do
+    has_unassigned_replicas "$pattern" && return 0
+  done
+  for pattern in "${DATA_STREAM_PATTERNS[@]}"; do
+    data_stream_has_unassigned_replicas "$pattern" && return 0
+  done
+
+  return 1
+}
+
+# get index templates, update with auto_expand_replicas=0-1, and PUT back. Keeping mappings/settings/aliases in-place
+update_system_templates() {
+  local pattern="$1"
+  local templates name response template auto_expand_replicas
+
+  if ! templates=$(query_es "_index_template/${pattern}" 2>/dev/null); then
+    return 0
+  fi
+  while read -r name; do
+    response=$(query_es "_index_template/${name}")
+    template=$(jq -c '.index_templates[0].index_template' <<<"$response")
+    auto_expand_replicas=$(jq -r '.template.settings["index.auto_expand_replicas"] // .template.settings.index.auto_expand_replicas // empty' <<<"$template")
+    [[ "$auto_expand_replicas" == "0-1" ]] && continue
+
+    template=$(jq '
+            if (.template.settings.index | type) == "object" then
+                .template.settings.index.auto_expand_replicas = "0-1"
+            else
+                .template.settings["index.auto_expand_replicas"] = "0-1"
+            end
+            | del(.created_date_millis, .modified_date_millis)
+        ' <<<"$template")
+    echo "Setting auto_expand_replicas to 0-1 on index template ${name}."
+    query_es "_index_template/${name}" -XPUT -d "$template" >/dev/null
+  done < <(jq -r '.index_templates[]?.name' <<<"$templates")
+}
+
+if [[ "${1:-}" == "--check" ]]; then
+  needs_patch
+  exit $?
+fi
+
+if [[ $# -ne 0 ]]; then
+  echo "Usage: $0 [--check]" >&2
+  exit 1
+fi
+
+patched=false
+for pattern in "${INDEX_PATTERNS[@]}"; do
+  if has_unassigned_replicas "$pattern"; then
+    update_system_indices "$pattern"
+    patched=true
+  fi
+done
+
+for pattern in "${DATA_STREAM_PATTERNS[@]}"; do
+  if data_stream_has_unassigned_replicas "$pattern"; then
+    update_system_ds "$pattern"
+    patched=true
+  fi
+done
+
+if [[ "$patched" == true ]]; then
+  for pattern in "${TEMPLATE_PATTERNS[@]}"; do
+    update_system_templates "$pattern"
+  done
+fi
@@ -4,11 +4,19 @@
 {%- set role = GLOBALS.role.split('-')[1] %}
 {%- from 'firewall/containers.map.jinja' import NODE_CONTAINERS %}

+{%- set NODE_NETWORKS = [] %}
+{%- for NETNAME, NETWORK in DOCKERMERGED.networks.items() %}
+{%-   if not NETWORK.get('manager_only') or GLOBALS.get('is_manager', False) %}
+{%-     do NODE_NETWORKS.append(NETNAME) %}
+{%-   endif %}
+{%- endfor %}
+
 {%- set PR = [] %}
 {%- set D1 = [] %}
 {%- set D2 = [] %}
 {%- for container in NODE_CONTAINERS %}
 {%-   set IP = DOCKERMERGED.containers[container].ip %}
+{%-   set BRIDGE = DOCKERMERGED.containers[container].network %}
 {%-   if DOCKERMERGED.containers[container].port_bindings is defined %}
 {%-     for binding in DOCKERMERGED.containers[container].port_bindings %}
 {#-       cant split int so we convert to string #}
@@ -35,11 +43,11 @@
 {%-         endif %}
 {%-         do PR.append("-A POSTROUTING -s " ~ DOCKERMERGED.containers[container].ip ~ "/32 -d " ~ DOCKERMERGED.containers[container].ip ~ "/32 -p " ~  proto ~ " -m " ~ proto  ~ " --dport " ~ containerPort ~ " -j MASQUERADE") %}
 {%-         if bindip | length and bindip != '0.0.0.0' %}
-{%-           do D1.append("-A DOCKER -d " ~ bindip ~ "/32 ! -i sobridge -p " ~ proto ~ " -m " ~ proto ~ " --dport " ~ hostPort ~ " -j DNAT --to-destination " ~ DOCKERMERGED.containers[container].ip ~ ":" ~ containerPort) %}
+{%-           do D1.append("-A DOCKER -d " ~ bindip ~ "/32 ! -i " ~ BRIDGE ~ " -p " ~ proto ~ " -m " ~ proto ~ " --dport " ~ hostPort ~ " -j DNAT --to-destination " ~ DOCKERMERGED.containers[container].ip ~ ":" ~ containerPort) %}
 {%-         else %}
-{%-           do D1.append("-A DOCKER ! -i sobridge -p " ~ proto ~ " -m " ~ proto ~ " --dport " ~ hostPort ~ " -j DNAT --to-destination " ~ DOCKERMERGED.containers[container].ip ~ ":" ~ containerPort) %}
+{%-           do D1.append("-A DOCKER ! -i " ~ BRIDGE ~ " -p " ~ proto ~ " -m " ~ proto ~ " --dport " ~ hostPort ~ " -j DNAT --to-destination " ~ DOCKERMERGED.containers[container].ip ~ ":" ~ containerPort) %}
 {%-         endif %}
-{%-         do D2.append("-A DOCKER -d " ~ DOCKERMERGED.containers[container].ip ~ "/32 ! -i sobridge -o sobridge -p " ~ proto ~ " -m " ~ proto ~ " --dport " ~ containerPort ~ " -j ACCEPT") %}
+{%-         do D2.append("-A DOCKER -d " ~ DOCKERMERGED.containers[container].ip ~ "/32 ! -i " ~ BRIDGE ~ " -o " ~ BRIDGE ~ " -p " ~ proto ~ " -m " ~ proto ~ " --dport " ~ containerPort ~ " -j ACCEPT") %}
 {%-     endfor %}
 {%-   endif %}
 {%- endfor %}
@@ -52,11 +60,15 @@
 :DOCKER - [0:0]
 -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
 -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s {{DOCKERMERGED.range}} ! -o sobridge -j MASQUERADE
+{%- for NETNAME in NODE_NETWORKS %}
+-A POSTROUTING -s {{ DOCKERMERGED.networks[NETNAME].range }} ! -o {{ NETNAME }} -j MASQUERADE
+{%- endfor %}
 {%- for rule in PR %}
 {{ rule }}
 {%- endfor %}
-A DOCKER -i sobridge -j RETURN
+{%- for NETNAME in NODE_NETWORKS %}
+-A DOCKER -i {{ NETNAME }} -j RETURN
+{%- endfor %}
 {%- for rule in D1 %}
 {{ rule }}
 {%- endfor %}
@@ -97,10 +109,12 @@ COMMIT
 {%- endif %}
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o sobridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o sobridge -j DOCKER
-A FORWARD -i sobridge ! -o sobridge -j ACCEPT
-A FORWARD -i sobridge -o sobridge -j ACCEPT
+{%- for NETNAME in NODE_NETWORKS %}
+-A FORWARD -o {{ NETNAME }} -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A FORWARD -o {{ NETNAME }} -j DOCKER
+-A FORWARD -i {{ NETNAME }} ! -o {{ NETNAME }} -j ACCEPT
+-A FORWARD -i {{ NETNAME }} -o {{ NETNAME }} -j ACCEPT
+{%- endfor %}
 -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A FORWARD -i lo -j ACCEPT
 -A FORWARD -m conntrack --ctstate INVALID -j DROP
@@ -112,13 +126,18 @@ COMMIT
 {%- for rule in D2 %}
 {{ rule }}
 {%- endfor %}
-
-A DOCKER-ISOLATION-STAGE-1 -i sobridge ! -o sobridge -j DOCKER-ISOLATION-STAGE-2
+{% for NETNAME in NODE_NETWORKS %}
+-A DOCKER-ISOLATION-STAGE-1 -i {{ NETNAME }} ! -o {{ NETNAME }} -j DOCKER-ISOLATION-STAGE-2
+{%- endfor %}
 -A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o sobridge -j DROP
+{%- for NETNAME in NODE_NETWORKS %}
+-A DOCKER-ISOLATION-STAGE-2 -o {{ NETNAME }} -j DROP
+{%- endfor %}
 -A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER ! -i sobridge -o sobridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER ! -i sobridge -o sobridge -j LOGGING
+{%- for NETNAME in NODE_NETWORKS %}
+-A DOCKER-USER ! -i {{ NETNAME }} -o {{ NETNAME }} -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-USER ! -i {{ NETNAME }} -o {{ NETNAME }} -j LOGGING
+{%- endfor %}
 -A DOCKER-USER -j RETURN
 -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-dropped: "
 -A LOGGING -j DROP
@@ -4,8 +4,12 @@

 {# add our ip to self #}
 {% do FIREWALL_DEFAULT.firewall.hostgroups.self.append(GLOBALS.node_ip) %}
-{# add dockernet range #}
-{% do FIREWALL_DEFAULT.firewall.hostgroups.dockernet.append(DOCKERMERGED.range) %}
+{# add dockernet ranges #}
+{% for NETNAME, NETWORK in DOCKERMERGED.networks.items() %}
+{%   if not NETWORK.get('manager_only') or GLOBALS.get('is_manager', False) %}
+{%     do FIREWALL_DEFAULT.firewall.hostgroups.dockernet.append(NETWORK.range) %}
+{%   endif %}
+{% endfor %}

 {% if GLOBALS.role == 'so-idh' %}
 {%   from 'idh/opencanary_config.map.jinja' import IDH_PORTGROUPS %}
@@ -26,8 +26,8 @@ so-hydra:
    - hostname: hydra
    - name: so-hydra
    - networks:
-      - sobridge:
-        - ipv4_address: {{ DOCKERMERGED.containers['so-hydra'].ip }}
+      - soauth:
+        - ipv4_address: {{ DOCKERMERGED.containers['so-hydra'].ips['soauth'] }}
    - binds:
      - /opt/so/conf/hydra/:/hydra-conf:ro
      - /opt/so/log/hydra/:/hydra-log:rw
@@ -73,7 +73,7 @@ delete_so-hydra_so-status.disabled:

 wait_for_hydra:
  http.wait_for_successful_query:
-    - name: 'http://{{ GLOBALS.manager }}:4444/health/alive'
+    - name: 'http://{{ DOCKERMERGED.containers['so-hydra'].ips['soauth'] }}:4444/health/alive'
    - ssl: True
    - verify_ssl: False
    - status:
@@ -22,7 +22,7 @@ kibana:
          - default
          - file
    migrations:
-      discardCorruptObjects: "9.3.7"
+      discardCorruptObjects: "9.4.5"
    telemetry:
      enabled: False
    xpack:
@@ -9,5 +9,5 @@ SESSIONCOOKIE=$(curl -K /opt/so/conf/elasticsearch/curl.config -c - -X GET http:
 # Disable certain Features from showing up in the Kibana UI
 echo
 echo "Setting up default Kibana Space:"
-curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -L -X PUT "localhost:5601/api/spaces/space/default" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d' {"id":"default","name":"Default","disabledFeatures":["ml","enterpriseSearch","logs","infrastructure","apm","uptime","monitoring","stackAlerts","actions","securitySolutionCasesV3","inventory","dataQuality","searchSynonyms","searchQueryRules","enterpriseSearchApplications","enterpriseSearchAnalytics","securitySolutionTimeline","securitySolutionNotes","securitySolutionRulesV1","entityManager","streams","cloudConnect","slo"]} ' >> /opt/so/log/kibana/misc.log
+curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -L -X PUT "localhost:5601/api/spaces/space/default" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d' {"id":"default","name":"Default","disabledFeatures":["ml","enterpriseSearch","logs","infrastructure","apm","uptime","securitySolutionCasesV3","inventory","searchSynonyms","searchQueryRules","enterpriseSearchApplications","enterpriseSearchAnalytics","securitySolutionTimeline","securitySolutionNotes","securitySolutionRulesV4","securitySolutionAlertsV1","entityManager","slo","streams","anonymization","searchInferenceEndpoints","cloudConnect","queryActivity","automatic_import","stackAlerts","monitoring","dataQuality","actions"]} ' >> /opt/so/log/kibana/misc.log
 echo
@@ -19,8 +19,8 @@ so-kratos:
    - hostname: kratos
    - name: so-kratos
    - networks:
-      - sobridge:
-        - ipv4_address: {{ DOCKERMERGED.containers['so-kratos'].ip }}
+      - soauth:
+        - ipv4_address: {{ DOCKERMERGED.containers['so-kratos'].ips['soauth'] }}
    - binds:
      - /opt/so/conf/kratos/:/kratos-conf:ro
      - /opt/so/log/kratos/:/kratos-log:rw
@@ -71,7 +71,7 @@ delete_so-kratos_so-status.disabled:

 wait_for_kratos:
  http.wait_for_successful_query:
-    - name: 'http://{{ GLOBALS.manager }}:4434/'
+    - name: 'http://{{ DOCKERMERGED.containers['so-kratos'].ips['soauth'] }}:4434/'
    - ssl: True
    - verify_ssl: False
    - status:
@@ -220,6 +220,26 @@ logrotate:
      - extension .log
      - dateext
      - dateyesterday
+    /opt/so/log/salt/virtual_node_manager:
+      - daily
+      - rotate 14
+      - missingok
+      - copytruncate
+      - compress
+      - create
+      - extension .log
+      - dateext
+      - dateyesterday
+    /opt/so/log/salt/so-salt-cloud:
+      - daily
+      - rotate 14
+      - missingok
+      - copytruncate
+      - compress
+      - create
+      - extension .log
+      - dateext
+      - dateyesterday
    /opt/so/log/salt/so-soup-grid-highstate:
      - daily
      - rotate 14
@@ -140,6 +140,20 @@ logrotate:
      multiline: True
      global: True
      forcedType: "[]string"
+    "/opt/so/log/salt/virtual_node_manager":
+      description: List of logrotate options for this file.
+      title: /opt/so/log/salt/virtual_node_manager
+      advanced: True
+      multiline: True
+      global: True
+      forcedType: "[]string"
+    "/opt/so/log/salt/so-salt-cloud":
+      description: List of logrotate options for this file.
+      title: /opt/so/log/salt/so-salt-cloud
+      advanced: True
+      multiline: True
+      global: True
+      forcedType: "[]string"
    "/opt/so/log/salt/so-soup-grid-highstate":
      description: List of logrotate options for this file.
      title: /opt/so/log/salt/so-soup-grid-highstate
@@ -3,9 +3,16 @@ beacons:
  postgres_pillar_beacon:
    - interval: {{ AUTOAPPLY.drain_interval }}
    - disable_during_state_run: False
-  rules_beacon:
+  local_files_beacon:
    - interval: {{ AUTOAPPLY.drain_interval }}
    - disable_during_state_run: False
+    # Tags are app names in salt/reactor/pillar_push_map.yaml.
+    # Allowlist on purpose: salt writes elsewhere under local/salt/ and would self-retrigger.
    - paths:
        /opt/so/saltstack/local/salt/suricata/rules: suricata
        /opt/so/saltstack/local/salt/strelka/rules/compiled: strelka
+        /opt/so/saltstack/local/salt/zeek/policy: zeek
+        /opt/so/saltstack/local/salt/zeek/zkg: zeek
+        /opt/so/saltstack/local/salt/elasticsearch/files/ingest: elasticsearch
+        /opt/so/saltstack/local/salt/elasticsearch/roles: elasticsearch
+        /opt/so/saltstack/local/salt/logstash/pipelines/config/custom: logstash
@@ -106,7 +106,8 @@ while [[ $# -gt 0 ]]; do
  esac
 done

-hydraUrl=${HYDRA_URL:-http://127.0.0.1:4445}
+hydraContainer=${HYDRA_CONTAINER:-so-hydra}
+hydraUrl=${HYDRA_URL:-http://localhost:4445}
 socRolesFile=${SOC_ROLES_FILE:-/opt/so/conf/soc/soc_clients_roles}
 soUID=${SOCORE_UID:-939}
 soGID=${SOCORE_GID:-939}
@@ -124,6 +125,10 @@ function fail() {
  exit 1
 }

+function hydraCurl() {
+  docker exec -i "$hydraContainer" curl "$@"
+}
+
 function require() {
  cmd=$1
  which "$1" 2>&1 > /dev/null
@@ -133,8 +138,8 @@ function require() {
 # Verify this environment is capable of running this script
 function verifyEnvironment() {
  require "jq"
-  require "curl"
-  response=$(curl -Ss -L ${hydraUrl}/health/alive)
+  require "docker"
+  response=$(hydraCurl -Ss -L ${hydraUrl}/health/alive)
  [[ "$response" != '{"status":"ok"}' ]] && fail "Unable to communicate with Hydra; specify URL via HYDRA_URL environment variable"
 }

@@ -164,7 +169,7 @@ function ensureRoleFileExists() {
 }

 function listClients() {
-  response=$(curl -Ss -L -f ${hydraUrl}/admin/clients)
+  response=$(hydraCurl -Ss -L -f ${hydraUrl}/admin/clients)
  [[ $? != 0 ]] && fail "Unable to communicate with Hydra"

  clientIds=$(echo "${response}" | jq -r ".[] | .client_id" | sort)
@@ -251,7 +256,7 @@ function createClient() {
 EOF
  )
  
-  response=$(curl -Ss -L --fail-with-body -X POST ${hydraUrl}/admin/clients -d "$body")
+  response=$(hydraCurl -Ss -L --fail-with-body -X POST ${hydraUrl}/admin/clients -d "$body")
  if [[ $? != 0 ]]; then
    error=$(echo $response | jq .error)
    fail "Failed to submit request to Hydra: $error"
@@ -283,7 +288,7 @@ function update() {
 EOF
  )
  
-  response=$(curl -Ss -L --fail-with-body -X PATCH ${hydraUrl}/admin/clients/$id -d "$body")
+  response=$(hydraCurl -Ss -L --fail-with-body -X PATCH ${hydraUrl}/admin/clients/$id -d "$body")
  if [[ $? != 0 ]]; then
    error=$(echo $response | jq .error)
    fail "Failed to submit request to Hydra: $error"
@@ -305,7 +310,7 @@ function generateSecret() {
 EOF
  )
  
-  response=$(curl -Ss -L --fail-with-body -X PATCH ${hydraUrl}/admin/clients/$id -d "$body")
+  response=$(hydraCurl -Ss -L --fail-with-body -X PATCH ${hydraUrl}/admin/clients/$id -d "$body")
  if [[ $? != 0 ]]; then
    error=$(echo $response | jq .error)
    fail "Failed to submit request to Hydra: $error"
@@ -317,7 +322,7 @@ function deleteClient() {

  [[ ${identityId} == "" ]] && fail "Client not found"

-  response=$(curl -Ss -XDELETE -L --fail-with-body "${hydraUrl}/admin/clients/$identityId")
+  response=$(hydraCurl -Ss -XDELETE -L --fail-with-body "${hydraUrl}/admin/clients/$identityId")
  if [[ $? != 0 ]]; then
    error=$(echo $response | jq .error)
    fail "Failed to submit request to Hydra: $error"
@@ -20,7 +20,7 @@ is older than debounce_seconds, this script:
    with the deduped actions list passed as pillar kwargs
  * deletes the contributed intent files on successful dispatch

-Reactor sls files (push_suricata, push_strelka, push_pillar) write intents
+Reactor sls files (push_files, push_pillar) write intents
 but never dispatch directly
 """

@@ -129,7 +129,8 @@ while [[ $# -gt 0 ]]; do
  esac
 done

-kratosUrl=${KRATOS_URL:-http://127.0.0.1:4434/admin}
+kratosContainer=${KRATOS_CONTAINER:-so-kratos}
+kratosUrl=${KRATOS_URL:-http://localhost:4434/admin}
 databasePath=${KRATOS_DB_PATH:-/nsm/kratos/db/db.sqlite}
 databaseTimeout=${KRATOS_DB_TIMEOUT:-5000}
 bcryptRounds=${BCRYPT_ROUNDS:-12}
@@ -154,6 +155,10 @@ function fail() {
  exit 1
 }

+function kratosCurl() {
+  docker exec -i "$kratosContainer" curl "$@"
+}
+
 function require() {
  cmd=$1
  which "$1" 2>&1 > /dev/null
@@ -164,18 +169,18 @@ function require() {
 function verifyEnvironment() {
  require "htpasswd"
  require "jq"
-  require "curl"
+  require "docker"
  require "openssl"
  require "sqlite3"
  [[ ! -f $databasePath ]] && fail "Unable to find database file; specify path via KRATOS_DB_PATH environment variable"
-  response=$(curl -Ss -L ${kratosUrl}/)
+  response=$(kratosCurl -Ss -L ${kratosUrl}/)
  [[ "$response" != "404 page not found" ]] && fail "Unable to communicate with Kratos; specify URL via KRATOS_URL environment variable"
 }

 function findIdByEmail() {
  email=${1,,}

-  response=$(curl -Ss -L ${kratosUrl}/identities)
+  response=$(kratosCurl -Ss -L ${kratosUrl}/identities)
  identityId=$(echo "${response}" | jq -r ".[] | select(.verifiable_addresses[0].value == \"$email\") | .id")
  echo $identityId
 }
@@ -416,7 +421,7 @@ function syncAll() {
 }

 function listUsers() {
-  response=$(curl -Ss -L ${kratosUrl}/identities)
+  response=$(kratosCurl -Ss -L ${kratosUrl}/identities)
  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"

  users=$(echo "${response}" | jq -r ".[] | .verifiable_addresses[0].value" | sort)
@@ -495,7 +500,7 @@ function createUser() {
 EOF
  )
  
-  response=$(curl -Ss -L ${kratosUrl}/identities -d "$addUserJson")
+  response=$(kratosCurl -Ss -L ${kratosUrl}/identities -d "$addUserJson")
  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"

  identityId=$(echo "${response}" | jq -r ".id")
@@ -518,7 +523,7 @@ function updateStatus() {
  identityId=$(findIdByEmail "$email")
  [[ ${identityId} == "" ]] && fail "User not found"

-  response=$(curl -Ss -L "${kratosUrl}/identities/$identityId")
+  response=$(kratosCurl -Ss -L "${kratosUrl}/identities/$identityId")
  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"

  schemaId=$(echo "$response" | jq -r .schema_id)
@@ -531,7 +536,7 @@ function updateStatus() {
    state="inactive"
  fi
  body="{ \"schema_id\": \"$schemaId\", \"state\": \"$state\", \"traits\": $traitBlock }"
-  response=$(curl -fSsL -XPUT -H "Content-Type: application/json" "${kratosUrl}/identities/$identityId" -d "$body")
+  response=$(kratosCurl -fSsL -XPUT -H "Content-Type: application/json" "${kratosUrl}/identities/$identityId" -d "$body")
  [[ $? != 0 ]] && fail "Unable to update user"
 }

@@ -550,7 +555,7 @@ function updateUserProfile() {
  identityId=$(findIdByEmail "$email")
  [[ ${identityId} == "" ]] && fail "User not found"

-  response=$(curl -Ss -L "${kratosUrl}/identities/$identityId")
+  response=$(kratosCurl -Ss -L "${kratosUrl}/identities/$identityId")
  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"

  schemaId=$(echo "$response" | jq -r .schema_id)
@@ -559,7 +564,7 @@ function updateUserProfile() {
  traitBlock="{\"email\":\"$email\",\"firstName\":\"$firstName\",\"lastName\":\"$lastName\",\"note\":\"$note\"}"

  body="{ \"schema_id\": \"$schemaId\", \"state\": \"$state\", \"traits\": $traitBlock }"
-  response=$(curl -fSsL -XPUT -H "Content-Type: application/json" "${kratosUrl}/identities/$identityId" -d "$body")
+  response=$(kratosCurl -fSsL -XPUT -H "Content-Type: application/json" "${kratosUrl}/identities/$identityId" -d "$body")
  [[ $? != 0 ]] && fail "Unable to update user"
 }

@@ -569,7 +574,7 @@ function deleteUser() {
  identityId=$(findIdByEmail "$email")
  [[ ${identityId} == "" ]] && fail "User not found"

-  response=$(curl -Ss -XDELETE -L "${kratosUrl}/identities/$identityId")
+  response=$(kratosCurl -Ss -XDELETE -L "${kratosUrl}/identities/$identityId")
  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"

  rolesTmpFile="${socRolesFile}.tmp"
@@ -28,6 +28,7 @@ INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk '{print $2}')
 # percentage like "25%"). Empty means so-soup-grid-highstate uses the salt:auto_apply:batch
 # pillar default.
 BATCHSIZE=
+DEFAULT_DOCKER_RANGE='172.17.1.0/24'
 SOUP_LOG=/root/soup.log
 SOUP_DEBUG_LOG=/root/soup-debug.log
 WHATWOULDYOUSAYYAHDOHERE=soup
@@ -331,29 +332,24 @@ check_pillar_items() {
 }

 check_cluster_health() {
+  # Require a 'green' cluster before upgrading
  echo "Checking Elasticsearch cluster health."
-  # Require a 'green' cluster before upgrading; anything less (yellow, red, or
-  # unreachable) blocks. Modeled on the wait used in so-elasticsearch-roles-load.
  if so-elasticsearch-query "_cluster/health?wait_for_status=green&timeout=120s" --fail > /dev/null 2>&1; then
    printf "\nThe Elasticsearch cluster is healthy (green). We can proceed with SOUP.\n\n"
-  else
-    printf "\nThe Elasticsearch cluster is not green. Please resolve the cluster health issue so the cluster is green before running SOUP again.\n\n"
-    exit 0
+    return
  fi
+
+  if command -v so-elasticsearch-troubleshoot > /dev/null 2>&1; then
+    printf "\nRunning so-elasticsearch-troubleshoot for additional detail.\n"
+    so-elasticsearch-troubleshoot || true
+  fi
+
+  printf "\nThe Elasticsearch cluster is not green. Resolve the cluster health issue so the cluster is green before running SOUP again.\n\n"
+  exit 0
 }

 check_fleet_server() {
  echo "Checking that Elastic Fleet Server is responding."
-  # Before checking fleet health, check for and fix known issue with elastic-agent container and fs.protected_symlinks
-  local protected_symlinks=$(sysctl -b fs.protected_symlinks)
-  if [[ "$protected_symlinks" == "1" ]]; then
-    # disable fs.protected_symlinks and restart elasticfleet
-    sysctl -w fs.protected_symlinks=0
-    docker stop so-elastic-fleet; docker rm -f so-elastic-fleet
-    printf "\nUpdated sysctl fs.protected_symlinks. Restarting fleet before running health check and continuing with soup.\n"
-    salt-call state.apply elasticfleet queue=True
-  fi
-  
  # Modeled on the wait_for_so-elastic-fleet state check in elasticfleet/enabled.sls,
  # which waits for HTTP 200 from the Fleet Server status API.
  if curl -sk --fail --retry 3 --retry-delay 10 --max-time 30 "https://localhost:8220/api/status" > /dev/null 2>&1; then
@@ -465,6 +461,19 @@ highstate() {
  salt-call state.highstate -l info queue=True
 }

+upgrade_searchnode_elasticsearch() {
+  # Run the elasticsearch state across the true elastic cluster (non-heavy) with a retry attempt
+  #   Excludes the manager, so that kibana & elasticfleet are not upgraded until searchnodes are upgraded.
+  echo "Getting ready to upgrade Elasticsearch across the grid. This may take a while..."
+  if salt -b 10% -C "I@elasticsearch:enabled and not G@role:so-heavynode and not G@id:${MINIONID}" state.apply elasticsearch queue=True; then
+    return 0
+  fi
+  
+  echo "Initial elasticsearch state attempt had a problem; retrying in 30 seconds."
+  sleep 30
+  salt -b 10% -C "I@elasticsearch:enabled and not G@role:so-heavynode and not G@id:${MINIONID}" state.apply elasticsearch queue=True
+}
+
 push_grid_highstate() {
  # Drive a batched, role-tiered highstate across the rest of the grid so remote minions
  # pick up this upgrade now instead of waiting up to ~2.5 hours for their own scheduled
@@ -493,11 +502,10 @@ push_grid_highstate() {
 masterlock() {
  echo "Locking Salt Master"
  mv -v $TOPFILE $BACKUPTOPFILE
-  # Render the real top file only for the host running soup; every other
-  # minion gets an empty top (no states) while the master is upgrading.
-  echo "{% if grains['id'] == '$MINIONID' %}" > $TOPFILE
-  cat $BACKUPTOPFILE >> $TOPFILE
-  echo "{% endif %}" >> $TOPFILE
+  echo "base:" > $TOPFILE
+  echo "  $MINIONID:" >> $TOPFILE
+  echo "    - ca" >> $TOPFILE
+  echo "    - elasticsearch" >> $TOPFILE
 }

 masterunlock() {
@@ -518,6 +526,7 @@ preupgrade_changes() {
    [[ "$INSTALLEDVERSION" == "3.0.0" ]] && up_to_3.1.0
    [[ "$INSTALLEDVERSION" == "3.1.0" ]] && up_to_3.2.0
    [[ "$INSTALLEDVERSION" == "3.2.0" ]] && up_to_3.3.0
+    [[ "$INSTALLEDVERSION" == "3.3.0" ]] && up_to_3.4.0
    true
 }

@@ -536,6 +545,7 @@ postupgrade_changes() {
    [[ "$POSTVERSION" == "3.0.0" ]] && post_to_3.1.0
    [[ "$POSTVERSION" == "3.1.0" ]] && post_to_3.2.0
    [[ "$POSTVERSION" == "3.2.0" ]] && post_to_3.3.0
+    [[ "$POSTVERSION" == "3.3.0" ]] && post_to_3.4.0
    # All applicable post-upgrade steps completed; clear the resume marker.
    rm -f "$POSTVERSION_FILE"
    true
@@ -1013,9 +1023,6 @@ update_kafka_metadata() {
 up_to_3.2.0() {
  ensure_salt_local_pillar

-  # download 9.3.7 elastic agent packages
-  determine_elastic_agent_upgrade
-
  fix_logstash_0013_lumberjack_pipeline_name

  pin_elasticsearch_data_retention_method
@@ -1034,10 +1041,6 @@ post_to_3.2.0() {
  # SOC database bootstrap is handled by the postgres.enabled highstate.
  scrub_postgres_log_passwords

-  # Generate 9.3.7 elastic agent installers
-  echo "Regenerating Elastic Agent Installers"
-  /sbin/so-elastic-agent-gen-installers
-
  kibana_backport_streams_index_template

  update_kafka_metadata "4.3"
@@ -1047,19 +1050,16 @@ post_to_3.2.0() {
 ### 3.2.0 End ###

 ### 3.3.0 Scripts ###
-
- # Sets fs.protected_symlinks=0
- #
- # Elastic Agent docker image chowns its directory to the running UID 
- # but does not chown the elastic-agent launcher symlink. 
- # Preventing non-root users from following that launcher symlink.
-disable_sysctl_fs_protected_symlink() {
-  salt -C 'I@stig:enabled' state.single sysctl.present name=fs.protected_symlinks value=0 config=/etc/sysctl.conf || true
-}
-    
 up_to_3.3.0() {
-  disable_sysctl_fs_protected_symlink
+  # download 9.4.5 elastic agent packages
+  determine_elastic_agent_upgrade
  
+  # remove existing (patched) elasticsearch index template to match integration naming change
+  if ! remove_elasticsearch_index_template "so-logs-sentinel_one_cloud_funnel.login" "sentinel_one_cloud_funnel.login changed to sentinel_one_cloud_funnel.logins"; then
+    FINAL_MESSAGE_QUEUE+=("WARNING: Unable to automatically remove the so-logs-sentinel_one_cloud_funnel.login index template. This step can be performed manually using the following command:")
+    FINAL_MESSAGE_QUEUE+=("    - sudo so-elasticsearch-query _index_template/so-logs-sentinel_one_cloud_funnel.login -XDELETE && so-checkin")
+  fi
+
  INSTALLEDVERSION=3.3.0
 }

@@ -1086,10 +1086,92 @@ post_to_3.3.0() {
  # Recollate again since some internal DBs were excluded during 3.2.0 soup
  recollate_postgres

+  # Generate 9.4.5 elastic agent installers
+  echo "Regenerating Elastic Agent Installers"
+  /sbin/so-elastic-agent-gen-installers
+
  telegraf_repair
+
+  set_postversion 3.3.0
 }
 ### 3.3.0 End ###

+### 3.4.0 Scripts ###
+up_to_3.4.0() {
+  set_soauth_range
+
+  echo "Removing so-kratos, so-hydra and so-soc so they are recreated on the soauth network."
+  docker rm -f so-kratos so-hydra so-soc >> $SOUP_LOG 2>&1
+
+  INSTALLEDVERSION=3.4.0
+}
+
+set_soauth_range() {
+  local pillar_file=/opt/so/saltstack/local/pillar/docker/soc_docker.sls
+  local current_range suggested authnet authgw input
+
+  [[ -f "$pillar_file" ]] || return 0
+
+  current_range=$(so-yaml.py get -r "$pillar_file" docker.range 2>/dev/null) || return 0
+
+  # A default range gets the 172.17.2.0/24 from docker/defaults.yaml, same as a fresh
+  # install, so there is nothing to ask about.
+  [[ -n "$current_range" && "$current_range" != "$DEFAULT_DOCKER_RANGE" ]] || return 0
+
+  if so-yaml.py get -r "$pillar_file" docker.networks.soauth.range >/dev/null 2>&1; then
+    return 0
+  fi
+
+  suggested=$(echo "${current_range%%/*}" | awk -F'.' '{ printf "%s.%s.%s.%s", $1, $2, ($3 + 1) % 256, $4 }')
+
+  if [[ -z $UNATTENDED ]]; then
+    echo ""
+    echo "This grid uses a custom Docker range ($current_range). The authentication"
+    echo "services are moving to their own isolated network, which needs a second /24"
+    echo "that does not overlap it."
+    echo ""
+    while :; do
+      read -rp "Enter the network without the /24 suffix, or press Enter for ${suggested}: " input
+      [[ -z "$input" ]] && input="$suggested"
+      if valid_soauth_range "$input" "$current_range"; then
+        authnet="$input"
+        break
+      fi
+      echo "That range must be a valid IPv4 network, must not be within 172.17.0.0/24, and must not overlap ${current_range}."
+    done
+  else
+    if ! valid_soauth_range "$suggested" "$current_range"; then
+      FINAL_MESSAGE_QUEUE+=("WARNING: Unable to pick a range for the authentication network alongside $current_range. Set it manually before the next highstate:")
+      FINAL_MESSAGE_QUEUE+=("    - so-yaml.py add $pillar_file docker.networks.soauth.range <network>/24")
+      FINAL_MESSAGE_QUEUE+=("    - so-yaml.py add $pillar_file docker.networks.soauth.gateway <gateway>")
+      return 0
+    fi
+    authnet="$suggested"
+    FINAL_MESSAGE_QUEUE+=("NOTE: The authentication services moved to an isolated Docker network and were assigned ${authnet}/24.")
+    FINAL_MESSAGE_QUEUE+=("    - If that conflicts with your environment, update docker.networks.soauth in $pillar_file and run so-checkin.")
+  fi
+
+  authgw=$(echo "$authnet" | awk -F'.' '{print $1,$2,$3,1}' OFS='.')
+
+  echo "Assigning the authentication network the range ${authnet}/24."
+  so-yaml.py add "$pillar_file" docker.networks.soauth.range "${authnet}/24" >> $SOUP_LOG 2>&1
+  so-yaml.py add "$pillar_file" docker.networks.soauth.gateway "$authgw" >> $SOUP_LOG 2>&1
+}
+
+valid_soauth_range() {
+  local candidate=$1 docker_range=$2
+
+  valid_ip4 "$candidate" || return 1
+  [[ $candidate =~ ^172\.17\.0\. ]] && return 1
+  [[ "${candidate}/24" == "$docker_range" ]] && return 1
+  return 0
+}
+
+post_to_3.4.0() {
+  set_postversion 3.4.0
+}
+### 3.4.0 End ###
+

 repo_sync() {
  echo "Sync the local repo."
@@ -1359,11 +1441,12 @@ verify_es_version_compatibility() {
    local is_active_intermediate_upgrade=1
    # supported upgrade paths for SO-ES versions
    declare -A es_upgrade_map=(
-        ["8.18.4"]="8.18.6 8.18.8 9.0.8"
+      ["8.18.4"]="8.18.6 8.18.8 9.0.8"
 	    ["8.18.6"]="8.18.8 9.0.8"
 	    ["8.18.8"]="9.0.8"
-        ["9.0.8"]="9.3.3 9.3.7"
-        ["9.3.3"]="9.3.7"
+      ["9.0.8"]="9.3.3 9.3.7 9.4.5"
+      ["9.3.3"]="9.3.7 9.4.5"
+      ["9.3.7"]="9.4.5"
    )

    # Elasticsearch MUST upgrade through these versions
@@ -2139,12 +2222,28 @@ main() {

    # ensure the mine is updated and populated before highstates run, following the salt-master restart
    update_salt_mine
-
+    
+    # kick off a searchnode elasticsearch upgrade
+    set +e
+    if [[ "$es_version" != "$target_es_version" ]]; then
+      if salt-key -L accepted | grep -q "_searchnode$" 2>/dev/null; then
+        # only run if there is atleast 1 searchnode
+        upgrade_searchnode_elasticsearch
+      fi
+    fi
+    set -e
+    
    highstate
    check_saltmaster_status
    postupgrade_changes
    [[ $is_airgap -eq 0 ]] && unmount_update
-
+    
+    if [[ "$es_version" != "$target_es_version" ]]; then
+      # Run final elasticsearch / fleet state on manager to ensure addon index templates are created/regenerated and loaded
+      echo "Running final Elastic states at $(date +"%T.%6N"), after upgrade to $NEWVERSION"
+      salt-call state.apply elasticsearch,elasticfleet queue=True
+    fi
+    
    echo ""
    echo "Upgrade to $NEWVERSION complete."

@@ -260,7 +260,7 @@ http {
 		}

 {%  if 'api' in salt['pillar.get']('features', []) %}
-		location ~* (^/oauth2/token.*|^.well-known/jwks.json|^.well-known/openid-configuration) {
+		location ~* (^/oauth2/token.*|^/\.well-known/jwks.json|^/\.well-known/openid-configuration) {
 			limit_req             zone=auth_throttle burst={{ NGINXMERGED.config.throttle_login_burst }} nodelay;
 			limit_req_status      429;
 			proxy_pass            http://{{ GLOBALS.manager }}:4444;
@@ -1,3 +1,6 @@
+# Read by push_pillar.sls (SOC config saves) and push_files.sls (local/salt file edits);
+# both key on the app name. An app missing here waits for the next scheduled highstate.
+#
 # One pillar directory can map to multiple (state, tgt) actions.
 # tgt is a raw salt compound expression. tgt_type is always "compound".
 # Per-action `batch` / `batch_wait` override the orch defaults (25% / 15s).
@@ -0,0 +1,131 @@
+#!py
+
+# Reactor invoked by local_files_beacon when a watched directory under
+# /opt/so/saltstack/local/salt/ changes. The beacon tag is an app name in
+# pillar_push_map.yaml, so file changes and pillar changes route through the same
+# table -- see salt/reactor/push_pillar.sls.
+#
+# The app comes from the event tag, not the payload: salt's beacon loop pops the
+# beacon's 'tag' key off the data and appends it to the event tag instead (see
+# salt/beacons/__init__.py). The reactor renderer sets both `tag` and `data` as
+# module globals.
+#
+# Reactors never dispatch directly. The so-push-drainer schedule picks up ready
+# intents, dedupes across pending files, and dispatches orch.push_batch.
+
+import fcntl
+import json
+import logging
+import os
+import time
+
+from salt.client import Caller
+import yaml
+
+LOG = logging.getLogger(__name__)
+
+PENDING_DIR = '/opt/so/state/push_pending'
+LOCK_FILE = os.path.join(PENDING_DIR, '.lock')
+MAX_PATHS = 20
+
+# The pillar_push_map.yaml is shipped via salt:// but the reactor runs on the
+# master, which mounts the default saltstack tree at this path.
+PUSH_MAP_PATH = '/opt/so/saltstack/default/salt/reactor/pillar_push_map.yaml'
+
+_PUSH_MAP_CACHE = {'mtime': 0, 'data': None}
+
+
+def _load_push_map():
+    try:
+        st = os.stat(PUSH_MAP_PATH)
+    except OSError:
+        LOG.warning('push_files: %s not found', PUSH_MAP_PATH)
+        return {}
+    if _PUSH_MAP_CACHE['mtime'] != st.st_mtime:
+        try:
+            with open(PUSH_MAP_PATH, 'r') as f:
+                _PUSH_MAP_CACHE['data'] = yaml.safe_load(f) or {}
+        except Exception:
+            LOG.exception('push_files: failed to load %s', PUSH_MAP_PATH)
+            _PUSH_MAP_CACHE['data'] = {}
+        _PUSH_MAP_CACHE['mtime'] = st.st_mtime
+    return _PUSH_MAP_CACHE['data'] or {}
+
+
+def _push_enabled():
+    try:
+        caller = Caller()
+        return bool(caller.cmd('pillar.get', 'salt:auto_apply:enabled', True))
+    except Exception:
+        LOG.exception('push_files: pillar.get salt:auto_apply:enabled failed, assuming enabled')
+        return True
+
+
+def _write_intent(key, actions, path):
+    now = time.time()
+    try:
+        os.makedirs(PENDING_DIR, exist_ok=True)
+    except OSError:
+        LOG.exception('push_files: cannot create %s', PENDING_DIR)
+        return
+
+    intent_path = os.path.join(PENDING_DIR, '{}.json'.format(key))
+    lock_fd = os.open(LOCK_FILE, os.O_CREAT | os.O_RDWR, 0o644)
+    try:
+        fcntl.flock(lock_fd, fcntl.LOCK_EX)
+
+        intent = {}
+        if os.path.exists(intent_path):
+            try:
+                with open(intent_path, 'r') as f:
+                    intent = json.load(f)
+            except (IOError, ValueError):
+                intent = {}
+
+        intent.setdefault('first_touch', now)
+        intent['last_touch'] = now
+        intent['actions'] = actions
+        paths = intent.get('paths', [])
+        if path and path not in paths:
+            paths.append(path)
+            paths = paths[-MAX_PATHS:]
+        intent['paths'] = paths
+
+        tmp_path = intent_path + '.tmp'
+        with open(tmp_path, 'w') as f:
+            json.dump(intent, f)
+        os.rename(tmp_path, intent_path)
+    except Exception:
+        LOG.exception('push_files: failed to write intent %s', intent_path)
+    finally:
+        try:
+            fcntl.flock(lock_fd, fcntl.LOCK_UN)
+        finally:
+            os.close(lock_fd)
+
+
+def run():
+    if not _push_enabled():
+        LOG.info('push_files: push disabled, skipping')
+        return {}
+
+    event = data.get('data', data)  # noqa: F821 -- data provided by reactor
+    path = event.get('path', '')
+    app = tag.rsplit('/', 1)[-1].strip()  # noqa: F821 -- tag provided by reactor
+
+    if not app:
+        LOG.debug('push_files: ignoring event with no app segment: tag=%s', tag)  # noqa: F821
+        return {}
+
+    entry = _load_push_map().get(app)
+    if not entry:
+        LOG.warning(
+            'push_files: app "%s" is not in pillar_push_map.yaml; change will be '
+            'picked up at the next scheduled highstate (path=%s)',
+            app, path,
+        )
+        return {}
+
+    _write_intent('files_{}'.format(app), list(entry), path)
+    LOG.info('push_files: intent updated for %s (path=%s)', app, path)
+    return {}
@@ -1,96 +0,0 @@
-#!py
-
-# Reactor invoked by the rules_beacon poll beacon (salt/_beacons/rules_beacon.py) on rule
-# file changes under /opt/so/saltstack/local/salt/strelka/rules/compiled/.
-#
-# Writes (or updates) a push intent at /opt/so/state/push_pending/rules_strelka.json
-# and returns {}. The so-push-drainer schedule picks up ready intents, dedupes
-# across pending files, and dispatches orch.push_batch. Reactors never dispatch
-# directly
-
-import fcntl
-import json
-import logging
-import os
-import time
-
-from salt.client import Caller
-
-LOG = logging.getLogger(__name__)
-
-PENDING_DIR = '/opt/so/state/push_pending'
-LOCK_FILE = os.path.join(PENDING_DIR, '.lock')
-MAX_PATHS = 20
-
-# Mirrors GLOBALS.sensor_roles in salt/vars/globals.map.jinja. Sensor-side
-# strelka runs on exactly these four roles; so-import gets strelka.manager
-# instead, which is not fired on pillar changes.
-SENSOR_ROLES = ['so-eval', 'so-heavynode', 'so-sensor', 'so-standalone']
-
-
-def _sensor_compound():
-    return ' or '.join('G@role:{}'.format(r) for r in SENSOR_ROLES)
-
-
-def _push_enabled():
-    try:
-        caller = Caller()
-        return bool(caller.cmd('pillar.get', 'salt:auto_apply:enabled', True))
-    except Exception:
-        LOG.exception('push_strelka: pillar.get salt:auto_apply:enabled failed, assuming enabled')
-        return True
-
-
-def _write_intent(key, actions, path):
-    now = time.time()
-    try:
-        os.makedirs(PENDING_DIR, exist_ok=True)
-    except OSError:
-        LOG.exception('push_strelka: cannot create %s', PENDING_DIR)
-        return
-
-    intent_path = os.path.join(PENDING_DIR, '{}.json'.format(key))
-    lock_fd = os.open(LOCK_FILE, os.O_CREAT | os.O_RDWR, 0o644)
-    try:
-        fcntl.flock(lock_fd, fcntl.LOCK_EX)
-
-        intent = {}
-        if os.path.exists(intent_path):
-            try:
-                with open(intent_path, 'r') as f:
-                    intent = json.load(f)
-            except (IOError, ValueError):
-                intent = {}
-
-        intent.setdefault('first_touch', now)
-        intent['last_touch'] = now
-        intent['actions'] = actions
-        paths = intent.get('paths', [])
-        if path and path not in paths:
-            paths.append(path)
-            paths = paths[-MAX_PATHS:]
-        intent['paths'] = paths
-
-        tmp_path = intent_path + '.tmp'
-        with open(tmp_path, 'w') as f:
-            json.dump(intent, f)
-        os.rename(tmp_path, intent_path)
-    except Exception:
-        LOG.exception('push_strelka: failed to write intent %s', intent_path)
-    finally:
-        try:
-            fcntl.flock(lock_fd, fcntl.LOCK_UN)
-        finally:
-            os.close(lock_fd)
-
-
-def run():
-    if not _push_enabled():
-        LOG.info('push_strelka: push disabled, skipping')
-        return {}
-
-    path = data.get('path', '')  # noqa: F821 -- data provided by reactor
-    actions = [{'state': 'strelka', 'tgt': _sensor_compound()}]
-    _write_intent('rules_strelka', actions, path)
-    LOG.info('push_strelka: intent updated for path=%s', path)
-    return {}
@@ -1,95 +0,0 @@
-#!py
-
-# Reactor invoked by the rules_beacon poll beacon (salt/_beacons/rules_beacon.py) on rule
-# file changes under /opt/so/saltstack/local/salt/suricata/rules/.
-#
-# Writes (or updates) a push intent at /opt/so/state/push_pending/rules_suricata.json
-# and returns {}. The so-push-drainer schedule picks up ready intents, dedupes
-# across pending files, and dispatches orch.push_batch. Reactors never dispatch
-# directly
-
-import fcntl
-import json
-import logging
-import os
-import time
-
-from salt.client import Caller
-
-LOG = logging.getLogger(__name__)
-
-PENDING_DIR = '/opt/so/state/push_pending'
-LOCK_FILE = os.path.join(PENDING_DIR, '.lock')
-MAX_PATHS = 20
-
-# Mirrors GLOBALS.sensor_roles in salt/vars/globals.map.jinja. Suricata also
-# runs on so-import per salt/top.sls, so that role is appended below.
-SENSOR_ROLES = ['so-eval', 'so-heavynode', 'so-sensor', 'so-standalone']
-
-
-def _sensor_compound_plus_import():
-    return ' or '.join('G@role:{}'.format(r) for r in SENSOR_ROLES) + ' or G@role:so-import'
-
-
-def _push_enabled():
-    try:
-        caller = Caller()
-        return bool(caller.cmd('pillar.get', 'salt:auto_apply:enabled', True))
-    except Exception:
-        LOG.exception('push_suricata: pillar.get salt:auto_apply:enabled failed, assuming enabled')
-        return True
-
-
-def _write_intent(key, actions, path):
-    now = time.time()
-    try:
-        os.makedirs(PENDING_DIR, exist_ok=True)
-    except OSError:
-        LOG.exception('push_suricata: cannot create %s', PENDING_DIR)
-        return
-
-    intent_path = os.path.join(PENDING_DIR, '{}.json'.format(key))
-    lock_fd = os.open(LOCK_FILE, os.O_CREAT | os.O_RDWR, 0o644)
-    try:
-        fcntl.flock(lock_fd, fcntl.LOCK_EX)
-
-        intent = {}
-        if os.path.exists(intent_path):
-            try:
-                with open(intent_path, 'r') as f:
-                    intent = json.load(f)
-            except (IOError, ValueError):
-                intent = {}
-
-        intent.setdefault('first_touch', now)
-        intent['last_touch'] = now
-        intent['actions'] = actions
-        paths = intent.get('paths', [])
-        if path and path not in paths:
-            paths.append(path)
-            paths = paths[-MAX_PATHS:]
-        intent['paths'] = paths
-
-        tmp_path = intent_path + '.tmp'
-        with open(tmp_path, 'w') as f:
-            json.dump(intent, f)
-        os.rename(tmp_path, intent_path)
-    except Exception:
-        LOG.exception('push_suricata: failed to write intent %s', intent_path)
-    finally:
-        try:
-            fcntl.flock(lock_fd, fcntl.LOCK_UN)
-        finally:
-            os.close(lock_fd)
-
-
-def run():
-    if not _push_enabled():
-        LOG.info('push_suricata: push disabled, skipping')
-        return {}
-
-    path = data.get('path', '')  # noqa: F821 -- data provided by reactor
-    actions = [{'state': 'suricata', 'tgt': _sensor_compound_plus_import()}]
-    _write_intent('rules_suricata', actions, path)
-    LOG.info('push_suricata: intent updated for path=%s', path)
-    return {}
@@ -1,7 +1,5 @@
 reactor:
-  - 'salt/beacon/*/rules_beacon/suricata':
-    - salt://reactor/push_suricata.sls
-  - 'salt/beacon/*/rules_beacon/strelka':
-    - salt://reactor/push_strelka.sls
+  - 'salt/beacon/*/local_files_beacon/*':
+    - salt://reactor/push_files.sls
  - 'salt/beacon/*/postgres_pillar_beacon/audit_settings':
    - salt://reactor/push_pillar.sls
@@ -14,6 +14,8 @@
 {%   do SOCDEFAULTS.soc.config.server.modules[module].update({'hostUrl': application_url}) %}
 {% endfor %}

+{% do SOCDEFAULTS.soc.config.server.modules.kratos.update({'publicHostUrl': 'http://' ~ DOCKERMERGED.containers['so-kratos'].ips['soauth'] ~ ':4433/'}) %}
+
 {# add all grid heavy nodes to soc.server.modules.elastic.remoteHostUrls #}
 {% for node_type, minions in salt['pillar.get']('elasticsearch:nodes', {}).items() %}
 {%   if node_type in ['heavynode'] %}
@@ -1380,6 +1380,7 @@ soc:
          retryFailureMaxAttempts: 5
        kratos:
          hostUrl:
+          publicHostUrl:
        hydra:
          hostUrl:
        elastalertengine:
@@ -1539,6 +1540,7 @@ soc:
            DetectionEngineer: gemma@SOAI
          useMemory: true
          useMemoryScanner: false
+          dontScanBefore: ""
          memoryScanIntervalSeconds: 300
          memoryProximityThreshold: 0.8
          messageProximityThreshold: 0.5
@@ -1546,9 +1548,15 @@ soc:
          maxGlobalMemoriesToInclude: 5
          maxUserMemoriesToReconcile: 20
          maxGlobalMemoriesToReconcile: 20
+          memoryExtractBatchSize: 5
+          maxMemoryRetries: 2
          memoryModel: gemma@SOAI
          embedModel: amazon.titan-embed-text-v2@SOAI
          reconcileModel: gemma@SOAI
+          memoryPersona: ""
+          reconcilePersona: ""
+          toolUseTurnAttempts: 12
+          toolUseTurnDelayMs: 175
        onionconfig:
          saltstackDir: /opt/so/saltstack
          bypassEnabled: false
@@ -23,7 +23,9 @@ so-soc:
    - name: so-soc
    - networks:
      - sobridge:
-        - ipv4_address: {{ DOCKERMERGED.containers['so-soc'].ip }}
+        - ipv4_address: {{ DOCKERMERGED.containers['so-soc'].ips['sobridge'] }}
+      - soauth:
+        - ipv4_address: {{ DOCKERMERGED.containers['so-soc'].ips['soauth'] }}
    - binds:
      - /nsm/rules:/nsm/rules:rw
      - /opt/so/conf/strelka:/opt/sensoroni/yara:rw
@@ -853,6 +853,12 @@ soc:
            description: Enables the memory scanner for automatic memory extraction from historical sessions.
            global: True
            forcedType: bool
+          dontScanBefore:
+            description: A date specifying how far back to scan for memories. Must be in UTC format (2026-08-31T22:05:48Z).
+            regex: '^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z)?$'
+            regexFailureMessage: Expecting date in RFC3339 format (2026-08-31T22:05:48Z)
+            global: True
+            advanced: True
          memoryScanIntervalSeconds:
            description: How long to wait in seconds between attempts to scan sessions for new memories.
            global: True
@@ -884,6 +890,32 @@ soc:
          reconcileModel:
            description: The model to use when reconciling memories that contain nearly the same content.
            global: True
+          memoryPersona:
+            description: Text appended to the built-in prompt of the memory extraction agent, managed from the Agent Studio. Use it to steer what is worth remembering.
+            global: True
+            readonlyUi: True
+            multiline: True
+          reconcilePersona:
+            description: Text appended to the built-in prompt of the memory reconciliation agent, managed from the Agent Studio. Use it to steer how new memories are merged with existing ones.
+            global: True
+            readonlyUi: True
+            multiline: True
+          toolUseTurnAttempts:
+            description: Number of times the API will retry looking up a tool request that has finished streaming but has not yet been saved to the database. Each retry waits toolUseTurnDelayMs before checking again. If tool auto-approval fails intermittently, increasing this value gives the message more time to be saved.
+            global: True
+            advanced: True
+          toolUseTurnDelayMs:
+            description: Milliseconds to wait between attempts to look up a tool request that has finished streaming but has not yet been saved to the database. The API retries up to toolUseTurnAttempts times, so this value times the attempt count is the maximum wait before auto-approval fails. Increase it if tool auto-approval fails intermittently.
+            global: True
+            advanced: True
+          memoryExtractBatchSize:
+            description: Max number of a session's messages that will be processed for memory extractions at a time. Larger values are more efficient but risk exceeding the memory extraction LLM's size limits, due to the variable and unpredictable size of each message.
+            global: True
+            advanced: True
+          maxMemoryRetries:
+            description: The number of times to retry extracting memories from a session if errors occur.
+            global: True
+            advanced: True
      client:
        assistant:
          enabled:
@@ -65,15 +65,6 @@ run_remediate:
    - success_retcodes:
      - 2

-# Elastic Agent docker image chowns its directory to the running UID but does not
-#  chown the elastic-agent launcher symlink. fs.protected_symlinks=1 then prevents
-#  non-root users from following that launcher symlink.
-{# OSCAP rule id: xccdf_org.ssgproject.content_rule_sysctl_fs_protected_symlinks #}
-fs.protected_symlinks:
-  sysctl.present:
-    - value: 0
-    - config: /etc/sysctl.conf
-
 {# OSCAP rule id: xccdf_org.ssgproject.content_rule_disable_ctrlaltdel_burstaction #}
 disable_ctrl_alt_del_action:
  file.replace:
@@ -1601,7 +1601,7 @@ DISA STIG for Oracle Linux 9 V1R3.</xccdf-1.2:description>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sudoers_validate_passwd" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_crypto_fips_enabled" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_fs_protected_hardlinks" selected="true"/>
-        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_fs_protected_symlinks" selected="false"/>
+        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_fs_protected_symlinks" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_kernel_core_pattern" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_kernel_dmesg_restrict" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_kernel_exec_shield" selected="true"/>
@@ -2202,7 +2202,7 @@ standard DISA STIG for Oracle Linux 9 profile.</xccdf-1.2:description>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sudoers_validate_passwd" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_crypto_fips_enabled" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_fs_protected_hardlinks" selected="true"/>
-        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_fs_protected_symlinks" selected="false"/>
+        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_fs_protected_symlinks" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_kernel_core_pattern" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_kernel_dmesg_restrict" selected="true"/>
        <xccdf-1.2:select idref="xccdf_org.ssgproject.content_rule_sysctl_kernel_exec_shield" selected="true"/>
@@ -335,7 +335,7 @@
 {%-   do TELEGRAFMERGED.scripts[GLOBALS.role.split('-')[1]].remove('sostatus.sh') %}
 [[inputs.exec]]
   commands = [
-      "/scripts/sostatus.sh"
+      ["/scripts/sostatus.sh"]
   ]
   data_format = "influx"
   timeout = "15s"
@@ -346,7 +346,7 @@
 [[inputs.exec]]
   commands = [
 {%-   for script in TELEGRAFMERGED.scripts[GLOBALS.role.split('-')[1]] %}
-      "/scripts/{{script}}"{% if not loop.last %},{% endif %}
+      ["/scripts/{{script}}"]{% if not loop.last %},{% endif %}
 {%-   endfor %}
   ]
  data_format = "influx"
@@ -375,7 +375,7 @@
 {%- if GLOBALS.is_manager or GLOBALS.role == 'so-heavynode' %}
 [[ inputs.exec ]]
  commands = [
-    "/scripts/esindexsize.sh"
+    ["/scripts/esindexsize.sh"]
  ]
  data_format = "influx"
  interval = "1h"
@@ -54,8 +54,8 @@
 {%
  do GLOBALS.update({
    'application_urls': {
-      'hydra': 'http://' ~ GLOBALS.manager ~ ':4445/',
-      'kratos': 'http://' ~ GLOBALS.manager ~ ':4434/',
+      'hydra': 'http://' ~ DOCKERMERGED.containers['so-hydra'].ips['soauth'] ~ ':4445/',
+      'kratos': 'http://' ~ DOCKERMERGED.containers['so-kratos'].ips['soauth'] ~ ':4434/',
      'elastic': 'https://' ~ GLOBALS.manager ~ ':9200/',
      'influxdb': 'https://' ~ GLOBALS.manager ~ ':8086/'
    }
@@ -276,9 +276,20 @@ collect_dockernet() {
 			whiptail_invalid_input
 			whiptail_dockernet_sosnet "$DOCKERNET"
 		done
+
+		whiptail_authnet_sosnet "$(adjacent_net "$DOCKERNET")"
+
+		while ! valid_ip4 "$AUTHNET" || [[ $AUTHNET =~ "172.17.0." ]] || [[ "$AUTHNET" == "$DOCKERNET" ]]; do
+			whiptail_invalid_input
+			whiptail_authnet_sosnet "$AUTHNET"
+		done
 	fi
 }

+adjacent_net() {
+	echo "$1" | awk -F'.' '{ printf "%s.%s.%s.%s", $1, $2, ($3 + 1) % 256, $4 }'
+}
+
 collect_gateway() {
 	whiptail_management_interface_gateway

@@ -1399,6 +1410,15 @@ docker_pillar() {
 			"docker:"\
 			"  range: '$DOCKERNET/24'"\
 			"  gateway: '$DOCKERGATEWAY'" > $docker_pillar_file
+
+		if [ ! -z "$AUTHNET" ]; then
+			AUTHGATEWAY=$(echo $AUTHNET | awk -F'.' '{print $1,$2,$3,1}' OFS='.')
+			printf '%s\n'\
+				"  networks:"\
+				"    soauth:"\
+				"      range: '$AUTHNET/24'"\
+				"      gateway: '$AUTHGATEWAY'" >> $docker_pillar_file
+		fi
 	fi
 }

@@ -365,6 +365,18 @@ whiptail_dockernet_sosnet() {

 }

+whiptail_authnet_sosnet() {
+
+	[ -n "$TESTING" ] && return
+
+	AUTHNET=$(whiptail --title "$whiptail_title" --inputbox \
+	"\nEnter a second /24 size network range WITHOUT the /24 suffix. The authentication services are isolated on their own network so that the identity provider is not reachable from other containers. It must not overlap the range you just entered, and any range within 172.17.0.0/24 cannot be used." 13 65 "$1" 3>&1 1>&2 2>&3)
+
+	local exitstatus=$?
+	whiptail_check_exitstatus $exitstatus
+
+}
+
 whiptail_end_settings() {
 	[ -n "$TESTING" ] && return

@@ -427,6 +439,7 @@ whiptail_end_settings() {
 	[[ -n $WEBUSER ]] && __append_end_msg "Web User: $WEBUSER"

 	[[ -n $DOCKERNET ]] && __append_end_msg "Docker network: $DOCKERNET/24"
+	[[ -n $AUTHNET ]] && __append_end_msg "Authentication network: $AUTHNET/24"
 	if [[ ${#ntp_servers[@]} -gt 0 ]]; then
 		__append_end_msg "NTP Servers:"
 		for server in "${ntp_servers[@]}"; do
Author	SHA1	Message	Date
Mike Reeves	866a046853	Annotate the soauth network leaves instead of the networks map SOC reported "Malformed config setting (docker.networks.soauth.range): Setting name 'networks' conflicts with another similarly named setting" and refused to render the config tree. soc_docker.yaml annotated docker.networks itself, and every key in that block was a scalar, so FlattenAnnotations registered docker.networks as a setting. defaults.yaml holds a nested map there, so FlattenPillar separately registered docker.networks.soauth.range, .gateway and .manager_only, and HydrateAnnotations unions the two sets. The config tree gives each id segment either a value or children, never both, so addToNode pushed docker.networks as a childless leaf and then threw when docker.networks.soauth.range tried to descend through it. The annotations move down to the three leaves that actually carry values, so docker.networks is a plain branch. This matches docker.containers, which is nested the same way and has never carried annotations of its own. docker.ulimits and the per-container networks list are unaffected: their pillar values are lists rather than maps, so they stay single settings.	2026-09-18 13:40:05 -04:00
Mike Reeves	65e261475d	Mention Hydra in the soauth network description The description was written alongside the Kratos change and named only the Kratos admin API, but so-hydra now sits on soauth too. The trailing note about rebuilding the firewall is dropped, since the setting is readonly and cannot be changed from SOC in the first place.	2026-09-18 10:17:20 -04:00
Mike Reeves	bbc28c88b7	Isolate the Hydra admin API on the soauth docker network The Hydra admin API creates OAuth clients and introspects tokens without authentication, relying on network isolation the same way Kratos does, but so-hydra published 0.0.0.0:4445:4445 and sat on sobridge. With userland-proxy left at its default of true, Docker ran a proxy listener on the host, so any container reached the admin API through the manager IP or the bridge gateway regardless of its docker network. so-hydra now sits alone on soauth and no longer publishes 4445. 4444 is still published, so the nginx proxy for /oauth2/token and the well-known endpoints is unchanged. so-soc is already dual homed from the Kratos change and reaches the admin API over soauth, so only its hostUrl moves. wait_for_hydra polls the container address instead of the host port. so-client reaches the admin API through docker exec, mirroring so-user. Exit codes still propagate, so the --fail-with-body error handling is unchanged. so-hydra is removed in up_to_3.4.0 alongside so-kratos and so-soc so the highstate recreates it with the correct network membership. The auth network range is already handled by set_soauth_range.	2026-09-17 16:58:49 -04:00
Mike Reeves	cf3a4ebc27	Isolate the Kratos admin API on a dedicated soauth docker network The Kratos admin API is unauthenticated by design and relies on network isolation, but so-kratos published 0.0.0.0:4434:4434 and daemon.json leaves userland-proxy at its default of true. Docker therefore ran a proxy listener on the host, and any container reached the admin API through the manager IP or the bridge gateway regardless of its docker network. so-kratos now sits alone on a new soauth network and no longer publishes 4434. 4433 is still published, so nginx is unchanged. so-soc is dual homed and reaches the admin API over soauth, with publicHostUrl set explicitly. so-user reaches it through docker exec, and wait_for_kratos polls the container address instead of the host port. firewall/iptables.jinja hardcoded sobridge in every generated DNAT, ACCEPT, masquerade and isolation rule, so it is now driven by a docker:networks map and a per-container networks list. Containers that do not declare one default to sobridge, and .ip still resolves to the primary network's address. soauth is only created on the roles that run so-kratos. Setup already allows a custom docker range, so it now also prompts for the auth network range and writes it to the docker pillar. Grids on the default range need no decision during soup, since they get the same 172.17.2.0/24 a fresh install would. Grids that set a custom range are prompted, defaulting to the adjacent /24, and unattended upgrades take that default with a notice rather than blocking. so-kratos and so-soc are removed in up_to_3.4.0 so the highstate recreates them with the correct network membership.	2026-09-09 16:09:00 -04:00
Mike Reeves	c49008a413	Merge pull request #16216 from Security-Onion-Solutions/TOoSmOotH-patch-1 Bump version from 3.3.0 to 3.4.0	2026-09-08 15:15:12 -04:00
Mike Reeves	fcbea1a1c4	Add 3.4.0 option to discussion template	2026-09-08 15:14:26 -04:00
Mike Reeves	6736f9c3a0	Bump version from 3.3.0 to 3.4.0	2026-09-08 15:13:40 -04:00
Mike Reeves	86ae51b2b5	Merge pull request #16214 from Security-Onion-Solutions/3.3.0 3.3.0	2026-09-08 14:54:21 -04:00
Mike Reeves	2cde9abec2	3.3.0	2026-09-08 14:52:40 -04:00
Mike Reeves	57ce2cea0d	Merge pull request #16213 from Security-Onion-Solutions/3.3.0 3.3.0	2026-09-08 14:51:38 -04:00
Mike Reeves	ff7555d93b	3.3.0	2026-09-08 14:49:39 -04:00
Jorge Reyes	65b84026a8	Merge pull request #16212 from Security-Onion-Solutions/reyesj2-patch-5 drop remapped sentinel_one_cloud_funnel.login index template	2026-09-08 08:49:09 -05:00
reyesj2	721b1d6207	drop remapped sentinel_one_cloud_funnel.login index template	2026-09-08 08:30:00 -05:00
Josh Brower	a74046fc24	Merge pull request #16210 from Security-Onion-Solutions/soup-fixes Fix soup	2026-09-04 17:53:50 -04:00
Josh Brower	ea539f8679	Fix soup	2026-09-04 17:50:32 -04:00
coreyogburn	54bb0e2c12	Merge pull request #16209 from Security-Onion-Solutions/cogburn/memory-fixes Batch Size and Retries	2026-09-04 12:52:19 -06:00
Corey Ogburn	67b4d82f62	Improved Description of MemoryExtractBatchSize	2026-09-04 12:49:29 -06:00
Corey Ogburn	30574fdbb9	Batch Size and Retries 2 new config fields. Batch size is used to limit how many message turns we put in the transcript when we ask the memory agent to extract facts. The retries helps limit how many times we ask the memory agent to process a problematic session.	2026-09-04 12:18:29 -06:00
Josh Brower	64771e9e1d	Merge pull request #16207 from Security-Onion-Solutions/soup-es-tshoot Run estshoot if cluster is not green	2026-09-03 12:09:14 -04:00
Josh Brower	01ca33b90a	Run estshoot if cluster is not green	2026-09-03 11:48:16 -04:00
coreyogburn	599d19215b	Merge pull request #16206 from Security-Onion-Solutions/cogburn/tool_turn_config Define ToolTurn Defaults	2026-09-02 11:49:54 -06:00
Corey Ogburn	ee671e7ec9	Define ToolTurn Defaults When auto approving tools, we might approve a tool_request before it's been saved to ES. These vars describe some leniency in retrying when the message can't be found before giving up.	2026-09-02 11:42:29 -06:00
coreyogburn	42d429a11e	Merge pull request #16205 from Security-Onion-Solutions/cogburn/fix-regex Fix Regex	2026-09-02 09:14:02 -06:00
Corey Ogburn	332a5d11bc	Fix Regex Double quoted stings in yaml allow for escape sequences like `\n` and `\t` but when used around a regex, salt will hang up on `\d` not being a valid escape sequence. Switching to single quotes so escapes aren't processed.	2026-09-02 09:08:46 -06:00
coreyogburn	42a62c90a5	Merge pull request #16203 from Security-Onion-Solutions/cogburn/memory-metrics Cogburn/memory metrics	2026-09-01 09:52:39 -06:00
Corey Ogburn	fb7e065590	Remove YYYY-MM-DD No more instances of YYYY-MM-DD	2026-09-01 09:32:26 -06:00
Corey Ogburn	bce6b0c1fe	Allow Timestamp in dontScanBefore Now storing UTC timestamp in the dontScanBefore field.	2026-08-31 16:11:29 -06:00
Corey Ogburn	b9ba7df80c	DontScanBefore New field that'll stop the memory scanner from scanning before an indicated date. Leaving it empty lets the scanner scan everything. The regex for it allows YYYY-MM-DD and ensures months only allow the max number of days (no June the 43rd).	2026-08-28 19:02:39 -06:00
Jorge Reyes	97fddc0719	Merge pull request #16201 from Security-Onion-Solutions/reyesj2-patch-1 fix salt batching command	2026-08-28 16:32:08 -05:00
Jorge Reyes	a5deee1444	fix salt batching command	2026-08-28 16:27:29 -05:00
Jorge Reyes	3585ccca79	Merge pull request #16190 from Security-Onion-Solutions/reyesj2/es945 UPGRADE: Elasticsearch 9.4.5	2026-08-28 16:18:13 -05:00
reyesj2	dd035beec4	include fleet state	2026-08-28 13:45:55 -05:00
reyesj2	0dbb7803ef	Merge branch 'reyesj2/reworksoup' into reyesj2/es945	2026-08-28 13:28:53 -05:00
reyesj2	30deb00277	use correct version variable	2026-08-28 13:27:44 -05:00
reyesj2	192363bc2f	Merge branch 'reyesj2/reworksoup' into reyesj2/es945	2026-08-28 12:19:20 -05:00
reyesj2	3d8f86883a	after an ES upgrade run a final elasticsearch state to create/regenerate any needed addon index templates	2026-08-28 12:16:08 -05:00
Josh Patterson	96bef89ba9	Merge pull request #16200 from Security-Onion-Solutions/rotatehype add log rotation for hypervisor logs	2026-08-28 09:54:59 -04:00
Josh Patterson	a244640539	Merge remote-tracking branch 'origin/3/dev' into rotatehype # Conflicts: # salt/logrotate/defaults.yaml # salt/logrotate/soc_logrotate.yaml	2026-08-28 09:26:36 -04:00
reyesj2	fdb975fdef	Merge branch 'reyesj2/reworksoup' into reyesj2/es945	2026-08-27 22:12:44 -05:00
reyesj2	f8401bef37	exclude elasticsearch indexing error during upgrade for temporarily outdated policies	2026-08-27 21:17:27 -05:00
Jason Ertel	ca96a15091	Merge pull request #16199 from Security-Onion-Solutions/jertel/wip fix well-known paths	2026-08-27 16:31:27 -04:00
Jason Ertel	1bac9a218e	fix well-known paths	2026-08-27 16:28:00 -04:00
reyesj2	4786d359fb	exclude telegraf error during elasticsearch upgrade / master election	2026-08-27 14:39:46 -05:00
reyesj2	d771fbc444	upgrade integration policies directly after integration package upgrade	2026-08-27 14:18:39 -05:00
reyesj2	85ab4c69e5	rename	2026-08-27 14:17:29 -05:00
reyesj2	cb8e576d6b	run elasticsearch state on remote minions when there is an ES upgrade. Prior to manager completing its first full highstate that includes kibana / elasticfleet	2026-08-27 12:52:59 -05:00
reyesj2	fae1754fec	clean elasticsearch transform prior to elasticsearch integration package upgrade to prevent fleet automatic rollback	2026-08-27 12:50:32 -05:00
reyesj2	d33eb70af6	reverts `83aaa76` #15985 - allow full highstate on manager when locked	2026-08-27 12:12:38 -05:00
Jorge Reyes	f45dcfdf73	Merge pull request #16195 from Security-Onion-Solutions/revert-16165-reyesj2-patch-stg Revert "patch issue with fs.protected_symlinks"	2026-08-27 09:36:36 -05:00
Jorge Reyes	62da505ea7	Revert "patch issue with fs.protected_symlinks"	2026-08-27 09:21:57 -05:00
Josh Patterson	7e5b6f276f	Merge pull request #16194 from Security-Onion-Solutions/fix/auto-state-apply-local-salt-files Detect hand-placed local/salt files in Auto State Apply	2026-08-27 09:58:32 -04:00
Matthew Wright	376d29e376	Merge pull request #16191 from Security-Onion-Solutions/mwright/agent-studio-memory Memory and Reconcile Persona Annotations	2026-08-27 09:34:32 -04:00
Josh Patterson	665772adb8	Merge remote-tracking branch 'origin/3/dev' into fix/auto-state-apply-local-salt-files	2026-08-26 15:16:21 -04:00
Josh Patterson	094b4d5e86	Detect hand-placed local/salt files in Auto State Apply Auto State Apply fires on SOC config saves and on suricata/strelka rule updates. Files a user creates or edits by hand under /opt/so/saltstack/local/salt/ change no pillar, so nothing fired and the change waited for the next scheduled highstate, now 120 minutes by default. That gap is the 3.2 Known Issue in the docs. Watch the directories the docs tell users to edit, and route them through the push pipeline that already exists: zeek/policy -> zeek (covers intel/ and custom/) zeek/zkg -> zeek elasticsearch/files/ingest -> elasticsearch elasticsearch/roles -> elasticsearch logstash/pipelines/config/custom -> logstash Tags are pillar_push_map.yaml app names, so the existing entries already carry the right state and compound target, and no map entry changes. Rename the beacon rules_beacon -> local_files_beacon. Rules are now one of five kinds of file it watches, and the new name matches how its sibling postgres_pillar_beacon is named: source, then what it watches. Replace push_suricata.sls and push_strelka.sls with one push_files.sls bound to salt/beacon//local_files_beacon/, which looks the tag up in pillar_push_map.yaml the same way push_pillar.sls does. The map's suricata and strelka targets match the compounds those two reactors hardcoded, so rule pushes are unchanged. The app comes from the event tag rather than the payload because salt's beacon loop pops the beacon's tag key off the data. Key watermarks by watched directory instead of by tag. zeek/policy and zeek/zkg both emit the tag zeek, and a shared watermark would make them overwrite each other's digest and emit on every poll. Prune .git from the fingerprint walk. zkg packages must be git clones with a clean working tree, so the watched tree carries full git metadata; walking it every 15s is wasted work and git's own index and ref mtime churn would fire a grid-wide zeek apply on its own. Placing or updating a package always touches working-tree files too, so detection is unaffected. The watch is an allowlist rather than the whole local salt tree because salt writes into that tree itself: hypervisor/hosts/ is rewritten continuously by virtual_node_manager.py and virtual_power_manager.py, libvirt/images/ holds multi-GB qcow2 files, and elasticfleet/files/so_agent-installers/, elasticsearch/files/users, ca/files/ and filebeat/files/ are all state-written. Watching any of them would either self-retrigger or make the 15s poll walk gigabytes.	2026-08-26 15:15:29 -04:00
Matthew Wright	3a3667996c	make personas non-advanced and readonlyui	2026-08-26 15:08:17 -04:00
Mike Reeves	dfa6f0b454	Merge pull request #16192 from Security-Onion-Solutions/TOoSmOotH/telegraf-exec-array-syntax Use argv arrays for telegraf inputs.exec commands	2026-08-26 14:14:02 -04:00
Mike Reeves	9f6679c043	Use argv arrays for telegraf inputs.exec commands Telegraf 1.39 deprecated bare string entries in inputs.exec commands and will drop them in 1.45, warning on every start: W! DeprecationWarning: Value "/scripts/esindexsize.sh" for option "command" of plugin "inputs.exec" deprecated since version 1.39.0 Each entry is now a single-element argv array. Array form skips shell parsing, which is fine here: every command is a bare script path from telegraf's scripts list, no args or shell metacharacters.	2026-08-26 13:42:37 -04:00
Matthew Wright	fb7d162de1	memory and reconcile persona annotations	2026-08-26 12:36:46 -04:00
reyesj2	0ee8aa8079	revert manager running two highstates	2026-08-26 09:11:39 -05:00
reyesj2	99c3c7f8aa	ES 9.4.5 soup es compatibility update	2026-08-21 12:46:46 -05:00
reyesj2	cef1dcfcee	add additional problematic indices / templates. Also only update index templates if the script patched any index / data stream	2026-08-21 12:16:41 -05:00
reyesj2	247d9cdb34	kibana spaces 9.4.5	2026-08-20 17:00:53 -05:00
reyesj2	088b761190	9.4.5 policy updates	2026-08-20 16:35:46 -05:00
reyesj2	2dcc81ea7d	run so-elasticsearch-systems-indices-patch script every highstate with no op if no unassigned replicas are found for known problematic indices	2026-08-20 14:30:53 -05:00
reyesj2	35f545a858	find known problematic system indices and add missing auto_expand_replicas configuration to prevent yellow cluster	2026-08-19 16:30:38 -05:00
reyesj2	c9a041ddb4	upstream sentinel_one_cloud_funnel integration patched data stream name ref: https://github.com/elastic/integrations/commit/9f1513423ca26e6fded414c4f5a3a89409efb736	2026-08-17 22:08:19 -05:00
reyesj2	de3306e73c	ES 9.4.5	2026-08-17 16:45:41 -05:00
Josh Patterson	52791204e4	add logrotate for virtual_node_manager and so-salt-cloud	2026-05-19 13:40:19 -04:00
@@ -1 +1 @@
 .3.0
 .4.0