fix: stop pip's patchelf 'ERROR' line from polluting sosetup.log

The cmd.run for psycopg2 install was already tolerating pip's non-zero exit with `|| true`, but pip's stderr — which contains the literal string "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'patchelf'" — was still being captured into salt's state-result dict. so-setup logs salt state output to /root/sosetup.log, and verify_setup() then greps for the substring "ERROR" to build /root/errors.log. The patchelf line then shows up at the end of every install as "WARNING: Errors detected during setup" even though the install is in fact green. Redirect pip's combined stdout/stderr to /opt/so/log/so_pillar/psycopg2_install.log so the noise lives in a dedicated, predictable triage location instead of leaking into salt's state result. The `unless: import psycopg2` check is still the actual readiness gate, so a real install failure (rather than just the patchelf RPATH-rewrite step that has no functional effect on the wheel) would still surface via the state being re-run on every apply and `import psycopg2` failing.
fix: tolerate pip's non-zero exit on psycopg2 patchelf step
2026-07-22 08:45:40 +02:00 · 2026-05-05 10:38:57 -04:00 · 2026-05-04 22:08:31 -04:00 · 2026-05-04 21:02:08 -04:00 · 2026-05-04 19:56:14 -04:00 · 2026-05-04 19:47:38 -04:00
199 changed files with 3758 additions and 7718 deletions
@@ -11,7 +11,6 @@ body:
        -
        - 3.0.0
        - 3.1.0
        - 3.2.0
        - Other (please provide detail below)
    validations:
      required: true
@@ -5,7 +5,6 @@ on:
    paths:
      - "salt/sensoroni/files/analyzers/**"
      - "salt/manager/tools/sbin/**"
      - "salt/_beacons/**"
 jobs:
  build:
@@ -15,7 +14,7 @@ jobs:
      fail-fast: false
      matrix:
        python-version: ["3.14"]
-        python-code-path: ["salt/sensoroni/files/analyzers", "salt/manager/tools/sbin", "salt/_beacons"]
+        python-code-path: ["salt/sensoroni/files/analyzers", "salt/manager/tools/sbin"]
    steps:
    - uses: actions/checkout@v3
@@ -1,17 +1,17 @@
-### 3.1.0-20260528 ISO image released on 2026/05/28
+### 3.0.0-20260331 ISO image released on 2026/03/31
 ### Download and Verify
-3.1.0-20260528 ISO image:  
+3.0.0-20260331 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-3.1.0-20260528.iso
+https://download.securityonion.net/file/securityonion/securityonion-3.0.0-20260331.iso
-MD5: 9D6FF58DEEE24089D722C73169765B3E  
+MD5: ECD318A1662A6FDE0EF213F5A9BD4B07  
-SHA1: 2B8B816B6CEC3B7F96B3C5E040EBF502DD2C412F  
+SHA1: E55BE314440CCF3392DC0B06BC5E270B43176D9C  
-SHA256: 62FAB57E247C843D6A04F0796D8162C732B65D82FC3E4A59D087135B9FD32912  
+SHA256: 7FC47405E335CBE5C2B6C51FE7AC60248F35CBE504907B8B5A33822B23F8F4D5  
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.1.0-20260528.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.0.0-20260331.iso.sig
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/main/KEYS  
@@ -25,22 +25,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.1.0-20260528.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.0.0-20260331.iso.sig
 ```
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-3.1.0-20260528.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-3.0.0-20260331.iso
 ```
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-3.1.0-20260528.iso.sig securityonion-3.1.0-20260528.iso
+gpg --verify securityonion-3.0.0-20260331.iso.sig securityonion-3.0.0-20260331.iso
 ```
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Wed 27 May 2026 03:03:59 PM EDT using RSA key ID FE507013
+gpg: Signature made Mon 30 Mar 2026 06:22:14 PM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
@@ -1 +0,0 @@
@@ -1 +1 @@
-3.2.0
+3.1.0
@@ -0,0 +1,59 @@
 #!/usr/bin/env bash
 # This script adds sensors/nodes/etc to the nodes tab
 default_salt_dir=/opt/so/saltstack/default
 local_salt_dir=/opt/so/saltstack/local
 TYPE=$1
 NAME=$2
 IPADDRESS=$3
 CPUS=$4
 GUID=$5
 MANINT=$6
 ROOTFS=$7
 NSM=$8
 MONINT=$9
 #NODETYPE=$10
 #HOTNAME=$11
 echo "Seeing if this host is already in here. If so delete it"
 if grep -q $NAME "$local_salt_dir/pillar/data/$TYPE.sls"; then
  echo "Node Already Present - Let's re-add it"
  awk -v blah="  $NAME:" 'BEGIN{ print_flag=1 }
 {
    if( $0 ~ blah )
    {
       print_flag=0;
       next
    }
    if( $0 ~ /^  [a-zA-Z0-9]+:$/ )
    {
        print_flag=1;
    }
    if ( print_flag == 1 )
        print $0
 } ' $local_salt_dir/pillar/data/$TYPE.sls > $local_salt_dir/pillar/data/tmp.$TYPE.sls
 mv $local_salt_dir/pillar/data/tmp.$TYPE.sls $local_salt_dir/pillar/data/$TYPE.sls
 echo "Deleted $NAME from the tab. Now adding it in again with updated info"
 fi
 echo "  $NAME:" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    ip: $IPADDRESS" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    manint: $MANINT" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    totalcpus: $CPUS" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    guid: $GUID" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    rootfs: $ROOTFS" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    nsmfs: $NSM" >> $local_salt_dir/pillar/data/$TYPE.sls
 if [ $TYPE == 'sensorstab' ]; then
  echo "    monint: bond0" >> $local_salt_dir/pillar/data/$TYPE.sls
 fi
 if [ $TYPE == 'evaltab' ] || [ $TYPE == 'standalonetab' ]; then
  echo "    monint: bond0" >> $local_salt_dir/pillar/data/$TYPE.sls
  if [ ! $10 ]; then
    salt-call state.apply utility queue=True
  fi
 fi
 if [ $TYPE == 'nodestab' ]; then
  salt-call state.apply elasticsearch queue=True
 #  echo "    nodetype: $NODETYPE" >> $local_salt_dir/pillar/data/$TYPE.sls
 #  echo "    hotname: $HOTNAME" >> $local_salt_dir/pillar/data/$TYPE.sls
 fi
@@ -3,8 +3,6 @@ base:
    - ca
    - global.soc_global
    - global.adv_global
    - salt.soc_salt
    - salt.adv_salt
    - docker.soc_docker
    - docker.adv_docker
    - influxdb.token
@@ -1,142 +0,0 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 # Custom salt beacon that watches the SOC audit_settings table in postgres for
 # new settings changes and emits a beacon event per new row. This replaces the
 # inotify watch on /opt/so/saltstack/local/pillar -- instead of monitoring pillar
 # files on disk, we monitor the securityonion.audit_settings table that SOC writes to.
 #
 # Detection is poll-based with a monotonic `id` watermark persisted to
 # WATERMARK_FILE: each pass selects rows with id greater than the last id seen,
 # which makes it self-healing (a missed poll simply catches up on the next one).
 #
 # Each emitted event carries setting_id and node_id; the push_pillar reactor maps
 # setting_id -> app via pillar_push_map.yaml and writes a push intent, after which
 # the existing so-push-drainer / orch.push_batch pipeline takes over unchanged.
 import logging
 import os
 import subprocess
 log = logging.getLogger(__name__)
 WATERMARK_FILE = '/opt/so/state/postgres_pillar_beacon_watch.id'
 CONTAINER = 'so-postgres'
 DATABASE = 'securityonion'
 # Unaligned, tuples-only psql output with a field separator that cannot appear in
 # an id/setting_id/node_id, so we can split each row reliably.
 FIELD_SEP = '\x1f'
 def __virtual__():
    return True
 def validate(config):
    return True, 'valid'
 def _read_watermark():
    # Returns the last processed id, or None if the watermark has not been seeded.
    try:
        with open(WATERMARK_FILE, 'r') as f:
            return int((f.read() or '').strip())
    except (IOError, ValueError):
        return None
 def _write_watermark(value):
    try:
        os.makedirs(os.path.dirname(WATERMARK_FILE), exist_ok=True)
        tmp = WATERMARK_FILE + '.tmp'
        with open(tmp, 'w') as f:
            f.write(str(int(value)))
        os.rename(tmp, WATERMARK_FILE)
    except OSError:
        log.exception('postgres_pillar_beacon: failed to persist watermark to %s', WATERMARK_FILE)
 def _query(sql):
    # Run a query against securityonion inside the so-postgres container over the unix
    # socket (trust auth, no password). Returns stdout on success, or None on any
    # failure so the caller can no-op and retry on the next interval.
    cmd = [
        'docker', 'exec', CONTAINER,
        'psql', '-U', 'postgres', '-d', DATABASE,
        '-tA', '-F', FIELD_SEP, '-c', sql,
    ]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
    except subprocess.TimeoutExpired:
        log.warning('postgres_pillar_beacon: psql timed out')
        return None
    except Exception:
        log.exception('postgres_pillar_beacon: failed to exec psql')
        return None
    if result.returncode != 0:
        log.warning('postgres_pillar_beacon: psql failed (rc=%s): %s',
                    result.returncode, (result.stderr or '').strip())
        return None
    return result.stdout
 def beacon(config):  # noqa: C901
    retval = []
    watermark = _read_watermark()
    # First run / missing watermark: seed to the current MAX(id) and emit nothing
    # so we never replay the entire settings history into a fleetwide push.
    if watermark is None:
        seed = _query('SELECT COALESCE(MAX(id), 0) FROM audit_settings;')
        if seed is None:
            return retval  # postgres not ready yet; retry next interval
        try:
            _write_watermark(int((seed or '0').strip() or 0))
        except ValueError:
            log.warning('postgres_pillar_beacon: could not parse MAX(id) seed: %r', seed)
        return retval
    rows = _query(
        "SELECT id, setting_id, COALESCE(node_id, '') FROM audit_settings "
        "WHERE id > %d ORDER BY id;" % watermark
    )
    if rows is None:
        return retval
    max_id = watermark
    for line in rows.splitlines():
        # Do NOT str.strip() the whole line: Python treats the \x1f field
        # separator (and \x1c-\x1e) as whitespace, so stripping would eat an
        # empty trailing node_id field and make the row look malformed.
        if not line.strip():
            continue
        parts = line.split(FIELD_SEP)
        if len(parts) < 3:
            log.warning('postgres_pillar_beacon: skipping malformed row: %r', line)
            continue
        try:
            row_id = int(parts[0])
        except ValueError:
            log.warning('postgres_pillar_beacon: skipping row with non-int id: %r', line)
            continue
        setting_id = parts[1]
        node_id = parts[2]
        retval.append({
            'tag': 'audit_settings',
            'id': row_id,
            'setting_id': setting_id,
            'node_id': node_id,
        })
        if row_id > max_id:
            max_id = row_id
    if max_id > watermark:
        _write_watermark(max_id)
        log.info('postgres_pillar_beacon: emitted %d change(s), watermark %d -> %d',
                 len(retval), watermark, max_id)
    return retval
@@ -1,165 +0,0 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 import os
 import shutil
 import subprocess
 import tempfile
 import unittest
 from unittest.mock import patch
 import postgres_pillar_beacon
 class TestPostgresPillarBeacon(unittest.TestCase):
    def setUp(self):
        # Point WATERMARK_FILE at a throwaway dir so the real read/write helpers
        # (and their os.makedirs/os.rename) run against actual files, then clean
        # it all up in tearDown.
        self.tmpdir = tempfile.mkdtemp()
        self.watermark = os.path.join(self.tmpdir, 'state', 'watch.id')
        patcher = patch.object(postgres_pillar_beacon, 'WATERMARK_FILE', self.watermark)
        patcher.start()
        self.addCleanup(patcher.stop)
    def tearDown(self):
        shutil.rmtree(self.tmpdir, ignore_errors=True)
    # -- trivial contract -------------------------------------------------
    def test_virtual_returns_true(self):
        self.assertTrue(postgres_pillar_beacon.__virtual__())
    def test_validate_returns_valid(self):
        self.assertEqual(postgres_pillar_beacon.validate({}), (True, 'valid'))
    # -- _read_watermark --------------------------------------------------
    def test_read_watermark_valid(self):
        postgres_pillar_beacon._write_watermark(42)
        self.assertEqual(postgres_pillar_beacon._read_watermark(), 42)
    def test_read_watermark_missing_file_returns_none(self):
        # tmp watermark file was never created
        self.assertIsNone(postgres_pillar_beacon._read_watermark())
    def test_read_watermark_garbage_returns_none(self):
        os.makedirs(os.path.dirname(self.watermark), exist_ok=True)
        with open(self.watermark, 'w') as f:
            f.write('nope')
        self.assertIsNone(postgres_pillar_beacon._read_watermark())
    # -- _write_watermark -------------------------------------------------
    def test_write_watermark_round_trip(self):
        postgres_pillar_beacon._write_watermark(7)
        with open(self.watermark) as f:
            self.assertEqual(f.read(), '7')
    def test_write_watermark_swallows_oserror(self):
        with patch.object(postgres_pillar_beacon.os, 'makedirs', side_effect=OSError):
            # Must not raise; failure is logged and the beacon retries next pass.
            postgres_pillar_beacon._write_watermark(5)
        self.assertFalse(os.path.exists(self.watermark))
    # -- _query -----------------------------------------------------------
    def test_query_success_returns_stdout_and_builds_argv(self):
        completed = subprocess.CompletedProcess(args=[], returncode=0, stdout='rows', stderr='')
        with patch.object(postgres_pillar_beacon.subprocess, 'run', return_value=completed) as mock_run:
            result = postgres_pillar_beacon._query('SELECT 1;')
        self.assertEqual(result, 'rows')
        argv = mock_run.call_args[0][0]
        self.assertEqual(argv[:5], ['docker', 'exec', 'so-postgres', 'psql', '-U'])
        self.assertIn('SELECT 1;', argv)
        self.assertFalse(mock_run.call_args[1].get('shell', False))
    def test_query_timeout_returns_none(self):
        with patch.object(postgres_pillar_beacon.subprocess, 'run',
                          side_effect=subprocess.TimeoutExpired(cmd='psql', timeout=30)):
            self.assertIsNone(postgres_pillar_beacon._query('SELECT 1;'))
    def test_query_generic_exception_returns_none(self):
        with patch.object(postgres_pillar_beacon.subprocess, 'run', side_effect=Exception('boom')):
            self.assertIsNone(postgres_pillar_beacon._query('SELECT 1;'))
    def test_query_nonzero_returncode_returns_none(self):
        completed = subprocess.CompletedProcess(args=[], returncode=1, stdout='', stderr='bad')
        with patch.object(postgres_pillar_beacon.subprocess, 'run', return_value=completed):
            self.assertIsNone(postgres_pillar_beacon._query('SELECT 1;'))
    # -- beacon: first run / seeding --------------------------------------
    def test_beacon_seeds_when_postgres_not_ready(self):
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=None), \
             patch.object(postgres_pillar_beacon, '_query', return_value=None), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            self.assertEqual(postgres_pillar_beacon.beacon({}), [])
        mock_write.assert_not_called()
    def test_beacon_seeds_to_max_id_and_emits_nothing(self):
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=None), \
             patch.object(postgres_pillar_beacon, '_query', return_value='7\n'), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            self.assertEqual(postgres_pillar_beacon.beacon({}), [])
        mock_write.assert_called_once_with(7)
    def test_beacon_seed_unparseable_is_swallowed(self):
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=None), \
             patch.object(postgres_pillar_beacon, '_query', return_value='abc'), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            self.assertEqual(postgres_pillar_beacon.beacon({}), [])
        mock_write.assert_not_called()
    # -- beacon: steady state ---------------------------------------------
    def test_beacon_query_failure_returns_empty(self):
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=10), \
             patch.object(postgres_pillar_beacon, '_query', return_value=None), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            self.assertEqual(postgres_pillar_beacon.beacon({}), [])
        mock_write.assert_not_called()
    def test_beacon_emits_events_and_advances_watermark(self):
        sep = postgres_pillar_beacon.FIELD_SEP
        rows = '11%s5%snode1\n12%s6%s\n' % (sep, sep, sep, sep)
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=10), \
             patch.object(postgres_pillar_beacon, '_query', return_value=rows), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            result = postgres_pillar_beacon.beacon({})
        self.assertEqual(result, [
            {'tag': 'audit_settings', 'id': 11, 'setting_id': '5', 'node_id': 'node1'},
            {'tag': 'audit_settings', 'id': 12, 'setting_id': '6', 'node_id': ''},
        ])
        mock_write.assert_called_once_with(12)
    def test_beacon_skips_malformed_blank_and_noninteger_rows(self):
        sep = postgres_pillar_beacon.FIELD_SEP
        rows = (
            '\n'                       # blank line -> skipped
            '13%s7\n'                  # too few fields -> skipped
            'abc%s8%snodeX\n'          # non-integer id -> skipped
            '14%s9%snodeY\n'           # the one good row
        ) % (sep, sep, sep, sep, sep)
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=10), \
             patch.object(postgres_pillar_beacon, '_query', return_value=rows), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            result = postgres_pillar_beacon.beacon({})
        self.assertEqual(result, [
            {'tag': 'audit_settings', 'id': 14, 'setting_id': '9', 'node_id': 'nodeY'},
        ])
        mock_write.assert_called_once_with(14)
    def test_beacon_no_new_rows_does_not_advance_watermark(self):
        with patch.object(postgres_pillar_beacon, '_read_watermark', return_value=10), \
             patch.object(postgres_pillar_beacon, '_query', return_value=''), \
             patch.object(postgres_pillar_beacon, '_write_watermark') as mock_write:
            self.assertEqual(postgres_pillar_beacon.beacon({}), [])
        mock_write.assert_not_called()
 if __name__ == '__main__':
    unittest.main()
@@ -1,139 +0,0 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 # Custom salt beacon that watches the suricata/strelka rule directories for changes
 # and emits a beacon event per changed directory. This replaces the stock salt
 # `inotify` beacon, which leaks a kernel inotify instance every time the minion
 # rebuilds the beacon loader's __context__ (orphaning the old pyinotify.Notifier
 # without closing it) until fs.inotify.max_user_instances is exhausted and the
 # beacon dies with EMFILE. Polling holds zero inotify instances, so the leak is
 # impossible, and it keeps firing during state runs (no blackout).
 #
 # Detection is poll-based with a per-directory fingerprint persisted to
 # WATERMARK_DIR: each pass walks the directory and hashes every file's
 # (relpath, st_mtime_ns, st_size), which catches content writes, additions,
 # moves, and deletions. A change in the digest emits one event; an unchanged
 # digest emits nothing. This makes it self-healing (a missed poll simply catches
 # up on the next one).
 #
 # Each emitted event carries the watched directory path under the configured tag
 # (e.g. salt/beacon/<minion>/rules_beacon/suricata); the push_suricata / push_strelka
 # reactors write a push intent, after which the existing so-push-drainer /
 # orch.push_batch pipeline takes over unchanged.
 import hashlib
 import logging
 import os
 import re
 log = logging.getLogger(__name__)
 WATERMARK_DIR = '/opt/so/state'
 # Temp/editor files that should not trigger a push. Mirrors the exclude regexes
 # the inotify beacon used. Matched against the full pathname.
 EXCLUDES = [
    re.compile(r'\.sw[a-z]$'),
    re.compile(r'~$'),
    re.compile(r'/4913$'),
    re.compile(r'/\.#'),
 ]
 def __virtual__():
    return True
 def validate(config):
    return True, 'valid'
 def _paths_from_config(config):
    # The beacon config arrives as a list of single-key dicts (salt beacon style).
    # Merge it and return the {dir: tag} mapping under the 'paths' key.
    merged = {}
    if isinstance(config, list):
        for item in config:
            if isinstance(item, dict):
                merged.update(item)
    elif isinstance(config, dict):
        merged = config
    paths = merged.get('paths', {})
    return paths if isinstance(paths, dict) else {}
 def _excluded(pathname):
    for pattern in EXCLUDES:
        if pattern.search(pathname):
            return True
    return False
 def _fingerprint(directory):
    # Stat-only walk; hash each file's (relpath, mtime_ns, size). Returns a hex
    # digest, or the digest of an empty tree if the directory does not exist.
    h = hashlib.sha1()
    if os.path.isdir(directory):
        entries = []
        for root, _dirs, files in os.walk(directory):
            for name in files:
                full = os.path.join(root, name)
                if _excluded(full):
                    continue
                try:
                    st = os.stat(full)
                except OSError:
                    continue
                rel = os.path.relpath(full, directory)
                entries.append('%s\0%d\0%d' % (rel, st.st_mtime_ns, st.st_size))
        for line in sorted(entries):
            h.update(line.encode('utf-8', 'surrogateescape'))
            h.update(b'\n')
    return h.hexdigest()
 def _watermark_file(tag):
    return os.path.join(WATERMARK_DIR, 'rules_beacon_%s.hash' % tag)
 def _read_watermark(tag):
    try:
        with open(_watermark_file(tag), 'r') as f:
            return (f.read() or '').strip() or None
    except IOError:
        return None
 def _write_watermark(tag, digest):
    path = _watermark_file(tag)
    try:
        os.makedirs(WATERMARK_DIR, exist_ok=True)
        tmp = path + '.tmp'
        with open(tmp, 'w') as f:
            f.write(digest)
        os.rename(tmp, path)
    except OSError:
        log.exception('rules_beacon: failed to persist watermark to %s', path)
 def beacon(config):
    retval = []
    for directory, tag in _paths_from_config(config).items():
        digest = _fingerprint(directory)
        previous = _read_watermark(tag)
        # First run / missing watermark: seed the digest and emit nothing so a
        # fresh host does not fire a spurious fleetwide push.
        if previous is None:
            _write_watermark(tag, digest)
            continue
        if digest != previous:
            _write_watermark(tag, digest)
            retval.append({'tag': tag, 'path': directory})
            log.info('rules_beacon: change detected in %s, emitting %s', directory, tag)
    return retval
@@ -1,172 +0,0 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 import hashlib
 import os
 import shutil
 import tempfile
 import unittest
 from unittest.mock import patch
 import rules_beacon
 class TestRulesBeacon(unittest.TestCase):
    def setUp(self):
        # Isolate all on-disk state (watermarks and the dirs we fingerprint) in a
        # throwaway tree, and point WATERMARK_DIR at it so the real read/write
        # helpers run against actual files.
        self.tmpdir = tempfile.mkdtemp()
        self.state = os.path.join(self.tmpdir, 'state')
        patcher = patch.object(rules_beacon, 'WATERMARK_DIR', self.state)
        patcher.start()
        self.addCleanup(patcher.stop)
    def tearDown(self):
        shutil.rmtree(self.tmpdir, ignore_errors=True)
    def _make_dir(self, name, files=None):
        path = os.path.join(self.tmpdir, name)
        os.makedirs(path, exist_ok=True)
        for fname, content in (files or {}).items():
            with open(os.path.join(path, fname), 'w') as f:
                f.write(content)
        return path
    # -- trivial contract -------------------------------------------------
    def test_virtual_returns_true(self):
        self.assertTrue(rules_beacon.__virtual__())
    def test_validate_returns_valid(self):
        self.assertEqual(rules_beacon.validate({}), (True, 'valid'))
    # -- _paths_from_config -----------------------------------------------
    def test_paths_from_config_list_of_dicts(self):
        config = [{'interval': 10}, {'paths': {'/a': 'suricata', '/b': 'strelka'}}]
        self.assertEqual(
            rules_beacon._paths_from_config(config),
            {'/a': 'suricata', '/b': 'strelka'},
        )
    def test_paths_from_config_plain_dict(self):
        self.assertEqual(
            rules_beacon._paths_from_config({'paths': {'/a': 'suricata'}}),
            {'/a': 'suricata'},
        )
    def test_paths_from_config_skips_non_dict_items(self):
        self.assertEqual(rules_beacon._paths_from_config(['bogus', 42]), {})
    def test_paths_from_config_paths_not_a_dict(self):
        self.assertEqual(rules_beacon._paths_from_config({'paths': 'nope'}), {})
    def test_paths_from_config_unexpected_type(self):
        self.assertEqual(rules_beacon._paths_from_config('nonsense'), {})
    # -- _excluded --------------------------------------------------------
    def test_excluded_matches_temp_and_editor_files(self):
        for pathname in ('/rules/foo.swp', '/rules/foo~', '/rules/4913', '/rules/.#foo'):
            self.assertTrue(rules_beacon._excluded(pathname), pathname)
    def test_excluded_allows_real_rule_files(self):
        self.assertFalse(rules_beacon._excluded('/rules/suricata.rules'))
    # -- _fingerprint -----------------------------------------------------
    def test_fingerprint_missing_dir_is_empty_tree_digest(self):
        missing = os.path.join(self.tmpdir, 'does-not-exist')
        self.assertEqual(rules_beacon._fingerprint(missing), hashlib.sha1().hexdigest())
    def test_fingerprint_changes_when_content_changes(self):
        d = self._make_dir('rules', {'a.rules': 'alert'})
        before = rules_beacon._fingerprint(d)
        with open(os.path.join(d, 'a.rules'), 'w') as f:
            f.write('alert tcp any any -> any any')  # different size
        self.assertNotEqual(rules_beacon._fingerprint(d), before)
    def test_fingerprint_ignores_excluded_files(self):
        d = self._make_dir('rules', {'a.rules': 'alert'})
        before = rules_beacon._fingerprint(d)
        with open(os.path.join(d, 'a.rules.swp'), 'w') as f:
            f.write('editor swap')
        self.assertEqual(rules_beacon._fingerprint(d), before)
    def test_fingerprint_skips_unstatable_entries(self):
        # A dangling symlink appears in os.walk's file list but os.stat raises
        # OSError, exercising the except-continue path.
        d = self._make_dir('rules', {'a.rules': 'alert'})
        good = rules_beacon._fingerprint(d)
        os.symlink(os.path.join(d, 'missing-target'), os.path.join(d, 'broken.link'))
        self.assertEqual(rules_beacon._fingerprint(d), good)
    # -- _read_watermark / _write_watermark -------------------------------
    def test_watermark_round_trip(self):
        rules_beacon._write_watermark('suricata', 'deadbeef')
        self.assertEqual(rules_beacon._read_watermark('suricata'), 'deadbeef')
    def test_read_watermark_missing_returns_none(self):
        self.assertIsNone(rules_beacon._read_watermark('suricata'))
    def test_read_watermark_empty_file_returns_none(self):
        os.makedirs(self.state, exist_ok=True)
        with open(rules_beacon._watermark_file('suricata'), 'w') as f:
            f.write('')
        self.assertIsNone(rules_beacon._read_watermark('suricata'))
    def test_write_watermark_swallows_oserror(self):
        with patch.object(rules_beacon.os, 'makedirs', side_effect=OSError):
            rules_beacon._write_watermark('suricata', 'deadbeef')
        self.assertIsNone(rules_beacon._read_watermark('suricata'))
    # -- beacon -----------------------------------------------------------
    def _config(self, mapping):
        return [{'paths': mapping}]
    def test_beacon_seeds_first_run_and_emits_nothing(self):
        with patch.object(rules_beacon, '_fingerprint', return_value='hash1'), \
             patch.object(rules_beacon, '_read_watermark', return_value=None), \
             patch.object(rules_beacon, '_write_watermark') as mock_write:
            result = rules_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
        self.assertEqual(result, [])
        mock_write.assert_called_once_with('suricata', 'hash1')
    def test_beacon_emits_on_change(self):
        with patch.object(rules_beacon, '_fingerprint', return_value='newhash'), \
             patch.object(rules_beacon, '_read_watermark', return_value='oldhash'), \
             patch.object(rules_beacon, '_write_watermark') as mock_write:
            result = rules_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
        self.assertEqual(result, [{'tag': 'suricata', 'path': '/rules/suricata'}])
        mock_write.assert_called_once_with('suricata', 'newhash')
    def test_beacon_no_change_emits_nothing(self):
        with patch.object(rules_beacon, '_fingerprint', return_value='samehash'), \
             patch.object(rules_beacon, '_read_watermark', return_value='samehash'), \
             patch.object(rules_beacon, '_write_watermark') as mock_write:
            result = rules_beacon.beacon(self._config({'/rules/suricata': 'suricata'}))
        self.assertEqual(result, [])
        mock_write.assert_not_called()
    def test_beacon_end_to_end_with_real_files(self):
        # Exercise the full stack (real fingerprint + real watermark files) across
        # two poll passes: first seeds silently, second fires after a write.
        d = self._make_dir('rules', {'a.rules': 'alert'})
        config = self._config({d: 'suricata'})
        self.assertEqual(rules_beacon.beacon(config), [])  # seed pass
        self.assertEqual(rules_beacon.beacon(config), [])  # unchanged pass
        with open(os.path.join(d, 'b.rules'), 'w') as f:
            f.write('alert tcp any any -> any any')
        self.assertEqual(rules_beacon.beacon(config), [{'tag': 'suricata', 'path': d}])
 if __name__ == '__main__':
    unittest.main()
@@ -4,7 +4,7 @@ import logging
 def status():
 cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl status'"
-    retval = __salt__['docker.run']('so-zeek', cmd)  # noqa: F821
+ retval = __salt__['docker.run']('so-zeek', cmd)
 logging.info('zeekctl_module: zeekctl.status retval: %s' % retval)
 return retval
@@ -14,7 +14,7 @@ def beacon(config):
  retval = []
-    is_enabled = __salt__['healthcheck.is_enabled']()  # noqa: F821
+  is_enabled = __salt__['healthcheck.is_enabled']()
  logging.info('zeek_beacon: healthcheck_is_enabled: %s' % is_enabled)
  if is_enabled:
@@ -25,8 +25,9 @@ def beacon(config):
    else:
     zeek_restart = False
-        __salt__['telegraf.send']('healthcheck zeek_restart=%s' % str(zeek_restart))  # noqa: F821
+    __salt__['telegraf.send']('healthcheck zeek_restart=%s' % str(zeek_restart))
    retval.append({'zeek_restart': zeek_restart})
    logging.info('zeek_beacon: retval: %s' % str(retval))
  return retval
@@ -1,59 +0,0 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 import unittest
 from unittest.mock import MagicMock
 import zeek
 ZEEKCTL_CMD = "runuser -l zeek -c '/opt/zeek/bin/zeekctl status'"
 class TestZeekBeacon(unittest.TestCase):
    def setUp(self):
        # zeek.py relies on the __salt__ dunder that Salt injects at load time.
        # Nothing defines it under test, so we attach a dict of mock loader
        # functions to the module and remove it again afterwards.
        self.salt = {
            'docker.run': MagicMock(return_value='Zeek is running'),
            'healthcheck.is_enabled': MagicMock(return_value=True),
            'telegraf.send': MagicMock(),
        }
        zeek.__salt__ = self.salt
        self.addCleanup(lambda: delattr(zeek, '__salt__'))
    # -- status -----------------------------------------------------------
    def test_status_runs_zeekctl_and_returns_output(self):
        self.salt['docker.run'].return_value = 'Zeek is running'
        result = zeek.status()
        self.assertEqual(result, 'Zeek is running')
        self.salt['docker.run'].assert_called_once_with('so-zeek', ZEEKCTL_CMD)
    # -- beacon -----------------------------------------------------------
    def test_beacon_disabled_returns_empty_and_skips_telegraf(self):
        self.salt['healthcheck.is_enabled'].return_value = False
        self.assertEqual(zeek.beacon({}), [])
        self.salt['telegraf.send'].assert_not_called()
    def test_beacon_running_reports_no_restart(self):
        self.salt['docker.run'].return_value = 'Zeek is running'
        self.assertEqual(zeek.beacon({}), [{'zeek_restart': False}])
        self.salt['telegraf.send'].assert_called_once_with('healthcheck zeek_restart=False')
    def test_beacon_unhealthy_status_triggers_restart(self):
        # Each of these status tokens should flag a restart (the or-chain in beacon).
        for status_text in ('Zeek is stopped', 'Zeek crashed', 'Zeek error state', 'Zeek error:'):
            with self.subTest(status=status_text):
                self.salt['docker.run'].return_value = status_text
                self.salt['telegraf.send'].reset_mock()
                self.assertEqual(zeek.beacon({}), [{'zeek_restart': True}])
                self.salt['telegraf.send'].assert_called_once_with('healthcheck zeek_restart=True')
 if __name__ == '__main__':
    unittest.main()
@@ -37,7 +37,8 @@
    'elasticfleet',
    'elasticfleet.manager',
    'elasticsearch.cluster',
-    'elastic-fleet-package-registry'
+    'elastic-fleet-package-registry',
    'utility'
 ] %}
 {% set sensor_states = [
@@ -25,11 +25,9 @@ if [ ! -f $BACKUPFILE ]; then
  # Create empty backup file
  tar -cf $BACKUPFILE -T /dev/null
-  # Loop through all paths defined in global.sls, and append them to backup file if they exist
+  # Loop through all paths defined in global.sls, and append them to backup file
  {%- for LOCATION in BACKUPLOCATIONS %}
  if [[ -d {{ LOCATION }} || -f {{ LOCATION }} ]]; then
  tar -rf $BACKUPFILE "${EXCLUSIONS[@]}" {{ LOCATION }}
  fi
  {%- endfor %}
 fi
@@ -130,17 +130,6 @@ common_sbin:
      - so-pcap-import
 {% endif %}
 # Pin physical NIC names by MAC (run-once) so a kernel upgrade can't renumber the
 # interfaces SO binds by name. The marker keeps it a one-time setup; an admin can
 # pre-create the marker to opt out.
 pin_nic_names:
  cmd.run:
    - name: /usr/sbin/so-nic-pin
    - unless: 'test -e /opt/so/state/nic_names_pinned'
    - require:
      - file: common_sbin
      - file: statedir
 common_sbin_jinja:
  file.recurse:
    - name: /usr/sbin
@@ -142,11 +142,6 @@ check_elastic_license() {
 	fi  
 }
 check_elasticsearch_responsive() {
    retry 3 15 "so-elasticsearch-query / --output /dev/null --fail" ||
        fail "Elasticsearch is not responding. Please review Elasticsearch logs /opt/so/log/elasticsearch/securityonion.log for more details. Additionally, consider running so-elasticsearch-troubleshoot."
 }
 check_salt_master_status() {
 	local count=0
    local attempts="${1:- 10}"
@@ -291,20 +286,6 @@ download_and_verify() {
 	fi
 }
 # check if container with name is running and optionally stop it
 docker_check_running() {
    # show running containers, only names
    if docker ps --format '{{.Names}}' | grep -q "^so-${1}$"; then
        if [[ "$2" == "--stop" ]]; then
            docker stop "so-${1}"
        fi
        return 0
    else
        return 1
    fi
 }
 elastic_license() {
 read -r -d '' message <<- EOM
@@ -602,6 +583,42 @@ run_check_net_err() {
 	fi
 }
 wait_for_salt_minion() {
    local minion="$1"
    local max_wait="${2:-30}"
    local interval="${3:-2}"
    local logfile="${4:-'/dev/stdout'}"
    local elapsed=0
 	echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - Waiting for salt-minion '$minion' to be ready..."
    while [ $elapsed -lt $max_wait ]; do
        # Check if service is running
 		echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - Check if salt-minion service is running"
        if ! systemctl is-active --quiet salt-minion; then
            echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - salt-minion service not running (elapsed: ${elapsed}s)"
            sleep $interval
            elapsed=$((elapsed + interval))
            continue
        fi
 		echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - salt-minion service is running"
        # Check if minion responds to ping
 		echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - Check if $minion responds to ping"
        if salt "$minion" test.ping --timeout=3 --out=json 2>> "$logfile" | grep -q "true"; then
            echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - salt-minion '$minion' is connected and ready!"
            return 0
        fi
        echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - Waiting... (${elapsed}s / ${max_wait}s)"
        sleep $interval
        elapsed=$((elapsed + interval))
    done
    echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - ERROR: salt-minion '$minion' not ready after $max_wait seconds"
    return 1
 }
 salt_minion_count() {
 	local MINIONDIR="/opt/so/saltstack/local/pillar/minions"
 	MINIONCOUNT=$(ls -la $MINIONDIR/*.sls | grep -v adv_ | wc -l)
@@ -666,7 +683,7 @@ systemctl_func() {
 	echo ""
 	echo "${echo_action^}ing $service_name service at $(date +"%T.%6N")"
-    systemctl $action $service_name && echo "Successfully ${echo_action}ed $service_name at $(date +"%T.%6N")." || echo "Failed to $action $service_name at $(date +"%T.%6N")."
+    systemctl $action $service_name && echo "Successfully ${echo_action}ed $service_name." || echo "Failed to $action $service_name."
 	echo ""
 }
@@ -1,243 +0,0 @@
 #!/bin/bash
 #
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 #
 # so-kernel-upgrade — install the UEK8 (6.x) kernel and make it the boot default.
 #
 # Security Onion is moving off the EL9 stock kernel (RHCK, 5.14) and UEK7 (5.15) onto UEK8
 # (6.x). Three things have to happen, and the tool has to drive each one:
 #
 #   1. Populate. The manager mirrors the UEK8 packages into /nsm/kernelrepo via so-repo-sync,
 #      and serves them to the grid over https://<manager>/kernelrepo. Until that sync runs the
 #      repo is valid but EMPTY -- dnf resolves it happily and installs nothing, with no error.
 #   2. Install. A node on RHCK has no kernel-uek* package at all, so there is nothing for
 #      'dnf update' to upgrade. A node on UEK7 does have kernel-uek installed, so
 #      'dnf install kernel-uek' reports "Nothing to do" and exits 0 without installing 6.x.
 #      Both cases need an explicit install of the UEK8 NEVRA.
 #   3. Boot it. Whether a newly installed UEK8 kernel becomes the boot default depends on the
 #      RUNNING kernel's flavor. kernel-install/grubby (with UPDATEDEFAULT=yes) only auto-promote
 #      within the running kernel's flavor lineage:
 #        - From UEK7 (5.x, kernel-uek) the install stays in the kernel-uek lineage and IS
 #          auto-promoted, so no grubby change is needed -- just make sure the repo is populated
 #          and install UEK8.
 #        - From the stock EL9 kernel (RHCK, 5.14, no UEK) it is a flavor CROSS that is NOT
 #          auto-promoted, so the box keeps booting RHCK until grubby is told otherwise.
 #      This tool inspects the running kernel and only runs 'grubby --set-default' for RHCK.
 #
 # Every one of those failure modes is silent by default. This tool handles each case and fails
 # loudly when it cannot, rather than reporting success while changing nothing.
 #
 # Manager vs minion: only the manager owns /nsm/kernelrepo, so only the manager can populate
 # it. If the repo is empty here, a manager runs so-repo-sync itself; a minion has no way to
 # fix it and exits non-zero telling the admin to sync the manager first.
 #
 # Idempotent: an already-installed, already-default UEK8 kernel is left alone. It only sets
 # the boot default; it does NOT reboot -- the admin reboots the node on their own schedule.
 . /usr/sbin/so-common
 # Client-side repo id (what dnf enables on this node, from repo/client/oracle.sls) vs the
 # reposync-side section in repodownload.conf that the manager mirrors from (mirrors the
 # securityonion/securityonionsync split for the main repo).
 KERNEL_REPO="securityonionkernel"
 KERNEL_REPO_SYNC="securityonionkernelsync"
 KERNEL_PKG="kernel-uek"
 KERNEL_REPO_DIR="/nsm/kernelrepo"
 REPOSYNC_CONF="/opt/so/conf/reposync/repodownload.conf"
 GLOBAL_PILLAR="/opt/so/saltstack/local/pillar/global/soc_global.sls"
 log() { echo "[so-kernel-upgrade] $*"; }
 die() { echo "[so-kernel-upgrade] ERROR: $*" >&2; exit 1; }
 command -v grubby >/dev/null 2>&1 || die "grubby not found"
 command -v dnf >/dev/null 2>&1 || die "dnf not found"
 ARCH="$(rpm -E '%{_arch}')"
 is_airgap() {
    [ -f "$GLOBAL_PILLAR" ] && grep -q 'airgap: *[Tt]rue' "$GLOBAL_PILLAR"
 }
 # Newest installed UEK8 (6.x) kernel known to the bootloader. UEK8 vmlinuz paths look like
 # /boot/vmlinuz-6.12.0-204.92.4.2.el9uek.x86_64; UEK7 (5.15) and RHCK (5.14) won't match.
 find_uek8() {
    grubby --info=ALL 2>/dev/null \
        | sed -n 's/^kernel="\(.*\)"$/\1/p' \
        | grep -E '/vmlinuz-6\.[0-9]+.*uek' \
        | sort -V | tail -1
 }
 # Classify the RUNNING kernel (uname -r) -- this, not what's installed, is what decides whether
 # a UEK8 install auto-promotes to the boot default:
 #   uek8  6.x UEK    already on the target line; nothing to do
 #   uek7  5.x UEK    a UEK8 install stays in the kernel-uek lineage and auto-promotes (no grubby)
 #   rhck  5.14 EL9   crossing into the UEK flavor does NOT auto-promote (needs grubby --set-default)
 running_flavor() {
    case "$(uname -r)" in
        6.*uek*) echo uek8 ;;
        *uek*)   echo uek7 ;;
        *)       echo rhck ;;
    esac
 }
 # Newest UEK8 kernel-uek NEVRA offered by the kernel repo, empty if the repo has none.
 # Restricted to the kernel repo so a UEK7 kernel-uek in the main repo can't be picked up,
 # and filtered to 6.x so we never "succeed" by reinstalling the 5.15 we already have.
 uek8_available() {
    dnf -q repoquery --disablerepo='*' --enablerepo="$KERNEL_REPO" \
        --arch="$ARCH" --latest-limit=1 \
        --qf '%{name}-%{evr}.%{arch}\n' "$KERNEL_PKG" 2>/dev/null \
        | grep -E "^${KERNEL_PKG}-6\." | tail -1
 }
 kernelrepo_rpm_count() {
    find "$KERNEL_REPO_DIR" -maxdepth 1 -name '*.rpm' 2>/dev/null | wc -l
 }
 # The kernel repo starts life as valid-but-empty (kernelrepo_init_empty in
 # salt/manager/init.sls) and is filled by so-repo-sync. During a soup, so-repo-sync runs
 # BEFORE the highstate deploys the [securityonionkernelsync] section into repodownload.conf, so
 # the first kernel-aware soup leaves the repo empty until the next nightly sync.
 sync_kernel_repo() {
    if is_airgap; then
        log "airgap install: $KERNEL_REPO_DIR is populated from the airgap ISO, not by so-repo-sync."
        return 1
    fi
    if ! grep -q "^\[${KERNEL_REPO_SYNC}\]" "$REPOSYNC_CONF" 2>/dev/null; then
        log "$REPOSYNC_CONF has no [${KERNEL_REPO_SYNC}] section -- run a highstate to deploy it."
        return 1
    fi
    log "populating $KERNEL_REPO_DIR with so-repo-sync (mirrors upstream; can take several minutes)"
    su socore -c '/usr/sbin/so-repo-sync' || { log "so-repo-sync failed"; return 1; }
    dnf -q clean expire-cache >/dev/null 2>&1
    return 0
 }
 # Make the kernel repo actually able to serve a UEK8 package, or fail trying.
 ensure_kernel_repo() {
    # The repo is assigned by the repo.client highstate, and only once NICs are pinned by MAC
    # (/opt/so/state/nic_names_pinned) so the kernel swap can't renumber interfaces SO binds
    # by name. skip_if_unavailable=1 means a broken repo is silently ignored, so check first.
    if ! dnf -q repolist --enabled 2>/dev/null | awk '{print $1}' | grep -qx "$KERNEL_REPO"; then
        log "repo '$KERNEL_REPO' is not enabled on this node."
        log "Run a highstate first; the repo is skipped until /opt/so/state/nic_names_pinned"
        log "exists (run so-nic-pin) and this node's salt matches the version this release ships."
        die "kernel repo unavailable"
    fi
    [ -n "$(uek8_available)" ] && return 0
    log "repo '$KERNEL_REPO' is enabled but offers no UEK8 $KERNEL_PKG package"
    if ! is_manager_node; then
        log "This is a minion; it consumes the kernel repo from the manager and cannot populate it."
        log "On the manager, run:  su socore -c /usr/sbin/so-repo-sync"
        log "then re-run this script here."
        die "manager's kernel repo is empty"
    fi
    log "this is a manager and $KERNEL_REPO_DIR holds $(kernelrepo_rpm_count) rpm(s)"
    sync_kernel_repo || die "could not populate $KERNEL_REPO_DIR"
    [ -n "$(uek8_available)" ] \
        || die "so-repo-sync completed but $KERNEL_REPO still offers no UEK8 $KERNEL_PKG"
 }
 reboot_notice() {
    [ "$(uname -r)" = "$(basename "$1" | sed 's/^vmlinuz-//')" ] \
        || log "REBOOT REQUIRED to start using the UEK8 kernel (currently running $(uname -r))."
 }
 # Keep future kernel updates on the UEK line rather than falling back to RHCK. Oracle ships
 # /etc/sysconfig/kernel; only rewrite it when it's actually pointing somewhere else.
 set_default_kernel_conf() {
    if [ -f /etc/sysconfig/kernel ] && ! grep -q '^DEFAULTKERNEL=kernel-uek-core$' /etc/sysconfig/kernel; then
        log "setting DEFAULTKERNEL=kernel-uek-core in /etc/sysconfig/kernel"
        sed -i 's/^DEFAULTKERNEL=.*/DEFAULTKERNEL=kernel-uek-core/' /etc/sysconfig/kernel
    fi
 }
 # Make sure a UEK8 kernel is installed, leaving its boot entry in INSTALLED_UEK8. If one is
 # already present we leave the repo alone -- it may be disabled or empty and we don't need it
 # just to flip the boot default. Otherwise install the explicit NEVRA, not the bare package
 # name: on a UEK7 node 'dnf install kernel-uek' sees 5.15 already present, prints "Nothing to
 # do" and exits 0 without installing 6.x.
 ensure_uek8_installed() {
    INSTALLED_UEK8="$(find_uek8)"
    if [ -n "$INSTALLED_UEK8" ]; then
        log "UEK8 kernel already installed: $INSTALLED_UEK8"
        return 0
    fi
    ensure_kernel_repo
    local nevra; nevra="$(uek8_available)"
    log "installing $nevra from $KERNEL_REPO"
    dnf -y install "$nevra" || die "failed to install $nevra"
    INSTALLED_UEK8="$(find_uek8)"
    [ -n "$INSTALLED_UEK8" ] || die "$nevra installed but no 6.x UEK boot entry appeared -- check 'grubby --info=ALL'"
    log "installed UEK8 kernel: $INSTALLED_UEK8"
 }
 case "$(running_flavor)" in
 uek8)
    # Already on the 6.x UEK line. A plain 'dnf update' keeps this node current within the
    # lineage and auto-promotes newer builds, so there is nothing for this tool to do.
    log "already running a UEK8 kernel ($(uname -r)); nothing to do."
    exit 0
    ;;
 uek7)
    # On a 5.x UEK kernel. Installing UEK8 stays inside the kernel-uek lineage, so dnf/grubby
    # (UPDATEDEFAULT=yes) auto-promote it and we do NOT touch grubby. A node still on UEK7
    # usually means the kernel repo was empty when it last updated, so populate it and install.
    log "running UEK7 kernel ($(uname -r)); the kernel repo was likely not yet populated when"
    log "this node last updated. Populating it and installing UEK8 -- the update stays on the"
    log "kernel-uek line, so it becomes the boot default automatically (no grubby change needed)."
    set_default_kernel_conf
    ensure_uek8_installed
    now="$(grubby --default-kernel 2>/dev/null)"
    if [ "$now" = "$INSTALLED_UEK8" ]; then
        log "boot default auto-promoted to UEK8 kernel: $INSTALLED_UEK8"
    else
        log "WARNING: expected the UEK8 kernel to auto-promote but the default is still"
        log "'${now:-unknown}'. Run 'grubby --set-default=$INSTALLED_UEK8' to force it."
    fi
    reboot_notice "$INSTALLED_UEK8"
    ;;
 rhck)
    # On the stock EL9 kernel (5.14, no UEK installed). Crossing from RHCK into the UEK flavor
    # does NOT auto-promote -- kernel-install/grubby only auto-promote within the running
    # kernel's flavor lineage -- so after installing we must set the boot default explicitly.
    log "running stock EL9 (RHCK) kernel ($(uname -r)); installing UEK8 and setting it as the"
    log "boot default explicitly (a RHCK->UEK flavor change does not auto-promote)."
    set_default_kernel_conf
    ensure_uek8_installed
    target="$INSTALLED_UEK8"
    current="$(grubby --default-kernel 2>/dev/null)"
    if [ "$current" = "$target" ]; then
        log "UEK8 kernel is already the boot default: $target"
        reboot_notice "$target"
        exit 0
    fi
    log "current default kernel: ${current:-unknown}"
    log "switching boot default to UEK8 kernel: $target"
    grubby --set-default="$target" || die "grubby --set-default failed for $target"
    # Verify the change actually took before claiming success.
    now="$(grubby --default-kernel 2>/dev/null)"
    [ "$now" = "$target" ] || die "default kernel is still '${now:-unknown}' after set-default"
    log "boot default is now $target"
    reboot_notice "$target"
    ;;
 esac
@@ -132,8 +132,6 @@ if [[ $EXCLUDE_STARTUP_ERRORS == 'Y' ]]; then
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|HTTP 404: Not Found"          # Salt loops until Kratos returns 200, during startup Kratos may not be ready
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|Cancelling deferred write event maybeFenceReplicas because the event queue is now closed" # Kafka controller log during shutdown/restart
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|Redis may have been restarted" # Redis likely restarted by salt
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|file already closed"          # Go logging race condition during container restart
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|relation \"audit_settings\" does not exist" # salt checking for changes before SOC starts
 fi
 if [[ $EXCLUDE_FALSE_POSITIVE_ERRORS == 'Y' ]]; then
@@ -167,8 +165,6 @@ if [[ $EXCLUDE_FALSE_POSITIVE_ERRORS == 'Y' ]]; then
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|upgrading component template"  # false positive (elasticsearch index or template names contain 'error')
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|upgrading composable template" # false positive (elasticsearch composable template names contain 'error')
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|Error while parsing document for index \[.ds-logs-kratos-so-.*object mapping for \[file\]" # false positive (mapping error occuring BEFORE kratos index has rolled over in 2.4.210)
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|No such container"            # false positive (telegraf trying to run stats on an old container)
    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|passwords do not match"       # false positive (automated hydra test)
 fi
 if [[ $EXCLUDE_KNOWN_ERRORS == 'Y' ]]; then
@@ -1,76 +0,0 @@
 #!/bin/bash
 #
 # so-nic-pin — pin physical NIC names by permanent MAC via classic by-MAC udev
 #              rules, so a kernel upgrade can't renumber them.
 #
 # Security Onion binds its management and monitor interfaces BY NAME in pillar
 # (host:mainint, sensor:mainint, and bond0 is built on a specific physical NIC).
 # A kernel upgrade can change the kernel/systemd-udevd predictable-naming output
 # and renumber those NICs (e.g. enp1s0 -> enp2s0), which breaks the grid: the
 # pillar references a name that no longer exists and bond/bridge bring-up fails.
 #
 # This writes /etc/udev/rules.d/70-persistent-net.rules pinning each PHYSICAL NIC
 # to its CURRENT name by its PERMANENT MAC, freezing the names across future kernel
 # changes. It only writes the rules file; it does NOT live-trigger a rename (the
 # rules apply on the next boot/kernel, and a live rename would be disruptive).
 #
 # Run-once: gated by the drop file /opt/so/state/nic_names_pinned. If the marker is
 # present the script does nothing, so an admin can pre-create it to opt out. Invoked
 # from the common state on every highstate; the marker keeps it a one-time setup.
 NET_RULES_FILE="/etc/udev/rules.d/70-persistent-net.rules"
 MARKER="/opt/so/state/nic_names_pinned"
 log() { echo -e "[so-nic-pin] $*"; }
 # Echo "<name> <permanent-mac>" for every PHYSICAL NIC. A physical NIC is backed by a
 # real device (has device/driver), which excludes bond0/sobridge/docker0/veth*/lo whose
 # MACs are dynamic and must never be pinned. The PERMANENT MAC is used (ethtool -P, with
 # fallbacks), not the current one: an enslaved bond member's current MAC is rewritten to
 # the bond's, so matching on it would be wrong/ambiguous.
 physical_nics() {
    local path n mac
    for path in /sys/class/net/*; do
        n="${path##*/}"
        [ "$n" = "lo" ] && continue
        [ -e "${path}/device/driver" ] || continue          # real device only
        mac="$(ethtool -P "$n" 2>/dev/null | awk '/Permanent address/{print $NF}')"
        case "$mac" in ""|00:00:00:00:00:00) mac="$(cat "${path}/bonding_slave/perm_hwaddr" 2>/dev/null)" ;; esac
        case "$mac" in ""|00:00:00:00:00:00) mac="$(cat "${path}/address" 2>/dev/null)" ;; esac
        case "$mac" in ""|00:00:00:00:00:00) continue ;; esac
        echo "$n $mac"
    done
 }
 # Turn "<name> <mac>" lines on stdin into classic by-MAC persistent-net udev rules.
 render_net_rules() {
    echo "# Generated by so-nic-pin: pin NIC names by MAC so kernel upgrades can't renumber them."
    echo "# Security Onion binds its management/monitor interfaces by name; do not hand-edit."
    local n mac
    while read -r n mac; do
        [ -n "$n" ] || continue
        printf 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="%s", NAME="%s"\n' \
            "$mac" "$n"
    done
 }
 [ "$(id -u)" -eq 0 ] || exit 0                   # salt runs us as root; bail quietly otherwise
 [ -e "${MARKER}" ] && exit 0                      # run-once guard (mirrors the state's unless)
 nics="$(physical_nics)"
 if [ -z "${nics}" ]; then
    log "no physical NICs detected — nothing to pin (will retry on next highstate)"
    exit 0                                         # do NOT drop the marker; let it retry later
 fi
 log "pinning physical NICs by permanent MAC:"
 echo "${nics}" | sed 's/^/    /'
 [ -f "${NET_RULES_FILE}" ] && cp -f "${NET_RULES_FILE}" "${NET_RULES_FILE}.bak"
 echo "${nics}" | render_net_rules > "${NET_RULES_FILE}" || {
    log "ERROR: failed to write ${NET_RULES_FILE}"
    exit 1
 }
 mkdir -p "$(dirname "${MARKER}")" && touch "${MARKER}"
 log "wrote ${NET_RULES_FILE} ($(grep -c '^SUBSYSTEM' "${NET_RULES_FILE}") NIC(s) pinned); dropped ${MARKER}"
@@ -5,44 +5,27 @@
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 # Usage: so-restart  kibana | playbook 
 . /usr/sbin/so-common
-usage() {
+if [ $# -ge 1 ]; then
    echo "Usage: $0 <component> [args]"
    echo ""
    echo "Supported args:"
    echo "  --force | -f          Force stop all Salt jobs before starting component."
    echo ""
    echo "Examples:"
    echo "  $0 kibana             Restart Kibana"
    echo "  $0 kibana --force     Force stop all Salt jobs before restarting Kibana"
    exit 1
 }
-if [[ $# -lt 1 ]]; then
+        echo $banner
-    usage
+        printf "Restarting $1...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
-fi
+        echo $banner
-#shellcheck disable=SC2154
+        if [ "$2" = "--force" ]; then
 echo "$banner"
 printf "Restarting %s...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n" "$1"
 echo "$banner"
 if [[ "$2" = "--force" ]] || [[ "$2" = "-f" ]]; then
                printf "\nForce-stopping all Salt jobs before proceeding\n\n"
                salt-call saltutil.kill_all_jobs
        fi
 case $1 in
    "elastic-fleet"|"elasticfleet")
        docker_check_running "elastic-fleet" "--stop"
        docker rm "so-elastic-fleet" 2> /dev/null
        # Removing the elastic fleet state directory, so that the next startup re-enrolls with a fresh policy
        rm -rf /opt/so/conf/elastic-fleet/state
-        salt-call state.apply elasticfleet queue=True
+        case $1 in
-        ;;
+                "elastic-fleet") docker stop so-elastic-fleet && docker rm so-elastic-fleet && salt-call state.apply elasticfleet queue=True;;
-    *)
+                *)  docker stop so-$1 ; docker rm so-$1 ; salt-call state.apply $1 queue=True;;
        docker_check_running "$1" "--stop"
        docker rm "so-${1}" 2> /dev/null
        salt-call state.apply "$1" queue=True
        ;;
        esac
 else
        echo -e "\nPlease provide an argument by running like so-restart $component, or by using the component-specific script.\nEx. so-restart logstash, or so-logstash-restart\n"
 fi
@@ -5,54 +5,27 @@
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
-# shellcheck disable=SC1091
+
 # Usage: so-start  all | kibana | playbook 
 . /usr/sbin/so-common
-usage() {
+if [ $# -ge 1 ]; then
-    echo "Usage: $0 <component> [args]"
+	echo $banner
-    echo ""
+	printf "Starting $1...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
-    echo "Supported args:"
+	echo $banner
    echo "  --force | -f          Force stop all Salt jobs before starting component."
    echo ""
    echo "Examples:"
    echo "  $0 kibana             Start Kibana"
    echo "  $0 kibana --force     Force stop all Salt jobs before starting Kibana"
    exit 1
 }
-if [[ $# -lt 1 ]]; then
+	if [ "$2" = "--force" ]; then
    usage
 fi
 #shellcheck disable=SC2154
 echo "$banner"
 printf "Starting %s...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n" "$1"
 echo "$banner"
 if [[ "$2" = "--force" ]] || [[ "$2" == "-f" ]]; then
   		printf "\nForce-stopping all Salt jobs before proceeding\n\n"
   		salt-call saltutil.kill_all_jobs
 	fi
-case "$1" in
+	case $1 in
-    "all")
+   		"all") salt-call state.highstate queue=True;;
-        salt-call state.highstate queue=True
+   		"elastic-fleet") if docker ps | grep -q so-$1; then printf "\n$1 is already running!\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply elasticfleet queue=True; fi ;; 
-        ;;
+   		*) if docker ps | grep -E -q '^so-$1$'; then printf "\n$1 is already running\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply $1 queue=True; fi ;; 
   	"elastic-fleet"|"elasticfleet")
        if docker_check_running "elastic-fleet"; then
            printf "\nso-%s is already running!\n\n" "elastic-fleet"
            /usr/sbin/so-status
        else
            docker rm "so-elastic-fleet" 2> /dev/null
            salt-call state.apply elasticfleet queue=True
        fi
        ;;
   	*)
        if docker_check_running "$1"; then
            printf "\nso-%s is already running\n\n" "$1"
            /usr/sbin/so-status
        else
            docker rm "so-${1}" 2> /dev/null
            salt-call state.apply "$1" queue=True
        fi
        ;;
 	esac
 else
 	echo -e "\nPlease provide an argument by running like so-start $component, or by using the component-specific script.\nEx. so-start logstash, or so-logstash-start\n"	
 fi
@@ -74,13 +74,13 @@ def output(options, console, code, data):
    summary = { "status_code": code, "containers": data }
    print(json.dumps(summary))
  elif "-q" not in options:
-    if code == 99:
+    if code == 2:
      console.print(" [bold yellow]:hourglass: [bold white]System appears to be starting. No highstate has completed since the system was restarted.")
    elif code == 99:
      console.print(" [bold red]:exclamation: [bold white]Installation does not appear to be complete. A highstate has not fully completed.")
    elif code == 100:
      console.print(" [bold red]:exclamation: [bold white]Installation encountered errors.")
    else:
      if code == 2:
        console.print(" [bold yellow]:hourglass: [bold white]System appears to be starting. No highstate has completed since the system was restarted. Container status is shown below.")
      table = Table(title = "Security Onion Status", show_edge = False, safe_box = True, box = box.MINIMAL)
      table.add_column("Container", justify="right", style="white", no_wrap=True)
      table.add_column("Status", justify="left", style="green", no_wrap=True)
@@ -154,14 +154,8 @@ def check_status(options, console):
  code = check_installation_status(options, console)
  if code == 0:
    code = check_system_status(options, console)
  # Containers now start on boot without a highstate, so gather/display their
  # status even when the system is still "starting" (code 2). Keep the starting
  # code as the exit/status_code so SOC keeps showing the "restarting" message
  # on the Grid until a highstate completes.
  if code == 0 or code == 2:
    container_code, container_list = check_container_status(options, console)
  if code == 0:
-      code = container_code
+    code, container_list = check_container_status(options, console)
  output(options, console, code, container_list)
  return code
@@ -186,3 +180,4 @@ def main():
 if __name__ == "__main__":
  main()
@@ -5,35 +5,21 @@
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
-# shellcheck disable=SC1091
+
 # Usage: so-stop kibana | playbook | thehive
 . /usr/sbin/so-common
-usage() {
+if [ $# -ge 1 ]; then
-    echo "Usage: $0 <component>"
+	echo $banner
-    echo ""
+	printf "Stopping $1...\n"
-    echo "Examples:"
+	echo $banner
    echo "  $0 kibana             Stop Kibana"
    exit 1
 }
-if [[ $# -lt 1 ]]; then
+	case $1 in
-    usage
+    		*) docker stop so-$1 ; docker rm so-$1 ;;
 	esac
 else
 	echo -e "\nPlease provide an argument by running like so-stop $component, or by using the component-specific script.\nEx. so-stop logstash, or so-logstash-stop\n"	
 fi
 #shellcheck disable=SC2154
 echo "$banner"
 printf "Stopping %s...\n" "$1"
 echo "$banner"
 case $1 in
 	"elasticfleet"|"elastic-fleet")
        docker_check_running "elastic-fleet" "--stop"
        docker rm "so-elastic-fleet" 2> /dev/null
        # Removing the elastic fleet state directory, so that the next startup re-enrolls with a fresh policy
        rm -rf /opt/so/conf/elastic-fleet/state
        ;;
    *)
        docker_check_running "$1" "--stop"
        docker rm "so-${1}" 2> /dev/null
    ;;
 esac
@@ -63,8 +63,7 @@ function status {
 function pcapinfo() {
  PCAP=$1
  ARGS=$2
-  docker run --rm -v "$PCAP:/input.pcap" --entrypoint capinfos {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} /input.pcap -ae $ARGS |\
+  docker run --rm -v "$PCAP:/input.pcap" --entrypoint capinfos {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} /input.pcap -ae $ARGS
    sed 's/First packet/Earliest packet/g' | sed 's/Last packet/Latest packet/g'
 }
 function pcapfix() {
@@ -1,3 +1,5 @@
 {% import_yaml 'salt/minion.defaults.yaml' as SALT_MINION_DEFAULTS -%}
 #!/bin/bash
 #
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
@@ -5,7 +7,7 @@
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
-{% from 'salt/schedule.map.jinja' import SCHEDULEMERGED %}
+
 # this script checks the time the file /opt/so/log/salt/state-apply-test was last modified and restarts the salt-minion service if it is outside a threshold date/time
 # the file is modified via file.touch using a scheduled job healthcheck.salt-minion.state-apply-test that runs a state.apply. 
@@ -18,14 +20,12 @@
 QUIET=false
 UPTIME_REQ=1800 #in seconds, how long the box has to be up before considering restarting salt-minion due to /opt/so/log/salt/state-apply-test not being touched
 HIGHSTATE_UPTIME_REQ=900 #in seconds; if the box has been up this long and no highstate has completed since boot, force one
 CURRENT_TIME=$(date +%s)
 SYSTEM_START_TIME=$(date -d "$(</proc/uptime awk '{print $1}') seconds ago" +%s)
 LAST_HIGHSTATE_END=$([ -e "/opt/so/log/salt/lasthighstate" ] && date -r /opt/so/log/salt/lasthighstate +%s || echo 0)
 LAST_HEALTHCHECK_STATE_APPLY=$([ -e "/opt/so/log/salt/state-apply-test" ] && date -r /opt/so/log/salt/state-apply-test +%s || echo 0)
 # SETTING THRESHOLD TO ANYTHING UNDER 600 seconds may cause a lot of salt-minion restarts since the job to touch the file occurs every 5-8 minutes by default
-# THRESHOLD is derived from the salt schedule highstate interval + 1 hour, so the minion-check grace period tracks the schedule automatically.
+THRESHOLD={{SALT_MINION_DEFAULTS.salt.minion.check_threshold}} #within how many seconds the file /opt/so/log/salt/state-apply-test must have been touched/modified before the salt minion is restarted
 THRESHOLD=$(( ({{ SCHEDULEMERGED.highstate_interval_hours }} + 1) * 3600 )) #within how many seconds the file /opt/so/log/salt/state-apply-test must have been touched/modified before the salt minion is restarted
 THRESHOLD_DATE=$((LAST_HEALTHCHECK_STATE_APPLY+THRESHOLD))
 logCmd() {
@@ -77,50 +77,24 @@ done
 log "running so-salt-minion-check"
 RESTARTED=false
 # Check 1 (minion-restart-check): if the minion has stopped applying states (the
 # state-apply-test healthcheck file has gone stale), restart the salt-minion service.
 if [ $CURRENT_TIME -ge $((SYSTEM_START_TIME+$UPTIME_REQ))  ]; then
  if [ $THRESHOLD_DATE -le $CURRENT_TIME ]; then
-    log "[minion-restart-check] salt-minion is unable to apply states; restarting salt-minion" E
+    log "salt-minion is unable to apply states" E
-    log "[minion-restart-check] state-apply-test not touched by required date `date -d @$THRESHOLD_DATE`, last touched `date -d @$LAST_HEALTHCHECK_STATE_APPLY`" I
+    log "/opt/so/log/salt/healthcheck-state-apply not touched by required date: `date -d @$THRESHOLD_DATE`, last touched: `date -d @$LAST_HEALTHCHECK_STATE_APPLY`" I
-    log "[minion-restart-check] last highstate completed at `date -d @$LAST_HIGHSTATE_END`" I
+    log "last highstate completed at `date -d @$LAST_HIGHSTATE_END`" I
-    log "[minion-restart-check] checking if any jobs are running" I
+    log "checking if any jobs are running" I
    logCmd "salt-call --local saltutil.running" I
-    log "[minion-restart-check] ensure salt.minion-state-apply-test is enabled" I
+    log "ensure salt.minion-state-apply-test is enabled" I
    logCmd "salt-call state.enable salt.minion-state-apply-test" I
-    log "[minion-restart-check] ensure highstate is enabled" I
+    log "ensure highstate is enabled" I
    logCmd "salt-call state.enable highstate" I
-    log "[minion-restart-check] killing all salt-minion processes" I
+    log "killing all salt-minion processes" I
    logCmd "pkill -9 -ef /usr/bin/salt-minion" I
-    log "[minion-restart-check] starting salt-minion service" I
+    log "starting salt-minion service" I
    logCmd "systemctl start salt-minion" I
    log "[minion-restart-check] waiting for salt-minion to become ready, then applying highstate in the background (queued)" I
    nohup bash -c '/usr/sbin/so-salt-minion-wait; salt-call state.highstate queue=True' >> "/opt/so/log/salt/so-salt-minion-check" 2>&1 &
    RESTARTED=true
  else
-    log "[minion-restart-check] healthy: state-apply-test last touched `date -d @$LAST_HEALTHCHECK_STATE_APPLY`, must go stale past `date -d @$THRESHOLD_DATE` to trigger a salt-minion restart" I
+    log "/opt/so/log/salt/healthcheck-state-apply last touched: `date -d @$LAST_HEALTHCHECK_STATE_APPLY` must be touched by `date -d @$THRESHOLD_DATE` to avoid salt-minion restart" I
  fi
 else
-  log "[minion-restart-check] skipped: system uptime $((CURRENT_TIME-SYSTEM_START_TIME))s is below the ${UPTIME_REQ}s minimum required before a salt-minion restart" I
+  log "system uptime only $((CURRENT_TIME-SYSTEM_START_TIME)) seconds does not meet $UPTIME_REQ second requirement." I
 fi
 # Check 2 (boot-highstate-check): if the host has been up long enough but no highstate
 # has completed since this boot, force one. This recovers a host whose boot highstate
 # (so-boot-highstate.service) failed or was skipped, even while the minion is otherwise
 # healthy (touching state-apply-test). We deliberately do NOT enable highstate here: if
 # soup has disabled it during an upgrade, Salt will refuse the highstate and we avoid
 # forcing one mid-upgrade.
 if $RESTARTED; then
  log "[boot-highstate-check] skipped: minion-restart-check already queued a highstate this run" I
 elif [ $CURRENT_TIME -lt $((SYSTEM_START_TIME+HIGHSTATE_UPTIME_REQ)) ]; then
  log "[boot-highstate-check] skipped: system uptime $((CURRENT_TIME-SYSTEM_START_TIME))s is below the ${HIGHSTATE_UPTIME_REQ}s minimum required before forcing a highstate" I
 elif [ $LAST_HIGHSTATE_END -ge $SYSTEM_START_TIME ]; then
  log "[boot-highstate-check] healthy: a highstate completed at `date -d @$LAST_HIGHSTATE_END`, after this boot at `date -d @$SYSTEM_START_TIME`" I
 elif salt-call --local saltutil.running 2>/dev/null | grep -q 'state.highstate'; then
  log "[boot-highstate-check] no highstate has completed since boot, but one is already running; skipping" I
 else
  log "[boot-highstate-check] no highstate has completed since boot after $((CURRENT_TIME-SYSTEM_START_TIME))s uptime; applying highstate" E
  nohup bash -c 'salt-call state.highstate -l info queue=True' >> "/opt/so/log/salt/so-salt-minion-check" 2>&1 &
 fi
@@ -9,8 +9,7 @@
 prune_images:
  cmd.run:
    - name: so-docker-prune
-    - onlyif: command -v /usr/sbin/so-docker-prune >/dev/null 2>&1
+    - order: last
    - order: 9000
 {% else %}
@@ -19,7 +19,6 @@ wait_for_elasticsearch:
 so-elastalert:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastalert:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: elastalert
    - name: so-elastalert
    - user: so-elastalert
@@ -15,7 +15,6 @@ include:
 so-elastic-fleet-package-registry:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-fleet-package-registry:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - name: so-elastic-fleet-package-registry
    - hostname: Fleet-package-reg-{{ GLOBALS.hostname }}
    - detach: True
@@ -16,7 +16,6 @@ include:
 so-elastic-agent:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - name: so-elastic-agent
    - hostname: {{ GLOBALS.hostname }}
    - detach: True
@@ -173,7 +173,7 @@ eaoptionalintegrationsdir:
 {% for minion in node_data %}
 {% set role = node_data[minion]["role"] %}
-{% if role in [ "eval","fleet","import","manager", "managerhype", "managersearch","standalone" ] %}
+{% if role in [ "eval","fleet","heavynode","import","manager", "managerhype", "managersearch","standalone" ] %}
 {% set optional_integrations = ELASTICFLEETMERGED.optional_integrations %}
 {% set integration_keys = optional_integrations.keys() %}
 fleet_server_integrations_{{ minion }}:
@@ -9,6 +9,7 @@
 {% set CORE_ESFLEET_PACKAGES = ELASTICFLEETDEFAULTS.get('elasticfleet', {}).get('packages', {}) %}
 {% set ADDON_CONTENT_INTEGRATION_DEFAULTS = {} %}
 {% set DEBUG_STUFF = {} %}
 {% for pkg in ADDON_CONTENT_PACKAGE_COMPONENTS %}
 {%   if pkg.name in CORE_ESFLEET_PACKAGES %}
@@ -1,5 +1,6 @@
 elasticfleet:
  enabled: False
  patch_version: 9.3.3+build202604082258 # Elastic Agent specific patch release.
  enable_manager_output: True
  config:
    server:
@@ -11,10 +11,6 @@
 {#   This value is generated during node install and stored in minion pillar #}
 {%   set SERVICETOKEN = salt['pillar.get']('elasticfleet:config:server:es_token','') %}
 {#   Prevent Elastic Agent from re-enrolling with a new agent.id everytime the container starts up.
       - if a fresh enrollment is needed use 'so-stop elasticfleet'
       #}
 {%   set ENROLLED = salt['file.file_exists']('/opt/so/conf/elastic-fleet/state/fleet.enc') %}
 include:
  - ca
@@ -30,9 +26,7 @@ include:
 wait_for_elasticsearch_elasticfleet:
  cmd.run:
    - name: so-elasticsearch-wait
 {% endif %}
 {% if GLOBALS.role == "so-fleet" %}
 # Sync Elastic Agent artifacts to Fleet Node
 elasticagent_syncartifacts:
  file.recurse:
@@ -46,7 +40,6 @@ elasticagent_syncartifacts:
 so-elastic-fleet:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - name: so-elastic-fleet
    - hostname: FleetServer-{{ GLOBALS.hostname }}
    - detach: True
@@ -70,7 +63,6 @@ so-elastic-fleet:
      - /etc/pki/elasticfleet-server.crt:/etc/pki/elasticfleet-server.crt:ro
      - /etc/pki/elasticfleet-server.key:/etc/pki/elasticfleet-server.key:ro
      - /etc/pki/tls/certs/intca.crt:/etc/pki/tls/certs/intca.crt:ro
      - /opt/so/conf/elastic-fleet/state:/usr/share/elastic-agent/state
      - /opt/so/log/elasticfleet:/usr/share/elastic-agent/logs 
     {% if DOCKERMERGED.containers['so-elastic-fleet'].custom_bind_mounts %}
        {% for BIND in DOCKERMERGED.containers['so-elastic-fleet'].custom_bind_mounts %}
@@ -78,7 +70,6 @@ so-elastic-fleet:
        {% endfor %}
      {% endif %}      
    - environment:
      {% if not ENROLLED %}
      - FLEET_SERVER_ENABLE=true
      - FLEET_URL=https://{{ GLOBALS.hostname }}:8220
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://{{ GLOBALS.manager }}:9200
@@ -88,9 +79,6 @@ so-elastic-fleet:
      - FLEET_SERVER_CERT_KEY=/etc/pki/elasticfleet-server.key
      - FLEET_CA=/etc/pki/tls/certs/intca.crt     
      - FLEET_SERVER_ELASTICSEARCH_CA=/etc/pki/tls/certs/intca.crt
      {% endif %}
      - STATE_PATH=/usr/share/elastic-agent/state
      - CONFIG_PATH=/usr/share/elastic-agent/state
      - LOGS_PATH=logs
      {% if DOCKERMERGED.containers['so-elastic-fleet'].extra_env %}
        {% for XTRAENV in DOCKERMERGED.containers['so-elastic-fleet'].extra_env %}
@@ -109,20 +97,8 @@ so-elastic-fleet:
      - x509: etc_elasticfleet_crt
    - require:
      - file: trusttheca
      - file: eastatedir
      - x509: etc_elasticfleet_key
      - x509: etc_elasticfleet_crt
 wait_for_so-elastic-fleet:
  http.wait_for_successful_query:
    - name: "https://localhost:8220/api/status"
    - ssl: True
    - verify_ssl: False
    - status: 200
    - wait_for: 300
    - request_interval: 15
    - require:
      - docker_container: so-elastic-fleet
 {%   endif %}
 delete_so-elastic-fleet_so-status.disabled:
@@ -5,7 +5,7 @@
  "package": {
    "name": "endpoint",
    "title": "Elastic Defend",
-    "version": "9.3.1",
+    "version": "9.3.0",
    "requires_root": true
  },
  "enabled": true,
@@ -29,7 +29,7 @@
              "\\.gz$"
            ],
            "include_files": [],
-            "processors": "- dissect:\n    tokenizer: \"/nsm/import/%{import.id}/evtx/%{import.file}\"\n    field: \"log.file.path\"\n    target_prefix: \"\"\n- decode_json_fields:\n    fields: [\"message\"]\n    target: \"\"\n- drop_fields:\n    fields: [\"host\"]\n    ignore_missing: true\n- add_fields:\n    target: data_stream\n    fields:\n      type: logs\n      dataset: system.security\n- add_fields:\n    target: event\n    fields:\n      dataset: system.security\n      module: system\n      imported: true\n- add_fields:\n    target: \"@metadata\"\n    fields:\n        pipeline: logs-system.security-2.20.0\n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-Sysmon/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.sysmon_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.sysmon_operational\n          module: windows\n          imported: true\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.sysmon_operational-3.8.3\n- if:\n    equals:\n      winlog.channel: 'Application'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.application-2.20.0\n- if:\n    equals:\n      winlog.channel: 'System'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.system-2.20.0\n    \n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-PowerShell/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.powershell_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.powershell_operational\n          module: windows\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.powershell_operational-3.8.3\n- add_fields:\n    target: data_stream\n    fields:\n        dataset: import",
+            "processors": "- dissect:\n    tokenizer: \"/nsm/import/%{import.id}/evtx/%{import.file}\"\n    field: \"log.file.path\"\n    target_prefix: \"\"\n- decode_json_fields:\n    fields: [\"message\"]\n    target: \"\"\n- drop_fields:\n    fields: [\"host\"]\n    ignore_missing: true\n- add_fields:\n    target: data_stream\n    fields:\n      type: logs\n      dataset: system.security\n- add_fields:\n    target: event\n    fields:\n      dataset: system.security\n      module: system\n      imported: true\n- add_fields:\n    target: \"@metadata\"\n    fields:\n        pipeline: logs-system.security-2.15.0\n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-Sysmon/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.sysmon_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.sysmon_operational\n          module: windows\n          imported: true\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.sysmon_operational-3.8.0\n- if:\n    equals:\n      winlog.channel: 'Application'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.application\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.application-2.15.0\n- if:\n    equals:\n      winlog.channel: 'System'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: event\n        fields:\n          dataset: system.system\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-system.system-2.15.0\n    \n- if:\n    equals:\n      winlog.channel: 'Microsoft-Windows-PowerShell/Operational'\n  then: \n    - add_fields:\n        target: data_stream\n        fields:\n          dataset: windows.powershell_operational\n    - add_fields:\n        target: event\n        fields:\n          dataset: windows.powershell_operational\n          module: windows\n    - add_fields:\n        target: \"@metadata\"\n        fields:\n            pipeline: logs-windows.powershell_operational-3.8.0\n- add_fields:\n    target: data_stream\n    fields:\n        dataset: import",
            "tags": [
              "import"
            ],
@@ -9,6 +9,7 @@
 {% set CORE_ESFLEET_PACKAGES = ELASTICFLEETDEFAULTS.get('elasticfleet', {}).get('packages', {}) %}
 {% set ADDON_INPUT_INTEGRATION_DEFAULTS = {} %}
 {% set DEBUG_STUFF = {} %}
 {% for pkg in ADDON_INPUT_PACKAGE_COMPONENTS %}
 {%   if pkg.name in CORE_ESFLEET_PACKAGES %}
@@ -115,6 +116,7 @@
 {%         do ADDON_INPUT_INTEGRATION_DEFAULTS.update({integration_key: integration_defaults}) %}
 {%         do DEBUG_STUFF.update({integration_key: "Generating defaults for "+ pkg.name })%}
 {%       endfor %}
 {%     endif %}
 {%   endif %}
@@ -10,15 +10,6 @@
 {% set AGENT_STATUS = salt['service.available']('elastic-agent') %}
 {% set AGENT_EXISTS = salt['file.file_exists']('/opt/Elastic/Agent/elastic-agent') %}
 so-elastic-agent-install:
  file.managed:
    - name: /usr/sbin/so-elastic-agent-install
    - source: salt://elasticfleet/tools/sbin/so-elastic-agent-install
    - user: 947
    - group: 939
    - mode: 755
    - show_changes: False
 {% if not AGENT_STATUS or not AGENT_EXISTS %}
 pull_agent_installer:
@@ -30,9 +21,11 @@ pull_agent_installer:
 run_installer:
  cmd.run:
-    - name: /usr/sbin/so-elastic-agent-install "{{ GRIDNODETOKEN }}"
+    - name: ./so-elastic-agent_linux_amd64 -token={{ GRIDNODETOKEN }} -force
-    - require:
+    - cwd: /opt/so
-      - file: pull_agent_installer
+    - retry:
        attempts: 3
        interval: 20
 cleanup_agent_installer:
  file.absent:
@@ -9,19 +9,15 @@
 include:
  - elasticfleet.config
  - kibana.enabled
 # If enabled, automatically update Fleet Logstash Outputs
-{% if ELASTICFLEETMERGED.config.server.enable_auto_configuration %}
+{% if ELASTICFLEETMERGED.config.server.enable_auto_configuration and grains.role not in ['so-import', 'so-eval'] %}
 {%   if grains.role not in ['so-import', 'so-eval']%}
 so-elastic-fleet-auto-configure-logstash-outputs:
  cmd.run:
    - name: /usr/sbin/so-elastic-fleet-outputs-update
    - retry:
        attempts: 4
        interval: 30
    - require:
      - http: wait_for_so-kibana
 {% endif %}
 # If enabled, automatically update Fleet Server URLs & ES Connection
@@ -31,9 +27,6 @@ so-elastic-fleet-auto-configure-server-urls:
    - retry:
        attempts: 4
        interval: 30
    - require:
      - http: wait_for_so-kibana
 {% endif %}
 # Automatically update Fleet Server Elasticsearch URLs & Agent Artifact URLs
 so-elastic-fleet-auto-configure-elasticsearch-urls:
@@ -42,8 +35,6 @@ so-elastic-fleet-auto-configure-elasticsearch-urls:
    - retry:
        attempts: 4
        interval: 30
    - require:
      - http: wait_for_so-kibana
 so-elastic-fleet-auto-configure-artifact-urls:
  cmd.run:
@@ -51,8 +42,6 @@ so-elastic-fleet-auto-configure-artifact-urls:
    - retry:
        attempts: 4
        interval: 30
    - require:
      - http: wait_for_so-kibana
 so-elastic-fleet-package-statefile:
  file.managed:
@@ -64,9 +53,9 @@ so-elastic-fleet-package-upgrade:
    - name: /usr/sbin/so-elastic-fleet-package-upgrade
    - retry:
        attempts: 3
-        interval: 30
+        interval: 10
-    - require:
+    - onchanges:
-      - http: wait_for_so-kibana
+      - file: /opt/so/state/elastic_fleet_packages.txt
 so-elastic-fleet-integrations:
  cmd.run:
@@ -74,8 +63,6 @@ so-elastic-fleet-integrations:
    - retry:
        attempts: 3
        interval: 10
    - require:
      - http: wait_for_so-kibana
 so-elastic-agent-grid-upgrade:
  cmd.run:
@@ -83,8 +70,6 @@ so-elastic-agent-grid-upgrade:
    - retry:
        attempts: 12
        interval: 5
    - require:
      - http: wait_for_so-kibana
 so-elastic-fleet-integration-upgrade:
  cmd.run:
@@ -92,22 +77,16 @@ so-elastic-fleet-integration-upgrade:
    - retry:
        attempts: 3
        interval: 10
    - require:
      - http: wait_for_so-kibana
 {# Optional integrations script doesn't need the retries like so-elastic-fleet-integration-upgrade which loads the default integrations #}
 so-elastic-fleet-addon-integrations:
  cmd.run:
    - name: /usr/sbin/so-elastic-fleet-optional-integrations-load
    - require:
      - http: wait_for_so-kibana
 {% if ELASTICFLEETMERGED.config.defend_filters.enable_auto_configuration %}
 so-elastic-defend-manage-filters-file-watch:
  cmd.run:
    - name: python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters/ &>> /opt/so/log/elasticfleet/elastic-defend-manage-filters.log
    - require:
      - http: wait_for_so-kibana
    - onchanges:
      - file: elasticdefendcustom
      - file: elasticdefenddisabled
@@ -1,100 +0,0 @@
 #!/bin/bash
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 . /usr/sbin/so-elastic-fleet-common
 # passed in as arg from elasticfleet/install_agent_grid.sls, else pulled from pillar later
 GRIDNODETOKEN="$1"
 LOGFILE="/opt/so/SO-Elastic-Agent_Installer_Health.log"
 check_agent_health() {
    timeout=300
    interval=10
    start=$SECONDS
    while (( SECONDS - start < timeout )); do
        agent_status=$(elastic-agent status 2>&1)
        echo -e "\n$(date)\n$agent_status\n" >> "$LOGFILE"
        if echo "$agent_status" | grep -A1 'elastic-agent$' | grep -q 'status: (HEALTHY)'; then
            return 0
        fi
        echo "The Elastic Agent is not yet healthy. Waiting for ${interval} seconds before checking again..."
        sleep "$interval"
    done
    echo "The Elastic Agent did not become healthy within ${timeout} seconds"
    return 1
 }
 uninstall_agent() {
    if command -v elastic-agent >/dev/null 2>&1; then
        elastic-agent uninstall -f
    fi
 }
 if [[ -z "$GRIDNODETOKEN" ]]; then
    noderole=$(so-yaml.py get -r /etc/salt/grains role)
    if [[ "$noderole" == "so-heavynode" ]]; then
        GRIDNODETOKEN=$(salt-call pillar.get global:fleet_grid_enrollment_token_heavy --out=newline_values_only)
    else
        GRIDNODETOKEN=$(salt-call pillar.get global:fleet_grid_enrollment_token_general --out=newline_values_only)
    fi
 fi
 if [[ -z "$GRIDNODETOKEN" ]]; then
    echo "Unable to determine Elastic Fleet enrollment token. Exiting."
    exit 1
 fi
 if [[ ! -x /opt/so/so-elastic-agent_linux_amd64 ]]; then
    echo "Downloading so-elastic-agent installer... This could take a while if another Salt job is running."
    # When running outside of elasticfleet/install_agent_grid.sls we need to download the installer independently.
    #  PYTHONWARNINGS="ignore" to avoid messages like the following when running salt-call:
    #  '/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py:129: TransportWarning: Unclosed transport! <salt.transport.zeromq.RequestClient object at 0x7fc5f0ee7a30>
    #    File "/bin/salt-call", line 12, in <module>
    #      sys.exit(salt_call())'
    PYTHONWARNINGS="ignore" salt-call state.single file.managed name=/opt/so/so-elastic-agent_linux_amd64 source=salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64 mode=755 makedirs=True queue=True
 fi
 if [[ -x /opt/so/so-elastic-agent_linux_amd64 ]]; then
    attempts=0
    cd /opt/so/ || exit 1
    truncate -s 0 "$LOGFILE"
    uninstall_agent
    while [[ $attempts -lt 3 ]]; do
        if ./so-elastic-agent_linux_amd64 -token="$GRIDNODETOKEN" -force && echo "Verifying Elastic Agent health..." && check_agent_health; then
            rm -f /opt/so/so-elastic-agent_linux_amd64
            elastic-agent status
            exit 0
        fi
        attempts=$((attempts + 1))
        if [[ $attempts -lt 3 ]]; then
            echo "Unable to verify Elastic Agent health... Retrying in 20 seconds..."
            sleep 20
        fi
    done
    uninstall_agent
    rm -f /opt/so/so-elastic-agent_linux_amd64
    echo "The so-elastic-agent installer failed after 3 attempts. Exiting."
    exit 1
 else
    echo "Unable to locate so-elastic-agent installer. Exiting."
    exit 1
 fi
@@ -30,94 +30,6 @@ fleet_api() {
    curl -sK /opt/so/conf/elasticsearch/curl.config -L "localhost:5601/api/fleet/${QUERYPATH}" "$@" --retry 3 --retry-delay 10 --fail 2>/dev/null
 }
 # Max number of concurrent Fleet write jobs (create/update). Override via env if needed.
 MAX_FLEET_JOBS=${MAX_FLEET_JOBS:-10}
 # Block until fewer than MAX_FLEET_JOBS background jobs are running.
 elastic_fleet_throttle() {
    while (( $(jobs -rp | wc -l) >= MAX_FLEET_JOBS )); do
        wait -n || true
    done
 }
 # Load every integration JSON in a directory into a single agent policy.
 # The agent policy is fetched ONCE (not per file), and the create/update writes
 # are dispatched as throttled background jobs.
 #   $1 AGENT_POLICY     - the agent policy id/name to load integrations into
 #   $2 DIR              - directory of integration *.json files
 #   $3 LABEL           - human-readable label for log output
 #   $4 SKIP_CREATE_NAME - (optional) integration name to skip when creating (still updated if present)
 # Returns 1 if the policy cannot be fetched or if any integration failed to create/update.
 elastic_fleet_load_integrations_dir() {
    local AGENT_POLICY=$1
    local DIR=$2
    local LABEL=$3
    local SKIP_CREATE_NAME=$4
    local POLICY_JSON FAIL_FILE OUT_DIR INTEGRATION NAME ID i
    FAIL_FILE=$(mktemp)
    # Each job buffers its full output (header + API response) into its own file so the
    # parent can print them grouped and in submission order after concurrent writes finish.
    OUT_DIR=$(mktemp -d)
    i=0
    # Fetch the agent policy a single time; we look up integration ids locally below.
    if ! POLICY_JSON=$(fleet_api "agent_policies/$AGENT_POLICY"); then
        echo "Error: Failed to retrieve agent policy '$AGENT_POLICY'."
        rm -f "$FAIL_FILE"
        rm -rf "$OUT_DIR"
        return 1
    fi
    if ! jq -e '.item.package_policies' <<<"$POLICY_JSON" >/dev/null 2>&1; then
        echo "Error: Invalid agent policy response for '$AGENT_POLICY'."
        rm -f "$FAIL_FILE"
        rm -rf "$OUT_DIR"
        return 1
    fi
    for INTEGRATION in "$DIR"/*.json; do
        [ -e "$INTEGRATION" ] || continue
        NAME=$(jq -r .name "$INTEGRATION")
        ID=$(jq -r --arg n "$NAME" '.item.package_policies[]? | select(.name==$n) | .id' <<<"$POLICY_JSON")
        elastic_fleet_throttle
        {
            local RESP
            if [ -n "$ID" ]; then
                printf "\n\n%s - Updating integration %s\n" "$LABEL" "$NAME"
                if ! RESP=$(elastic_fleet_integration_update "$ID" "@$INTEGRATION"); then
                    flock 9; echo "update ${INTEGRATION##*/}" >&9
                fi
                printf '%s\n' "$RESP"
            elif [ -n "$SKIP_CREATE_NAME" ] && [ "$NAME" == "$SKIP_CREATE_NAME" ]; then
                printf "\n\n%s - Skipping creation of %s\n" "$LABEL" "$NAME"
            else
                printf "\n\n%s - Creating integration %s\n" "$LABEL" "$NAME"
                if ! RESP=$(elastic_fleet_integration_create "@$INTEGRATION"); then
                    flock 9; echo "create ${INTEGRATION##*/}" >&9
                fi
                printf '%s\n' "$RESP"
            fi
        } >"$OUT_DIR/$(printf '%03d' "$i")" 9>>"$FAIL_FILE" &
        i=$((i+1))
    done
    wait || true
    # Emit per-integration output grouped and in submission order (glob sorts numerically).
    cat "$OUT_DIR"/* 2>/dev/null
    rm -rf "$OUT_DIR"
    local rc=0
    if [ -s "$FAIL_FILE" ]; then
        printf "\n%s: failed integrations:\n" "$LABEL"
        cat "$FAIL_FILE"
        rc=1
    fi
    rm -f "$FAIL_FILE"
    return $rc
 }
 elastic_fleet_integration_check() {
    AGENT_POLICY=$1
@@ -134,9 +46,7 @@ elastic_fleet_integration_create() {
    JSON_STRING=$1
-    # --retry-all-errors so transient 409 conflicts (concurrent writes to the same agent
+    if ! fleet_api "package_policies" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST -d "$JSON_STRING"; then
    # policy) are retried; curl --retry alone does not retry 409.
    if ! fleet_api "package_policies" --retry-all-errors -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST -d "$JSON_STRING"; then
        return 1
    fi
 }
@@ -167,9 +77,7 @@ elastic_fleet_integration_update() {
    JSON_STRING=$2
-    # --retry-all-errors so transient 409 conflicts (concurrent writes to the same agent
+    if ! fleet_api "package_policies/$UPDATE_ID" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPUT -d "$JSON_STRING"; then
    # policy) are retried; curl --retry alone does not retry 409.
    if ! fleet_api "package_policies/$UPDATE_ID" --retry-all-errors -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPUT -d "$JSON_STRING"; then
        return 1
    fi
 }
@@ -9,45 +9,108 @@
 RETURN_CODE=0
 if [ ! -f /opt/so/state/eaintegrations.txt ]; then
  # First, check for any package upgrades
  /usr/sbin/so-elastic-fleet-package-upgrade
-  # update Fleet Server policies
+  # Second, update Fleet Server policies
  /usr/sbin/so-elastic-fleet-integration-policy-elastic-fleet-server
-  # configure Elastic Defend Integration separately
+  # Third, configure Elastic Defend Integration seperately
  /usr/sbin/so-elastic-fleet-integration-policy-elastic-defend
  # Each group fetches its agent policy once and dispatches create/update writes concurrently.
  # Initial Endpoints
-  elastic_fleet_load_integrations_dir "endpoints-initial" \
+  for INTEGRATION in /opt/so/conf/elastic-fleet/integrations/endpoints-initial/*.json; do
-    /opt/so/conf/elastic-fleet/integrations/endpoints-initial "Initial Endpoints Policy" || RETURN_CODE=1
+    printf "\n\nInitial Endpoints Policy - Loading $INTEGRATION\n"
    elastic_fleet_integration_check "endpoints-initial" "$INTEGRATION"
    if [ -n "$INTEGRATION_ID" ]; then
      printf "\n\nIntegration $NAME exists - Updating integration\n"
      if ! elastic_fleet_integration_update "$INTEGRATION_ID" "@$INTEGRATION"; then
        echo -e "\nFailed to update integration for ${INTEGRATION##*/}"
        RETURN_CODE=1
        continue
      fi
    else
      printf "\n\nIntegration does not exist - Creating integration\n"
      if ! elastic_fleet_integration_create "@$INTEGRATION"; then
        echo -e "\nFailed to create integration for ${INTEGRATION##*/}"
        RETURN_CODE=1
        continue
      fi
    fi
  done
  # Grid Nodes - General
-  elastic_fleet_load_integrations_dir "so-grid-nodes_general" \
+  for INTEGRATION in /opt/so/conf/elastic-fleet/integrations/grid-nodes_general/*.json; do
-    /opt/so/conf/elastic-fleet/integrations/grid-nodes_general "Grid Nodes Policy_General" || RETURN_CODE=1
+    printf "\n\nGrid Nodes Policy_General - Loading $INTEGRATION\n"
    elastic_fleet_integration_check "so-grid-nodes_general" "$INTEGRATION"
    if [ -n "$INTEGRATION_ID" ]; then
      printf "\n\nIntegration $NAME exists - Updating integration\n"
      if ! elastic_fleet_integration_update "$INTEGRATION_ID" "@$INTEGRATION"; then
        echo -e "\nFailed to update integration for ${INTEGRATION##*/}"
        RETURN_CODE=1
        continue
      fi
    else
      printf "\n\nIntegration does not exist - Creating integration\n"
      if ! elastic_fleet_integration_create "@$INTEGRATION"; then
        echo -e "\nFailed to create integration for ${INTEGRATION##*/}"
        RETURN_CODE=1
        continue
      fi
    fi
  done
  # Grid Nodes - Heavy
-  elastic_fleet_load_integrations_dir "so-grid-nodes_heavy" \
+  for INTEGRATION in /opt/so/conf/elastic-fleet/integrations/grid-nodes_heavy/*.json; do
-    /opt/so/conf/elastic-fleet/integrations/grid-nodes_heavy "Grid Nodes Policy_Heavy" || RETURN_CODE=1
+    printf "\n\nGrid Nodes Policy_Heavy - Loading $INTEGRATION\n"
    elastic_fleet_integration_check "so-grid-nodes_heavy" "$INTEGRATION"
    if [ -n "$INTEGRATION_ID" ]; then
      printf "\n\nIntegration $NAME exists - Updating integration\n"
      if ! elastic_fleet_integration_update "$INTEGRATION_ID" "@$INTEGRATION"; then
        echo -e "\nFailed to update integration for ${INTEGRATION##*/}"
        RETURN_CODE=1
        continue
      fi
    else
      printf "\n\nIntegration does not exist - Creating integration\n"
      if ! elastic_fleet_integration_create "@$INTEGRATION"; then
        echo -e "\nFailed to create integration for ${INTEGRATION##*/}"
        RETURN_CODE=1
        continue
      fi
    fi
  done
-  # Fleet Server - Optional integrations (adds integration configuration to a given FleetServer_ policy)
+  # Fleet Server - Optional integrations
-  for FLEET_DIR in /opt/so/conf/elastic-fleet/integrations-optional/FleetServer*/; do
+  for INTEGRATION in /opt/so/conf/elastic-fleet/integrations-optional/FleetServer*/*.json; do
-    [ -d "$FLEET_DIR" ] || continue
+    if ! [ "$INTEGRATION" == "/opt/so/conf/elastic-fleet/integrations-optional/FleetServer*/*.json" ]; then
-    INTEGRATIONS=("${FLEET_DIR%/}"/*.json)
+      FLEET_POLICY=`echo "$INTEGRATION"| cut -d'/' -f7`
-    [ -e "${INTEGRATIONS[0]}" ] || continue
+      printf "\n\nFleet Server Policy - Loading $INTEGRATION\n"
-
+      elastic_fleet_integration_check "$FLEET_POLICY" "$INTEGRATION"
-    FLEET_POLICY=$(basename "$FLEET_DIR")
+      if [ -n "$INTEGRATION_ID" ]; then
-    elastic_fleet_load_integrations_dir "$FLEET_POLICY" \
+        printf "\n\nIntegration $NAME exists - Updating integration\n"
-      "${FLEET_DIR%/}" "Fleet Server Policy" "elasticsearch-logs" || RETURN_CODE=1
+        if ! elastic_fleet_integration_update "$INTEGRATION_ID" "@$INTEGRATION"; then
          echo -e "\nFailed to update integration for ${INTEGRATION##*/}"
          RETURN_CODE=1
          continue
        fi
      else
        printf "\n\nIntegration does not exist - Creating integration\n"
        if [ "$NAME" != "elasticsearch-logs" ]; then
          if ! elastic_fleet_integration_create "@$INTEGRATION"; then
            echo -e "\nFailed to create integration for ${INTEGRATION##*/}"
            RETURN_CODE=1
            continue
          fi
        fi
      fi
    fi
  done
  # Only create the state file if all policies were created/updated successfully
-  if [[ $RETURN_CODE -eq 0 ]]; then
+  if [[ "$RETURN_CODE" != "1" ]]; then
    touch /opt/so/state/eaintegrations.txt
  else
    exit 1
  fi
 else
-  echo "Fleet integration policies already loaded."
+  exit $RETURN_CODE
  exit 0
 fi
@@ -30,7 +30,7 @@ done
 if [[ -z $FLEETHOST ]] || [[ -z $ENROLLMENTOKEN ]]; then 
  printf "\nFleet Host URL, Enrollment Token or Elastic Version empty - exiting..."
  printf "\nFleet Host: $FLEETHOST, Enrollment Token: $ENROLLMENTOKEN\n"
-  exit 1
+  exit 
 fi
 OSARCH=( "linux-x86_64" "windows-x86_64" "darwin-x86_64" "darwin-aarch64" )
@@ -62,54 +62,31 @@ do
 done 
 GOTARGETOS=( "linux" "windows" "darwin" "darwin/arm64" )
 printf "\n### Generating OS packages using the cleaned up tarballs"
 for GOOS in "${GOTARGETOS[@]}"; do
 GOARCH="amd64"
 printf "\n### Generating OS packages using the cleaned up tarballs"
 for GOOS in "${GOTARGETOS[@]}"
 do
    if [[ $GOOS == 'darwin/arm64' ]]; then GOOS="darwin" && GOARCH="arm64"; fi
    printf "\n\n### Generating $GOOS/$GOARCH Installer...\n"
    docker run -e CGO_ENABLED=0 -e GOOS=$GOOS -e GOARCH=$GOARCH \
    --mount type=bind,source=/etc/pki/tls/certs/,target=/workspace/files/cert/ \
    --mount type=bind,source=/nsm/elastic-agent-workspace/,target=/workspace/files/elastic-agent/ \
-    --mount type=bind,source=/opt/so/saltstack/local/salt/elasticfleet/files/,target=/output/ \
+    --mount type=bind,source=/opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/,target=/output/ \
    {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent-builder:{{ GLOBALS.so_version }} go build -ldflags "-X main.fleetHostURLsList=$FLEETHOST -X main.enrollmentToken=$ENROLLMENTOKEN"  -o /output/so-elastic-agent_${GOOS}_${GOARCH}
    printf "\n### $GOOS/$GOARCH Installer Generated...\n"
 done
 printf "\n\n### Generating MSI...\n"
-cp /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64 /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64.exe
+cp /opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/so-elastic-agent_windows_amd64 /opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/so-elastic-agent_windows_amd64.exe
 docker run \
--mount type=bind,source=/opt/so/saltstack/local/salt/elasticfleet/files/,target=/output/ -w /output \
+--mount type=bind,source=/opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/,target=/output/ -w /output \
 {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent-builder:{{ GLOBALS.so_version }} wixl -o so-elastic-agent_windows_amd64_msi --arch x64 /workspace/so-elastic-agent.wxs
 printf "\n### MSI Generated...\n"
 # Verify installers were created
 for GOOS in "${GOTARGETOS[@]}"; do
    GOARCH="amd64"
    if [[ $GOOS == 'darwin/arm64' ]]; then GOOS="darwin"; GOARCH="arm64"; fi
    if [[ ! -f /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_${GOOS}_${GOARCH} ]]; then
        printf "\n### ERROR: Installer for %s/%s was not generated. Exiting...\n" "$GOOS" "$GOARCH"
        exit 1
    fi
    # After verifying new installer was generated, move it to so_agent-installers directory
    mv /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_${GOOS}_${GOARCH} /opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/
 done
 # Verify MSI installer
 if [[ ! -f /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64_msi ]]; then
    printf "\n### ERROR: Installer MSI was not generated. Exiting...\n"
    exit 1
 else
    # After verifying new installer MSI was generated, move it to so_agent-installers directory
    mv /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64_msi /opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/
 fi
 printf "\n### Cleaning up temp files \n"
 rm -rf /nsm/elastic-agent-workspace
-rm -rf /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64.exe
+rm -rf /opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/so-elastic-agent_windows_amd64.exe
 printf "\n### Copying so_agent-installers to /nsm/elastic-fleet/ for nginx.\n"
 \cp -vr /opt/so/saltstack/local/salt/elasticfleet/files/so_agent-installers/ /nsm/elastic-fleet/
 chmod 644 /nsm/elastic-fleet/so_agent-installers/*
 # if we got here all installers have been generated successfully
 exit 0
@@ -23,90 +23,73 @@ if [ $? -ne 0 ]; then
 fi
 default_packages=({% for pkg in SUPPORTED_PACKAGES %}"{{ pkg }}"{% if not loop.last %} {% endif %}{% endfor %})
 # JSON array of the default packages, used by the jq filter below.
 default_packages_json=$(printf '%s\n' "${default_packages[@]}" | jq -R . | jq -s '.')
 # Output lock (serializes concurrent job output) and failure file (one marker line per
 # failed integration). Mirrors the pattern used by elastic_fleet_load_integrations_dir.
 OUTPUT_LOCK=$(mktemp)
 FAIL_FILE=$(mktemp)
 trap 'rm -f "$OUTPUT_LOCK" "$FAIL_FILE"' EXIT
 # Cache of package name -> latest available version, so the same package is only looked up
 # once instead of once per (policy, integration).
 declare -A LATEST_VERSION_CACHE
 ERROR=false
 for AGENT_POLICY in $agent_policies; do
-    # Fetch the agent policy a single time; package name/version and integration id are all
+    if ! integrations=$(elastic_fleet_integration_policy_names "$AGENT_POLICY"); then
    # extracted locally below instead of re-fetching the same policy per integration.
    if ! POLICY_JSON=$(fleet_api "agent_policies/$AGENT_POLICY"); then
        # this script upgrades default integration packages, exit 1 and let salt handle retrying
        exit 1
    fi
-
+    for INTEGRATION in $integrations; do
-    # One jq pass emits name/package.name/package.version/id for every eligible integration.
+        if ! [[ "$INTEGRATION" == "elastic-defend-endpoints" ]] && ! [[ "$INTEGRATION" == "fleet_server-"* ]]; then
-    # The endpoint/fleet_server skips and the default-package gate are applied here in jq.
+            # Get package name so we know what package to look for when checking the current and latest available version
-    # $defaults (not $def, a jq reserved keyword) holds the default package list.
+            if ! PACKAGE_NAME=$(elastic_fleet_integration_policy_package_name "$AGENT_POLICY" "$INTEGRATION"); then
    while IFS=$'\t' read -r INTEGRATION PACKAGE_NAME PACKAGE_VERSION INTEGRATION_ID; do
        [ -n "$INTEGRATION" ] || continue
        # Look up the latest available version once per package, then memoize it.
        if [[ -z "${LATEST_VERSION_CACHE[$PACKAGE_NAME]+set}" ]]; then
            if ! AVAILABLE_VERSION=$(elastic_fleet_package_latest_version_check "$PACKAGE_NAME"); then
                echo "Error: Failed getting latest version for $PACKAGE_NAME"
                exit 1
            fi
-            LATEST_VERSION_CACHE[$PACKAGE_NAME]=$AVAILABLE_VERSION
+            {%- if not AUTO_UPGRADE_INTEGRATIONS %}
            if [[ " ${default_packages[@]} " =~ " $PACKAGE_NAME " ]]; then
            {%- endif %}
                # Get currently installed version of package
                attempt=0
                max_attempts=3
                while [ $attempt -lt $max_attempts ]; do
                    if PACKAGE_VERSION=$(elastic_fleet_integration_policy_package_version "$AGENT_POLICY" "$INTEGRATION") && AVAILABLE_VERSION=$(elastic_fleet_package_latest_version_check "$PACKAGE_NAME"); then
                        break
                    fi
                    attempt=$((attempt + 1))
                done
                if [ $attempt -eq $max_attempts ]; then
                    echo "Error: Failed getting $PACKAGE_VERSION or $AVAILABLE_VERSION"
                    exit 1
                fi
                # Get integration ID
                if ! INTEGRATION_ID=$(elastic_fleet_integration_id "$AGENT_POLICY" "$INTEGRATION"); then
                    exit 1
                fi
        AVAILABLE_VERSION=${LATEST_VERSION_CACHE[$PACKAGE_NAME]}
                if [[ "$PACKAGE_VERSION" != "$AVAILABLE_VERSION" ]]; then
-            # Dry run, then (if clean) the actual upgrade, dispatched as a throttled background
+                    # Dry run of the upgrade
-            # job. Each job builds its full log into one block, then flushes it under a single
+                    echo ""
-            # shared lock (OUTPUT_LOCK) so concurrent jobs never interleave on stdout; a failed
+                    echo "Current $PACKAGE_NAME package version ($PACKAGE_VERSION) is not the same as the latest available package ($AVAILABLE_VERSION)..."
-            # job also appends a marker line to FAIL_FILE while holding that same lock.
+                    echo "Upgrading $INTEGRATION..."
-            elastic_fleet_throttle
+                    echo "Starting dry run..."
            {
                block=$'\n'"Current $PACKAGE_NAME package version ($PACKAGE_VERSION) is not the same as the latest available package ($AVAILABLE_VERSION)..."$'\n'
                block+="Upgrading $INTEGRATION..."$'\n'"Starting dry run..."$'\n'
                fail=""
                    if ! DRYRUN_OUTPUT=$(elastic_fleet_integration_policy_dryrun_upgrade "$INTEGRATION_ID"); then
-                    block+="Error: Failed to complete dry run for '$INTEGRATION_ID'."$'\n'
+                        exit 1
-                    fail="dryrun $INTEGRATION"
+                    fi
-                elif [[ "$(jq .[].hasErrors <<<"$DRYRUN_OUTPUT")" == "false" ]]; then
+                    DRYRUN_ERRORS=$(echo "$DRYRUN_OUTPUT" | jq .[].hasErrors)
-                    block+="No errors detected. Proceeding with upgrade..."$'\n'
+
                    # If no errors with dry run, proceed with actual upgrade
                    if [[ "$DRYRUN_ERRORS" == "false" ]]; then
                        echo "No errors detected. Proceeding with upgrade..."
                        if ! elastic_fleet_integration_policy_upgrade "$INTEGRATION_ID"; then
-                        block+="Error: Upgrade failed for $PACKAGE_NAME with integration ID '$INTEGRATION_ID'."$'\n'
+                            echo "Error: Upgrade failed for $PACKAGE_NAME with integration ID '$INTEGRATION_ID'."
-                        fail="upgrade $INTEGRATION"
+                            ERROR=true
                            continue
                        fi
                    else
-                    block+="Errors detected during dry run for $PACKAGE_NAME policy upgrade..."$'\n'
+                        echo "Errors detected during dry run for $PACKAGE_NAME policy upgrade..."
-                    fail="dryrun-errors $INTEGRATION"
+                        ERROR=true
                        continue
                    fi
                {
                    flock 9
                    printf '%s' "$block"
                    [ -n "$fail" ] && printf '%s\n' "$fail" >>"$FAIL_FILE"
                } 9>>"$OUTPUT_LOCK"
            } &
                fi
    done < <(jq -r --argjson defaults "$default_packages_json" '
        .item.package_policies[]
        | select(.name != "elastic-defend-endpoints")
        | select(.name | startswith("fleet_server-") | not)
            {%- if not AUTO_UPGRADE_INTEGRATIONS %}
-        | select(.package.name | IN($defaults[]))
+            fi
            {%- endif %}
-        | [.name, .package.name, .package.version, .id] | @tsv
+        fi
    ' <<<"$POLICY_JSON")
    done
-
+done
-# Barrier: wait for every dispatched dry-run/upgrade job to finish.
+if [[ "$ERROR" == "true" ]]; then
 wait
 if [ -s "$FAIL_FILE" ]; then
    printf '\nFailed integration upgrades:\n'
    cat "$FAIL_FILE"
    exit 1
 fi
 echo
@@ -16,6 +16,7 @@
 STATE_FILE_SUCCESS=/opt/so/state/estemplates.txt
 INSTALLED_PACKAGE_LIST=/tmp/esfleet_installed_packages.json
 BULK_INSTALL_PACKAGE_LIST=/tmp/esfleet_bulk_install.json
 BULK_INSTALL_PACKAGE_TMP=/tmp/esfleet_bulk_install_tmp.json
 BULK_INSTALL_OUTPUT=/opt/so/state/esfleet_bulk_install_results.json
 INTEGRATION_PACKAGE_COMPONENTS=/opt/so/state/esfleet_package_components.json
 INPUT_PACKAGE_COMPONENTS=/opt/so/state/esfleet_input_package_components.json
@@ -28,6 +29,29 @@ PENDING_UPDATE=false
 #   Requiring some level of manual Elastic Stack configuration before installation
 EXCLUDED_INTEGRATIONS=('apm')
 version_conversion(){
    version=$1
    echo "$version" | awk -F '.' '{ printf("%d%03d%03d\n", $1, $2, $3); }'
 }
 compare_versions() {
    version1=$1
    version2=$2
    # Convert versions to numbers
    num1=$(version_conversion "$version1")
    num2=$(version_conversion "$version2")
    # Compare using bc
    if (( $(echo "$num1 < $num2" | bc -l) )); then
        echo "less"
    elif (( $(echo "$num1 > $num2" | bc -l) )); then
        echo "greater"
    else
        echo "equal"
    fi
 }
 IFS=$'\n'
 agent_policies=$(elastic_fleet_agent_policy_ids)
 if [ $? -ne 0 ]; then
@@ -39,23 +63,23 @@ default_packages=({% for pkg in SUPPORTED_PACKAGES %}"{{ pkg }}"{% if not loop.l
 in_use_integrations=()
 # Fetch each agent policy once; its package_policies[] already contain both the integration name
 #  and the .package.name, so extract all non-default package names locally in a single jq instead
 #  of re-fetching the same policy per integration.
 default_packages_json=$(printf '%s\n' "${default_packages[@]}" | jq -R . | jq -s '.')
 for AGENT_POLICY in $agent_policies; do
-    if ! policy_json=$(fleet_api "agent_policies/$AGENT_POLICY"); then
+    if ! integrations=$(elastic_fleet_integration_policy_names "$AGENT_POLICY"); then
        # skip the agent policy if we can't get required info, let salt retry. Integrations loaded by this script are non-default integrations.
        echo "Skipping $AGENT_POLICY.. "
        continue
    fi
    for INTEGRATION in $integrations; do
        if ! PACKAGE_NAME=$(elastic_fleet_integration_policy_package_name "$AGENT_POLICY" "$INTEGRATION"); then
            echo  "Not adding $INTEGRATION, couldn't get package name"
            continue
        fi
        # non-default integrations that are in-use in any policy
-    while IFS= read -r PACKAGE_NAME; do
+        if ! [[ " ${default_packages[@]} " =~ " $PACKAGE_NAME " ]]; then
-        [ -n "$PACKAGE_NAME" ] && in_use_integrations+=("$PACKAGE_NAME")
+            in_use_integrations+=("$PACKAGE_NAME")
-    done < <(jq -r --argjson defaults "$default_packages_json" \
+        fi
-        '.item.package_policies[].package.name | select(. as $n | ($defaults | index($n)) | not)' \
+    done
        <<<"$policy_json")
 done
 if [[ -f $STATE_FILE_SUCCESS  ]]; then
@@ -66,55 +90,72 @@ if [[ -f $STATE_FILE_SUCCESS  ]]; then
        rm -f $INSTALLED_PACKAGE_LIST
        echo $latest_package_list | jq '{packages: [.items[] | {name: .name, latest_version: .version, installed_version: .installationInfo.version, subscription: .conditions.elastic.subscription }]}' >> $INSTALLED_PACKAGE_LIST
-        # Build the bulk install list and the per-package status messages with two jq passes
+        while read -r package; do
-        #  instead of a per-package bash loop. The old loop forked ~10 processes per package
+            # get package details
-        #  (5 jq + awk/bc for the version compare) and re-parsed/rewrote a growing JSON file on
+            package_name=$(echo "$package" | jq -r '.name')
-        #  every add (O(n^2)). Selection and messages below are identical to that logic.
+            latest_version=$(echo "$package" | jq -r '.latest_version')
-        SUB={% if SUB %}true{% else %}false{% endif %}
+            installed_version=$(echo "$package" | jq -r '.installed_version')
-        AUTOUP={% if AUTO_UPGRADE_INTEGRATIONS %}true{% else %}false{% endif %}
+            subscription=$(echo "$package" | jq -r '.subscription')
-        EXCLUDED_JSON=$(printf '%s\n' "${EXCLUDED_INTEGRATIONS[@]}" | jq -R 'select(length>0)' | jq -s '.')
+            bulk_package=$(echo "$package" | jq '{name: .name, version: .latest_version}' )
        INUSE_JSON=$(printf '%s\n' "${in_use_integrations[@]}" | jq -R 'select(length>0)' | jq -s 'unique')
-        # vnum replicates the previous version_conversion (%d%03d%03d of the first three dotted
+            if [[ ! "${EXCLUDED_INTEGRATIONS[@]}" =~ "$package_name" ]]; then
-        #  fields); needs() replicates the excluded/subscription/installed/upgrade/in-use logic.
+            {% if not SUB %}
-        JQ_DECISION='
+                if [[ "$subscription" != "basic" && "$subscription" != "null" && -n "$subscription" ]]; then
-def vnum:
+                    # pass over integrations that require non-basic elastic license
-  [ (split(".")|.[0:3][] | gsub("[^0-9].*";"") | (if .=="" then "0" else . end) | tonumber) ]
+                    echo "$package_name integration requires an Elastic license of $subscription or greater... skipping"
-  | (.[0]//0)*1000000 + (.[1]//0)*1000 + (.[2]//0);
+                    continue
-def needs($sub;$autoup;$excluded;$inuse):
+                else
-  .name as $n
+                    if [[ "$installed_version" == "null" || -z "$installed_version" ]]; then
-  | ($n | IN($excluded[]) | not)
+                        echo "$package_name is not installed... Adding to next update."
-  and ( $sub or (.subscription==null or .subscription=="basic" or .subscription=="") )
+                        jq --argjson package "$bulk_package" '.packages += [$package]' $BULK_INSTALL_PACKAGE_LIST > $BULK_INSTALL_PACKAGE_TMP && mv $BULK_INSTALL_PACKAGE_TMP $BULK_INSTALL_PACKAGE_LIST
  and ( (.installed_version==null or .installed_version=="")
        or ( ((.latest_version|vnum) > (.installed_version|vnum))
             and ( $autoup or ($n | IN($inuse[]) | not) ) ) );'
        JQ_ARGS=(--argjson sub "$SUB" --argjson autoup "$AUTOUP" --argjson excluded "$EXCLUDED_JSON" --argjson inuse "$INUSE_JSON")
        # (a) Per-package status messages (parity with the previous echo output).
        jq -r "${JQ_ARGS[@]}" "$JQ_DECISION"'
          .packages[]
          | .name as $n
          | if ($n|IN($excluded[])) then "Skipping \($n)..."
            elif (($sub|not) and (.subscription!=null and .subscription!="basic" and .subscription!="")) then
                 "\($n) integration requires an Elastic license of \(.subscription) or greater... skipping"
            elif (.installed_version==null or .installed_version=="") then
                 "\($n) is not installed... Adding to next update."
            elif ((.latest_version|vnum) > (.installed_version|vnum)) then
                 (if ($autoup or ($n|IN($inuse[])|not))
                  then "\($n) is at version \(.installed_version) latest version is \(.latest_version)... Adding to next update."
                  else "skipping available upgrade for in use integration - \($n)." end)
            else empty end
        ' "$INSTALLED_PACKAGE_LIST"
        # (b) The bulk install list, built in a single pass.
        jq "${JQ_ARGS[@]}" "$JQ_DECISION"'
          {packages: [ .packages[] | select(needs($sub;$autoup;$excluded;$inuse)) | {name, version: .latest_version} ]}
        ' "$INSTALLED_PACKAGE_LIST" > "$BULK_INSTALL_PACKAGE_LIST"
        if jq -e '.packages | length > 0' "$BULK_INSTALL_PACKAGE_LIST" >/dev/null; then
                        PENDING_UPDATE=true
                    else
                        results=$(compare_versions "$latest_version" "$installed_version")
                        if [ $results == "greater" ]; then
                            {#- When auto_upgrade_integrations is false, skip upgrading in_use_integrations  #}
                            {%- if not AUTO_UPGRADE_INTEGRATIONS %}
                            if ! [[ " ${in_use_integrations[@]} " =~ " $package_name " ]]; then
                            {%- endif %}
                                echo "$package_name is at version $installed_version latest version is $latest_version... Adding to next update."
                                jq --argjson package "$bulk_package" '.packages += [$package]' $BULK_INSTALL_PACKAGE_LIST > $BULK_INSTALL_PACKAGE_TMP && mv $BULK_INSTALL_PACKAGE_TMP $BULK_INSTALL_PACKAGE_LIST
                                PENDING_UPDATE=true
                            {%- if not AUTO_UPGRADE_INTEGRATIONS %}
                            else
                                echo "skipping available upgrade for in use integration - $package_name."
                            fi
                            {%- endif %}
                        fi
                    fi
                fi
            {% else %}
                if [[ "$installed_version" == "null" || -z "$installed_version" ]]; then
                    echo "$package_name is not installed... Adding to next update."
                    jq --argjson package "$bulk_package" '.packages += [$package]' $BULK_INSTALL_PACKAGE_LIST > $BULK_INSTALL_PACKAGE_TMP && mv $BULK_INSTALL_PACKAGE_TMP $BULK_INSTALL_PACKAGE_LIST
                    PENDING_UPDATE=true
                else
                    results=$(compare_versions "$latest_version" "$installed_version")
                    if [ $results == "greater" ]; then
                        {#- When auto_upgrade_integrations is false, skip upgrading in_use_integrations  #}
                        {%- if not AUTO_UPGRADE_INTEGRATIONS %}
                        if ! [[ " ${in_use_integrations[@]} " =~ " $package_name " ]]; then
                        {%- endif %}
                            echo "$package_name is at version $installed_version latest version is $latest_version... Adding to next update."
                            jq --argjson package "$bulk_package" '.packages += [$package]' $BULK_INSTALL_PACKAGE_LIST > $BULK_INSTALL_PACKAGE_TMP && mv $BULK_INSTALL_PACKAGE_TMP $BULK_INSTALL_PACKAGE_LIST
                            PENDING_UPDATE=true
                        {%- if not AUTO_UPGRADE_INTEGRATIONS %}
                        else
                            echo "skipping available upgrade for in use integration - $package_name."
                        fi
                        {%- endif %}
                    fi
                fi
            {% endif %}
            else
                echo "Skipping $package_name..."
            fi
        done <<< "$(jq -c '.packages[]' "$INSTALLED_PACKAGE_LIST")"
        if [ "$PENDING_UPDATE" = true ]; then
            # Run chunked install of packages
@@ -8,35 +8,18 @@
 . /usr/sbin/so-elastic-fleet-common
 PKG_LOAD_FAILURES=0
 PKG_LOAD_FAILURES_NAMES=()
 {%- for PACKAGE in SUPPORTED_PACKAGES %}
-if INSTALLED_VERSION=$(elastic_fleet_package_version_check "{{ PACKAGE }}") && LATEST_VERSION=$(elastic_fleet_package_latest_version_check "{{ PACKAGE }}"); then
+echo "Upgrading {{ PACKAGE }} package..."
-
+if VERSION=$(elastic_fleet_package_latest_version_check "{{ PACKAGE }}"); then
-    if [ "$INSTALLED_VERSION" == "$LATEST_VERSION" ]; then
+    if ! elastic_fleet_package_install "{{ PACKAGE }}" "$VERSION"; then
        echo "{{ PACKAGE }} integration version $INSTALLED_VERSION is already at the reported latest version $LATEST_VERSION, skipping upgrade."
    else
        echo "Upgrading {{ PACKAGE }} package to version $LATEST_VERSION..."
        if ! elastic_fleet_package_install "{{ PACKAGE }}" "$LATEST_VERSION"; then
            PKG_LOAD_FAILURES=$((PKG_LOAD_FAILURES + 1))
            PKG_LOAD_FAILURES_NAMES+=("{{ PACKAGE }}")
        fi
    fi
 else
    echo "ERROR: Failed to get version information for integration {{ PACKAGE }}"
    PKG_LOAD_FAILURES=$((PKG_LOAD_FAILURES + 1))
    PKG_LOAD_FAILURES_NAMES+=("{{ PACKAGE }}")
 fi
 {%- endfor %}
 if [ $PKG_LOAD_FAILURES -gt 0 ]; then
    echo "ERROR: Failed to upgrade $PKG_LOAD_FAILURES package(s):"
    for PKG in "${PKG_LOAD_FAILURES_NAMES[@]}"; do
        echo " - $PKG"
    done
        # exit 1 on failure to upgrade a default package, allow salt to handle retries
        echo -e "\nERROR: Failed to upgrade $PACKAGE to version: $VERSION"
        exit 1
 else
    echo "Successfully upgraded all packages."
    fi
 else
    echo -e "\nERROR: Failed to get version information for integration $PACKAGE"
 fi
 echo
 {%- endfor %}
 echo
 /usr/sbin/so-elasticsearch-templates-load
@@ -181,9 +181,6 @@ if ! elastic_fleet_policy_create "so-grid-nodes_heavy" "SO Grid Nodes - Heavy No
    exit 1
 fi
 # Check for package upgrades
 so-elastic-fleet-package-upgrade
 # Load Integrations for default policies
 so-elastic-fleet-integration-policy-load
@@ -244,37 +241,11 @@ printf '%s\n'\
    "" >> "$global_pillar_file"
 # Call Elastic-Fleet Salt State
-printf "\nApplying elasticfleet state\n"
+printf "\nApplying elasticfleet state"
-for state_attempt in {1..3}; do
+salt-call state.apply elasticfleet queue=True
    if salt-call state.apply elasticfleet queue=True; then
        break
    elif [[ $state_attempt -lt 3 ]]; then
        printf "\nElasticfleet state did not complete successfully... Attempt (%s/3). Retrying...\n" "$state_attempt"
        sleep 10
    else
        printf "\nFailure(s) in elasticfleet state... Exiting...\n"
        exit 1
    fi
 done
 printf "\nRunning so-elastic-agent-gen-installers\n"
 # Generate installers & install Elastic Agent on the node
-for agent_gen_attempt in {1..3}; do
+so-elastic-agent-gen-installers
-    if so-elastic-agent-gen-installers; then
+printf "\nApplying elasticfleet.install_agent_grid state"
-        break
+salt-call state.apply elasticfleet.install_agent_grid queue=True
-    elif [[ $agent_gen_attempt -lt 3 ]]; then
+exit 0
        printf "\nUnable to generate Elastic Agent installers... Attempt (%s/3). Retrying...\n" "$agent_gen_attempt"
        sleep 10
    else
        printf "\nFailed to generate Elastic Agent installers after 3 attempts. Exiting...\n"
        exit 1
    fi
 done
 printf "\nApplying elasticfleet.install_agent_grid state\n"
 if ! salt-call state.apply elasticfleet.install_agent_grid queue=True; then
    printf "\nFailure(s) in elasticfleet.install_agent_grid state... Exiting...\n"
    exit 1
 fi
 printf "\nElastic Fleet setup completed successfully\n"
@@ -9,12 +9,9 @@
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {%   from 'elasticsearch/template.map.jinja' import ES_INDEX_SETTINGS, SO_MANAGED_INDICES %}
 {%   if GLOBALS.role != 'so-heavynode' %}
-{%     from 'elasticsearch/template.map.jinja' import ALL_ADDON_SETTINGS, ADDON_INDICES %}
+{%     from 'elasticsearch/template.map.jinja' import ALL_ADDON_SETTINGS %}
 {%   endif %}
 include:
  - elasticsearch.enabled
 escomponenttemplates:
  file.recurse:
    - name: /opt/so/conf/elasticsearch/templates/component
@@ -38,20 +35,6 @@ so_index_template_dir:
      {%- endfor %}
    {%- endif %}
 {%  if GLOBALS.role != "so-heavynode" %}
 # Clean up legacy and non-SO managed templates from the elasticsearch/templates/addon-index/ directory
 addon_index_template_dir:
  file.directory:
    - name: /opt/so/conf/elasticsearch/templates/addon-index
    - clean: True
    {%- if ADDON_INDICES %}
    - require:
      {%- for index in ADDON_INDICES %}
      - file: addon_index_template_{{index}}
      {%- endfor %}
    {%- endif %}
 {%  endif %}
 # Auto-generate index templates for SO managed indices (directly defined in elasticsearch/defaults.yaml)
 #   These index templates are for the core SO datasets and are always required
 {%  for index, settings in ES_INDEX_SETTINGS.items() %}
@@ -133,18 +116,6 @@ so-elasticsearch-templates:
      - docker_container: so-elasticsearch
      - file: elasticsearch_sbin_jinja
 so-elasticsearch-dlm-apply:
  cmd.run:
    - name: /usr/sbin/so-elasticsearch-dlm-apply
    - cwd: /opt/so
    - require:
      - docker_container: so-elasticsearch
      - file: elasticsearch_sbin_jinja
      - cmd: so-elasticsearch-templates
    - retry:
        attempts: 3
        interval: 10
 so-elasticsearch-pipelines:
  cmd.run:
    - name: /usr/sbin/so-elasticsearch-pipelines {{ GLOBALS.hostname }}
@@ -165,8 +136,7 @@ so-elasticsearch-roles-load:
 {%    set ap = "absent" %}
 {%  endif %}
 {%  if grains.role in ['so-eval', 'so-standalone', 'so-heavynode'] %}
-{#    Remove so-elasticsearch-indices-delete script when using DLM #}
+{%    if ELASTICSEARCHMERGED.index_clean %}
 {%    if ELASTICSEARCHMERGED.index_clean and ELASTICSEARCHMERGED.data_retention_method == "ILM" %}
 {%      set ap = "present" %}
 {%    else %}
 {%      set ap = "absent" %}
@@ -1,8 +1,7 @@
 elasticsearch:
  enabled: false
-  version: 9.3.7
+  version: 9.3.3
  index_clean: true
  data_retention_method: DLM
  vm:
    max_map_count: 1048576
  config:
@@ -64,8 +63,6 @@ elasticsearch:
            verification_mode: none
  index_settings:
    global_overrides:
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        template:
          settings:
@@ -146,8 +143,6 @@ elasticsearch:
                order: desc
    so-common:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -309,8 +304,6 @@ elasticsearch:
              number_of_shards: 1
    so-assistant-chat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: ""
      index_template:
        composed_of:
        - assistant-chat-mappings
@@ -351,8 +344,6 @@ elasticsearch:
            min_age: 0ms
    so-assistant-session:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: ""
      index_template:
        composed_of:
        - assistant-session-mappings
@@ -506,8 +497,6 @@ elasticsearch:
            min_age: 30d
    so-idh:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -616,8 +605,6 @@ elasticsearch:
            min_age: 30d
    so-import:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -800,8 +787,6 @@ elasticsearch:
            min_age: 0ms
    so-kismet:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - kismet-mappings
@@ -851,8 +836,6 @@ elasticsearch:
            min_age: 30d
    so-kratos:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -921,8 +904,6 @@ elasticsearch:
            min_age: 30d
    so-hydra:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -1068,8 +1049,6 @@ elasticsearch:
            min_age: 0ms
    so-logs:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - so-data-streams-mappings
@@ -1150,8 +1129,6 @@ elasticsearch:
            min_age: 30d
    so-logs-detections_x_alerts:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - so-data-streams-mappings
@@ -1215,8 +1192,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1332,8 +1307,6 @@ elasticsearch:
            min_age: 30d
    so-elastic-agent-monitor:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1396,8 +1369,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_apm_server:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-elastic_agent.apm_server@package
@@ -1462,8 +1433,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_auditbeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-elastic_agent.auditbeat@package
@@ -1528,8 +1497,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_cloudbeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-elastic_agent.cloudbeat@package
@@ -1594,8 +1561,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_endpoint_security:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1655,8 +1620,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_filebeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1716,8 +1679,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_fleet_server:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1774,8 +1735,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_heartbeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-elastic_agent.heartbeat@package
@@ -1840,8 +1799,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_metricbeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1901,8 +1858,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_osquerybeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -1962,8 +1917,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elastic_agent_x_packetbeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-elastic_agent.packetbeat@package
@@ -2028,8 +1981,6 @@ elasticsearch:
            min_age: 30d
    so-logs-elasticsearch_x_server:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-elasticsearch.server@package
@@ -2094,13 +2045,10 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_actions:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - .logs-endpoint.actions@package
        - .logs-endpoint.actions@custom
        - endpoint@custom
        - event-mappings
        - so-fleet_integrations.ip_mappings-1
        - so-fleet_globals-1
@@ -2110,9 +2058,8 @@ elasticsearch:
          hidden: false
        ignore_missing_component_templates:
        - .logs-endpoint.actions@custom
        - endpoint@custom
        index_patterns:
-        - .logs-endpoint.actions-*
+        - logs-endpoint.actions-*
        priority: 501
        template:
          settings:
@@ -2157,13 +2104,10 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_action_x_responses:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - .logs-endpoint.action.responses@package
        - .logs-endpoint.action.responses@custom
        - endpoint@custom
        - event-mappings
        - so-fleet_integrations.ip_mappings-1
        - so-fleet_globals-1
@@ -2173,15 +2117,14 @@ elasticsearch:
          hidden: false
        ignore_missing_component_templates:
        - .logs-endpoint.action.responses@custom
        - endpoint@custom
        index_patterns:
-        - .logs-endpoint.action.responses-*
+        - logs-endpoint.action.responses-*
        priority: 501
        template:
          settings:
            index:
              lifecycle:
-                name: so-logs-endpoint.action.responses-logs
+                name: so-logs-endpoint.actions-logs
              mapping:
                total_fields:
                  limit: 5000
@@ -2220,8 +2163,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_alerts:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.alerts@package
@@ -2281,8 +2222,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_diagnostic_x_collection:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - .logs-endpoint.diagnostic.collection@package
@@ -2358,8 +2297,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_api:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.api@package
@@ -2419,8 +2356,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_file:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.file@package
@@ -2480,8 +2415,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_library:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.library@package
@@ -2541,8 +2474,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_network:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.network@package
@@ -2602,8 +2533,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_process:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.process@package
@@ -2663,8 +2592,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_registry:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.registry@package
@@ -2724,8 +2651,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_events_x_security:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-endpoint.events.security@package
@@ -2785,8 +2710,6 @@ elasticsearch:
            min_age: 30d
    so-logs-endpoint_x_heartbeat:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - .logs-endpoint.heartbeat@package
@@ -2846,8 +2769,6 @@ elasticsearch:
            min_age: 30d
    so-logs-http_endpoint_x_generic:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-http_endpoint.generic@package
@@ -2896,8 +2817,6 @@ elasticsearch:
            min_age: 30d
    so-logs-httpjson_x_generic:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-httpjson.generic@package
@@ -2963,8 +2882,6 @@ elasticsearch:
              number_of_replicas: 0
    so-logs-osquery-manager_x_action_x_responses:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        _meta:
          managed: true
@@ -3036,8 +2953,6 @@ elasticsearch:
              number_of_replicas: 0
    so-logs-osquery-manager_x_result:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        _meta:
          managed: true
@@ -3090,8 +3005,6 @@ elasticsearch:
            min_age: 30d
    so-logs-soc:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -3200,8 +3113,6 @@ elasticsearch:
            min_age: 30d
    so-logs-system_x_application:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -3251,8 +3162,6 @@ elasticsearch:
            min_age: 30d
    so-logs-system_x_auth:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -3302,8 +3211,6 @@ elasticsearch:
            min_age: 30d
    so-logs-system_x_security:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -3353,8 +3260,6 @@ elasticsearch:
            min_age: 30d
    so-logs-system_x_syslog:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -3404,8 +3309,6 @@ elasticsearch:
            min_age: 30d
    so-logs-system_x_system:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - event-mappings
@@ -3455,8 +3358,6 @@ elasticsearch:
            min_age: 30d
    so-logs-windows_x_forwarded:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-windows.forwarded@package
@@ -3504,8 +3405,6 @@ elasticsearch:
            min_age: 30d
    so-logs-windows_x_powershell:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-windows.powershell@package
@@ -3553,8 +3452,6 @@ elasticsearch:
            min_age: 30d
    so-logs-windows_x_powershell_operational:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-windows.powershell_operational@package
@@ -3602,8 +3499,6 @@ elasticsearch:
            min_age: 30d
    so-logs-windows_x_sysmon_operational:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-windows.sysmon_operational@package
@@ -3651,8 +3546,6 @@ elasticsearch:
            min_age: 30d
    so-logs-winlog_x_winlog:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - logs-winlog.winlog@package
@@ -3701,8 +3594,6 @@ elasticsearch:
            min_age: 30d
    so-logstash:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -3818,8 +3709,6 @@ elasticsearch:
            min_age: 30d
    so-metrics-endpoint_x_metadata:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - metrics-endpoint.metadata@package
@@ -3867,8 +3756,6 @@ elasticsearch:
            min_age: 30d
    so-metrics-endpoint_x_metrics:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - metrics-endpoint.metrics@package
@@ -3916,8 +3803,6 @@ elasticsearch:
            min_age: 30d
    so-metrics-endpoint_x_policy:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - metrics-endpoint.policy@package
@@ -3965,8 +3850,6 @@ elasticsearch:
            min_age: 30d
    so-metrics-fleet_server_x_agent_status:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - metrics@tsdb-settings
@@ -3991,8 +3874,6 @@ elasticsearch:
              number_of_replicas: 0
    so-metrics-fleet_server_x_agent_versions:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - metrics@tsdb-settings
@@ -4017,8 +3898,6 @@ elasticsearch:
              number_of_replicas: 0
    so-redis:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -4134,8 +4013,6 @@ elasticsearch:
            min_age: 30d
    so-strelka:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -4253,8 +4130,6 @@ elasticsearch:
            min_age: 30d
    so-suricata:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -4371,8 +4246,6 @@ elasticsearch:
            min_age: 30d
    so-suricata_x_alerts:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -4489,8 +4362,6 @@ elasticsearch:
            min_age: 30d
    so-syslog:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -4607,8 +4478,6 @@ elasticsearch:
            min_age: 30d
    so-zeek:
      index_sorting: false
      data_stream_lifecycle:
        data_retention: 90d
      index_template:
        composed_of:
        - agent-mappings
@@ -24,7 +24,6 @@ include:
 so-elasticsearch:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elasticsearch:{{ ELASTICSEARCHMERGED.version }}
    - restart_policy: unless-stopped
    - hostname: elasticsearch
    - name: so-elasticsearch
    - user: elasticsearch
@@ -63,8 +63,7 @@
    { "set":             { "if": "ctx.event?.dataset != null && !ctx.event.dataset.contains('.')", "field": "event.dataset", "value": "{{event.module}}.{{event.dataset}}" } },
    { "split":           { "if": "ctx.event?.dataset != null && ctx.event.dataset.contains('.')", "field": "event.dataset", "separator": "\\.", "target_field": "dataset_tag_temp" } },
    { "append":          { "if": "ctx.dataset_tag_temp != null", "field": "tags", "value": "{{dataset_tag_temp.1}}"  } },
-    { "grok":            { "if": "ctx.http?.response?.status_code instanceof String", "field": "http.response.status_code", "patterns": ["%{NUMBER:http.response.status_code:long}(?:\\s+%{GREEDYDATA})?"], "ignore_failure": true } },
+    { "grok":            { "if": "ctx.http?.response?.status_code != null", "field": "http.response.status_code", "patterns": ["%{NUMBER:http.response.status_code:long} %{GREEDYDATA}"]} },
    { "convert":         { "if": "ctx.http?.response?.status_code != null && !(ctx.http.response.status_code instanceof Number)", "field": "http.response.status_code", "type": "long", "ignore_failure": true } },
    { "set":             { "if": "ctx?.metadata?.kafka != null" , "field": "kafka.id", "value": "{{metadata.kafka.partition}}{{metadata.kafka.offset}}{{metadata.kafka.timestamp}}", "ignore_failure": true } },
    { "remove":          { "field": [ "message2", "type", "fields", "category", "module", "dataset", "dataset_tag_temp", "event.dataset_temp" ], "ignore_missing": true, "ignore_failure": true } },
    { "pipeline": { "name": "global@custom", "ignore_missing_pipeline": true, "description": "[Fleet] Global pipeline for all data streams" } }
@@ -177,84 +177,12 @@
                "description": "Extract IPs from Elastic Agent events (host.ip) and adds them to related.ip"
            }
        },
        {
            "script": {
                "description": "Snapshot event.ingested into _tmp.event_ingested_pre_fleet before .fleet_final_pipeline-1 overwrites it with ES ingest time",
                "lang": "painless",
                "if": "ctx.event?.ingested != null && ctx.event?.created == null",
                "ignore_failure": true,
                "source": "ctx.putIfAbsent('_tmp', [:]); ctx._tmp.event_ingested_pre_fleet = ctx.event.ingested;"
            }
        },
        {
            "pipeline": {
                "name": ".fleet_final_pipeline-1",
                "ignore_missing_pipeline": true
            }
        },
        {
            "script": {
                "description": "Calculate time from Elastic Agent to Logstash.",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_from_agent != null",
                "ignore_failure": true,
                "source": "ZonedDateTime start = ctx._tmp.event_ingested_pre_fleet != null ? ZonedDateTime.parse(ctx._tmp.event_ingested_pre_fleet) : ZonedDateTime.parse(ctx['@timestamp']); ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_elasticagent_to_logstash = ChronoUnit.SECONDS.between(start, ZonedDateTime.parse(ctx._tmp.logstash_from_agent));"
            }
        },
        {
            "script": {
                "description": "Calculate time from Logstash to Redis",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_from_agent != null && ctx._tmp?.logstash_to_redis != null",
                "ignore_failure": true,
                "source": "ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_logstash_to_redis = ChronoUnit.SECONDS.between(ZonedDateTime.parse(ctx._tmp.logstash_from_agent), ZonedDateTime.parse(ctx._tmp.logstash_to_redis));"
            }
        },
        {
            "script": {
                "description": "Calculate time message spends in redis queue (logstash delay in pulling event).",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_to_redis != null && ctx._tmp?.logstash_from_redis != null",
                "ignore_failure": true,
                "source": "ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_redis_to_logstash = ChronoUnit.SECONDS.between(ZonedDateTime.parse(ctx._tmp.logstash_to_redis), ZonedDateTime.parse(ctx._tmp.logstash_from_redis));"
            }
        },
        {
            "script": {
                "description": "Calculate time from Logstash to Elasticsearch (after read from Redis).",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_from_redis != null",
                "ignore_failure": true,
                "source": "ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_logstash_to_elasticsearch = ChronoUnit.SECONDS.between(ZonedDateTime.parse(ctx._tmp.logstash_from_redis), metadata().now);"
            }
        },
        {
            "script": {
                "description": "Calculate time from Elastic Agent to Kafka.",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_from_kafka != null && ctx._tmp?.logstash_from_agent == null",
                "ignore_failure": true,
                "source": "ZonedDateTime start = ctx._tmp.event_ingested_pre_fleet != null ? ZonedDateTime.parse(ctx._tmp.event_ingested_pre_fleet) : ZonedDateTime.parse(ctx['@timestamp']); ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_elasticagent_to_kafka = ChronoUnit.SECONDS.between(start, ZonedDateTime.parse(ctx._tmp.logstash_from_kafka));"
            }
        },
        {
            "script": {
                "description": "Calculate time message spends in Kafka queue (logstash delay in pulling event).",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_from_kafka != null && ctx.metadata?.kafka?.timestamp != null && ctx._tmp?.logstash_from_agent == null",
                "ignore_failure": true,
                "source": "ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_kafka_queue = ChronoUnit.SECONDS.between(ZonedDateTime.ofInstant(Instant.ofEpochMilli(Long.parseLong(ctx.metadata.kafka.timestamp.toString())), ZoneId.of('UTC')), ZonedDateTime.parse(ctx._tmp.logstash_from_kafka));"
            }
        },
        {
            "script": {
                "description": "Calculate time from Logstash to Elasticsearch (after read from Kafka).",
                "lang": "painless",
                "if": "ctx._tmp?.logstash_from_kafka != null && ctx._tmp?.logstash_from_agent == null",
                "ignore_failure": true,
                "source": "ctx.event.putIfAbsent('ingestion', [:]); ctx.event.ingestion.latency_kafka_to_elasticsearch = ChronoUnit.SECONDS.between(ZonedDateTime.parse(ctx._tmp.logstash_from_kafka), metadata().now);"
            }
        },
        {
            "remove": {
                "field": "event.agent_id_status",
@@ -274,8 +202,7 @@
                    "event.dataset_temp",
                    "dataset_tag_temp",
                    "module_temp",
-                    "datastream_dataset_temp",
+                    "datastream_dataset_temp"
                    "_tmp"
                ],
                "ignore_missing": true,
                "ignore_failure": true
@@ -118,70 +118,70 @@
    {
      "pipeline": {
        "tag": "pipeline_e16851a7",
-        "name": "logs-pfsense.log-1.25.4-firewall",
+        "name": "logs-pfsense.log-1.25.2-firewall",
        "if": "ctx.event.provider == 'filterlog'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_828590b5",
-        "name": "logs-pfsense.log-1.25.4-openvpn",
+        "name": "logs-pfsense.log-1.25.2-openvpn",
        "if": "ctx.event.provider == 'openvpn'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_9d37039c",
-        "name": "logs-pfsense.log-1.25.4-ipsec",
+        "name": "logs-pfsense.log-1.25.2-ipsec",
        "if": "ctx.event.provider == 'charon'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_ad56bbca",
-        "name": "logs-pfsense.log-1.25.4-dhcp",
+        "name": "logs-pfsense.log-1.25.2-dhcp",
        "if": "[\"dhcpd\", \"dhclient\", \"dhcp6c\", \"dnsmasq-dhcp\"].contains(ctx.event.provider)"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_dd85553d",
-        "name": "logs-pfsense.log-1.25.4-unbound",
+        "name": "logs-pfsense.log-1.25.2-unbound",
        "if": "ctx.event.provider == 'unbound'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_720ed255",
-        "name": "logs-pfsense.log-1.25.4-haproxy",
+        "name": "logs-pfsense.log-1.25.2-haproxy",
        "if": "ctx.event.provider == 'haproxy'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_456beba5",
-        "name": "logs-pfsense.log-1.25.4-php-fpm",
+        "name": "logs-pfsense.log-1.25.2-php-fpm",
        "if": "ctx.event.provider == 'php-fpm'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_a0d89375",
-        "name": "logs-pfsense.log-1.25.4-squid",
+        "name": "logs-pfsense.log-1.25.2-squid",
        "if": "ctx.event.provider == 'squid'"
      }
    },
    {
      "pipeline": {
        "tag": "pipeline_c2f1ed55",
-        "name": "logs-pfsense.log-1.25.4-snort",
+        "name": "logs-pfsense.log-1.25.2-snort",
        "if": "ctx.event.provider == 'snort'"
      }
    },
    {
      "pipeline": {
        "tag":"pipeline_33db1c9e",
-        "name": "logs-pfsense.log-1.25.4-suricata",
+        "name": "logs-pfsense.log-1.25.2-suricata",
        "if": "ctx.event.provider == 'suricata'"
      }
    },
@@ -1,71 +0,0 @@
 {
    "description": "zeek.ja4d",
    "processors": [
        {
            "set": {
                "field": "event.dataset",
                "value": "ja4d"
            }
        },
        {
            "remove": {
                "field": [
                    "host"
                ],
                "ignore_failure": true
            }
        },
        {
            "json": {
                "field": "message",
                "target_field": "message2",
                "ignore_failure": true
            }
        },
        {
            "rename": {
                "field": "message2.ja4d",
                "target_field": "hash.ja4d",
                "ignore_missing": true,
                "if": "ctx?.message2?.ja4d != null && ctx.message2.ja4d.length() > 0"
            }
        },
        {
            "rename": {
                "field": "message2.client_mac",
                "target_field": "host.mac",
                "ignore_missing": true,
                "if": "ctx?.message2?.client_mac != null && ctx.message2.client_mac.length() > 0"
            }
        },
        {
            "rename": {
                "field": "message2.hostname",
                "target_field": "host.hostname",
                "ignore_missing": true,
                "if": "ctx?.message2?.hostname != null && ctx.message2.hostname.length() > 0"
            }
        },
        {
            "rename": {
                "field": "message2.requested_ip",
                "target_field": "dhcp.requested_address",
                "ignore_missing": true,
                "if": "ctx?.message2?.requested_ip != null && ctx.message2.requested_ip.length() > 0"
            }
        },
        {
            "rename": {
                "field": "message2.vendor_class_id",
                "target_field": "zeek.ja4d.vendor_class_id",
                "ignore_missing": true,
                "if": "ctx?.message2?.vendor_class_id != null && ctx.message2.vendor_class_id.length() > 0"
            }
        },
        {
            "pipeline": {
                "name": "zeek.common"
            }
        }
    ]
 }
@@ -5,7 +5,6 @@
    { "remove":   { "field": ["host"],     "ignore_failure": true } },
    { "json":             { "field": "message",                 "target_field": "message2",             "ignore_failure": true  } },
    { "rename":         { "field": "message2.version",          "target_field": "ssl.version",          "ignore_missing": true  } },
    { "set":      { "description": "Set transport for the community_id processor", "if": "ctx.ssl?.version == null || !ctx.ssl.version.startsWith('DTLS')", "field": "network.transport", "value": "tcp", "ignore_failure": true } },
    { "rename":         { "field": "message2.cipher",           "target_field": "ssl.cipher",           "ignore_missing": true  } },
    { "rename":         { "field": "message2.curve",            "target_field": "ssl.curve",            "ignore_missing": true  } },
    { "rename":         { "field": "message2.server_name",      "target_field": "ssl.server_name",              "ignore_missing": true  } },
@@ -4,13 +4,6 @@ elasticsearch:
    forcedType: bool
    advanced: True
    helpLink: elasticsearch
  data_retention_method:
    description: Method for data retention. Options are ILM or DLM. For single node deployments and most distributed grid users, DLM will be the recommended option for simplified management. Those with more complex use cases may prefer ILM. The latter allows for more granular control, but requires more management overhead.
    options:
    - ILM
    - DLM
    forcedType: string
    global: True
  version:
    description: "This specifies the version of the following containers: so-elastic-fleet-package-registry, so-elastic-agent, so-elastic-fleet, so-kibana, so-logstash and so-elasticsearch. Modifying this value in the Elasticsearch defaults.yaml will result in catastrophic grid failure."
    readonly: True
@@ -20,7 +13,7 @@ elasticsearch:
    description: Specify the memory heap size in (m)egabytes for Elasticsearch.
    helpLink: elasticsearch
  index_clean:
-    description: Determines if indices should be considered for deletion by available disk space in the cluster. Otherwise, data is retained by the configured lifecycle settings. This setting only applies to EVAL, STANDALONE, and HEAVY NODE installations. Other installations use lifecycle settings only.
+    description: Determines if indices should be considered for deletion by available disk space in the cluster. Otherwise, indices will only be deleted by the age defined in the ILM settings. This setting only applies to EVAL, STANDALONE, and HEAVY NODE installations. Other installations can only use ILM settings.
    forcedType: bool
    helpLink: elasticsearch
  vm:
@@ -146,23 +139,6 @@ elasticsearch:
    custom010: *pipelines
  index_settings:
    global_overrides:
      data_stream_lifecycle:
        data_retention:
          description: |
            The retention period for all data streams. Retention does not define the period that the data will be removed, but the minimum time period they will be kept.
            Use a number followed by a time unit, such as 7d. Leave blank for indefinite retention where supported.
            Configured retention period also affects the frequency of rolling over data streams.
              - If retention is less than or equal to 1 day, max_age will be 1 hour
              - If retention is less than or equal to 14 days, max_age will be 1 day
              - If retention is less than or equal to 90 days, max_age will be 7 days
              - If retention is greater than 90 days, max_age will be 30 days
          forcedType: string
          allowedNodeTypes:
            - heavynode
          regex: ^$|^[0-9]{1,5}(?:d|h|m|s)$
          regexFailureMessage: Must be blank or a number followed by d, h, m, or s, such as 7d.
      index_template:
        template:
          settings:
@@ -335,30 +311,13 @@ elasticsearch:
              forcedType: string
              global: True
              helpLink: elasticsearch
-    so-logs: &dataStreamSettings
+    so-logs: &indexSettings
      index_sorting:
        description: Sorts the index by event time, at the cost of additional processing resource consumption.
        forcedType: bool
        global: True
        advanced: True
        helpLink: elasticsearch
      data_stream_lifecycle:
        data_retention:
          description: |
            The retention period for this data stream. Retention does not define the period that the data will be removed, but the minimum time period it will be kept.
            Use a number followed by a time unit, such as 7d. Leave blank for indefinite retention where supported.
            Configured retention period also affects the frequency of rolling over this data stream.
              - If retention is less than or equal to 1 day, max_age will be 1 hour
              - If retention is less than or equal to 14 days, max_age will be 1 day
              - If retention is less than or equal to 90 days, max_age will be 7 days
              - If retention is greater than 90 days, max_age will be 30 days
          forcedType: string
          allowedNodeTypes:
            - heavynode
          regex: ^$|^[0-9]{1,5}(?:d|h|m|s)$
          regexFailureMessage: Must be blank or a number followed by d, h, m, or s, such as 7d.
      index_template:
        index_patterns:
          description: Patterns for matching multiple indices or tables.
@@ -376,14 +335,6 @@ elasticsearch:
                global: True
                advanced: True
                helpLink: elasticsearch
              auto_expand_replicas:
                description: Automatically expand the number of replicas based on the number of data nodes in the cluster. This can help ensure high availability as the cluster scales up or down.
                forcedType: string
                regex: "^(0-[1-9]|1-[2-9]|2-[3-9]|3-[4-9]|4-[5-9]|5-[6-9]|6-[7-9]|7-[89]|8-9|[0-9]-all|false)$"
                regexFailureMessage: Must be in the format of "x-y" where x is minimum number of replicas and y is maximum number of replicas, or "0-all" to specify a minimum of 0 and no maximum, or "false" to disable automatic replica expansion.
                global: True
                advanced: True
                helpLink: elasticsearch
              mapping:
                total_fields:
                  limit:
@@ -645,350 +596,65 @@ elasticsearch:
            global: True
            advanced: True
            helpLink: elasticsearch
-    so-logs-soc: *dataStreamSettings
+    so-logs-system_x_auth: *indexSettings
-    so-logs-system_x_auth: *dataStreamSettings
+    so-logs-system_x_syslog: *indexSettings
-    so-logs-system_x_syslog: *dataStreamSettings
+    so-logs-system_x_system: *indexSettings
-    so-logs-system_x_system: *dataStreamSettings
+    so-logs-system_x_application: *indexSettings
-    so-logs-system_x_application: *dataStreamSettings
+    so-logs-system_x_security: *indexSettings
-    so-logs-system_x_security: *dataStreamSettings
+    so-logs-windows_x_forwarded: *indexSettings
-    so-logs-windows_x_forwarded: *dataStreamSettings
+    so-logs-windows_x_powershell: *indexSettings
-    so-logs-windows_x_powershell: *dataStreamSettings
+    so-logs-windows_x_powershell_operational: *indexSettings
-    so-logs-windows_x_powershell_operational: *dataStreamSettings
+    so-logs-windows_x_sysmon_operational: *indexSettings
-    so-logs-windows_x_sysmon_operational: *dataStreamSettings
+    so-logs-winlog_x_winlog: *indexSettings
-    so-logs-winlog_x_winlog: *dataStreamSettings
+    so-logs-detections_x_alerts: *indexSettings
-    so-logs-detections_x_alerts: *dataStreamSettings
+    so-logs-http_endpoint_x_generic: *indexSettings
-    so-logs-http_endpoint_x_generic: *dataStreamSettings
+    so-logs-httpjson_x_generic: *indexSettings
-    so-logs-httpjson_x_generic: *dataStreamSettings
+    so-logs-osquery-manager-actions: *indexSettings
-    so-logs-osquery-manager-actions: *dataStreamSettings
+    so-logs-osquery-manager-action_x_responses: *indexSettings
-    so-logs-osquery-manager-action_x_responses: *dataStreamSettings
+    so-logs-osquery-manager_x_action_x_responses: *indexSettings
-    so-logs-osquery-manager_x_action_x_responses: *dataStreamSettings
+    so-logs-osquery-manager_x_result: *indexSettings
-    so-logs-osquery-manager_x_result: *dataStreamSettings
+    so-logs-elastic_agent_x_apm_server: *indexSettings
-    so-logs-elastic_agent_x_apm_server: *dataStreamSettings
+    so-logs-elastic_agent_x_auditbeat: *indexSettings
-    so-logs-elastic_agent_x_auditbeat: *dataStreamSettings
+    so-logs-elastic_agent_x_cloudbeat: *indexSettings
-    so-logs-elastic_agent_x_cloudbeat: *dataStreamSettings
+    so-logs-elastic_agent_x_endpoint_security: *indexSettings
-    so-logs-elastic_agent_x_endpoint_security: *dataStreamSettings
+    so-logs-endpoint_x_alerts: *indexSettings
-    so-logs-endpoint_x_alerts: *dataStreamSettings
+    so-logs-endpoint_x_events_x_api: *indexSettings
-    so-logs-endpoint_x_events_x_api: *dataStreamSettings
+    so-logs-endpoint_x_events_x_file: *indexSettings
-    so-logs-endpoint_x_events_x_file: *dataStreamSettings
+    so-logs-endpoint_x_events_x_library: *indexSettings
-    so-logs-endpoint_x_events_x_library: *dataStreamSettings
+    so-logs-endpoint_x_events_x_network: *indexSettings
-    so-logs-endpoint_x_events_x_network: *dataStreamSettings
+    so-logs-endpoint_x_events_x_process: *indexSettings
-    so-logs-endpoint_x_events_x_process: *dataStreamSettings
+    so-logs-endpoint_x_events_x_registry: *indexSettings
-    so-logs-endpoint_x_events_x_registry: *dataStreamSettings
+    so-logs-endpoint_x_events_x_security: *indexSettings
-    so-logs-endpoint_x_events_x_security: *dataStreamSettings
+    so-logs-elastic_agent_x_filebeat: *indexSettings
-    so-logs-elastic_agent_x_filebeat: *dataStreamSettings
+    so-logs-elastic_agent_x_fleet_server: *indexSettings
-    so-logs-elastic_agent_x_fleet_server: *dataStreamSettings
+    so-logs-elastic_agent_x_heartbeat: *indexSettings
-    so-logs-elastic_agent_x_heartbeat: *dataStreamSettings
+    so-logs-elastic_agent: *indexSettings
-    so-logs-elastic_agent: *dataStreamSettings
+    so-logs-elastic_agent_x_metricbeat: *indexSettings
-    so-logs-elastic_agent_x_metricbeat: *dataStreamSettings
+    so-logs-elastic_agent_x_osquerybeat: *indexSettings
-    so-logs-elastic_agent_x_osquerybeat: *dataStreamSettings
+    so-logs-elastic_agent_x_packetbeat: *indexSettings
-    so-logs-elastic_agent_x_packetbeat: *dataStreamSettings
+    so-logs-elasticsearch_x_server: *indexSettings
-    so-logs-elasticsearch_x_server: *dataStreamSettings
+    so-metrics-endpoint_x_metadata: *indexSettings
-    so-metrics-endpoint_x_metadata: *dataStreamSettings
+    so-metrics-endpoint_x_metrics: *indexSettings
-    so-metrics-endpoint_x_metrics: *dataStreamSettings
+    so-metrics-endpoint_x_policy: *indexSettings
-    so-metrics-endpoint_x_policy: *dataStreamSettings
+    so-metrics-nginx_x_stubstatus: *indexSettings
-    so-metrics-nginx_x_stubstatus: *dataStreamSettings
+    so-metrics-vsphere_x_datastore: *indexSettings
-    so-metrics-vsphere_x_datastore: *dataStreamSettings
+    so-metrics-vsphere_x_host: *indexSettings
-    so-metrics-vsphere_x_host: *dataStreamSettings
+    so-metrics-vsphere_x_virtualmachine: *indexSettings
-    so-metrics-vsphere_x_virtualmachine: *dataStreamSettings
+    so-case: *indexSettings
-    so-common: *dataStreamSettings
+    so-common: *indexSettings
-    so-endgame: *dataStreamSettings
+    so-endgame: *indexSettings
-    so-idh: *dataStreamSettings
+    so-idh: *indexSettings
-    so-suricata: *dataStreamSettings
+    so-suricata: *indexSettings
-    so-suricata_x_alerts: *dataStreamSettings
+    so-suricata_x_alerts: *indexSettings
-    so-import: *dataStreamSettings
+    so-import: *indexSettings
-    so-kratos: *dataStreamSettings
+    so-kratos: *indexSettings
-    so-hydra: *dataStreamSettings
+    so-hydra: *indexSettings
-    so-kismet: *dataStreamSettings
+    so-kismet: *indexSettings
-    so-logstash: *dataStreamSettings
+    so-logstash: *indexSettings
-    so-redis: *dataStreamSettings
+    so-redis: *indexSettings
-    so-strelka: *dataStreamSettings
+    so-strelka: *indexSettings
-    so-syslog: *dataStreamSettings
+    so-syslog: *indexSettings
-    so-zeek: *dataStreamSettings
+    so-zeek: *indexSettings
    # Managed SOC integration annotations are inserted below this line. Referencing '*dataStreamSettings'
    so-case: &indexSettings
      index_sorting:
        description: Sorts the index by event time, at the cost of additional processing resource consumption.
        forcedType: bool
        global: True
        advanced: True
        helpLink: elasticsearch
      index_template:
        index_patterns:
          description: Patterns for matching multiple indices or tables.
          forcedType: "[]string"
          multiline: True
          global: True
          advanced: True
          helpLink: elasticsearch
        template:
          settings:
            index:
              number_of_replicas:
                description: Number of replicas required for this index. Multiple replicas protects against data loss, but also increases storage costs.
                forcedType: int
                global: True
                advanced: True
                helpLink: elasticsearch
              auto_expand_replicas:
                description: Automatically expand the number of replicas based on the number of data nodes in the cluster. This can help ensure high availability as the cluster scales up or down.
                forcedType: string
                regex: "^(0-[1-9]|1-[2-9]|2-[3-9]|3-[4-9]|4-[5-9]|5-[6-9]|6-[7-9]|7-[89]|8-9|[0-9]-all|false)$"
                regexFailureMessage: Must be in the format of "x-y" where x is minimum number of replicas and y is maximum number of replicas, or "0-all" to specify a minimum of 0 and no maximum, or "false" to disable automatic replica expansion.
                global: True
                advanced: True
                helpLink: elasticsearch
              mapping:
                total_fields:
                  limit:
                    description: Max number of fields that can exist on a single index. Larger values will consume more resources.
                    global: True
                    advanced: True
                    helpLink: elasticsearch
              refresh_interval:
                description: Seconds between index refreshes. Shorter intervals can cause query performance to suffer since this is a synchronous and resource-intensive operation.
                global: True
                advanced: True
                helpLink: elasticsearch
              number_of_shards:
                description: Number of shards required for this index. Using multiple shards increases fault tolerance, but also increases storage and network costs.
                global: True
                advanced: True
                helpLink: elasticsearch
              sort:
                field:
                  description: The field to sort by. Must set index_sorting to True.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
                order:
                  description: The order to sort by. Must set index_sorting to True.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
          mappings:
            _meta:
              package:
                name:
                  description: Meta settings for the mapping.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              managed_by:
                  description: Meta settings for the mapping.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              managed:
                  description: Meta settings for the mapping.
                  forcedType: bool
                  global: True
                  advanced: True
                  helpLink: elasticsearch
        composed_of:
          description: The index template is composed of these component templates.
          forcedType: "[]string"
          global: True
          advanced: True
          helpLink: elasticsearch
        priority:
          description: The priority of the index template.
          forcedType: int
          global: True
          advanced: True
          helpLink: elasticsearch
      policy:
        phases:
          hot:
            min_age:
              description: Minimum age of index. This determines when the index should be moved to the hot tier.
              global: True
              advanced: True
              helpLink: elasticsearch
            actions:
              set_priority:
                priority:
                  description: Priority of index. This is used for recovery after a node restart. Indices with higher priorities are recovered before indices with lower priorities.
                  forcedType: int
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              rollover:
                max_age:
                  description: Maximum age of index.  Once an index reaches this limit, it will be rolled over into a new index.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
                max_primary_shard_size:
                  description: Maximum primary shard size. Once an index reaches this limit, it will be rolled over into a new index.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              shrink:
                method:
                  description: Shrink the index to a new index with fewer primary shards. Shrink operation is by count or size.
                  options:
                  - COUNT
                  - SIZE
                  global: True
                  advanced: True
                  forcedType: string
                number_of_shards:
                  title: shard count
                  description: Desired shard count. Note that this value is only used when the shrink method selected is 'COUNT'.
                  global: True
                  forcedType: int
                  advanced: True
                max_primary_shard_size:
                  title: max shard size
                  description: Desired shard size in gb/tb/pb eg. 100gb. Note that this value is only used when the shrink method selected is 'SIZE'.
                  regex: ^[0-9]+(?:gb|tb|pb)$
                  global: True
                  forcedType: string
                  advanced: True
                allow_write_after_shrink:
                  description: Allow writes after shrink.
                  global: True
                  forcedType: bool
                  default: False
                  advanced: True
              forcemerge:
                max_num_segments:
                  description: Reduce the number of segments in each index shard and clean up deleted documents.
                  global: True
                  forcedType: int
                  advanced: True
                index_codec:
                  title: compression
                  description: Use higher compression for stored fields at the cost of slower performance.
                  forcedType: bool
                  global: True
                  default: False
                  advanced: True
          warm:
            min_age:
              description: Minimum age of index. ex. 30d - This determines when the index should be moved to the warm tier. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. It’s important to note that this is calculated relative to the rollover date (NOT the original creation date of the index). For example, if you have an index that is set to rollover after 30 days and warm min_age set to 30 then there will be 30 days from index creation to rollover and then an additional 30 days before moving to warm tier.
              regex: ^[0-9]{1,5}d$
              forcedType: string
              global: True
              advanced: True
              helpLink: elasticsearch
            actions:
              set_priority:
                priority:
                  description: Priority of index. This is used for recovery after a node restart. Indices with higher priorities are recovered before indices with lower priorities.
                  forcedType: int
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              rollover:
                max_age:
                  description: Maximum age of index.  Once an index reaches this limit, it will be rolled over into a new index.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
                max_primary_shard_size:
                  description: Maximum primary shard size. Once an index reaches this limit, it will be rolled over into a new index.
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              shrink:
                method:
                  description: Shrink the index to a new index with fewer primary shards. Shrink operation is by count or size.
                  options:
                  - COUNT
                  - SIZE
                  global: True
                  advanced: True
                number_of_shards:
                  title: shard count
                  description: Desired shard count. Note that this value is only used when the shrink method selected is 'COUNT'.
                  global: True
                  forcedType: int
                  advanced: True
                max_primary_shard_size:
                  title: max shard size
                  description: Desired shard size in gb/tb/pb eg. 100gb. Note that this value is only used when the shrink method selected is 'SIZE'.
                  regex: ^[0-9]+(?:gb|tb|pb)$
                  global: True
                  forcedType: string
                  advanced: True
                allow_write_after_shrink:
                  description: Allow writes after shrink.
                  global: True
                  forcedType: bool
                  default: False
                  advanced: True
              forcemerge:
                max_num_segments:
                  description: Reduce the number of segments in each index shard and clean up deleted documents.
                  global: True
                  forcedType: int
                  advanced: True
                index_codec:
                  title: compression
                  description: Use higher compression for stored fields at the cost of slower performance.
                  forcedType: bool
                  global: True
                  default: False
                  advanced: True
              allocate:
                number_of_replicas:
                  description: Set the number of replicas. Remains the same as the previous phase by default.
                  forcedType: int
                  global: True
                  advanced: True
          cold:
            min_age:
              description: Minimum age of index. ex. 60d - This determines when the index should be moved to the cold tier.  While still searchable, this tier is typically optimized for lower storage costs rather than search speed. It’s important to note that this is calculated relative to the rollover date (NOT the original creation date of the index). For example, if you have an index that is set to rollover after 30 days and cold min_age set to 60 then there will be 30 days from index creation to rollover and then an additional 60 days before moving to cold tier.
              regex: ^[0-9]{1,5}d$
              forcedType: string
              global: True
              advanced: True
              helpLink: elasticsearch
            actions:
              set_priority:
                priority:
                  description: Used for index recovery after a node restart. Indices with higher priorities are recovered before indices with lower priorities.
                  forcedType: int
                  global: True
                  advanced: True
                  helpLink: elasticsearch
              allocate:
                number_of_replicas:
                  description: Set the number of replicas. Remains the same as the previous phase by default.
                  forcedType: int
                  global: True
                  advanced: True
          delete:
            min_age:
              description: Minimum age of index. ex. 90d - This determines when the index should be deleted. It’s important to note that this is calculated relative to the rollover date (NOT the original creation date of the index). For example, if you have an index that is set to rollover after 30 days and delete min_age set to 90 then there will be 30 days from index creation to rollover and then an additional 90 days before deletion.
              regex: ^[0-9]{1,5}d$
              forcedType: string
              global: True
              advanced: True
              helpLink: elasticsearch
        _meta:
          package:
            name:
              description: Meta settings for the mapping.
              global: True
              advanced: True
              helpLink: elasticsearch
          managed_by:
            description: Meta settings for the mapping.
            global: True
            advanced: True
            helpLink: elasticsearch
          managed:
            description: Meta settings for the mapping.
            forcedType: bool
            global: True
            advanced: True
            helpLink: elasticsearch
    sos-backup: *indexSettings
    so-detection: *indexSettings
    so-assistant-chat: *indexSettings
    so-assistant-session: *indexSettings
    so-metrics-fleet_server_x_agent_status: &fleetMetricsSettings
      index_sorting:
        description: Sorts the index by event time, at the cost of additional processing resource consumption.
@@ -4,11 +4,7 @@
   Elastic License 2.0. #}
 {% import_yaml 'elasticsearch/defaults.yaml' as ELASTICSEARCHDEFAULTS %}
 {# ELASTICSEARCHMERGED only used here to collect data_retention_method. This file intentionally works with ELASTICSEARCHDEFAULTS #}
 {% from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {% set DEFAULT_GLOBAL_OVERRIDES = ELASTICSEARCHDEFAULTS.elasticsearch.index_settings.pop('global_overrides') %}
 {% set DATA_RETENTION_METHOD = ELASTICSEARCHMERGED.data_retention_method %}
 {% set PILLAR_GLOBAL_OVERRIDES = {} %}
 {% set ES_INDEX_PILLAR = salt['pillar.get']('elasticsearch:index_settings', {}) %}
@@ -65,25 +61,15 @@
 {% if ALL_ADDON_SETTINGS_ORIG.keys() | length > 0 %}
 {%   for index in ALL_ADDON_SETTINGS_ORIG.keys() %}
 {%     do ALL_ADDON_SETTINGS_GLOBAL_OVERRIDES.update({index: salt['defaults.merge'](ALL_ADDON_SETTINGS_ORIG[index], PILLAR_GLOBAL_OVERRIDES, in_place=False)}) %}
 {#     Explicitly excluding addon indices from ES_INDEX_SETTINGS_ORIG
         When manager.soc_managed_annotations runs, new entries are added to the salt/elasticsearch/defaults.yaml file to support 'revert to default' functionality.
         Subsequent map renders will then incorrectly include 'integration X' in 'ES_INDEX_SETTINGS_ORIG' due to being in the defaults.yaml file. #}
 {%     if index in ES_INDEX_SETTINGS_ORIG.keys() %}
 {%       do ES_INDEX_SETTINGS_ORIG.pop(index) %}
 {%     endif %}
 {%   endfor %}
 {% endif %}
 {% set ES_INDEX_SETTINGS = {} %}
-{% macro create_final_index_template(DEFINED_SETTINGS, GLOBAL_OVERRIDES, FINAL_INDEX_SETTINGS, EXCLUDE_INDICES=[]) %}
+{% macro create_final_index_template(DEFINED_SETTINGS, GLOBAL_OVERRIDES, FINAL_INDEX_SETTINGS) %}
 {% do GLOBAL_OVERRIDES.update(salt['defaults.merge'](GLOBAL_OVERRIDES, ES_INDEX_PILLAR, in_place=False)) %}
 {% for index, settings in GLOBAL_OVERRIDES.items() %}
 {%   if index in EXCLUDE_INDICES %}
 {%     continue %}
 {%   endif %}
 {#   prevent this action from being performed on custom defined indices. #}
 {#   the custom defined index is not present in either of the dictionaries and fails to reder. #}
 {%   if index in DEFINED_SETTINGS and index in GLOBAL_OVERRIDES %}
@@ -109,17 +95,6 @@
 {%     if not settings.get('index_sorting', False) | to_bool and settings.index_template.template.settings.index.sort is defined %}
 {%       do settings.index_template.template.settings.index.pop('sort') %}
 {%     endif %}
 {%     if DATA_RETENTION_METHOD == 'DLM' and settings.index_template.data_stream is defined and settings.data_stream_lifecycle is defined %}
 {%       if settings.data_stream_lifecycle.data_retention is defined and settings.data_stream_lifecycle.data_retention %}
 {%         do settings.index_template.template.update({'lifecycle': {'data_retention': settings.data_stream_lifecycle.data_retention}}) %}
 {%       else %}
 {%         do settings.index_template.template.update({'lifecycle': {}}) %}
 {%       endif %}
 {%       if settings.index_template.template.settings.index.lifecycle is not defined %}
 {%         do settings.index_template.template.settings.index.update({'lifecycle': {}}) %}
 {%       endif %}
 {%       do settings.index_template.template.settings.index.lifecycle.update({'prefer_ilm': false}) %}
 {%     endif %}
 {%   endif %}
 {# advanced ilm actions #}
@@ -175,19 +150,10 @@
 {% endfor %}
 {% endmacro %}
-{# Exclude addon integrations from final ES_INDEX_SETTINGS #}
+{{ create_final_index_template(ES_INDEX_SETTINGS_ORIG, ES_INDEX_SETTINGS_GLOBAL_OVERRIDES, ES_INDEX_SETTINGS) }}
-{{ create_final_index_template(ES_INDEX_SETTINGS_ORIG, ES_INDEX_SETTINGS_GLOBAL_OVERRIDES, ES_INDEX_SETTINGS, ALL_ADDON_SETTINGS_ORIG.keys() | list ) }}
+{{ create_final_index_template(ALL_ADDON_SETTINGS_ORIG, ALL_ADDON_SETTINGS_GLOBAL_OVERRIDES, ALL_ADDON_SETTINGS) }}
 {# Exclude SO managed indices, otherwise ALL_ADDON_SETTINGS will include pillar values
  of core integrations without merging defaults, resulting in an overlapping, but bad index template being generated. #}
 {{ create_final_index_template(ALL_ADDON_SETTINGS_ORIG, ALL_ADDON_SETTINGS_GLOBAL_OVERRIDES, ALL_ADDON_SETTINGS, ES_INDEX_SETTINGS_ORIG.keys() | list ) }}
 {% set SO_MANAGED_INDICES = [] %}
 {% for index, settings in ES_INDEX_SETTINGS.items() %}
 {%   do SO_MANAGED_INDICES.append(index) %}
 {% endfor %}
 {% set ADDON_INDICES = [] %}
 {% for index, settings in ALL_ADDON_SETTINGS.items() %}
 {%   do ADDON_INDICES.append(index) %}
 {% endfor %}
@@ -11,8 +11,10 @@ ADDON_STATEFILE_SUCCESS=/opt/so/state/addon_estemplates.txt
 ELASTICSEARCH_TEMPLATES_DIR="/opt/so/conf/elasticsearch/templates"
 SO_TEMPLATES_DIR="${ELASTICSEARCH_TEMPLATES_DIR}/index"
 ADDON_TEMPLATES_DIR="${ELASTICSEARCH_TEMPLATES_DIR}/addon-index"
-FAILED_NAMES=()
+SO_LOAD_FAILURES=0
-FAILED_COUNT=0
+ADDON_LOAD_FAILURES=0
 SO_LOAD_FAILURES_NAMES=()
 ADDON_LOAD_FAILURES_NAMES=()
 IS_HEAVYNODE="false"
 FORCE="false"
 VERBOSE="false"
@@ -44,86 +46,20 @@ while [[ $# -gt 0 ]]; do
    shift
 done
 # Max number of concurrent template PUT jobs. Override via env if needed.
 MAX_TEMPLATE_JOBS=${MAX_TEMPLATE_JOBS:-10}
 # Block until fewer than MAX_TEMPLATE_JOBS background jobs are running.
 template_throttle() {
    while (( $(jobs -rp | wc -l) >= MAX_TEMPLATE_JOBS )); do
        wait -n
    done
 }
 # Per-job failure markers and an output lock for serializing parallel job output.
 # Each failed load drops one file (named after the template) into FAIL_DIR; the
 # output of each job is flushed as a single block under flock so concurrent jobs
 # never interleave their (chatty) retry output.
 FAIL_DIR=$(mktemp -d)
 OUTPUT_LOCK="${FAIL_DIR}/.output.lock"
 : > "$OUTPUT_LOCK"
 trap 'rm -rf "$FAIL_DIR"' EXIT
 # Record a failure: $1 = the template name/path to report later. Slashes are
 # encoded so the path becomes a safe single filename.
 record_failure() {
    local marker="${1//\//__}"
    : > "${FAIL_DIR}/fail.${marker}"
 }
 # Populate FAILED_NAMES and FAILED_COUNT from the current phase's markers.
 # Must run in the current shell (not a command substitution) so the array sticks.
 collect_failures() {
    FAILED_NAMES=()
    FAILED_COUNT=0
    local f name
    shopt -s nullglob
    for f in "${FAIL_DIR}"/fail.*; do
        name="${f##*/fail.}"
        name="${name//__//}"
        FAILED_NAMES+=("$name")
        FAILED_COUNT=$((FAILED_COUNT + 1))
    done
    shopt -u nullglob
 }
 # Clear markers and names between phases so SO and addon counts stay independent.
 reset_failures() {
    shopt -s nullglob
    rm -f "${FAIL_DIR}"/fail.*
    shopt -u nullglob
    FAILED_NAMES=()
    FAILED_COUNT=0
 }
 # Print a block of text atomically (under the shared output lock) so the output
 # of concurrent background jobs is not interleaved.
 locked_echo() {
    { flock 9; printf '%s\n' "$1"; } 9>>"$OUTPUT_LOCK"
 }
 # Loads one template file via PUT. Intended to be dispatched as a background job.
 #   $1 uri          - e.g. _component_template/foo or _index_template/foo
 #   $2 file         - path to the template JSON
 #   $3 report_name  - name/path to record if this load fails
 load_template() {
    local uri="$1"
    local file="$2"
    local report_name="$3"
    local out rc=0 block
-    # Capture everything (including retry's diagnostic chatter) into one block so
+    echo "Loading template file $file"
-    # concurrent jobs never interleave; the whole block is flushed under one flock.
+    if ! output=$(retry 3 3 "so-elasticsearch-query $uri -d@$file -XPUT" "{\"acknowledged\":true}"); then
-    block="Loading template file $file"$'\n'
+        echo "$output"
-    if ! out=$(retry 3 3 "so-elasticsearch-query $uri -d@$file -XPUT" "{\"acknowledged\":true}" 2>&1); then
+
-        block+="$out"$'\n'
+        return 1
-        rc=1
+
    elif [[ "$VERBOSE" == "true" ]]; then
-        block+="$out"$'\n'
+        echo "$output"
    fi
    { flock 9; printf '%s' "$block"; } 9>>"$OUTPUT_LOCK"
    (( rc != 0 )) && record_failure "$report_name"
 }
 check_required_component_template_exists() {
@@ -174,9 +110,6 @@ load_component_templates() {
        return
    fi
    # Dispatch loads as throttled background jobs. The barrier (wait) happens in
    # the caller after all component groups have been dispatched, since index
    # templates must not load until every component template is in place.
    for component in "$pattern"/*.json; do
        tmpl_name=$(basename "${component%.json}")
@@ -185,11 +118,21 @@ load_component_templates() {
            tmpl_name="${tmpl_name%-mappings}-mappings"
        fi
-        template_throttle
+        if ! load_template "_component_template/${tmpl_name}" "$component"; then
-        load_template "_component_template/${tmpl_name}" "$component" "$component" &
+            SO_LOAD_FAILURES=$((SO_LOAD_FAILURES + 1))
            SO_LOAD_FAILURES_NAMES+=("$component")
        fi
    done
 }
 check_elasticsearch_responsive() {
    # Cannot load templates if Elasticsearch is not responding.
    #  NOTE: Slightly faster exit w/ failure than previous "retry 240 1" if there is a problem with Elasticsearch the
    #    script should exit sooner rather than hang at the 'so-elasticsearch-templates' salt state.
    retry 3 15 "so-elasticsearch-query / --output /dev/null --fail" ||
        fail "Elasticsearch is not responding. Please review Elasticsearch logs /opt/so/log/elasticsearch/securityonion.log for more details. Additionally, consider running so-elasticsearch-troubleshoot."
 }
 index_templates_exist() {
    local templates_dir="$1"
@@ -237,9 +180,6 @@ if [[ "$FORCE" == "true" || ! -f "$SO_STATEFILE_SUCCESS" ]] && index_templates_e
    load_component_templates "Elastic Agent" "elastic-agent"
    load_component_templates "Security Onion" "so"
    # Barrier: every component template PUT must complete before we snapshot the
    # component template list and start loading index templates that depend on them.
    wait
    component_templates=$(so-elasticsearch-component-templates-list)
    echo -e "Loading Security Onion index templates...\n"
    for so_idx_tmpl in "${SO_TEMPLATES_DIR}"/*.json; do
@@ -249,7 +189,7 @@ if [[ "$FORCE" == "true" || ! -f "$SO_STATEFILE_SUCCESS" ]] && index_templates_e
            # TODO: Better way to load only heavynode specific templates
            if ! check_heavynode_compatiable_index_template "$tmpl_name"; then
                if [[ "$VERBOSE" == "true" ]]; then
-                    locked_echo "Skipping over $so_idx_tmpl, template is not a heavynode specific index template."
+                    echo "Skipping over $so_idx_tmpl, template is not a heavynode specific index template."
                fi
                continue
@@ -257,34 +197,32 @@ if [[ "$FORCE" == "true" || ! -f "$SO_STATEFILE_SUCCESS" ]] && index_templates_e
        fi
        if check_required_component_template_exists "$so_idx_tmpl"; then
-            template_throttle
+            if ! load_template "_index_template/$tmpl_name" "$so_idx_tmpl"; then
-            load_template "_index_template/$tmpl_name" "$so_idx_tmpl" "$so_idx_tmpl" &
+                SO_LOAD_FAILURES=$((SO_LOAD_FAILURES + 1))
                SO_LOAD_FAILURES_NAMES+=("$so_idx_tmpl")
            fi
        else
-            locked_echo "Skipping over $so_idx_tmpl due to missing required component template(s)."
+            echo "Skipping over $so_idx_tmpl due to missing required component template(s)."
-            record_failure "$so_idx_tmpl"
+            SO_LOAD_FAILURES=$((SO_LOAD_FAILURES + 1))
            SO_LOAD_FAILURES_NAMES+=("$so_idx_tmpl")
            continue
        fi
    done
-    # Barrier: all SO index template PUTs must finish before tallying failures.
+    if [[ $SO_LOAD_FAILURES -eq 0 ]]; then
    wait
    collect_failures
    if [[ $FAILED_COUNT -eq 0 ]]; then
        echo "All Security Onion core templates loaded successfully."
        touch "$SO_STATEFILE_SUCCESS"
    else
-        echo "Encountered $FAILED_COUNT failure(s) loading templates:"
+        echo "Encountered $SO_LOAD_FAILURES failure(s) loading templates:"
-        for failed_template in "${FAILED_NAMES[@]}"; do
+        for failed_template in "${SO_LOAD_FAILURES_NAMES[@]}"; do
            echo "  - $failed_template"
        done
        if [[ "$SHOULD_EXIT_ON_FAILURE" == "true" ]]; then
            fail "Failed to load all Security Onion core templates successfully."
        fi
    fi
    reset_failures
 elif ! index_templates_exist "$SO_TEMPLATES_DIR"; then
    echo "No Security Onion core index templates found in ${SO_TEMPLATES_DIR}, skipping."
 elif [[ -f "$SO_STATEFILE_SUCCESS" ]]; then
@@ -303,27 +241,26 @@ if should_load_addon_templates; then
        tmpl_name=$(basename "${addon_idx_tmpl%-template.json}")
        if check_required_component_template_exists "$addon_idx_tmpl"; then
-            template_throttle
+            if ! load_template "_index_template/${tmpl_name}" "$addon_idx_tmpl"; then
-            load_template "_index_template/${tmpl_name}" "$addon_idx_tmpl" "$addon_idx_tmpl" &
+                ADDON_LOAD_FAILURES=$((ADDON_LOAD_FAILURES + 1))
                ADDON_LOAD_FAILURES_NAMES+=("$addon_idx_tmpl")
            fi
        else
-            locked_echo "Skipping over $addon_idx_tmpl due to missing required component template(s)."
+            echo "Skipping over $addon_idx_tmpl due to missing required component template(s)."
-            record_failure "$addon_idx_tmpl"
+            ADDON_LOAD_FAILURES=$((ADDON_LOAD_FAILURES + 1))
            ADDON_LOAD_FAILURES_NAMES+=("$addon_idx_tmpl")
            continue
        fi
    done
-    # Barrier: all addon index template PUTs must finish before tallying failures.
+    if [[ $ADDON_LOAD_FAILURES -eq 0 ]]; then
    wait
    collect_failures
    if [[ $FAILED_COUNT -eq 0 ]]; then
        echo "All addon integration templates loaded successfully."
        touch "$ADDON_STATEFILE_SUCCESS"
    else
-        echo "Encountered $FAILED_COUNT failure(s) loading addon integration templates:"
+        echo "Encountered $ADDON_LOAD_FAILURES failure(s) loading addon integration templates:"
-        for failed_template in "${FAILED_NAMES[@]}"; do
+        for failed_template in "${ADDON_LOAD_FAILURES_NAMES[@]}"; do
            echo "  - $failed_template"
        done
        if [[ "$SHOULD_EXIT_ON_FAILURE" == "true" ]]; then
@@ -1,175 +0,0 @@
 #!/bin/bash
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 . /usr/sbin/so-common
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {%- set DATA_RETENTION_METHOD = ELASTICSEARCHMERGED.data_retention_method %}
 ELASTICSEARCH_TEMPLATES_DIR="${ELASTICSEARCH_TEMPLATES_DIR:-/opt/so/conf/elasticsearch/templates}"
 TEMPLATE_DIRS=(
    "${ELASTICSEARCH_TEMPLATES_DIR}/index"
    "${ELASTICSEARCH_TEMPLATES_DIR}/addon-index"
 )
 DATA_RETENTION_METHOD=$(cat <<'EOF'
 {{ DATA_RETENTION_METHOD }}
 EOF
 )
 DLM_FAILURES=0
 DLM_FAILURE_NAMES=()
 if [[ "$DATA_RETENTION_METHOD" != "DLM" && "$DATA_RETENTION_METHOD" != "ILM" ]]; then
    echo "Unsupported data retention method $DATA_RETENTION_METHOD. Expected DLM or ILM."
    exit 1
 fi
 validate_template_file() {
    local template_file="$1"
    if ! jq -e 'type == "object" and (.data_stream == null or (.data_stream | type == "object")) and (.template.lifecycle == null or (.template.lifecycle | type == "object")) and (.template.lifecycle.data_retention == null or (.template.lifecycle.data_retention | type == "string"))' >/dev/null 2>&1 "$template_file"; then
        echo "Invalid index template JSON: $template_file"
        return 1
    fi
 }
 is_data_stream_template() {
    jq -e '.data_stream | type == "object"' >/dev/null 2>&1 "$1"
 }
 has_data_stream_lifecycle() {
    jq -e '.template.lifecycle | type == "object"' >/dev/null 2>&1 "$1"
 }
 get_data_retention() {
    jq -r '.template.lifecycle.data_retention // ""' "$1"
 }
 find_template_file() {
    local template="$1"
    local template_dir
    local template_file
    for template_dir in "${TEMPLATE_DIRS[@]}"; do
        template_file="${template_dir}/${template}-template.json"
        if [[ -f "$template_file" ]]; then
            echo "$template_file"
            return 0
        fi
    done
    return 1
 }
 set_data_stream_lifecycle() {
    local data_stream="$1"
    local data_retention="$2"
    local body
    local output
    if [[ -n "$data_retention" ]]; then
        if jq -e --arg data_stream "$data_stream" --arg data_retention "$data_retention" '.data_streams[]? | select(.name == $data_stream and .lifecycle.enabled == true and .lifecycle.data_retention == $data_retention)' >/dev/null 2>&1 <<< "$data_streams"; then
            echo "DLM lifecycle already set for $data_stream with data_retention $data_retention, skipping."
            return 0
        fi
    elif jq -e --arg data_stream "$data_stream" '.data_streams[]? | select(.name == $data_stream and .lifecycle.enabled == true and (.lifecycle.data_retention == null))' >/dev/null 2>&1 <<< "$data_streams"; then
        echo "DLM lifecycle already set for $data_stream with indefinite retention, skipping."
        return 0
    fi
    if [[ -n "$data_retention" ]]; then
        body=$(jq -cn --arg data_retention "$data_retention" '{data_retention: $data_retention}')
    else
        # Setting indefinite retention
        body='{}'
    fi
    if ! output=$(so-elasticsearch-query "_data_stream/${data_stream}/_lifecycle" -XPUT -d "$body" --retry 3 --retry-delay 5 --fail); then
        echo "Failed to set data stream lifecycle for $data_stream."
        return 1
    fi
    if [[ -n "$data_retention" ]]; then
        echo "Set DLM lifecycle for $data_stream with data_retention $data_retention."
    else
        echo "Set DLM lifecycle for $data_stream with indefinite retention."
    fi
 }
 disable_data_stream_lifecycle() {
    local data_stream="$1"
    local body='{"enabled":false}'
    local output
    if ! jq -e --arg data_stream "$data_stream" '.data_streams[]? | select(.name == $data_stream and .lifecycle != null and .lifecycle.enabled != false)' >/dev/null 2>&1 <<< "$data_streams"; then
        # No action needed
        return 0
    fi
    if ! output=$(so-elasticsearch-query "_data_stream/${data_stream}/_lifecycle" -XPUT -d "$body" --retry 3 --retry-delay 5 --fail); then
        echo "Failed to disable data stream lifecycle for $data_stream."
        return 1
    fi
    echo "Disabled DLM lifecycle for $data_stream."
 }
 process_data_stream() {
    local data_stream="$1"
    local data_retention="$2"
    if [[ "$DATA_RETENTION_METHOD" == "DLM" ]]; then
        set_data_stream_lifecycle "$data_stream" "$data_retention"
    else
        disable_data_stream_lifecycle "$data_stream"
    fi
 }
 check_elasticsearch_responsive
 if ! data_streams=$(so-elasticsearch-query "_data_stream?format=json" --retry 3 --retry-delay 5 --fail); then
    echo "Failed to retrieve data streams."
    exit 1
 fi
 while read -r data_stream_config; do
    data_stream=$(jq -r '.name' <<< "$data_stream_config")
    template=$(jq -r '.template' <<< "$data_stream_config")
    if ! template_file=$(find_template_file "$template"); then
        echo "Skipping $data_stream: index template file not found for $template."
        continue
    fi
    validate_template_file "$template_file" || exit 1
    if ! is_data_stream_template "$template_file"; then
        echo "Skipping $data_stream: $template_file is not a data stream template."
        continue
    fi
    if [[ "$DATA_RETENTION_METHOD" == "DLM" ]] && ! has_data_stream_lifecycle "$template_file"; then
        echo "Skipping $data_stream: $template_file does not define data stream lifecycle."
        continue
    fi
    data_retention=$(get_data_retention "$template_file")
    if ! process_data_stream "$data_stream" "$data_retention"; then
        DLM_FAILURES=$((DLM_FAILURES + 1))
        DLM_FAILURE_NAMES+=("$data_stream")
    fi
 done < <(jq -c '.data_streams[]' <<< "$data_streams")
 if [[ $DLM_FAILURES -eq 0 ]]; then
    echo "Data stream lifecycle updates completed successfully."
 else
    echo "Encountered $DLM_FAILURES failure(s) updating data stream lifecycle:"
    for failed_data_stream in "${DLM_FAILURE_NAMES[@]}"; do
        echo "  - $failed_data_stream"
    done
    exit 1
 fi
@@ -6,48 +6,6 @@
 . /usr/sbin/so-common
 MAX_JOBS=${MAX_ILM_JOBS:-10}
 # Lock used to serialize block writes so concurrent jobs never interleave their output.
 ILM_OUTPUT_LOCK=$(mktemp)
 ILM_FAIL_FILE=$(mktemp)
 trap 'rm -f "$ILM_OUTPUT_LOCK" "$ILM_FAIL_FILE"' EXIT
 # Policies are loaded concurrently (up to MAX_JOBS at a time) for speed. Each policy's block is
 # printed the moment its curl returns, so output appears in COMPLETION ORDER, not the order
 # policies are defined in configuration.
 echo "Loading ILM policies concurrently; output below appears in completion order, not configuration order."
 echo
 put_policy() {
  local desc="$1" policyname="$2" data="$3" result rc=0
  if ! result=$(curl -K /opt/so/conf/elasticsearch/curl.config -s -k -L --fail \
    -X PUT "https://localhost:9200/_ilm/policy/${policyname}" \
    -H 'Content-Type: application/json' -d"${data}" 2>&1); then
    rc=1
  elif ! jq -e '.acknowledged == true' <<<"$result" >/dev/null 2>&1; then
    rc=1
  fi
  # curl above ran in parallel; serialize just this block write so concurrent jobs never interleave.
  {
    flock 200
    printf 'Setting up %s policy...\n%s\n\n' "${desc}" "${result}"
    if (( rc != 0 )); then
      printf '%s\n' "${policyname}" >>"$ILM_FAIL_FILE"
    fi
  } 200>>"${ILM_OUTPUT_LOCK}"
  return "$rc"
 }
 # Block until fewer than MAX_JOBS background curls are running.
 throttle() {
  while (( $(jobs -rp | wc -l) >= MAX_JOBS )); do
    wait -n || true
  done
 }
 {%- from 'elasticsearch/template.map.jinja' import ES_INDEX_SETTINGS %}
 {%- if GLOBALS.role != "so-heavynode" %}
 {%-   from 'elasticsearch/template.map.jinja' import ALL_ADDON_SETTINGS %}
@@ -56,36 +14,35 @@ throttle() {
 {%- for index, settings in ES_INDEX_SETTINGS.items() %}
 {%-   if settings.policy is defined %}
 {%-     if index == 'so-logs-detections.alerts' %}
-  throttle
+  echo
-  put_policy "so-logs-detections.alerts-so" "{{ index }}-so" '{ "policy": {{ settings.policy | tojson(true) }} }' &
+  echo "Setting up so-logs-detections.alerts-so policy..."
  curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -s -k -L -X PUT "https://localhost:9200/_ilm/policy/{{ index }}-so" -H 'Content-Type: application/json' -d'{ "policy": {{ settings.policy | tojson(true) }} }'
  echo
 {%-     elif index == 'so-logs-soc' %}
-  throttle
+  echo
-  put_policy "so-soc-logs" "so-soc-logs" '{ "policy": {{ settings.policy | tojson(true) }} }' &
+  echo "Setting up so-soc-logs policy..."
-  throttle
+  curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -s -k -L -X PUT "https://localhost:9200/_ilm/policy/so-soc-logs" -H 'Content-Type: application/json' -d'{ "policy": {{ settings.policy | tojson(true) }} }'
-  put_policy "{{ index }}-logs" "{{ index }}-logs" '{ "policy": {{ settings.policy | tojson(true) }} }' &
+  echo
  echo
  echo "Setting up {{ index }}-logs policy..."
  curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -s -k -L -X PUT "https://localhost:9200/_ilm/policy/{{ index }}-logs" -H 'Content-Type: application/json' -d'{ "policy": {{ settings.policy | tojson(true) }} }'
  echo
 {%-     else %}
-  throttle
+  echo
-  put_policy "{{ index }}-logs" "{{ index }}-logs" '{ "policy": {{ settings.policy | tojson(true) }} }' &
+  echo "Setting up {{ index }}-logs policy..."
  curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -s -k -L -X PUT "https://localhost:9200/_ilm/policy/{{ index }}-logs" -H 'Content-Type: application/json' -d'{ "policy": {{ settings.policy | tojson(true) }} }'
  echo
 {%-     endif %}
 {%-   endif %}
 {%- endfor %}
 echo
 {%- if GLOBALS.role != "so-heavynode" %}
 {%-   for index, settings in ALL_ADDON_SETTINGS.items() %}
 {%-     if settings.policy is defined %}
-  throttle
+  echo
-  put_policy "{{ index }}-logs" "{{ index }}-logs" '{ "policy": {{ settings.policy | tojson(true) }} }' &
+  echo "Setting up {{ index }}-logs policy..."
  curl -K /opt/so/conf/elasticsearch/curl.config -b "sid=$SESSIONCOOKIE" -s -k -L -X PUT "https://localhost:9200/_ilm/policy/{{ index }}-logs" -H 'Content-Type: application/json' -d'{ "policy": {{ settings.policy | tojson(true) }} }'
  echo
 {%-     endif %}
 {%-   endfor %}
 {%- endif %}
 wait || true
 if [[ -s "$ILM_FAIL_FILE" ]]; then
  echo "ERROR: Failed to load ILM policy(s):"
  while read -r POLICY; do
    echo " - $POLICY"
  done < "$ILM_FAIL_FILE"
  exit 1
 else
  echo "Successfully loaded all ILM policies."
 fi
@@ -22,7 +22,6 @@ include:
 so-hydra:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-hydra:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: hydra
    - name: so-hydra
    - networks:
@@ -59,6 +58,7 @@ so-hydra:
      - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
    {%   endfor %}
    {% endif %}
    - restart_policy: unless-stopped
    - watch:
      - file: hydraconfig
    - require:
@@ -15,7 +15,6 @@ include:
 so-idh:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-idh:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - name: so-idh
    - detach: True
    - network_mode: host
@@ -18,7 +18,6 @@ include:
 so-influxdb:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-influxdb:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: influxdb
    - networks:
      - sobridge:
@@ -27,13 +27,12 @@ include:
 so-kafka:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-kafka:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: so-kafka
    - name: so-kafka
    - networks:
      - sobridge:
        - ipv4_address: {{ DOCKERMERGED.containers['so-kafka'].ip }}
-    - user: "960"
+    - user: kafka
    - environment:
        KAFKA_HEAP_OPTS: -Xmx2G -Xms1G
        KAFKA_OPTS: "-javaagent:/opt/jolokia/agents/jolokia-agent-jvm-javaagent.jar=port=8778,host={{ DOCKERMERGED.containers['so-kafka'].ip }},policyLocation=file:/opt/jolokia/jolokia.xml {%- if KAFKA_EXTERNAL_ACCESS %} -Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf {% endif -%}"
@@ -22,7 +22,7 @@ kibana:
          - default
          - file
    migrations:
-      discardCorruptObjects: "9.3.7"
+      discardCorruptObjects: "9.3.3"
    telemetry:
      enabled: False
    xpack:
@@ -6,7 +6,6 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
 include:
@@ -17,9 +16,8 @@ include:
 so-kibana:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-kibana:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: kibana
-    - user: "932:0"
+    - user: kibana
    - networks:
      - sobridge:
        - ipv4_address: {{ DOCKERMERGED.containers['so-kibana'].ip }}
@@ -62,19 +60,6 @@ so-kibana:
    - watch:
      - file: kibanaconfig
 wait_for_so-kibana:
  http.wait_for_successful_query:
    - name: "http://localhost:5601/api/status"
    - username: 'so_elastic'
    - password: '{{ ELASTICSEARCHMERGED.auth.users.so_elastic_user.pass }}'
    - ssl: True
    - verify_ssl: False
    - status: 200
    - wait_for: 600
    - request_interval: 15
    - require:
      - docker_container: so-kibana
 delete_so-kibana_so-status.disabled:
  file.uncomment:
    - name: /opt/so/conf/so-status/so-status.conf
@@ -15,7 +15,6 @@ include:
 so-kratos:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-kratos:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: kratos
    - name: so-kratos
    - networks:
@@ -52,6 +51,7 @@ so-kratos:
      - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
    {%   endfor %}
    {% endif %}
    - restart_policy: unless-stopped
    - watch:
      - file: kratosschema
      - file: kratosconfig
@@ -103,7 +103,7 @@ kratos:
  config:
    session:
      lifespan: 
-        description: Defines the length of a login session before it will timeout, and require a new login.
+        description: Defines the length of a login session.
        global: True
        helpLink: kratos
      whoami:
@@ -1,67 +0,0 @@
 # This state is designed to run on a development manager running in a libvirt VM. It will map the default pillar and salt directories
 # from /opt/so/saltstack/default to your local development machine as the source path.
 # The VM requires a filesystem to be added. Only the source path should be changed to your development codebase
 #  Driver: virtio-9p
 #  Source path: ~/project/securityonion
 #  Target path: saltDev
 # If you want a directory to be RW, then kvm must have group privileges.
 # ll /home/user/projects/securityonion/salt/hypervisor
 # total 48
 # drwxrwxr-x  3 user kvm 4096 Feb 13 11:18 ./
 # drwxrwxr-x 64 user user 4096 Feb 13 10:32 ../
 # -rw-rw-r--  1 user kvm 2238 Feb 12 15:06 defaults.yaml
 # -rw-rw-r--  1 user kvm 1467 Feb 12 15:06 init.sls
 # -rw-rw-r--  1 user kvm   70 Feb 13 09:37 soc_hypervisor.yaml
 # drwxrwxr-x  3 user kvm 4096 Feb 12 15:06 tools/
 # Ensure required kernel modules are configured for loading
 /etc/modules-load.d/virtio-9p.conf:
  file.managed:
    - contents: |
        9pnet_virtio
        9pnet
        9p
    - mode: 644
    - user: root
    - group: root
 # Load the kernel modules immediately (in the correct order)
 load_9p_modules:
  cmd.run:
    - names:
      - modprobe 9pnet_virtio
      - modprobe 9pnet
      - modprobe 9p
    - unless: lsmod | grep -E '9pnet_virtio|9pnet|9p'
 # Ensure mount point exists
 /opt/so/saltstack/default:
  file.directory:
    - user: root
    - group: root
    - mode: 755
    - makedirs: True
 # Configure fstab entry using mount.fstab_present
 # Configure fstab entry using mount.fstab_present
 saltdev_fstab:
  mount.fstab_present:
    - name: saltDev
    - fs_file: /opt/so/saltstack/default
    - fs_vfstype: 9p
    - fs_mntops: _netdev,trans=virtio,version=9p2000.L
    - fs_freq: 0
    - fs_passno: 0
 # Mount the filesystem if not already mounted
 mount_saltdev:
  mount.mounted:
    - name: /opt/so/saltstack/default
    - device: saltDev
    - fstype: 9p
    - opts: _netdev,trans=virtio,version=9p2000.L
    - require:
      - file: /opt/so/saltstack/default
      - mount: saltdev_fstab
      - cmd: load_9p_modules
@@ -150,16 +150,6 @@ logrotate:
      - extension .log
      - dateext
      - dateyesterday
    /opt/so/log/postgres/*_x_log:
      - daily
      - rotate 14
      - missingok
      - copytruncate
      - compress
      - create
      - extension .log
      - dateext
      - dateyesterday
    /opt/so/log/telegraf/*_x_log:
      - daily
      - rotate 14
@@ -91,13 +91,6 @@ logrotate:
      multiline: True
      global: True
      forcedType: "[]string"
    "/opt/so/log/postgres/*_x_log":
      description: List of logrotate options for this file.
      title: /opt/so/log/postgres/*.log
      advanced: True
      multiline: True
      global: True
      forcedType: "[]string"
    "/opt/so/log/telegraf/*_x_log":
      description: List of logrotate options for this file.
      title: /opt/so/log/telegraf/*.log
@@ -26,12 +26,12 @@ logstash:
    manager:
      - so/0011_input_endgame.conf
      - so/0012_input_elastic_agent.conf.jinja
-      - so/0013_input_lumberjack_fleet.conf.jinja
+      - so/0013_input_lumberjack_fleet.conf
      - so/9999_output_redis.conf.jinja
    receiver:
      - so/0011_input_endgame.conf
      - so/0012_input_elastic_agent.conf.jinja
-      - so/0013_input_lumberjack_fleet.conf.jinja
+      - so/0013_input_lumberjack_fleet.conf
      - so/9999_output_redis.conf.jinja
    search:
      - so/0900_input_redis.conf.jinja
@@ -69,5 +69,4 @@ logstash:
    pipeline_x_batch_x_size: 125
    pipeline_x_ecs_compatibility: disabled
  dmz_nodes: []
  latency_metrics: False
@@ -28,13 +28,12 @@ include:
 so-logstash:
  docker_container.running:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-logstash:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - hostname: so-logstash
    - name: so-logstash
    - networks:
      - sobridge:
        - ipv4_address: {{ DOCKERMERGED.containers['so-logstash'].ip }}
-    - user: "931:0"
+    - user: logstash
    - extra_hosts:
    {% for node in LOGSTASH_NODES %}
    {%   for hostname, ip in node.items() %}
@@ -1,4 +1,3 @@
 {%- from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 input {
  elastic_agent {
    port => 5055
@@ -12,11 +11,6 @@ input {
  }
 }
 filter {
  {% if LOGSTASH_MERGED.get('latency_metrics', False) %}
  ruby {
    code => "event.set('[_tmp][logstash_from_agent]', Time.now().utc.iso8601(3));"
  }
  {% endif %}
 if ![metadata] {
  mutate {
    rename => {"@metadata" => "metadata"}
@@ -0,0 +1,23 @@
 input {
  elastic_agent {
    port => 5056
    tags => [ "elastic-agent", "fleet-lumberjack-input" ]
    ssl_enabled => true
    ssl_certificate => "/usr/share/logstash/elasticfleet-lumberjack.crt"
    ssl_key => "/usr/share/logstash/elasticfleet-lumberjack.key"
    ecs_compatibility => v8
    id => "fleet-lumberjack-in"  
    codec => "json"
  }
 }
 filter {
 if ![metadata] {
  mutate {
    rename => {"@metadata" => "metadata"}
  }
 }
 }
@@ -1,26 +0,0 @@
 {%- from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 input {
  elastic_agent {
    port => 5056
    tags => [ "elastic-agent", "fleet-lumberjack-input" ]
    ssl_enabled => true
    ssl_certificate => "/usr/share/logstash/elasticfleet-lumberjack.crt"
    ssl_key => "/usr/share/logstash/elasticfleet-lumberjack.key"
    ecs_compatibility => v8
    id => "fleet-lumberjack-in"
    codec => "json"
  }
 }
 filter {
  {% if LOGSTASH_MERGED.get('latency_metrics', False) %}
  ruby {
    code => "event.set('[_tmp][logstash_from_fleet]', Time.now().utc.iso8601(3));"
  }
  {% endif %}
  if ![metadata] {
    mutate {
      rename => {"@metadata" => "metadata"}
    }
  }
 }
@@ -1,4 +1,3 @@
 {%- from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 {%- set kafka_password = salt['pillar.get']('kafka:config:password') %}
 {%- set kafka_trustpass = salt['pillar.get']('kafka:config:trustpass') %}
 {%- set kafka_brokers = salt['pillar.get']('kafka:nodes', {}) %}
@@ -31,11 +30,6 @@ input {
    }
 }
 filter {
  {% if LOGSTASH_MERGED.get('latency_metrics', False) %}
  ruby {
    code => "event.set('[_tmp][logstash_from_kafka]', Time.now().utc.iso8601(3));"
  }
  {% endif %}
  if ![metadata] {
    mutate {
      rename => { "@metadata" => "metadata" }
@@ -1,4 +1,4 @@
-{%- from 'logstash/map.jinja' import LOGSTASH_REDIS_NODES, LOGSTASH_MERGED %}
+{%- from 'logstash/map.jinja' import LOGSTASH_REDIS_NODES with context %}
 {%- set REDIS_PASS = salt['pillar.get']('redis:config:requirepass') %}
 {%- for index in range(LOGSTASH_REDIS_NODES|length) %}
@@ -18,10 +18,3 @@ input {
 }
 {%   endfor %}
 {% endfor -%}
 filter {
  {% if LOGSTASH_MERGED.get('latency_metrics', False) %}
  ruby {
    code => "event.set('[_tmp][logstash_from_redis]', Time.now().utc.iso8601(3));"
  }
  {% endif %}
 }
@@ -1,11 +1,3 @@
 {%- from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 {% if LOGSTASH_MERGED.get('latency_metrics', False) %}
 filter {
  ruby {
    code => "event.set('[_tmp][logstash_to_elasticsearch]', Time.now().utc.iso8601(3));"
  }
 }
 {% endif %}
 output {
  if "elastic-agent" in [tags] and "so-ip-mappings" in [tags] {
    elasticsearch {
@@ -13,14 +13,7 @@ filter {
                    add_tag => "fleet-lumberjack-{{ GLOBALS.hostname }}"
          }
  }
-{%- from 'logstash/map.jinja' import LOGSTASH_MERGED %}
+
 {% if LOGSTASH_MERGED.get('latency_metrics', False) %}
 filter {
  ruby {
    code => "event.set('[_tmp][fleet_to_logstash]', Time.now().utc.iso8601(3));"
  }
 }
 {% endif %}
 output { 
    lumberjack { 
        codec => json 
@@ -1,17 +1,10 @@
 {%- from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 {%- if grains.role in ['so-heavynode', 'so-receiver'] %}
  {%- set HOST = GLOBALS.hostname %}
 {%- else %}
  {%- set HOST = GLOBALS.manager %}
 {%- endif %}
 {%- set REDIS_PASS = salt['pillar.get']('redis:config:requirepass') %}
-{% if LOGSTASH_MERGED.get('latency_metrics', False) %}
+
 filter {
  ruby {
    code => "event.set('[_tmp][logstash_to_redis]', Time.now().utc.iso8601(3));"
  }
 }
 {% endif %}
 output {
 	redis {
 		host => '{{ HOST }}'
@@ -86,8 +86,3 @@ logstash:
    multiline: True
    advanced: True
    forcedType: "[]string"
  latency_metrics:
    description: Enable latency metrics within events processed by logstash. Useful for pinpointing log ingest delay.
    forcedType: bool
    global: False
    advanced: True
@@ -1,21 +0,0 @@
 {% from 'vars/globals.map.jinja' import GLOBALS %}
 {% from 'salt/auto_apply.map.jinja' import AUTOAPPLY %}
 include:
  - salt.minion
 {% if GLOBALS.is_manager and AUTOAPPLY.enabled %}
 salt_beacons_pushstate:
  file.managed:
    - name: /etc/salt/minion.d/beacons_pushstate.conf
    - source: salt://manager/files/beacons_pushstate.conf.jinja
    - template: jinja
    - watch_in:
      - service: salt_minion_service
 {% else %}
 salt_beacons_pushstate:
  file.absent:
    - name: /etc/salt/minion.d/beacons_pushstate.conf
    - watch_in:
      - service: salt_minion_service
 {% endif %}
@@ -1,11 +0,0 @@
 {% from 'salt/auto_apply.map.jinja' import AUTOAPPLY %}
 beacons:
  postgres_pillar_beacon:
    - interval: {{ AUTOAPPLY.drain_interval }}
    - disable_during_state_run: False
  rules_beacon:
    - interval: {{ AUTOAPPLY.drain_interval }}
    - disable_during_state_run: False
    - paths:
        /opt/so/saltstack/local/salt/suricata/rules: suricata
        /opt/so/saltstack/local/salt/strelka/rules/compiled: strelka
@@ -1,2 +0,0 @@
 https://repo.securityonion.net/file/so-repo/prod/3/oracle/9-uek8
 https://repo-alt.securityonion.net/prod/3/oracle/9-uek8
@@ -11,8 +11,3 @@ name=Security Onion Repo repo
 mirrorlist=file:///opt/so/conf/reposync/mirror.txt
 enabled=1
 gpgcheck=1
 [securityonionkernelsync]
 name=Security Onion Kernel Repo repo
 mirrorlist=file:///opt/so/conf/reposync/mirror-kernel.txt
 enabled=1
 gpgcheck=1
@@ -15,7 +15,6 @@ include:
  - manager.elasticsearch
  - manager.kibana
  - manager.managed_soc_annotations
  - manager.beacons
 repo_log_dir:
  file.directory:
@@ -87,28 +86,6 @@ repo_dir:
      - group
    - show_changes: False
 kernelrepo_dir:
  file.directory:
    - name: /nsm/kernelrepo
    - user: socore
    - group: socore
    - recurse:
      - user
      - group
    - show_changes: False
 # Ensure /nsm/kernelrepo is always a valid (if empty) repo before it is ever assigned to
 # a client. Without repodata/repomd.xml an enabled file:///nsm/kernelrepo repo makes every
 # dnf operation fail; so-repo-sync only populates it after the highstate, so seed an empty
 # repo here. Only runs when repodata is missing, so it won't clobber a synced repo.
 kernelrepo_init_empty:
  cmd.run:
    - name: createrepo /nsm/kernelrepo
    - unless: 'test -e /nsm/kernelrepo/repodata/repomd.xml'
    - require:
      - file: kernelrepo_dir
      - pkg: install_createrepo
 manager_sbin:
  file.recurse:
    - name: /usr/sbin
@@ -145,13 +122,6 @@ so-repo-mirrorlist:
    - user: socore
    - group: socore
 so-repo-kernel-mirrorlist:
  file.managed:
    - name: /opt/so/conf/reposync/mirror-kernel.txt
    - source: salt://manager/files/mirror-kernel.txt
    - user: socore
    - group: socore
 so-repo-sync:
  {%     if MANAGERMERGED.reposync.enabled %}
  cron.present:
@@ -261,7 +231,6 @@ surifiltersrules:
    - user: 939
    - group: 939
 {% else %}
 {{sls}}_state_not_allowed:
@@ -16,35 +16,40 @@
 {%       endif %}
 {%     endfor %}
 {%   endfor %}
 {%   set soc_annotation_lines = [] %}
 {%   set defaults_lines = [] %}
 {%   for k in matched_integration_names %}
 {%     do soc_annotation_lines.append('    ' ~ k ~ ': *dataStreamSettings') %}
 {%     do defaults_lines.append('    ' ~ k ~ ':') %}
 {%     set defaults_yaml = salt['slsutil.serialize']('yaml', ADDON_INTEGRATION_DEFAULTS[k], default_flow_style=False).strip() %}
 {%     for line in defaults_yaml.splitlines() %}
 {%       do defaults_lines.append('      ' ~ line) %}
 {%     endfor %}
 {%   endfor %}
 {%   set es_soc_annotations = '/opt/so/saltstack/default/salt/elasticsearch/soc_elasticsearch.yaml' %}
-manage_soc_annotations:
+{{   es_soc_annotations }}:
-  file.blockreplace:
+     file.serialize:
-    - name: {{ es_soc_annotations }}
+       - dataset:
-    - marker_start: '    # START managed SOC integration annotations'
+           {% set data = salt['file.read'](es_soc_annotations) | load_yaml %}
-    - marker_end: '    # END managed SOC integration annotations'
+           {% set es = data.get('elasticsearch', {}) %}
-    - content: {{ soc_annotation_lines | join('\n') | tojson }}
+           {% set index_settings = es.get('index_settings', {}) %}
-    - insert_after_match: '^    # Managed SOC integration annotations are inserted below this line\.'
+           {% set input = index_settings.get('so-logs', {}) %}
-    - append_if_not_found: False
+           {% for k in matched_integration_names %}
-    - show_changes: True
+           {%   do index_settings.update({k: input}) %}
           {% endfor %}
           {% for k in addon_integration_keys %}
           {%   if k not in matched_integration_names and k in index_settings %}
           {%     do index_settings.pop(k) %}
           {%   endif %}
           {% endfor %}
           {{ data }}
 {#   Managed elasticsearch/defaults.yaml file for enabling 'Revert to default' via SOC UI for newly added config items #}
 {%   set es_defaults = '/opt/so/saltstack/default/salt/elasticsearch/defaults.yaml' %}
 {{   es_defaults }}:
-  file.blockreplace:
+     file.serialize:
-    - marker_start: '    # START managed SOC integration defaults'
+       - dataset:
-    - marker_end: '    # END managed SOC integration defaults'
+           {% set data = salt['file.read'](es_defaults) | load_yaml %}
-    - content: {{ defaults_lines | join('\n') | tojson }}
+           {% set es = data.get('elasticsearch', {}) %}
-    - insert_after_match: '^  index_settings:$'
+           {% set index_settings = es.get('index_settings', {}) %}
-    - append_if_not_found: False
+           {% for k in matched_integration_names %}
-    - show_changes: True
+           {%   set input = ADDON_INTEGRATION_DEFAULTS[k] %}
           {%     do index_settings.update({k: input})%}
           {% endfor %}
           {% for k in addon_integration_keys %}
           {%   if k not in matched_integration_names and k in index_settings %}
           {%     do index_settings.pop(k) %}
           {%   endif %}
           {% endfor %}
           {{ data }}
 {% endif %}
@@ -31,13 +31,11 @@ sync_es_users:
      - http: wait_for_kratos
      - file: so-user.lock # require so-user.lock file to be missing
-# we dont want this added too early in setup, so the onlyif gates on the
+# we dont want this added too early in setup, so we add the onlyif to verify 'startup_states: highstate'
-# /opt/so/state/setup-complete marker. The marker is written by
+# is in the minion config. That line is added before the final highstate during setup
 # mark_setup_complete in setup/so-functions just before the final setup
 # highstate (and by an upgrade-path state for systems set up under the old gate).
 so-user_sync:
  cron.present:
    - user: root
    - name: 'PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin /usr/sbin/so-user sync &>> /opt/so/log/soc/sync.log'
    - identifier: so-user_sync
-    - onlyif: "test -e /opt/so/state/setup-complete"
+    - onlyif: "grep -x 'startup_states: highstate' /etc/salt/minion"
@@ -1,117 +0,0 @@
 #!/bin/bash
 #
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 # Runs once per boot on managers (via so-boot-mine-update.service), before
 # so-boot-highstate.service. Waits for the responsive minion set to settle, pushes
 # mine.update, waits until every up minion has actually reported to the mine, then
 # warms the master's per-minion pillar cache so the mine-backed node pillars (node
 # IPs, ES/Redis/Logstash/hypervisor discovery -- some glob- and some pillar/grain-
 # targeted) are complete before the boot highstate renders them. Otherwise a node
 # that is up but not yet fully reported gets dropped from those pillars and torn
 # out of the configs they build (e.g. so-elasticsearch ExtraHosts -> container recreate).
 MAX_WAIT=${MINE_UPDATE_MAX_WAIT:-180}   # hard backstop only
 INTERVAL=10
 STABLE_CHECKS=3                          # up-count must hold steady this many polls
 elapsed=0
 prev=-1
 stable=0
 up=0
 # Wait for the *reachable* minion set to settle rather than for every accepted
 # key to report up: an operator may accept a minion's key and then intentionally
 # power off that host, so requiring up >= accepted would never be satisfied and
 # we'd always burn the full MAX_WAIT. Once the responsive count stops growing we
 # stop waiting and run mine.update against whoever is up.
 while [ "$elapsed" -lt "$MAX_WAIT" ]; do
  up=$(/usr/bin/salt-run manage.up --out=json 2>/dev/null \
    | python3 -c 'import sys,json; print(len(json.load(sys.stdin)))' 2>/dev/null)
  up=${up:-0}
  if [ "$up" -gt 0 ] && [ "$up" -eq "$prev" ]; then
    stable=$((stable + 1))
    [ "$stable" -ge "$STABLE_CHECKS" ] && break
  else
    stable=0
  fi
  prev=$up
  sleep "$INTERVAL"
  elapsed=$((elapsed + INTERVAL))
 done
 echo "so-boot-mine-update: ${up} minions up (settled after ${elapsed}s); running mine.update"
 /usr/bin/salt '*' mine.update --out=txt
 # A node that is up but has not yet re-reported network.ip_addrs to the mine is
 # silently dropped from mine-backed pillars (elasticsearch:nodes, node_data, ...)
 # when highstate recompiles them -- which e.g. removes it from so-elasticsearch
 # ExtraHosts and forces a container recreate. After the broad mine.update above,
 # wait until every up minion actually has network.ip_addrs in the mine, re-pushing
 # mine.update to stragglers, before releasing the boot highstate. Bounded by the
 # same MAX_WAIT backstop so a slow/down node never blocks boot indefinitely.
 missing=""
 while [ "$elapsed" -lt "$MAX_WAIT" ]; do
  up_json=$(/usr/bin/salt-run manage.up --out=json 2>/dev/null)
  mine_json=$(/usr/bin/salt-run mine.get '*' network.ip_addrs tgt_type=glob --out=json 2>/dev/null)
  missing=$(printf '%s' "$up_json" | python3 -c '
 import sys, json
 up = set(json.load(sys.stdin) or [])
 mine = {k for k, v in (json.loads(sys.argv[1]) or {}).items() if v}
 print("\n".join(sorted(up - mine)))
 ' "$mine_json" 2>/dev/null)
  if [ -z "$missing" ]; then
    echo "so-boot-mine-update: mine complete for all up minions after ${elapsed}s"
    break
  fi
  echo "so-boot-mine-update: mine missing up minion(s): $(echo $missing); re-running mine.update"
  for m in $missing; do /usr/bin/salt "$m" mine.update --out=txt; done
  sleep "$INTERVAL"
  elapsed=$((elapsed + INTERVAL))
 done
 [ -n "$missing" ] && echo "so-boot-mine-update: WARNING ${MAX_WAIT}s backstop hit; up minion(s) still absent from mine: $(echo $missing); highstate may drop them from configs"
 # The pillar/compound-targeted node pillars (elasticsearch:nodes, redis:nodes,
 # logstash:nodes, hypervisor:nodes) resolve their target against the master's
 # per-minion data cache (grains+pillar in .../minions/<id>/data.p), populated only
 # when a minion's pillar is (re)compiled -- separately from the mine. A freshly
 # booted node can be in the mine (glob/node_data sees it) yet absent from that
 # cache, so it is dropped from those pillars and from the configs they build (e.g.
 # so-elasticsearch ExtraHosts). Force a synchronous pillar refresh so the master
 # caches every up node's pillar; refresh_pillar wait=True returns only once the
 # pillar is recompiled (and thus cached for matching). Retry stragglers <= MAX_WAIT.
 echo "so-boot-mine-update: warming master pillar cache for pillar/grain-targeted node pillars"
 /usr/bin/salt '*' saltutil.refresh_pillar wait=True --out=txt
 missing=""
 while [ "$elapsed" -lt "$MAX_WAIT" ]; do
  up_json=$(/usr/bin/salt-run manage.up --out=json 2>/dev/null)
  cached_json=$(/usr/bin/salt-run cache.pillar tgt='*' --out=json 2>/dev/null)
  missing=$(printf '%s' "$up_json" | python3 -c '
 import sys, json
 up = set(json.load(sys.stdin) or [])
 cached = {k for k, v in (json.loads(sys.argv[1]) or {}).items() if v}
 print("\n".join(sorted(up - cached)))
 ' "$cached_json" 2>/dev/null)
  if [ -z "$missing" ]; then
    echo "so-boot-mine-update: pillar cache warm for all up minions after ${elapsed}s"
    break
  fi
  echo "so-boot-mine-update: pillar not yet cached for: $(echo $missing); refreshing"
  for m in $missing; do /usr/bin/salt "$m" saltutil.refresh_pillar wait=True --out=txt; done
  sleep "$INTERVAL"
  elapsed=$((elapsed + INTERVAL))
 done
 [ -n "$missing" ] && echo "so-boot-mine-update: WARNING ${MAX_WAIT}s backstop hit; pillar not cached for: $(echo $missing); pillar-targeted pillars may drop them"
 # Log what the mine-backed pillars render so the boot-time state is inspectable.
 /usr/bin/salt-call saltutil.refresh_pillar >/dev/null 2>&1
 sleep 2
 for key in node_data elasticsearch:nodes; do
  rendered=$(/usr/bin/salt-call --out=json pillar.get "$key" 2>/dev/null \
    | python3 -c 'import sys,json; print(json.dumps(json.load(sys.stdin).get("local"), indent=2, sort_keys=True))' 2>/dev/null)
  echo "so-boot-mine-update: ${key} rendered as:"
  echo "${rendered:-null}"
 done
 exit 0
@@ -1,381 +0,0 @@
 #!/usr/bin/env python3
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 # Imports detection overrides (e.g. from so-detections-backup) into the so-detection
 # index. Reads <publicId>.<ext> files (NDJSON, one override per line) from a source
 # directory, looks up the matching detection by publicId+engine, validates each
 # override against the same rules SOC enforces, dedupes against existing overrides
 # (operational fields only), and appends new ones.
 import argparse
 import ipaddress
 import json
 import os
 import re
 import sys
 from datetime import datetime
 import requests
 from requests.auth import HTTPBasicAuth
 import urllib3
 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 DEFAULT_INDEX = "so-detection"
 AUTH_FILE = "/opt/so/conf/elasticsearch/curl.config"
 ES_URL = "https://localhost:9200"
 # Engines we know how to handle and the file extension the backup script writes.
 ENGINES = {
    "suricata": "txt",
 }
 # Standard Suricata variables that ship with Security Onion. Anything else
 # referenced in an override is "custom" and the user needs to make sure it
 # exists in SOC Config before the override will function.
 BUILTIN_SURICATA_VARS = {
    "$HOME_NET", "$EXTERNAL_NET",
    "$HTTP_SERVERS", "$DNS_SERVERS", "$SQL_SERVERS", "$SMTP_SERVERS",
    "$TELNET_SERVERS", "$AIM_SERVERS", "$DC_SERVERS", "$MODBUS_SERVER",
    "$MODBUS_CLIENT", "$ENIP_CLIENT", "$ENIP_SERVER",
    "$HTTP_PORTS", "$SHELLCODE_PORTS", "$ORACLE_PORTS", "$SSH_PORTS",
    "$FTP_PORTS", "$FILE_DATA_PORTS",
 }
 VAR_PATTERN = re.compile(r"\$[A-Z_][A-Z0-9_]*")
 # Canonical valid values, per securityonion-soc/model/detection.go.
 SURICATA_OVERRIDE_TYPES = {"suppress", "threshold", "modify"}
 SUPPRESS_TRACKS = {"by_src", "by_dst", "by_either"}
 THRESHOLD_TRACKS = {"by_src", "by_dst", "by_both"}
 THRESHOLD_TYPES = {"limit", "threshold", "both"}
 STALE_WARNING = """\
 WARNING: so-detections-backup does not remove backup files when overrides are
 deleted via the Security Onion web UI. As a result, files in the source
 directory may represent overrides that were intentionally deleted and should
 NOT be re-imported.
 Before continuing, verify that the source directory reflects the overrides you
 actually want imported. Remove any files corresponding to overrides you previously deleted.
 """
 def make_session(auth_file):
    with open(auth_file, "r") as f:
        for line in f:
            if line.startswith("user ="):
                creds = line.split("=", 1)[1].strip().replace('"', "")
                user, _, password = creds.partition(":")
                session = requests.Session()
                session.auth = HTTPBasicAuth(user, password)
                session.headers.update({"Content-Type": "application/json"})
                session.verify = False
                return session
    raise RuntimeError(f"Could not find 'user =' line in {auth_file}")
 def find_detection(session, index, public_id, engine):
    query = {
        "query": {"bool": {"must": [
            {"term": {"so_detection.publicId": public_id}},
            {"term": {"so_detection.engine": engine}},
        ]}},
        "size": 2,
    }
    r = session.get(f"{ES_URL}/{index}/_search", json=query)
    r.raise_for_status()
    hits = r.json().get("hits", {}).get("hits", [])
    if not hits:
        return None, None, None
    if len(hits) > 1:
        # Shouldn't happen — publicId is unique per engine — but flag it.
        print(f"  WARN: {len(hits)} detections matched publicId={public_id} engine={engine}; using first")
    hit = hits[0]
    existing = hit["_source"].get("so_detection", {}).get("overrides") or []
    return hit["_id"], hit["_index"], existing
 def update_overrides(session, doc_index, doc_id, overrides):
    body = {"doc": {"so_detection": {"overrides": overrides}}}
    r = session.post(f"{ES_URL}/{doc_index}/_update/{doc_id}", json=body)
    r.raise_for_status()
    return r.json()
 def dedupe_key(override):
    """Operational fields only, per Override.Equal() in detection.go.
    Excludes timestamps and isEnabled so re-imports don't appear unique."""
    t = override.get("type")
    if t == "suppress":
        return (t, override.get("track"), override.get("ip"))
    if t == "threshold":
        return (t, override.get("thresholdType"), override.get("track"),
                override.get("count"), override.get("seconds"))
    if t == "modify":
        return (t, override.get("regex"), override.get("value"))
 def _validate_suricata_ip(ip):
    if not ip:
        return "ip cannot be empty"
    if ip.startswith("$"):
        return None
    if ip.startswith("[") and ip.endswith("]"):
        for part in ip[1:-1].split(","):
            err = _validate_single_ip(part.strip())
            if err:
                return f"invalid IP in list: {err}"
        return None
    return _validate_single_ip(ip)
 def _validate_single_ip(ip):
    try:
        if "/" in ip:
            ipaddress.ip_network(ip, strict=False)
        else:
            ipaddress.ip_address(ip)
    except ValueError:
        return f"invalid IP/CIDR {ip!r}"
    return None
 def validate_override(override, engine):
    """Mirror Override.Validate() from securityonion-soc/model/detection.go.
    Returns None on success, an error string otherwise."""
    t = override.get("type")
    if not t:
        return "override type is required"
    if t not in SURICATA_OVERRIDE_TYPES:
        return f"invalid type {t!r}: must be one of {sorted(SURICATA_OVERRIDE_TYPES)}"
    has = {k: override.get(k) is not None for k in
           ("regex", "value", "thresholdType", "track", "ip", "count", "seconds", "customFilter")}
    if t == "suppress":
        if not has["ip"] or not has["track"]:
            return "suppress requires 'ip' and 'track'"
        if any(has[k] for k in ("regex", "value", "thresholdType", "count", "seconds", "customFilter")):
            return "suppress has unnecessary fields"
        if override["track"] not in SUPPRESS_TRACKS:
            return f"invalid track {override['track']!r}: must be one of {sorted(SUPPRESS_TRACKS)}"
        return _validate_suricata_ip(override["ip"])
    if t == "threshold":
        if not all(has[k] for k in ("thresholdType", "track", "count", "seconds")):
            return "threshold requires 'thresholdType', 'track', 'count', 'seconds'"
        if any(has[k] for k in ("regex", "value", "customFilter")):
            return "threshold has unnecessary fields"
        if override["thresholdType"] not in THRESHOLD_TYPES:
            return f"invalid thresholdType {override['thresholdType']!r}: must be one of {sorted(THRESHOLD_TYPES)}"
        if override["track"] not in THRESHOLD_TRACKS:
            return f"invalid track {override['track']!r}: must be one of {sorted(THRESHOLD_TRACKS)}"
        if not isinstance(override["count"], int) or override["count"] <= 0:
            return f"count must be a positive integer, got {override['count']!r}"
        if not isinstance(override["seconds"], int) or override["seconds"] <= 0:
            return f"seconds must be a positive integer, got {override['seconds']!r}"
        return None
    if t == "modify":
        if not has["regex"] or not has["value"]:
            return "modify requires 'regex' and 'value'"
        if any(has[k] for k in ("thresholdType", "track", "count", "seconds", "customFilter")):
            return "modify has unnecessary fields"
        try:
            re.compile(override["regex"])
        except re.error as e:
            return f"invalid regex: {e}"
        return None
 def parse_overrides_file(path):
    """Parse a file written by so-detections-backup.py: NDJSON, one override
    per line. Returns a list of (override_dict, line_number)."""
    overrides = []
    with open(path, "r") as f:
        for i, line in enumerate(f, start=1):
            line = line.strip()
            if not line:
                continue
            overrides.append((json.loads(line), i))
    return overrides
 def describe(override):
    """Human-readable summary of the operational fields for a given override type."""
    t = override.get("type")
    if t == "suppress":
        return f"type=suppress track={override.get('track')} ip={override.get('ip')}"
    if t == "threshold":
        return (f"type=threshold track={override.get('track')} "
                f"thresholdType={override.get('thresholdType')} "
                f"count={override.get('count')} seconds={override.get('seconds')}")
    if t == "modify":
        return f"type=modify regex={override.get('regex')!r}"
 def collect_custom_vars(override):
    found = set()
    for value in override.values():
        if isinstance(value, str):
            for match in VAR_PATTERN.findall(value):
                if match not in BUILTIN_SURICATA_VARS:
                    found.add(match)
    return found
 def parse_args():
    p = argparse.ArgumentParser(
        description="Import detection overrides into the so-detection index.",
    )
    p.add_argument("--source", "-s", required=True,
                   help="Source directory containing <publicId>.<ext> override files.")
    p.add_argument("--engine", "-e", default="suricata", choices=list(ENGINES.keys()),
                   help="Detection engine (default: suricata).")
    p.add_argument("--dry-run", "-n", action="store_true",
                   help="Print what would happen without writing to Elasticsearch.")
    p.add_argument("--no-import-note", action="store_true",
                   help="Do not prepend '[Imported YYYY-MM-DD] ' to the override note.")
    p.add_argument("--index", "-i", default=DEFAULT_INDEX,
                   help=f"Elasticsearch index to update (default: {DEFAULT_INDEX}).")
    return p.parse_args()
 def confirm_proceed(args):
    """Show the stale-backup warning. Dry-run prints it and continues. Real
    runs require the user typing 'yes' at the prompt."""
    print(STALE_WARNING)
    if args.dry_run:
        print("(dry-run: no acknowledgement required)\n")
        return True
    answer = input("Type 'yes' to acknowledge and continue: ").strip().lower()
    print()
    return answer == "yes"
 def main():
    args = parse_args()
    if not os.path.isdir(args.source):
        print(f"ERROR: source directory not found: {args.source}", file=sys.stderr)
        sys.exit(1)
    extension = ENGINES[args.engine]
    files = sorted(f for f in os.listdir(args.source) if f.endswith(f".{extension}"))
    if not files:
        print(f"No *.{extension} files found in {args.source}")
        sys.exit(0)
    if not confirm_proceed(args):
        print("Aborted.")
        sys.exit(1)
    session = make_session(AUTH_FILE)
    today = datetime.now().strftime("%Y-%m-%d")
    note_prefix = "" if args.no_import_note else f"[Imported {today}] "
    counts = {"added": 0, "skipped_dedupe": 0, "skipped_not_found": 0, "invalid": 0, "error": 0}
    custom_vars = set()
    mode = "DRY-RUN" if args.dry_run else "IMPORT"
    print(f"[{mode}] engine={args.engine} source={args.source} index={args.index}\n")
    for filename in files:
        public_id = os.path.splitext(filename)[0]
        path = os.path.join(args.source, filename)
        print(f"{public_id}:")
        try:
            new_overrides = parse_overrides_file(path)
        except (json.JSONDecodeError, OSError) as e:
            print(f"  ERROR: could not parse {filename}: {e}")
            counts["error"] += 1
            continue
        if not new_overrides:
            print("  SKIP: empty file")
            continue
        try:
            doc_id, doc_index, existing = find_detection(session, args.index, public_id, args.engine)
        except requests.HTTPError as e:
            print(f"  ERROR: search failed: {e}")
            counts["error"] += 1
            continue
        if doc_id is None:
            print(f"  WARN: no detection found for publicId={public_id} engine={args.engine}; skipping")
            counts["skipped_not_found"] += len(new_overrides)
            continue
        existing_keys = {dedupe_key(o) for o in existing}
        merged = list(existing)
        added_this_file = 0
        for override, line_no in new_overrides:
            err = validate_override(override, args.engine)
            if err:
                print(f"  INVALID (line {line_no}): {err}")
                counts["invalid"] += 1
                continue
            custom_vars.update(collect_custom_vars(override))
            key = dedupe_key(override)
            if key in existing_keys:
                print(f"  SKIP (line {line_no}): duplicate of existing override [{describe(override)}]")
                counts["skipped_dedupe"] += 1
                continue
            if note_prefix:
                override = dict(override)
                override["note"] = note_prefix + (override.get("note") or "")
            merged.append(override)
            existing_keys.add(key)
            added_this_file += 1
            print(f"  ADD (line {line_no}): {describe(override)}")
        if added_this_file == 0:
            continue
        if args.dry_run:
            print(f"  DRY-RUN: would update {doc_index}/{doc_id} "
                  f"({len(existing)} existing → {len(merged)} total)")
            counts["added"] += added_this_file
            continue
        try:
            update_overrides(session, doc_index, doc_id, merged)
            print(f"  UPDATED {doc_index}/{doc_id} ({len(existing)} → {len(merged)})")
            counts["added"] += added_this_file
        except requests.HTTPError as e:
            print(f"  ERROR: update failed: {e}")
            counts["error"] += 1
    print()
    print("=" * 60)
    print(f"Summary ({mode}):")
    print(f"  Overrides added:           {counts['added']}")
    print(f"  Skipped (already present): {counts['skipped_dedupe']}")
    print(f"  Skipped (no detection):    {counts['skipped_not_found']}")
    print(f"  Invalid (failed checks):   {counts['invalid']}")
    print(f"  Errors:                    {counts['error']}")
    if custom_vars:
        print()
        print("WARNING: detected custom Suricata variables in imported overrides:")
        for v in sorted(custom_vars):
            print(f"  {v}")
        print("If any of these are not already defined in SOC Config (Suricata variables),")
        print("you must add them manually before the rules will function correctly.")
    sys.exit(0 if counts["error"] == 0 and counts["invalid"] == 0 else 1)
 if __name__ == "__main__":
    main()
@@ -1,588 +0,0 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 import importlib.util
 import json
 import os
 import shutil
 import sys
 import tempfile
 import unittest
 from importlib.machinery import SourceFileLoader
 from io import StringIO
 from unittest.mock import MagicMock, patch
 import requests
 # The script has no .py extension; spec_from_file_location can't auto-detect a
 # loader, so we hand it a SourceFileLoader explicitly. (load_module() is
 # deprecated in 3.14 and slated for removal in 3.15.)
 HERE = os.path.dirname(os.path.abspath(__file__))
 SCRIPT = os.path.join(HERE, "so-detections-overrides-import")
 _loader = SourceFileLoader("so_overrides_import", SCRIPT)
 _spec = importlib.util.spec_from_loader("so_overrides_import", _loader)
 soi = importlib.util.module_from_spec(_spec)
 _loader.exec_module(soi)
 class TestValidateSuppress(unittest.TestCase):
    def test_valid(self):
        self.assertIsNone(soi.validate_override(
            {"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}, "suricata"))
    def test_valid_var(self):
        self.assertIsNone(soi.validate_override(
            {"type": "suppress", "track": "by_either", "ip": "$HOME_NET"}, "suricata"))
    def test_valid_cidr(self):
        self.assertIsNone(soi.validate_override(
            {"type": "suppress", "track": "by_dst", "ip": "10.0.0.0/8"}, "suricata"))
    def test_valid_bracket_list(self):
        self.assertIsNone(soi.validate_override(
            {"type": "suppress", "track": "by_src", "ip": "[1.2.3.4,10.0.0.0/8]"}, "suricata"))
    def test_missing_ip(self):
        err = soi.validate_override({"type": "suppress", "track": "by_src"}, "suricata")
        self.assertIn("requires", err)
    def test_missing_track(self):
        err = soi.validate_override({"type": "suppress", "ip": "1.2.3.4"}, "suricata")
        self.assertIn("requires", err)
    def test_invalid_track(self):
        err = soi.validate_override(
            {"type": "suppress", "track": "by_both", "ip": "1.2.3.4"}, "suricata")
        self.assertIn("invalid track", err)
    def test_invalid_ip(self):
        err = soi.validate_override(
            {"type": "suppress", "track": "by_src", "ip": "not-an-ip"}, "suricata")
        self.assertIn("invalid IP", err)
    def test_unnecessary_field(self):
        err = soi.validate_override(
            {"type": "suppress", "track": "by_src", "ip": "1.2.3.4", "count": 5}, "suricata")
        self.assertIn("unnecessary fields", err)
 class TestValidateThreshold(unittest.TestCase):
    def test_valid(self):
        self.assertIsNone(soi.validate_override({
            "type": "threshold", "track": "by_src",
            "thresholdType": "limit", "count": 10, "seconds": 60,
        }, "suricata"))
    def test_valid_by_both(self):
        self.assertIsNone(soi.validate_override({
            "type": "threshold", "track": "by_both",
            "thresholdType": "both", "count": 1, "seconds": 1,
        }, "suricata"))
    def test_track_by_either_invalid(self):
        err = soi.validate_override({
            "type": "threshold", "track": "by_either",
            "thresholdType": "limit", "count": 10, "seconds": 60,
        }, "suricata")
        self.assertIn("invalid track", err)
    def test_invalid_threshold_type(self):
        err = soi.validate_override({
            "type": "threshold", "track": "by_src",
            "thresholdType": "bogus", "count": 10, "seconds": 60,
        }, "suricata")
        self.assertIn("invalid thresholdType", err)
    def test_zero_count(self):
        err = soi.validate_override({
            "type": "threshold", "track": "by_src",
            "thresholdType": "limit", "count": 0, "seconds": 60,
        }, "suricata")
        self.assertIn("count", err)
    def test_negative_seconds(self):
        err = soi.validate_override({
            "type": "threshold", "track": "by_src",
            "thresholdType": "limit", "count": 10, "seconds": -1,
        }, "suricata")
        self.assertIn("seconds", err)
    def test_missing_field(self):
        err = soi.validate_override({
            "type": "threshold", "track": "by_src",
            "thresholdType": "limit", "count": 10,  # missing seconds
        }, "suricata")
        self.assertIn("requires", err)
    def test_unnecessary_field(self):
        err = soi.validate_override({
            "type": "threshold", "track": "by_src",
            "thresholdType": "limit", "count": 10, "seconds": 60,
            "regex": "foo",
        }, "suricata")
        self.assertIn("unnecessary fields", err)
 class TestValidateModify(unittest.TestCase):
    def test_valid(self):
        self.assertIsNone(soi.validate_override(
            {"type": "modify", "regex": r"content:\"foo\"", "value": "content:bar"}, "suricata"))
    def test_invalid_regex(self):
        err = soi.validate_override(
            {"type": "modify", "regex": "(unbalanced", "value": "x"}, "suricata")
        self.assertIn("invalid regex", err)
    def test_missing_value(self):
        err = soi.validate_override({"type": "modify", "regex": "x"}, "suricata")
        self.assertIn("requires", err)
    def test_unnecessary_field(self):
        err = soi.validate_override(
            {"type": "modify", "regex": "x", "value": "y", "track": "by_src"}, "suricata")
        self.assertIn("unnecessary fields", err)
 class TestValidateMisc(unittest.TestCase):
    def test_unknown_type(self):
        err = soi.validate_override({"type": "suppresss", "track": "by_src", "ip": "1.2.3.4"}, "suricata")
        self.assertIn("invalid type", err)
    def test_missing_type(self):
        err = soi.validate_override({"track": "by_src"}, "suricata")
        self.assertIn("type is required", err)
 class TestValidateIP(unittest.TestCase):
    def test_plain_ipv4(self):
        self.assertIsNone(soi._validate_suricata_ip("1.2.3.4"))
    def test_plain_ipv6(self):
        self.assertIsNone(soi._validate_suricata_ip("::1"))
    def test_cidr(self):
        self.assertIsNone(soi._validate_suricata_ip("10.0.0.0/8"))
    def test_var(self):
        self.assertIsNone(soi._validate_suricata_ip("$CONCOURSEWORKERS"))
    def test_bracket_list(self):
        self.assertIsNone(soi._validate_suricata_ip("[1.2.3.4, 10.0.0.0/8]"))
    def test_bracket_list_bad_member(self):
        err = soi._validate_suricata_ip("[1.2.3.4,nope]")
        self.assertIn("invalid IP in list", err)
    def test_empty(self):
        self.assertIn("empty", soi._validate_suricata_ip(""))
    def test_invalid(self):
        self.assertIn("invalid", soi._validate_suricata_ip("999.999.999.999"))
 class TestDedupeKey(unittest.TestCase):
    def test_suppress(self):
        a = {"type": "suppress", "track": "by_src", "ip": "1.2.3.4", "count": 99}
        b = {"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}
        # count is irrelevant for suppress dedupe
        self.assertEqual(soi.dedupe_key(a), soi.dedupe_key(b))
    def test_suppress_differs_on_ip(self):
        a = {"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}
        b = {"type": "suppress", "track": "by_src", "ip": "5.6.7.8"}
        self.assertNotEqual(soi.dedupe_key(a), soi.dedupe_key(b))
    def test_threshold(self):
        a = {"type": "threshold", "track": "by_src", "thresholdType": "limit",
             "count": 10, "seconds": 60, "ip": "ignored"}
        b = {"type": "threshold", "track": "by_src", "thresholdType": "limit",
             "count": 10, "seconds": 60}
        self.assertEqual(soi.dedupe_key(a), soi.dedupe_key(b))
    def test_threshold_differs_on_count(self):
        a = {"type": "threshold", "track": "by_src", "thresholdType": "limit",
             "count": 10, "seconds": 60}
        b = {"type": "threshold", "track": "by_src", "thresholdType": "limit",
             "count": 20, "seconds": 60}
        self.assertNotEqual(soi.dedupe_key(a), soi.dedupe_key(b))
    def test_modify(self):
        a = {"type": "modify", "regex": "x", "value": "y"}
        b = {"type": "modify", "regex": "x", "value": "y"}
        self.assertEqual(soi.dedupe_key(a), soi.dedupe_key(b))
 class TestDescribe(unittest.TestCase):
    def test_suppress(self):
        s = soi.describe({"type": "suppress", "track": "by_src", "ip": "1.2.3.4"})
        self.assertIn("suppress", s)
        self.assertIn("by_src", s)
        self.assertIn("1.2.3.4", s)
    def test_threshold_includes_count(self):
        s = soi.describe({"type": "threshold", "track": "by_src",
                          "thresholdType": "limit", "count": 10, "seconds": 60})
        self.assertIn("count=10", s)
        self.assertIn("seconds=60", s)
    def test_modify(self):
        s = soi.describe({"type": "modify", "regex": "foo"})
        self.assertIn("modify", s)
        self.assertIn("foo", s)
 class TestParseOverridesFile(unittest.TestCase):
    def _write(self, content):
        fd, path = tempfile.mkstemp(suffix=".txt")
        os.close(fd)
        with open(path, "w") as f:
            f.write(content)
        self.addCleanup(os.unlink, path)
        return path
    def test_single_line(self):
        path = self._write('{"type":"suppress","track":"by_src","ip":"1.2.3.4"}')
        result = soi.parse_overrides_file(path)
        self.assertEqual(len(result), 1)
        self.assertEqual(result[0][0]["type"], "suppress")
        self.assertEqual(result[0][1], 1)
    def test_ndjson(self):
        path = self._write(
            '{"type":"suppress","track":"by_src","ip":"1.2.3.4"}\n'
            '{"type":"suppress","track":"by_dst","ip":"5.6.7.8"}\n'
        )
        result = soi.parse_overrides_file(path)
        self.assertEqual(len(result), 2)
        self.assertEqual(result[1][1], 2)
    def test_empty(self):
        path = self._write("")
        self.assertEqual(soi.parse_overrides_file(path), [])
    def test_blank_lines_skipped(self):
        path = self._write('\n{"type":"suppress","track":"by_src","ip":"1.2.3.4"}\n\n')
        result = soi.parse_overrides_file(path)
        self.assertEqual(len(result), 1)
        self.assertEqual(result[0][1], 2)  # line number reflects original position
    def test_invalid_raises(self):
        path = self._write("not json")
        with self.assertRaises(json.JSONDecodeError):
            soi.parse_overrides_file(path)
 class TestCollectCustomVars(unittest.TestCase):
    def test_finds_custom(self):
        v = soi.collect_custom_vars({"ip": "$CONCOURSEWORKERS"})
        self.assertEqual(v, {"$CONCOURSEWORKERS"})
    def test_filters_builtins(self):
        v = soi.collect_custom_vars({"ip": "$HOME_NET"})
        self.assertEqual(v, set())
    def test_mixed(self):
        v = soi.collect_custom_vars({"ip": "[$HOME_NET,$MYNET]"})
        self.assertEqual(v, {"$MYNET"})
    def test_non_string_fields_ignored(self):
        v = soi.collect_custom_vars({"count": 10, "isEnabled": True})
        self.assertEqual(v, set())
 class TestMakeSession(unittest.TestCase):
    def _write(self, content):
        fd, path = tempfile.mkstemp()
        os.close(fd)
        with open(path, "w") as f:
            f.write(content)
        self.addCleanup(os.unlink, path)
        return path
    def test_valid_auth_file(self):
        path = self._write('user = "admin:secret"\n')
        session = soi.make_session(path)
        self.assertEqual(session.auth.username, "admin")
        self.assertEqual(session.auth.password, "secret")
        self.assertFalse(session.verify)
    def test_missing_user_line(self):
        path = self._write("# no user line here\n")
        with self.assertRaises(RuntimeError):
            soi.make_session(path)
 class TestFindDetection(unittest.TestCase):
    def _session_with_response(self, payload):
        session = MagicMock()
        response = MagicMock()
        response.json.return_value = payload
        response.raise_for_status.return_value = None
        session.get.return_value = response
        return session
    def test_found(self):
        session = self._session_with_response({"hits": {"hits": [{
            "_id": "abc", "_index": "so-detection",
            "_source": {"so_detection": {"overrides": [{"type": "suppress"}]}},
        }]}})
        doc_id, idx, existing = soi.find_detection(session, "so-detection", "2049201", "suricata")
        self.assertEqual(doc_id, "abc")
        self.assertEqual(idx, "so-detection")
        self.assertEqual(len(existing), 1)
    def test_not_found(self):
        session = self._session_with_response({"hits": {"hits": []}})
        doc_id, idx, existing = soi.find_detection(session, "so-detection", "x", "suricata")
        self.assertIsNone(doc_id)
        self.assertIsNone(idx)
        self.assertIsNone(existing)
    def test_no_overrides_field(self):
        session = self._session_with_response({"hits": {"hits": [{
            "_id": "abc", "_index": "so-detection",
            "_source": {"so_detection": {}},
        }]}})
        _, _, existing = soi.find_detection(session, "so-detection", "x", "suricata")
        self.assertEqual(existing, [])
    def test_multiple_hits_warns(self):
        session = self._session_with_response({"hits": {"hits": [
            {"_id": "a", "_index": "i", "_source": {"so_detection": {"overrides": []}}},
            {"_id": "b", "_index": "i", "_source": {"so_detection": {"overrides": []}}},
        ]}})
        with patch("sys.stdout", new=StringIO()) as out:
            doc_id, _, _ = soi.find_detection(session, "i", "x", "suricata")
        self.assertEqual(doc_id, "a")
        self.assertIn("WARN", out.getvalue())
 class TestUpdateOverrides(unittest.TestCase):
    def test_posts_to_update_endpoint(self):
        session = MagicMock()
        response = MagicMock()
        response.raise_for_status.return_value = None
        response.json.return_value = {"result": "updated"}
        session.post.return_value = response
        result = soi.update_overrides(session, "so-detection", "abc", [{"type": "suppress"}])
        self.assertEqual(result, {"result": "updated"})
        url = session.post.call_args[0][0]
        self.assertIn("/_update/abc", url)
        body = session.post.call_args[1]["json"]
        self.assertEqual(body["doc"]["so_detection"]["overrides"], [{"type": "suppress"}])
 class TestConfirmProceed(unittest.TestCase):
    def test_dry_run_skips_prompt(self):
        args = MagicMock(dry_run=True)
        with patch("sys.stdout", new=StringIO()):
            self.assertTrue(soi.confirm_proceed(args))
    def test_yes_input(self):
        args = MagicMock(dry_run=False)
        with patch("sys.stdout", new=StringIO()):
            with patch("builtins.input", return_value="yes"):
                self.assertTrue(soi.confirm_proceed(args))
    def test_yes_input_case_insensitive(self):
        args = MagicMock(dry_run=False)
        with patch("sys.stdout", new=StringIO()):
            with patch("builtins.input", return_value="YES"):
                self.assertTrue(soi.confirm_proceed(args))
    def test_no_input_aborts(self):
        args = MagicMock(dry_run=False)
        with patch("sys.stdout", new=StringIO()):
            with patch("builtins.input", return_value="no"):
                self.assertFalse(soi.confirm_proceed(args))
    def test_empty_input_aborts(self):
        args = MagicMock(dry_run=False)
        with patch("sys.stdout", new=StringIO()):
            with patch("builtins.input", return_value=""):
                self.assertFalse(soi.confirm_proceed(args))
 class TestParseArgs(unittest.TestCase):
    def test_defaults(self):
        with patch.object(sys, "argv", ["cmd", "--source", "/some/path"]):
            args = soi.parse_args()
        self.assertEqual(args.source, "/some/path")
        self.assertEqual(args.engine, "suricata")
        self.assertFalse(args.dry_run)
        self.assertFalse(args.no_import_note)
        self.assertEqual(args.index, soi.DEFAULT_INDEX)
    def test_all_options(self):
        argv = ["cmd", "-s", "/x", "-e", "suricata", "-n",
                "--no-import-note", "-i", "alt-index"]
        with patch.object(sys, "argv", argv):
            args = soi.parse_args()
        self.assertEqual(args.source, "/x")
        self.assertTrue(args.dry_run)
        self.assertTrue(args.no_import_note)
        self.assertEqual(args.index, "alt-index")
 class TestMain(unittest.TestCase):
    def setUp(self):
        self.tmpdir = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
        # Stub make_session so tests don't need /opt/so/conf/elasticsearch/curl.config.
        p = patch.object(soi, "make_session", return_value=MagicMock())
        p.start()
        self.addCleanup(p.stop)
    def _write_file(self, public_id, overrides, ext="txt"):
        """Write an NDJSON override file. Entries may be dicts or raw strings (for malformed input)."""
        path = os.path.join(self.tmpdir, f"{public_id}.{ext}")
        with open(path, "w") as f:
            for o in overrides:
                f.write(o if isinstance(o, str) else json.dumps(o))
                f.write("\n")
        return path
    def _run_main(self, *extra_argv, input_response="yes"):
        """Run main() with stdout/stderr captured and input mocked. Returns (stdout, stderr, exit_code)."""
        argv = ["cmd", "--source", self.tmpdir, *extra_argv]
        out, err = StringIO(), StringIO()
        with patch.object(sys, "argv", argv), \
                patch("sys.stdout", new=out), \
                patch("sys.stderr", new=err), \
                patch("builtins.input", return_value=input_response):
            with self.assertRaises(SystemExit) as cm:
                soi.main()
        return out.getvalue(), err.getvalue(), cm.exception.code
    def test_source_dir_missing(self):
        argv = ["cmd", "--source", "/no/such/path/here"]
        err = StringIO()
        with patch.object(sys, "argv", argv), patch("sys.stderr", new=err):
            with self.assertRaises(SystemExit) as cm:
                soi.main()
        self.assertEqual(cm.exception.code, 1)
        self.assertIn("source directory not found", err.getvalue())
    def test_no_files_found(self):
        out, _, code = self._run_main()
        self.assertEqual(code, 0)
        self.assertIn("No *.txt files found", out)
    def test_user_aborts(self):
        self._write_file("1001", [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}])
        out, _, code = self._run_main(input_response="no")
        self.assertEqual(code, 1)
        self.assertIn("Aborted", out)
    def test_parse_error_increments_error(self):
        # Malformed JSON line — parse_overrides_file raises JSONDecodeError.
        self._write_file("1002", ["not json"])
        out, _, code = self._run_main("--dry-run")
        self.assertEqual(code, 1)  # invalid+error → non-zero
        self.assertIn("could not parse", out)
        self.assertIn("Errors:                    1", out)
    def test_empty_file_skipped(self):
        # Blank lines only — parse_overrides_file returns []; main reports "empty file" and continues.
        path = os.path.join(self.tmpdir, "1003.txt")
        with open(path, "w") as f:
            f.write("\n\n")
        out, _, code = self._run_main("--dry-run")
        self.assertEqual(code, 0)
        self.assertIn("empty file", out)
    @patch.object(soi, "find_detection")
    def test_search_http_error(self, mock_find):
        mock_find.side_effect = requests.HTTPError("boom")
        self._write_file("1004", [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}])
        out, _, code = self._run_main("--dry-run")
        self.assertEqual(code, 1)
        self.assertIn("search failed", out)
    @patch.object(soi, "find_detection")
    def test_no_detection_found(self, mock_find):
        mock_find.return_value = (None, None, None)
        self._write_file("1005", [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}])
        out, _, code = self._run_main("--dry-run")
        self.assertEqual(code, 0)
        self.assertIn("no detection found", out)
        self.assertIn("Skipped (no detection):    1", out)
    @patch.object(soi, "find_detection")
    def test_all_duplicates_no_update(self, mock_find):
        existing = [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}]
        mock_find.return_value = ("doc1", "so-detection", existing)
        self._write_file("1006", [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}])
        out, _, code = self._run_main("--dry-run")
        self.assertEqual(code, 0)
        self.assertIn("SKIP", out)
        self.assertNotIn("DRY-RUN: would update", out)  # added_this_file == 0 branch
    @patch.object(soi, "update_overrides")
    @patch.object(soi, "find_detection")
    def test_happy_path_full(self, mock_find, mock_update):
        # Exercises: ADD, dedupe SKIP, INVALID, note prefix, UPDATE, custom-vars warning, exit=1 (invalid present)
        existing = [{"type": "suppress", "track": "by_src", "ip": "9.9.9.9"}]
        mock_find.return_value = ("doc1", "so-detection", existing)
        mock_update.return_value = {"result": "updated"}
        self._write_file("1007", [
            {"type": "suppress", "track": "by_src", "ip": "1.2.3.4"},                # ADD
            {"type": "suppress", "track": "by_src", "ip": "9.9.9.9"},                # SKIP (dupe of existing)
            {"type": "suppress", "track": "bogus",  "ip": "1.2.3.4"},                # INVALID
            {"type": "suppress", "track": "by_src", "ip": "$CONCOURSEWORKERS"},      # ADD + custom var
        ])
        out, _, code = self._run_main()
        self.assertEqual(code, 1)  # one invalid -> non-zero
        mock_update.assert_called_once()
        merged = mock_update.call_args[0][3]
        self.assertEqual(len(merged), 3)  # 1 existing + 2 new
        new_notes = [o.get("note", "") for o in merged if o.get("ip") in ("1.2.3.4", "$CONCOURSEWORKERS")]
        self.assertTrue(all(n.startswith("[Imported ") for n in new_notes))
        self.assertIn("ADD", out)
        self.assertIn("SKIP", out)
        self.assertIn("INVALID", out)
        self.assertIn("UPDATED", out)
        self.assertIn("$CONCOURSEWORKERS", out)
    @patch.object(soi, "update_overrides")
    @patch.object(soi, "find_detection")
    def test_no_import_note_preserves_note(self, mock_find, mock_update):
        mock_find.return_value = ("doc1", "so-detection", [])
        mock_update.return_value = {"result": "updated"}
        self._write_file("1008", [
            {"type": "suppress", "track": "by_src", "ip": "1.2.3.4", "note": "original"},
        ])
        _, _, code = self._run_main("--no-import-note")
        self.assertEqual(code, 0)
        merged = mock_update.call_args[0][3]
        self.assertEqual(merged[0]["note"], "original")  # no prefix applied
    @patch.object(soi, "find_detection")
    def test_dry_run_skips_update(self, mock_find):
        mock_find.return_value = ("doc1", "so-detection", [])
        self._write_file("1009", [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}])
        with patch.object(soi, "update_overrides") as mock_update:
            out, _, code = self._run_main("--dry-run")
        self.assertEqual(code, 0)
        mock_update.assert_not_called()
        self.assertIn("DRY-RUN: would update", out)
    @patch.object(soi, "update_overrides")
    @patch.object(soi, "find_detection")
    def test_update_http_error(self, mock_find, mock_update):
        mock_find.return_value = ("doc1", "so-detection", [])
        mock_update.side_effect = requests.HTTPError("nope")
        self._write_file("1010", [{"type": "suppress", "track": "by_src", "ip": "1.2.3.4"}])
        out, _, code = self._run_main()
        self.assertEqual(code, 1)
        self.assertIn("update failed", out)
 if __name__ == "__main__":
    unittest.main()
@@ -0,0 +1,329 @@
 #!/usr/bin/env python3
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 """
 so-pillar-import — populate the so_pillar.* schema in so-postgres from the
 on-disk Salt pillar tree.
 Reads /opt/so/saltstack/local/pillar/, decomposes each .sls file into a
 (scope, role|minion_id, pillar_path, data) tuple, and UPSERTs it into
 so_pillar.pillar_entry. Idempotent — re-running with no SLS edits produces
 no version bumps because the audit trigger only writes a row when data
 actually changes.
 Bootstrap and mine-driven files are skipped (see EXCLUDE_BASENAMES /
 EXCLUDE_PREFIXES below). Files containing Jinja templates ({% or {{) are
 also skipped — those stay disk-authoritative and ext_pillar_first: False
 means they render before the PG overlay anyway.
 All SQL goes through `docker exec so-postgres psql` so no separate DSN
 config is required at first-install time. Designed to be called by
 salt/postgres/schema_pillar.sls (initial seed) and by salt/manager/tools/
 sbin/so-minion (per-minion sync on add/delete).
 """
 import argparse
 import json
 import os
 import shlex
 import subprocess
 import sys
 from pathlib import Path
 import yaml
 PILLAR_LOCAL_ROOT = Path("/opt/so/saltstack/local/pillar")
 PILLAR_DEFAULT_ROOT = Path("/opt/so/saltstack/default/pillar")
 DOCKER_CONTAINER = "so-postgres"
 PG_SUPERUSER = "postgres"
 PG_DATABASE = "securityonion"
 # Files that must NEVER move to Postgres. These are read by Salt before
 # Postgres is reachable, or contain renderer-time computed values (mine, etc.).
 EXCLUDE_BASENAMES = {
    "secrets.sls",
    "auth.sls",          # postgres/auth.sls bootstrap
    "top.sls",
 }
 # Filename prefixes to skip — these are renderer-time computed pillars
 # (Salt mine, file_exists guards, etc.) that have to stay on disk.
 EXCLUDE_PATH_FRAGMENTS = (
    "/elasticsearch/nodes.sls",
    "/redis/nodes.sls",
    "/kafka/nodes.sls",
    "/hypervisor/nodes.sls",
    "/logstash/nodes.sls",
    "/node_data/ips.sls",
    "/postgres/auth.sls",
    "/elasticsearch/auth.sls",
    "/kibana/secrets.sls",
 )
 def log(level, msg):
    print(f"[{level}] {msg}", file=sys.stderr)
 def is_jinja_templated(content_bytes):
    return b"{%" in content_bytes or b"{{" in content_bytes
 def classify(path):
    """Return (scope, role_name, minion_id, pillar_path) for a pillar file
    or None to skip it. role_name is None for now — the importer leaves role
    membership to the so_pillar.minion trigger and the salt/auth reactor."""
    rel_str = str(path)
    if path.name in EXCLUDE_BASENAMES:
        return None
    for frag in EXCLUDE_PATH_FRAGMENTS:
        if frag in rel_str:
            return None
    # /local/pillar/minions/<id>.sls  or  adv_<id>.sls
    if path.parent.name == "minions":
        stem = path.stem  # filename without .sls
        if stem.startswith("adv_"):
            mid = stem[4:]
            return ("minion", None, mid, f"minions.adv_{mid}")
        return ("minion", None, stem, f"minions.{stem}")
    # /local/pillar/<section>/<file>.sls
    if path.parent.parent == PILLAR_LOCAL_ROOT or path.parent.parent == PILLAR_DEFAULT_ROOT:
        section = path.parent.name
        stem = path.stem
        # Only soc_<section>.sls and adv_<section>.sls are SOC-managed pillar
        # surfaces. Other files (e.g. nodes.sls, auth.sls, *.token) are
        # either covered by EXCLUDE_PATH_FRAGMENTS or are bootstrap surfaces
        # we leave alone for now.
        if stem.startswith("soc_") or stem.startswith("adv_"):
            return ("global", None, None, f"{section}.{stem}")
        return None
    return None
 def parse_yaml_file(path):
    with open(path, "rb") as f:
        content = f.read()
    if not content.strip():
        return {}
    if is_jinja_templated(content):
        return None
    data = yaml.safe_load(content)
    if data is None:
        return {}
    if not isinstance(data, dict):
        return {"_raw": data}
    return data
 def derive_node_type(minion_id):
    """Conventional Security Onion minion ids are <host>_<role>. Take the
    last underscore-delimited token as the canonical role suffix."""
    parts = minion_id.rsplit("_", 1)
    if len(parts) == 2:
        return parts[1]
    return None
 def docker_psql(sql, *, db=PG_DATABASE, user=PG_SUPERUSER, on_error_stop=True, capture=True):
    """Run sql via docker exec ... psql. Returns stdout as str."""
    args = [
        "docker", "exec", "-i", DOCKER_CONTAINER,
        "psql", "-U", user, "-d", db, "-tA", "-q",
    ]
    if on_error_stop:
        args += ["-v", "ON_ERROR_STOP=1"]
    proc = subprocess.run(
        args, input=sql.encode(),
        capture_output=capture, check=False,
    )
    if proc.returncode != 0:
        sys.stderr.write(proc.stderr.decode(errors="replace"))
        raise RuntimeError(f"docker exec psql failed (rc={proc.returncode})")
    return proc.stdout.decode(errors="replace")
 def upsert_minion(minion_id, node_type):
    sql = (
        "INSERT INTO so_pillar.minion (minion_id, node_type) "
        f"VALUES ({pg_str(minion_id)}, {pg_str(node_type) if node_type else 'NULL'}) "
        "ON CONFLICT (minion_id) DO UPDATE SET node_type = EXCLUDED.node_type;"
    )
    docker_psql(sql)
 def delete_minion(minion_id):
    """CASCADE removes pillar_entry + role_member rows."""
    sql = f"DELETE FROM so_pillar.minion WHERE minion_id = {pg_str(minion_id)};"
    docker_psql(sql)
 def upsert_pillar_entry(scope, role_name, minion_id, pillar_path, data, reason):
    """Insert or update the row keyed by the partial unique index that
    matches scope. Audit trigger handles history; versioning trigger bumps
    version only when data changes."""
    data_json = json.dumps(data)
    role_sql = pg_str(role_name) if role_name else "NULL"
    minion_sql = pg_str(minion_id) if minion_id else "NULL"
    reason_sql = pg_str(reason)
    if scope == "global":
        conflict = "(pillar_path) WHERE scope='global'"
    elif scope == "role":
        conflict = "(role_name, pillar_path) WHERE scope='role'"
    elif scope == "minion":
        conflict = "(minion_id, pillar_path) WHERE scope='minion'"
    else:
        raise ValueError(f"unknown scope {scope!r}")
    sql = (
        "BEGIN;\n"
        f"SELECT set_config('so_pillar.change_reason', {reason_sql}, true);\n"
        f"INSERT INTO so_pillar.pillar_entry "
        f"(scope, role_name, minion_id, pillar_path, data, change_reason) "
        f"VALUES ({pg_str(scope)}, {role_sql}, {minion_sql}, {pg_str(pillar_path)}, {pg_jsonb(data_json)}, {reason_sql}) "
        f"ON CONFLICT {conflict} DO UPDATE "
        f"SET data = EXCLUDED.data, change_reason = EXCLUDED.change_reason;\n"
        "COMMIT;\n"
    )
    docker_psql(sql)
 def pg_str(s):
    """Escape a Python str for inclusion in literal SQL. Pillar content has
    already been validated as YAML; we just need standard SQL escaping."""
    if s is None:
        return "NULL"
    return "'" + str(s).replace("'", "''") + "'"
 def pg_jsonb(json_str):
    return pg_str(json_str) + "::jsonb"
 def walk_pillar_root(root, paths):
    if not root.is_dir():
        return
    for path in root.rglob("*.sls"):
        if path.is_file():
            paths.append(path)
 def import_minion(minion_id, node_type, dry_run, reason):
    """Re-import every pillar file for a single minion."""
    if not minion_id:
        raise ValueError("minion_id required for --scope minion")
    upsert_minion(minion_id, node_type)
    log("INFO", f"Upserted minion row {minion_id} (node_type={node_type})")
    targets = [
        PILLAR_LOCAL_ROOT / "minions" / f"{minion_id}.sls",
        PILLAR_LOCAL_ROOT / "minions" / f"adv_{minion_id}.sls",
    ]
    for path in targets:
        if not path.exists():
            log("INFO", f"  (no file at {path})")
            continue
        klass = classify(path)
        if not klass:
            log("INFO", f"  skip {path} (excluded)")
            continue
        scope, role, mid, pillar_path = klass
        data = parse_yaml_file(path)
        if data is None:
            log("WARN", f"  skip {path} (Jinja-templated; stays disk-only)")
            continue
        if dry_run:
            log("DRY", f"  would upsert {scope}/{pillar_path} = {len(json.dumps(data))} bytes")
            continue
        upsert_pillar_entry(scope, role, mid, pillar_path, data, reason)
        log("INFO", f"  imported {scope}/{pillar_path}")
 def import_all(dry_run, reason):
    """Walk the entire local pillar tree and import every eligible file."""
    paths = []
    walk_pillar_root(PILLAR_LOCAL_ROOT, paths)
    imported = 0
    skipped = 0
    minions_seen = set()
    for path in sorted(paths):
        klass = classify(path)
        if not klass:
            skipped += 1
            continue
        scope, role, minion_id, pillar_path = klass
        data = parse_yaml_file(path)
        if data is None:
            log("WARN", f"skip {path} (Jinja-templated; stays disk-only)")
            skipped += 1
            continue
        if scope == "minion" and minion_id not in minions_seen:
            node_type = derive_node_type(minion_id)
            if not dry_run:
                upsert_minion(minion_id, node_type)
            minions_seen.add(minion_id)
        if dry_run:
            log("DRY", f"would upsert {scope}/{pillar_path} ({len(json.dumps(data))} bytes)")
        else:
            upsert_pillar_entry(scope, role, minion_id, pillar_path, data, reason)
            log("INFO", f"imported {scope}/{pillar_path}")
        imported += 1
    log("INFO", f"done: {imported} imported, {skipped} skipped")
 def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--scope", choices=("global", "role", "minion", "all"), default="all")
    ap.add_argument("--minion-id")
    ap.add_argument("--node-type", help="override node_type for --scope minion (default: derived from minion_id)")
    ap.add_argument("--delete", action="store_true",
                    help="With --scope minion, remove the minion row (and its pillar rows via CASCADE)")
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--diff", action="store_true",
                    help="(reserved) print structural diffs vs current DB content")
    ap.add_argument("--yes", action="store_true",
                    help="Skip confirmation prompts (currently unused; reserved)")
    ap.add_argument("--reason", default="so-pillar-import",
                    help="change_reason recorded in pillar_entry_history")
    args = ap.parse_args()
    try:
        if args.scope == "minion":
            if not args.minion_id:
                ap.error("--minion-id required when --scope minion")
            if args.delete:
                if args.dry_run:
                    log("DRY", f"would delete {args.minion_id}")
                else:
                    delete_minion(args.minion_id)
                    log("INFO", f"deleted {args.minion_id}")
            else:
                node_type = args.node_type or derive_node_type(args.minion_id)
                import_minion(args.minion_id, node_type, args.dry_run, args.reason)
        elif args.scope == "all":
            import_all(args.dry_run, args.reason)
        else:
            log("ERROR", f"--scope {args.scope} not yet implemented; use --scope all or --scope minion")
            return 2
    except Exception as e:
        log("ERROR", str(e))
        return 1
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -1,231 +0,0 @@
 #!/opt/saltstack/salt/bin/python3
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 """
 so-push-drainer
 ===============
 Scheduled drainer for the active-push feature. Runs on the manager every
 drain_interval seconds (default 15) via a salt schedule in salt/salt/push_drain_schedule.sls.
 For each intent file under /opt/so/state/push_pending/*.json whose last_touch
 is older than debounce_seconds, this script:
  * concatenates the actions lists from every ready intent
  * dedupes by (state or __highstate__, tgt, tgt_type)
  * dispatches a single `salt-run state.orchestrate orch.push_batch --async`
    with the deduped actions list passed as pillar kwargs
  * deletes the contributed intent files on successful dispatch
 Reactor sls files (push_suricata, push_strelka, push_pillar) write intents
 but never dispatch directly
 """
 import fcntl
 import glob
 import json
 import logging
 import logging.handlers
 import os
 import subprocess
 import sys
 import time
 import salt.client
 PENDING_DIR = '/opt/so/state/push_pending'
 LOCK_FILE = os.path.join(PENDING_DIR, '.lock')
 LOG_FILE = '/opt/so/log/salt/so-push-drainer.log'
 HIGHSTATE_SENTINEL = '__highstate__'
 def _make_logger():
    logger = logging.getLogger('so-push-drainer')
    logger.setLevel(logging.INFO)
    if not logger.handlers:
        os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
        handler = logging.handlers.RotatingFileHandler(
            LOG_FILE, maxBytes=5 * 1024 * 1024, backupCount=3,
        )
        handler.setFormatter(logging.Formatter(
            '%(asctime)s | %(levelname)s | %(message)s',
        ))
        logger.addHandler(handler)
    return logger
 def _load_push_cfg():
    """Read the salt:auto_apply pillar subtree via salt-call. Returns a dict."""
    caller = salt.client.Caller()
    cfg = caller.cmd('pillar.get', 'salt:auto_apply', {})
    return cfg if isinstance(cfg, dict) else {}
 def _read_intent(path, log):
    try:
        with open(path, 'r') as f:
            return json.load(f)
    except (IOError, ValueError) as exc:
        log.warning('cannot read intent %s: %s', path, exc)
        return None
    except Exception:
        log.exception('unexpected error reading %s', path)
        return None
 def _dedupe_actions(actions):
    seen = set()
    deduped = []
    for action in actions:
        if not isinstance(action, dict):
            continue
        state_key = HIGHSTATE_SENTINEL if action.get('highstate') else action.get('state')
        tgt = action.get('tgt')
        tgt_type = action.get('tgt_type', 'compound')
        if not state_key or not tgt:
            continue
        key = (state_key, tgt, tgt_type)
        if key in seen:
            continue
        seen.add(key)
        deduped.append(action)
    return deduped
 def _dispatch(actions, log):
    pillar_arg = json.dumps({'actions': actions})
    cmd = [
        'salt-run',
        'state.orchestrate',
        'orch.push_batch',
        'pillar={}'.format(pillar_arg),
        '--async',
    ]
    log.info('dispatching: %s', ' '.join(cmd[:3]) + ' pillar=<{} actions>'.format(len(actions)))
    try:
        result = subprocess.run(
            cmd, check=True, capture_output=True, text=True, timeout=60,
        )
    except subprocess.CalledProcessError as exc:
        log.error('dispatch failed (rc=%s): stdout=%s stderr=%s',
                  exc.returncode, exc.stdout, exc.stderr)
        return False
    except subprocess.TimeoutExpired:
        log.error('dispatch timed out after 60s')
        return False
    except Exception:
        log.exception('dispatch raised')
        return False
    log.info('dispatch accepted: %s', (result.stdout or '').strip())
    return True
 def main():
    log = _make_logger()
    if not os.path.isdir(PENDING_DIR):
        # Nothing to do; reactors create the dir on first use.
        return 0
    try:
        push = _load_push_cfg()
    except Exception:
        log.exception('failed to read salt:auto_apply pillar; aborting drain pass')
        return 1
    if not push.get('enabled', True):
        log.debug('push disabled; exiting')
        return 0
    debounce_seconds = int(push.get('debounce_seconds', 30))
    os.makedirs(PENDING_DIR, exist_ok=True)
    lock_fd = os.open(LOCK_FILE, os.O_CREAT | os.O_RDWR, 0o644)
    try:
        fcntl.flock(lock_fd, fcntl.LOCK_EX)
        intent_files = [
            p for p in sorted(glob.glob(os.path.join(PENDING_DIR, '*.json')))
            if os.path.basename(p) != '.lock'
        ]
        if not intent_files:
            return 0
        now = time.time()
        ready = []
        skipped = 0
        broken = []
        for path in intent_files:
            intent = _read_intent(path, log)
            if not isinstance(intent, dict):
                broken.append(path)
                continue
            last_touch = intent.get('last_touch', 0)
            if now - last_touch < debounce_seconds:
                skipped += 1
                continue
            ready.append((path, intent))
        for path in broken:
            try:
                os.unlink(path)
            except OSError:
                pass
        if not ready:
            if skipped:
                log.debug('no ready intents (%d still in debounce window)', skipped)
            return 0
        combined_actions = []
        oldest_first_touch = now
        all_paths = []
        for path, intent in ready:
            combined_actions.extend(intent.get('actions', []) or [])
            first = intent.get('first_touch', now)
            if first < oldest_first_touch:
                oldest_first_touch = first
            all_paths.extend(intent.get('paths', []) or [])
        deduped = _dedupe_actions(combined_actions)
        if not deduped:
            log.warning('%d intent(s) had no usable actions; clearing', len(ready))
            for path, _ in ready:
                try:
                    os.unlink(path)
                except OSError:
                    pass
            return 0
        debounce_duration = now - oldest_first_touch
        log.info(
            'draining %d intent(s): %d action(s) after dedupe (raw=%d), '
            'debounce_duration=%.1fs, paths=%s',
            len(ready), len(deduped), len(combined_actions),
            debounce_duration, all_paths[:20],
        )
        if not _dispatch(deduped, log):
            log.warning('dispatch failed; leaving intent files in place for retry')
            return 1
        for path, _ in ready:
            try:
                os.unlink(path)
            except OSError:
                log.exception('failed to remove drained intent %s', path)
        return 0
    finally:
        try:
            fcntl.flock(lock_fd, fcntl.LOCK_UN)
        finally:
            os.close(lock_fd)
 if __name__ == '__main__':
    sys.exit(main())
@@ -10,16 +10,5 @@ NOROOT=1
 set -e
 curl --retry 5 --retry-delay 60 -A "reposync/$(sync_options)" https://sigs.securityonion.net/checkup --output /tmp/checkup
 dnf reposync --norepopath -g --delete -m -c /opt/so/conf/reposync/repodownload.conf --repoid=securityonionsync --download-metadata -p /nsm/repo/
 createrepo /nsm/repo
 # The kernel repo section is deployed to repodownload.conf by the manager highstate, which
 # runs AFTER this script during soup. On the first upgrade to a kernel-aware version the
 # on-disk config still predates the section, so guard on its presence to avoid dnf's
 # "Unknown repo: 'securityonionkernelsync'" aborting the sync (set -e). The next sync after the
 # highstate deploys the section will pick it up.
 if grep -q '^\[securityonionkernelsync\]' /opt/so/conf/reposync/repodownload.conf; then
  dnf reposync --norepopath -g --delete -m -c /opt/so/conf/reposync/repodownload.conf --repoid=securityonionkernelsync --download-metadata -p /nsm/kernelrepo/
  createrepo /nsm/kernelrepo
 fi
@@ -13,6 +13,64 @@ import json
 lockFile = "/tmp/so-yaml.lock"
 # postsalt: so-yaml supports three backend modes for PG-managed pillar paths:
 #
 #   dual      — write disk + mirror to so_pillar.*. Reads from disk.
 #               Used during the migration transition when disk is still
 #               canonical and PG runs as a shadow.
 #   postgres  — write to so_pillar.* only. Reads from so_pillar.*. No disk
 #               file is touched. The end state once cutover is complete.
 #   disk      — disk only, no PG. Emergency rollback escape hatch.
 #
 # Bootstrap and mine-driven files (secrets.sls, ca/init.sls, */nodes.sls,
 # top.sls, etc.) are always handled on disk regardless of mode — those paths
 # are explicitly excluded by so_yaml_postgres.locate() raising SkipPath.
 #
 # Mode resolution: SO_YAML_BACKEND env var, then /opt/so/conf/so-yaml/mode,
 # then default 'dual' (safe upgrade behavior — flipping to 'postgres' is
 # done by schema_pillar.sls after the schema is in place and the importer
 # has run at least once).
 MODE_FILE = "/opt/so/conf/so-yaml/mode"
 VALID_MODES = ("dual", "postgres", "disk")
 DEFAULT_MODE = "dual"
 try:
    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
    import so_yaml_postgres
    _SO_YAML_PG_AVAILABLE = True
 except Exception as _exc:
    _SO_YAML_PG_AVAILABLE = False
 def _resolveBackendMode():
    env = os.environ.get("SO_YAML_BACKEND")
    if env and env in VALID_MODES:
        return env
    try:
        with open(MODE_FILE, "r") as fh:
            value = fh.read().strip()
        if value in VALID_MODES:
            return value
    except (IOError, OSError):
        pass
    return DEFAULT_MODE
 _BACKEND_MODE = _resolveBackendMode()
 def _isPgManaged(filename):
    """True when so-yaml should route this file's reads/writes through
    so_pillar.*. False for bootstrap/mine-driven files that always live on
    disk, and for arbitrary YAML paths outside the pillar tree."""
    if not _SO_YAML_PG_AVAILABLE:
        return False
    try:
        return so_yaml_postgres.is_pg_managed(filename)
    except Exception:
        return False
 def showUsage(args):
    print('Usage: {} <COMMAND> <YAML_FILE> [ARGS...]'.format(sys.argv[0]), file=sys.stderr)
@@ -25,8 +83,14 @@ def showUsage(args):
    print('    get [-r]         - Displays (to stdout) the value stored in the given key. Requires KEY arg. Use -r for raw output without YAML formatting.', file=sys.stderr)
    print('    remove           - Removes a yaml key, if it exists. Requires KEY arg.', file=sys.stderr)
    print('    replace          - Replaces (or adds) a new key and set its value. Requires KEY and VALUE args.', file=sys.stderr)
    print('    purge            - Delete the YAML file from disk and remove its rows from so_pillar.* (no KEY arg).', file=sys.stderr)
    print('    help             - Prints this usage information.', file=sys.stderr)
    print('', file=sys.stderr)
    print('  Backend mode:', file=sys.stderr)
    print('    Resolved from $SO_YAML_BACKEND, then /opt/so/conf/so-yaml/mode, default "dual".', file=sys.stderr)
    print('    Valid values: dual | postgres | disk. Bootstrap pillar files (secrets, ca, *.nodes.sls)', file=sys.stderr)
    print('    are always handled on disk regardless of mode.', file=sys.stderr)
    print('', file=sys.stderr)
    print('  Where:', file=sys.stderr)
    print('   YAML_FILE          - Path to the file that will be modified. Ex: /opt/so/conf/service/conf.yaml', file=sys.stderr)
    print('   KEY                - YAML key, does not support \' or " characters at this time. Ex: level1.level2', file=sys.stderr)
@@ -39,6 +103,24 @@ def showUsage(args):
 def loadYaml(filename):
    """Load a YAML file's content as a dict.
    PG-canonical mode (`postgres`): for PG-managed paths, read from
    so_pillar.pillar_entry. A missing row is treated as an empty dict so
    that `replace`/`add` on a fresh path can populate it from scratch.
    Other modes / non-PG-managed paths: read from disk as today.
    """
    if _BACKEND_MODE == "postgres" and _isPgManaged(filename):
        try:
            data = so_yaml_postgres.read_yaml(filename)
        except so_yaml_postgres.SkipPath:
            data = None
        except Exception as e:
            print(f"so-yaml: pg read failed for {filename}: {e}", file=sys.stderr)
            sys.exit(1)
        return data if data is not None else {}
    try:
        with open(filename, "r") as file:
            content = file.read()
@@ -52,8 +134,97 @@ def loadYaml(filename):
 def writeYaml(filename, content):
    """Persist `content` for `filename`.
    PG-canonical mode + PG-managed path: write only to so_pillar.*. A PG
    failure is fatal (no disk fallback) — caller must retry.
    Dual mode: write disk, then mirror to PG (failures are warnings).
    Disk mode or non-PG-managed path: write disk only.
    """
    if _BACKEND_MODE == "postgres" and _isPgManaged(filename):
        if not _SO_YAML_PG_AVAILABLE:
            print("so-yaml: PG-canonical mode requires so_yaml_postgres module", file=sys.stderr)
            sys.exit(1)
        ok, msg = so_yaml_postgres.write_yaml(
            filename, content,
            reason="so-yaml " + " ".join(sys.argv[1:2]))
        if not ok:
            print(f"so-yaml: pg write failed for {filename}: {msg}", file=sys.stderr)
            sys.exit(1)
        return None
    file = open(filename, "w")
-    return yaml.safe_dump(content, file)
+    result = yaml.safe_dump(content, file)
    file.close()
    if _BACKEND_MODE == "dual":
        _mirrorToPostgres(filename, content)
    return result
 def _mirrorToPostgres(filename, content):
    """Best-effort dual-write of a YAML mutation into so_pillar.*. Skips
    files outside the PG-managed pillar surface (secrets.sls,
    elasticsearch/nodes.sls, etc.) and silently degrades when so-postgres
    is unreachable. Disk write is canonical in dual mode; this never
    raises.
    Only real PG failures (`pg write failed: ...`) are logged so the
    common cases (skipped path, postgres not running) don't pollute
    stderr."""
    if not _SO_YAML_PG_AVAILABLE:
        return
    try:
        ok, msg = so_yaml_postgres.write_yaml(filename, content,
                                              reason="so-yaml " + " ".join(sys.argv[1:2]))
        if not ok and msg.startswith("pg write failed"):
            print(f"so-yaml: {msg}", file=sys.stderr)
    except Exception as e:  # pragma: no cover — defensive: never break disk write
        print(f"so-yaml: pg mirror exception: {e}", file=sys.stderr)
 def purgeFile(filename):
    """Delete a YAML file from disk and remove the matching rows from
    so_pillar.*. Idempotent — missing file/row counts as success.
    PG-canonical mode + PG-managed path: PG delete is canonical. If a stale
    disk file from the dual-write era happens to still exist, it's removed
    too as a cleanup courtesy. PG failure is fatal in this mode.
    Dual / disk modes: remove disk first; PG cleanup is best-effort."""
    if _BACKEND_MODE == "postgres" and _isPgManaged(filename):
        if not _SO_YAML_PG_AVAILABLE:
            print("so-yaml: PG-canonical mode requires so_yaml_postgres module", file=sys.stderr)
            return 1
        ok, msg = so_yaml_postgres.purge_yaml(filename, reason="so-yaml purge")
        if not ok:
            print(f"so-yaml: pg purge failed for {filename}: {msg}", file=sys.stderr)
            return 1
        if os.path.exists(filename):
            try:
                os.remove(filename)
            except Exception as e:
                print(f"so-yaml: warn — could not remove stale disk file {filename}: {e}", file=sys.stderr)
        return 0
    if os.path.exists(filename):
        try:
            os.remove(filename)
        except Exception as e:
            print(f"Failed to remove {filename}: {e}", file=sys.stderr)
            return 1
    if _BACKEND_MODE == "dual" and _SO_YAML_PG_AVAILABLE:
        try:
            ok, msg = so_yaml_postgres.purge_yaml(filename,
                                                  reason="so-yaml purge")
            if not ok and msg.startswith("pg purge failed"):
                print(f"so-yaml: {msg}", file=sys.stderr)
        except Exception as e:
            print(f"so-yaml: pg purge exception: {e}", file=sys.stderr)
    return 0
 def appendItem(content, key, listItem):
@@ -371,6 +542,18 @@ def get(args):
    return 0
 def purge(args):
    """purge YAML_FILE — delete the file from disk and remove the matching
    rows from so_pillar.* in so-postgres. Used by so-minion's delete path
    (in place of `rm -f`) so the audit log captures the deletion and
    role_member rows get cleaned up via FK CASCADE on so_pillar.minion."""
    if len(args) != 1:
        print('Missing filename arg', file=sys.stderr)
        showUsage(None)
        return 1
    return purgeFile(args[0])
 def main():
    args = sys.argv[1:]
@@ -388,6 +571,7 @@ def main():
        "get": get,
        "remove": remove,
        "replace": replace,
        "purge": purge,
    }
    code = 1
@@ -991,3 +991,329 @@ class TestLoadYaml(unittest.TestCase):
                    soyaml.loadYaml("/tmp/so-yaml_test-unreadable.yaml")
                    sysmock.assert_called_with(1)
                    self.assertIn("Error reading file", mock_stderr.getvalue())
 class TestPurge(unittest.TestCase):
    def test_purge_missing_arg(self):
        # showUsage calls sys.exit(1); patch it like the other tests do.
        with patch('sys.exit', new=MagicMock()):
            with patch('sys.stderr', new=StringIO()) as mock_stderr:
                rc = soyaml.purge([])
                self.assertEqual(rc, 1)
                self.assertIn("Missing filename", mock_stderr.getvalue())
    def test_purge_existing_file(self):
        filename = "/tmp/so-yaml_test_purge.yaml"
        with open(filename, "w") as f:
            f.write("key: value\n")
        # Disable PG mirror so the test doesn't shell out to docker.
        with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', False):
            rc = soyaml.purge([filename])
        self.assertEqual(rc, 0)
        import os as _os
        self.assertFalse(_os.path.exists(filename))
    def test_purge_missing_file_idempotent(self):
        filename = "/tmp/so-yaml_test_purge_missing.yaml"
        import os as _os
        if _os.path.exists(filename):
            _os.remove(filename)
        with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', False):
            rc = soyaml.purge([filename])
        self.assertEqual(rc, 0)
 class TestSoYamlPostgres(unittest.TestCase):
    """Tests the path-locator and write/purge contract of the dual-write
    backend module without actually contacting Postgres."""
    def setUp(self):
        import importlib
        self.mod = importlib.import_module("so_yaml_postgres")
    def test_locate_global_soc(self):
        scope, role, mid, path = self.mod.locate(
            "/opt/so/saltstack/local/pillar/soc/soc_soc.sls")
        self.assertEqual(scope, "global")
        self.assertIsNone(role)
        self.assertIsNone(mid)
        self.assertEqual(path, "soc.soc_soc")
    def test_locate_global_advanced(self):
        scope, role, mid, path = self.mod.locate(
            "/opt/so/saltstack/local/pillar/soc/adv_soc.sls")
        self.assertEqual(scope, "global")
        self.assertEqual(path, "soc.adv_soc")
    def test_locate_minion(self):
        scope, role, mid, path = self.mod.locate(
            "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls")
        self.assertEqual(scope, "minion")
        self.assertEqual(mid, "h1_sensor")
        self.assertEqual(path, "minions.h1_sensor")
    def test_locate_minion_advanced(self):
        scope, role, mid, path = self.mod.locate(
            "/opt/so/saltstack/local/pillar/minions/adv_h1_sensor.sls")
        self.assertEqual(scope, "minion")
        self.assertEqual(mid, "h1_sensor")
        self.assertEqual(path, "minions.adv_h1_sensor")
    def test_locate_skip_secrets(self):
        with self.assertRaises(self.mod.SkipPath):
            self.mod.locate("/opt/so/saltstack/local/pillar/secrets.sls")
    def test_locate_skip_postgres_auth(self):
        with self.assertRaises(self.mod.SkipPath):
            self.mod.locate("/opt/so/saltstack/local/pillar/postgres/auth.sls")
    def test_locate_skip_mine_driven(self):
        with self.assertRaises(self.mod.SkipPath):
            self.mod.locate("/opt/so/saltstack/local/pillar/elasticsearch/nodes.sls")
    def test_locate_skip_top(self):
        with self.assertRaises(self.mod.SkipPath):
            self.mod.locate("/opt/so/saltstack/local/pillar/top.sls")
    def test_locate_skip_unrelated(self):
        with self.assertRaises(self.mod.SkipPath):
            self.mod.locate("/etc/hostname")
    def test_pg_str_escapes(self):
        self.assertEqual(self.mod._pg_str("a'b"), "'a''b'")
        self.assertEqual(self.mod._pg_str(None), "NULL")
    def test_conflict_target(self):
        self.assertIn("scope='global'", self.mod._conflict_target("global"))
        self.assertIn("scope='role'", self.mod._conflict_target("role"))
        self.assertIn("scope='minion'", self.mod._conflict_target("minion"))
        with self.assertRaises(ValueError):
            self.mod._conflict_target("bogus")
    def test_write_yaml_skips_disk_only_path(self):
        with patch.object(self.mod, '_is_enabled', return_value=True):
            ok, msg = self.mod.write_yaml(
                "/opt/so/saltstack/local/pillar/secrets.sls",
                {"secrets": {"foo": "bar"}})
        self.assertFalse(ok)
        self.assertIn("disk-only", msg)
    def test_write_yaml_unreachable(self):
        with patch.object(self.mod, '_is_enabled', return_value=False):
            ok, msg = self.mod.write_yaml(
                "/opt/so/saltstack/local/pillar/soc/soc_soc.sls",
                {"soc": {"foo": "bar"}})
        self.assertFalse(ok)
        self.assertEqual(msg, "postgres unreachable")
    def test_is_pg_managed_true(self):
        self.assertTrue(self.mod.is_pg_managed(
            "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls"))
        self.assertTrue(self.mod.is_pg_managed(
            "/opt/so/saltstack/local/pillar/soc/soc_soc.sls"))
    def test_is_pg_managed_false_for_bootstrap(self):
        self.assertFalse(self.mod.is_pg_managed(
            "/opt/so/saltstack/local/pillar/secrets.sls"))
        self.assertFalse(self.mod.is_pg_managed(
            "/opt/so/saltstack/local/pillar/postgres/auth.sls"))
        self.assertFalse(self.mod.is_pg_managed(
            "/opt/so/saltstack/local/pillar/elasticsearch/nodes.sls"))
    def test_read_yaml_unreachable(self):
        with patch.object(self.mod, '_is_enabled', return_value=False):
            self.assertIsNone(self.mod.read_yaml(
                "/opt/so/saltstack/local/pillar/soc/soc_soc.sls"))
    def test_read_yaml_skips_disk_only(self):
        with patch.object(self.mod, '_is_enabled', return_value=True):
            with self.assertRaises(self.mod.SkipPath):
                self.mod.read_yaml(
                    "/opt/so/saltstack/local/pillar/secrets.sls")
    def test_read_yaml_returns_data(self):
        with patch.object(self.mod, '_is_enabled', return_value=True):
            with patch.object(self.mod, '_docker_psql',
                              return_value='{"soc": {"foo": "bar"}}\n'):
                data = self.mod.read_yaml(
                    "/opt/so/saltstack/local/pillar/soc/soc_soc.sls")
        self.assertEqual(data, {"soc": {"foo": "bar"}})
    def test_read_yaml_returns_none_when_no_row(self):
        with patch.object(self.mod, '_is_enabled', return_value=True):
            with patch.object(self.mod, '_docker_psql', return_value=''):
                data = self.mod.read_yaml(
                    "/opt/so/saltstack/local/pillar/soc/soc_soc.sls")
        self.assertIsNone(data)
    def test_read_yaml_minion_query_shape(self):
        captured = {}
        def fake_psql(sql):
            captured['sql'] = sql
            return '{"host": {"mainip": "10.0.0.1"}}'
        with patch.object(self.mod, '_is_enabled', return_value=True):
            with patch.object(self.mod, '_docker_psql', side_effect=fake_psql):
                data = self.mod.read_yaml(
                    "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls")
        self.assertEqual(data, {"host": {"mainip": "10.0.0.1"}})
        self.assertIn("scope='minion'", captured['sql'])
        self.assertIn("'h1_sensor'", captured['sql'])
        self.assertIn("'minions.h1_sensor'", captured['sql'])
    def test_is_enabled_public_alias(self):
        with patch.object(self.mod, '_is_enabled', return_value=True):
            self.assertTrue(self.mod.is_enabled())
        with patch.object(self.mod, '_is_enabled', return_value=False):
            self.assertFalse(self.mod.is_enabled())
 class TestSoYamlBackendMode(unittest.TestCase):
    """Tests so-yaml's backend-mode resolution and PG-canonical routing
    for read/write/purge. The PG calls themselves are stubbed; what we're
    asserting is that the right backend is chosen for each (mode, path)
    combination."""
    def test_resolve_mode_env_overrides_file(self):
        with patch.dict('os.environ', {'SO_YAML_BACKEND': 'postgres'}):
            self.assertEqual(soyaml._resolveBackendMode(), 'postgres')
        with patch.dict('os.environ', {'SO_YAML_BACKEND': 'disk'}):
            self.assertEqual(soyaml._resolveBackendMode(), 'disk')
    def test_resolve_mode_invalid_env_falls_back(self):
        with patch.dict('os.environ', {'SO_YAML_BACKEND': 'garbage'}, clear=False):
            with patch('builtins.open', side_effect=IOError):
                self.assertEqual(soyaml._resolveBackendMode(), 'dual')
    def test_resolve_mode_default_dual(self):
        env = {k: v for k, v in __import__('os').environ.items()
               if k != 'SO_YAML_BACKEND'}
        with patch.dict('os.environ', env, clear=True):
            with patch('builtins.open', side_effect=IOError):
                self.assertEqual(soyaml._resolveBackendMode(), 'dual')
    def test_is_pg_managed_proxies(self):
        with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
            self.assertTrue(soyaml._isPgManaged(
                "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls"))
            self.assertFalse(soyaml._isPgManaged(
                "/opt/so/saltstack/local/pillar/secrets.sls"))
    def test_is_pg_managed_false_when_module_unavailable(self):
        with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', False):
            self.assertFalse(soyaml._isPgManaged(
                "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls"))
    def test_load_yaml_postgres_mode_reads_pg(self):
        with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
            with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                with patch.object(soyaml.so_yaml_postgres, 'is_pg_managed',
                                  return_value=True):
                    with patch.object(soyaml.so_yaml_postgres, 'read_yaml',
                                      return_value={"a": 1}):
                        result = soyaml.loadYaml(
                            "/opt/so/saltstack/local/pillar/soc/soc_soc.sls")
        self.assertEqual(result, {"a": 1})
    def test_load_yaml_postgres_mode_returns_empty_when_no_row(self):
        with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
            with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                with patch.object(soyaml.so_yaml_postgres, 'is_pg_managed',
                                  return_value=True):
                    with patch.object(soyaml.so_yaml_postgres, 'read_yaml',
                                      return_value=None):
                        result = soyaml.loadYaml(
                            "/opt/so/saltstack/local/pillar/soc/soc_soc.sls")
        self.assertEqual(result, {})
    def test_load_yaml_postgres_mode_reads_disk_for_bootstrap(self):
        import tempfile, os as _os
        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
            f.write("foo: bar\n")
            tmp = f.name
        try:
            with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
                with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                    with patch.object(soyaml.so_yaml_postgres,
                                      'is_pg_managed', return_value=False):
                        result = soyaml.loadYaml(tmp)
            self.assertEqual(result, {"foo": "bar"})
        finally:
            _os.unlink(tmp)
    def test_write_yaml_postgres_mode_skips_disk(self):
        import tempfile, os as _os
        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
            tmp = f.name
        _os.unlink(tmp)
        try:
            with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
                with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                    with patch.object(soyaml.so_yaml_postgres, 'is_pg_managed',
                                      return_value=True):
                        with patch.object(soyaml.so_yaml_postgres, 'write_yaml',
                                          return_value=(True, 'ok')) as mock_w:
                            soyaml.writeYaml(tmp, {"x": 1})
            self.assertFalse(_os.path.exists(tmp))
            mock_w.assert_called_once()
        finally:
            if _os.path.exists(tmp):
                _os.unlink(tmp)
    def test_write_yaml_postgres_mode_failure_is_fatal(self):
        with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
            with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                with patch.object(soyaml.so_yaml_postgres, 'is_pg_managed',
                                  return_value=True):
                    with patch.object(soyaml.so_yaml_postgres, 'write_yaml',
                                      return_value=(False, 'pg write failed: connection refused')):
                        with patch('sys.exit', new=MagicMock()) as sysmock:
                            with patch('sys.stderr', new=StringIO()) as mock_err:
                                soyaml.writeYaml(
                                    "/opt/so/saltstack/local/pillar/soc/soc_soc.sls",
                                    {"x": 1})
        sysmock.assert_called_with(1)
    def test_write_yaml_disk_mode_skips_pg(self):
        import tempfile, os as _os
        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
            tmp = f.name
        try:
            with patch.object(soyaml, '_BACKEND_MODE', 'disk'):
                with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                    with patch.object(soyaml.so_yaml_postgres, 'write_yaml') as mock_w:
                        soyaml.writeYaml(tmp, {"x": 1})
            mock_w.assert_not_called()
            with open(tmp) as f:
                self.assertIn('x: 1', f.read())
        finally:
            _os.unlink(tmp)
    def test_purge_postgres_mode_calls_pg_only(self):
        import tempfile, os as _os
        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
            tmp = f.name
        _os.unlink(tmp)
        with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
            with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                with patch.object(soyaml.so_yaml_postgres, 'is_pg_managed',
                                  return_value=True):
                    with patch.object(soyaml.so_yaml_postgres, 'purge_yaml',
                                      return_value=(True, 'ok')) as mock_p:
                        rc = soyaml.purgeFile(tmp)
        self.assertEqual(rc, 0)
        mock_p.assert_called_once()
    def test_purge_postgres_mode_failure_returns_nonzero(self):
        with patch.object(soyaml, '_BACKEND_MODE', 'postgres'):
            with patch.object(soyaml, '_SO_YAML_PG_AVAILABLE', True):
                with patch.object(soyaml.so_yaml_postgres, 'is_pg_managed',
                                  return_value=True):
                    with patch.object(soyaml.so_yaml_postgres, 'purge_yaml',
                                      return_value=(False, 'pg purge failed: x')):
                        with patch('sys.stderr', new=StringIO()):
                            rc = soyaml.purgeFile(
                                "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls")
        self.assertEqual(rc, 1)
@@ -0,0 +1,320 @@
 # Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
 # or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
 # https://securityonion.net/license; you may not use this file except in compliance with the
 # Elastic License 2.0.
 """
 so_yaml_postgres — Postgres-backed dual-write helpers for so-yaml.py.
 so-yaml.py writes YAML pillar files on disk; this module mirrors those
 writes into so_pillar.* in so-postgres so ext_pillar and the SOC
 PostgresConfigstore see the same data. During the postsalt transition
 disk is canonical; PG writes are best-effort and never fail the disk
 operation.
 Connection: shells out to `docker exec so-postgres psql -U postgres -d
 securityonion`. Same pattern so-pillar-import uses; avoids needing a
 separate DSN config at install time. Performance is fine because so-yaml
 is invoked from infrequent code paths (setup scripts, so-minion,
 so-firewall); SOC's hot path uses the in-process pgxpool in
 PostgresConfigstore, not so-yaml.
 Path-to-row mapping mirrors PostgresConfigstore.locateSetting in
 securityonion-soc:
  /opt/so/saltstack/local/pillar/<section>/soc_<section>.sls
        -> scope=global, pillar_path=<section>.soc_<section>
  /opt/so/saltstack/local/pillar/<section>/adv_<section>.sls
        -> scope=global, pillar_path=<section>.adv_<section>
  /opt/so/saltstack/local/pillar/minions/<id>.sls
        -> scope=minion, minion_id=<id>, pillar_path=minions.<id>
  /opt/so/saltstack/local/pillar/minions/adv_<id>.sls
        -> scope=minion, minion_id=<id>, pillar_path=minions.adv_<id>
 Files outside that mapping (notably secrets.sls, postgres/auth.sls,
 elasticsearch/nodes.sls, etc.) are skipped — they stay disk-only forever
 or render dynamically and don't belong in PG.
 """
 import json
 import os
 import shlex
 import subprocess
 import sys
 DOCKER_CONTAINER = os.environ.get("SO_PILLAR_PG_CONTAINER", "so-postgres")
 PG_DATABASE = os.environ.get("SO_PILLAR_PG_DATABASE", "securityonion")
 PG_USER = os.environ.get("SO_PILLAR_PG_USER", "postgres")
 # File paths whose mutations stay disk-only forever. Mirrors EXCLUDE_*
 # in so-pillar-import.
 DISK_ONLY_PATHS = (
    "/opt/so/saltstack/local/pillar/secrets.sls",
    "/opt/so/saltstack/local/pillar/postgres/auth.sls",
    "/opt/so/saltstack/local/pillar/elasticsearch/auth.sls",
    "/opt/so/saltstack/local/pillar/kibana/secrets.sls",
 )
 DISK_ONLY_FRAGMENTS = (
    "/elasticsearch/nodes.sls",
    "/redis/nodes.sls",
    "/kafka/nodes.sls",
    "/hypervisor/nodes.sls",
    "/logstash/nodes.sls",
    "/node_data/ips.sls",
    "/top.sls",
 )
 class SkipPath(Exception):
    """Raised when a file path is intentionally not mirrored to PG."""
 def is_enabled():
    """Public alias for callers that want to probe PG reachability without
    relying on a leading-underscore private name."""
    return _is_enabled()
 def _is_enabled():
    """PG dual-write only fires if so-postgres is reachable. Cheap probe.
    Returns True when docker exec succeeds, False otherwise. We never
    want a PG hiccup to fail a disk write on a manager whose Postgres is
    momentarily unreachable."""
    try:
        proc = subprocess.run(
            ["docker", "exec", DOCKER_CONTAINER,
             "pg_isready", "-h", "127.0.0.1", "-U", PG_USER, "-q"],
            capture_output=True, timeout=5, check=False,
        )
        return proc.returncode == 0
    except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
        return False
 def locate(path):
    """Translate a so-yaml file path to (scope, role_name, minion_id, pillar_path).
    Raises SkipPath when the file is not part of the PG-managed surface."""
    norm = os.path.normpath(path)
    if norm in DISK_ONLY_PATHS:
        raise SkipPath(f"{path}: explicit disk-only allowlist")
    for frag in DISK_ONLY_FRAGMENTS:
        if frag in norm:
            raise SkipPath(f"{path}: matches disk-only fragment {frag}")
    parent = os.path.basename(os.path.dirname(norm))
    grandparent = os.path.basename(os.path.dirname(os.path.dirname(norm)))
    name = os.path.basename(norm)
    if not name.endswith(".sls"):
        raise SkipPath(f"{path}: not a .sls file")
    stem = name[:-4]
    if parent == "minions":
        if stem.startswith("adv_"):
            mid = stem[4:]
            return ("minion", None, mid, f"minions.adv_{mid}")
        return ("minion", None, stem, f"minions.{stem}")
    # /local/pillar/<section>/<file>.sls
    if grandparent == "pillar" and parent and parent != "":
        if stem.startswith("soc_") or stem.startswith("adv_"):
            return ("global", None, None, f"{parent}.{stem}")
        raise SkipPath(f"{path}: <section>/{stem}.sls is not a soc_/adv_ file")
    raise SkipPath(f"{path}: unrecognised pillar layout")
 def _pg_str(s):
    if s is None:
        return "NULL"
    return "'" + str(s).replace("'", "''") + "'"
 def _docker_psql(sql):
    """Run sql via docker exec ... psql. Returns stdout. Caller catches
    exceptions and downgrades to a warning."""
    proc = subprocess.run(
        ["docker", "exec", "-i", DOCKER_CONTAINER,
         "psql", "-U", PG_USER, "-d", PG_DATABASE,
         "-tA", "-q", "-v", "ON_ERROR_STOP=1"],
        input=sql.encode(), capture_output=True, check=False, timeout=30,
    )
    if proc.returncode != 0:
        raise RuntimeError(proc.stderr.decode(errors="replace") or
                           f"docker exec psql exit {proc.returncode}")
    return proc.stdout.decode(errors="replace")
 def _conflict_target(scope):
    if scope == "global":
        return "(pillar_path) WHERE scope='global'"
    if scope == "role":
        return "(role_name, pillar_path) WHERE scope='role'"
    if scope == "minion":
        return "(minion_id, pillar_path) WHERE scope='minion'"
    raise ValueError(f"unknown scope {scope!r}")
 def is_pg_managed(path):
    """True if this path maps to a so_pillar.* row (locate() succeeds).
    Bootstrap and mine-driven files return False — they always live on
    disk regardless of so-yaml's backend mode."""
    try:
        locate(path)
        return True
    except SkipPath:
        return False
 def read_yaml(path):
    """Return the content dict stored in so_pillar.pillar_entry for `path`,
    or None when no row exists. Raises SkipPath when `path` is not part of
    the PG-managed surface (caller should read disk in that case).
    Used by so-yaml.py PG-canonical mode so `replace`, `get`, etc. resolve
    against the database rather than a stale (or absent) disk file."""
    if not _is_enabled():
        return None
    scope, role, minion_id, pillar_path = locate(path)
    if scope == "minion":
        sql = ("SELECT data FROM so_pillar.pillar_entry "
               "WHERE scope='minion' "
               f"AND minion_id={_pg_str(minion_id)} "
               f"AND pillar_path={_pg_str(pillar_path)}")
    elif scope == "role":
        sql = ("SELECT data FROM so_pillar.pillar_entry "
               "WHERE scope='role' "
               f"AND role_name={_pg_str(role)} "
               f"AND pillar_path={_pg_str(pillar_path)}")
    else:
        sql = ("SELECT data FROM so_pillar.pillar_entry "
               "WHERE scope='global' "
               f"AND pillar_path={_pg_str(pillar_path)}")
    try:
        out = _docker_psql(sql).strip()
    except Exception:
        return None
    if not out:
        return None
    try:
        return json.loads(out)
    except (ValueError, TypeError):
        return None
 def write_yaml(path, content_dict, *, reason="so-yaml dual-write"):
    """Mirror the disk write at `path` (whose content was just rendered as
    `content_dict`) into so_pillar.pillar_entry. Best-effort: any failure
    is swallowed so the caller (so-yaml.py) does not see it as a fatal."""
    if not _is_enabled():
        return False, "postgres unreachable"
    try:
        scope, role, minion_id, pillar_path = locate(path)
    except SkipPath as e:
        return False, str(e)
    data_json = json.dumps(content_dict if content_dict is not None else {})
    role_sql = _pg_str(role)
    minion_sql = _pg_str(minion_id)
    reason_sql = _pg_str(reason)
    conflict = _conflict_target(scope)
    sql_parts = []
    if scope == "minion":
        # FK requires the minion row before pillar_entry can reference it.
        sql_parts.append(
            f"INSERT INTO so_pillar.minion (minion_id) VALUES ({minion_sql}) "
            "ON CONFLICT (minion_id) DO NOTHING;"
        )
    sql_parts.append(
        "BEGIN;\n"
        f"SELECT set_config('so_pillar.change_reason', {reason_sql}, true);\n"
        "INSERT INTO so_pillar.pillar_entry "
        "(scope, role_name, minion_id, pillar_path, data, change_reason) "
        f"VALUES ({_pg_str(scope)}, {role_sql}, {minion_sql}, "
        f"{_pg_str(pillar_path)}, {_pg_str(data_json)}::jsonb, {reason_sql}) "
        f"ON CONFLICT {conflict} DO UPDATE "
        "SET data = EXCLUDED.data, change_reason = EXCLUDED.change_reason;\n"
        "COMMIT;\n"
    )
    try:
        _docker_psql("\n".join(sql_parts))
    except Exception as e:
        return False, f"pg write failed: {e}"
    return True, "ok"
 def purge_yaml(path, *, reason="so-yaml purge"):
    """Mirror the disk file deletion at `path` by deleting the matching
    pillar_entry rows. For minion files also deletes the so_pillar.minion
    row (CASCADE removes pillar_entry + role_member rows)."""
    if not _is_enabled():
        return False, "postgres unreachable"
    try:
        scope, role, minion_id, pillar_path = locate(path)
    except SkipPath as e:
        return False, str(e)
    reason_sql = _pg_str(reason)
    parts = ["BEGIN;",
             f"SELECT set_config('so_pillar.change_reason', {reason_sql}, true);"]
    if scope == "minion":
        # If both <id>.sls and adv_<id>.sls are gone the trigger / CASCADE
        # cleans up role_member; otherwise we just remove this one row.
        parts.append(
            f"DELETE FROM so_pillar.pillar_entry "
            f"WHERE scope='minion' AND minion_id={_pg_str(minion_id)} "
            f"AND pillar_path={_pg_str(pillar_path)};"
        )
        parts.append(
            f"DELETE FROM so_pillar.minion WHERE minion_id={_pg_str(minion_id)} "
            "AND NOT EXISTS (SELECT 1 FROM so_pillar.pillar_entry "
            f"WHERE minion_id={_pg_str(minion_id)});"
        )
    else:
        parts.append(
            f"DELETE FROM so_pillar.pillar_entry "
            f"WHERE scope={_pg_str(scope)} AND pillar_path={_pg_str(pillar_path)};"
        )
    parts.append("COMMIT;")
    try:
        _docker_psql("\n".join(parts))
    except Exception as e:
        return False, f"pg purge failed: {e}"
    return True, "ok"
 # CLI for diagnostics. Not exercised by so-yaml.py itself.
 def _main(argv):
    import argparse
    ap = argparse.ArgumentParser()
    ap.add_argument("op", choices=("locate", "ping"))
    ap.add_argument("path", nargs="?")
    args = ap.parse_args(argv)
    if args.op == "ping":
        ok = _is_enabled()
        print("ok" if ok else "unreachable")
        return 0 if ok else 1
    if args.op == "locate":
        if not args.path:
            ap.error("locate requires PATH")
        try:
            scope, role, minion_id, pillar_path = locate(args.path)
            print(f"scope={scope} role={role} minion_id={minion_id} pillar_path={pillar_path}")
            return 0
        except SkipPath as e:
            print(f"SKIP: {e}", file=sys.stderr)
            return 2
    return 1
 if __name__ == "__main__":
    sys.exit(_main(sys.argv[1:]))
@@ -12,31 +12,16 @@
 UPDATE_DIR=/tmp/sogh/securityonion
 DEFAULT_SALT_DIR=/opt/so/saltstack/default
 INSTALLEDVERSION=$(cat /etc/soversion)
 # /etc/sopostversion is a soup-owned marker (no salt state manages it) tracking how
 # far the post-upgrade walk has progressed. Its presence means a prior upgrade did
 # not finish its post-upgrade steps; its contents are the resume point. It is read
 # here before preupgrade_changes mutates INSTALLEDVERSION and before any highstate
 # stamps /etc/soversion from the pillar.
 POSTVERSION_FILE=/etc/sopostversion
 if [ -f "$POSTVERSION_FILE" ]; then
  POSTVERSION=$(cat "$POSTVERSION_FILE")
 else
 POSTVERSION=$INSTALLEDVERSION
 fi
 INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk '{print $2}')
 BATCHSIZE=5
 SOUP_LOG=/root/soup.log
 SOUP_DEBUG_LOG=/root/soup-debug.log
 WHATWOULDYOUSAYYAHDOHERE=soup
 whiptail_title='Security Onion UPdater'
 NOTIFYCUSTOMELASTICCONFIG=false
 TOPFILE=/opt/so/saltstack/default/salt/top.sls
 BACKUPTOPFILE=/opt/so/saltstack/default/salt/top.sls.backup
 SALTUPGRADED=false
 # Set true once soup begins modifying the system (past the pre-flight checks), so the
 # EXIT trap can tell the user the update did not finish and must be re-run. Only the
 # pre-flight gates (ES compatibility, disk, network) fail before this is set.
 SOUP_UPGRADE_STARTED=false
 SALT_CLOUD_INSTALLED=false
 SALT_CLOUD_CONFIGURED=false
 # Check if salt-cloud is installed
@@ -49,7 +34,6 @@ if [[ -f /etc/salt/cloud.profiles.d/socloud.conf ]]; then
 fi
 # used to display messages to the user at the end of soup
 declare -a FINAL_MESSAGE_QUEUE=()
 SOUP_ERR_CONTEXT=
 check_err() {
@@ -130,74 +114,11 @@ check_err() {
      echo "$err_msg"
    fi
    if [[ -n $SOUP_ERR_CONTEXT ]]; then
      echo ""
      printf '%s\n' "$SOUP_ERR_CONTEXT"
    fi
    echo "SOUP XTRACE debug log (if enabled) at $SOUP_DEBUG_LOG. Re-run soup with SOUP_DEBUG=1 to create $SOUP_DEBUG_LOG"
    # If soup had already started modifying the system, make it unmistakable that the
    # update is incomplete and must be re-run. soup is resumable: a version upgrade
    # picks up from the /etc/sopostversion marker, and a hotfix re-applies because
    # /etc/sohotfix is only advanced after a successful highstate.
    if [[ "$SOUP_UPGRADE_STARTED" == "true" ]]; then
      echo ""
      echo "=============================================================================="
      echo " UPGRADE INCOMPLETE"
      echo "=============================================================================="
      echo " This soup run did NOT finish. Your Security Onion installation may be in a"
      echo " partially-updated state and is not yet fully upgraded."
      echo ""
      echo " Review the error above and $SOUP_LOG, resolve the underlying problem, then"
      echo " run soup again to resume and complete the update:"
      echo ""
      echo "     sudo soup"
      echo ""
      echo " soup is resumable -- re-running it continues from where this run stopped."
      echo "=============================================================================="
      echo ""
    fi
    exit $exit_code
  fi
 }
 # Collect bash error context before passing off to check_err()
 on_err() {
  local exit_code=$?
  # Ignore failures in blocks that explicitly disabled errexit with `set +e`.
  [[ $- == *e* ]] || return $exit_code
  # turn off xtrace to prevent added noise in debug log
  set +x 2>/dev/null || true
  # Use first error context, multiple errors can happen with command substitutions or nested functions. We just need context from the initial error.
  [[ -n $SOUP_ERR_CONTEXT ]] && return $exit_code
  local cmd=$BASH_COMMAND
  local line=${BASH_LINENO[0]}
  local function=${FUNCNAME[1]:-main}
  local source=${BASH_SOURCE[1]##*/}
  local -a err_lines=(
    "ERROR on: ${cmd}"
    "  source: ${source}:${line} in ${function}()"
  )
  local i caller_line caller_src caller_func
  for ((i=2; i<${#FUNCNAME[@]}-1; i++)); do
    caller_line=${BASH_LINENO[$((i-1))]}
    [[ -n $caller_line && $caller_line -gt 0 ]] || continue
    caller_src=${BASH_SOURCE[$i]##*/}
    caller_func=${FUNCNAME[$i]:-main}
    err_lines+=("  called by: ${caller_src}:${caller_line} in ${caller_func}()")
  done
  SOUP_ERR_CONTEXT=$(printf '%s\n' "${err_lines[@]}")
  return $exit_code
 }
 airgap_mounted() {
  # Let's see if the ISO is already mounted.
  if [[ -f /tmp/soagupdate/SecurityOnion/VERSION ]]; then
@@ -267,6 +188,13 @@ airgap_update_dockers() {
  fi
 }
 backup_old_states_pillars() {
 	tar czf /nsm/backup/$(echo $INSTALLEDVERSION)_$(date +%Y%m%d-%H%M%S)_soup_default_states_pillars.tar.gz /opt/so/saltstack/default/
 	tar czf /nsm/backup/$(echo $INSTALLEDVERSION)_$(date +%Y%m%d-%H%M%S)_soup_local_states_pillars.tar.gz /opt/so/saltstack/local/
 }
 update_registry() {
  docker stop so-dockerregistry
  docker rm so-dockerregistry
@@ -281,7 +209,6 @@ check_airgap() {
      UPDATE_DIR=/tmp/soagupdate/SecurityOnion
      AGDOCKER=/tmp/soagupdate/docker
      AGREPO=/tmp/soagupdate/minimal/Packages
      AGUEKREPO=/tmp/soagupdate/uek/Packages
  else
      is_airgap=1
  fi
@@ -327,30 +254,6 @@ check_pillar_items() {
  fi
 }
 check_cluster_health() {
  echo "Checking Elasticsearch cluster health."
  # Require a 'green' cluster before upgrading; anything less (yellow, red, or
  # unreachable) blocks. Modeled on the wait used in so-elasticsearch-roles-load.
  if so-elasticsearch-query "_cluster/health?wait_for_status=green&timeout=120s" --fail > /dev/null 2>&1; then
    printf "\nThe Elasticsearch cluster is healthy (green). We can proceed with SOUP.\n\n"
  else
    printf "\nThe Elasticsearch cluster is not green. Please resolve the cluster health issue so the cluster is green before running SOUP again.\n\n"
    exit 0
  fi
 }
 check_fleet_server() {
  echo "Checking that Elastic Fleet Server is responding."
  # Modeled on the wait_for_so-elastic-fleet state check in elasticfleet/enabled.sls,
  # which waits for HTTP 200 from the Fleet Server status API.
  if curl -sk --fail --retry 3 --retry-delay 10 --max-time 30 "https://localhost:8220/api/status" > /dev/null 2>&1; then
    printf "\nElastic Fleet Server is responding. We can proceed with SOUP.\n\n"
  else
    printf "\nElastic Fleet Server is not responding at https://localhost:8220/api/status. Please ensure Elastic Fleet is healthy before running SOUP again.\n\n"
    exit 0
  fi
 }
 check_saltmaster_status() {
  set +e
  echo "Waiting on the Salt Master service to be ready."
@@ -437,8 +340,6 @@ get_soup_script_hashes() {
  GITIMGCMN=$(md5sum $UPDATE_DIR/salt/common/tools/sbin/so-image-common | awk '{print $1}')
  CURRENTSOFIREWALL=$(md5sum /usr/sbin/so-firewall | awk '{print $1}')
  GITSOFIREWALL=$(md5sum $UPDATE_DIR/salt/manager/tools/sbin/so-firewall | awk '{print $1}')
  CURRENTSOYAML=$(md5sum /usr/sbin/so-yaml.py | awk '{print $1}')
  GITSOYAML=$(md5sum $UPDATE_DIR/salt/manager/tools/sbin/so-yaml.py | awk '{print $1}')
 }
 highstate() {
@@ -449,11 +350,10 @@ highstate() {
 masterlock() {
  echo "Locking Salt Master"
  mv -v $TOPFILE $BACKUPTOPFILE
-  # Render the real top file only for the host running soup; every other
+  echo "base:" > $TOPFILE
-  # minion gets an empty top (no states) while the master is upgrading.
+  echo "  $MINIONID:" >> $TOPFILE
-  echo "{% if grains['id'] == '$MINIONID' %}" > $TOPFILE
+  echo "    - ca" >> $TOPFILE
-  cat $BACKUPTOPFILE >> $TOPFILE
+  echo "    - elasticsearch" >> $TOPFILE
  echo "{% endif %}" >> $TOPFILE
 }
 masterunlock() {
@@ -472,26 +372,15 @@ preupgrade_changes() {
    [[ "$INSTALLEDVERSION" =~ ^2\.4\.21[0-9]+$ ]] && up_to_3.0.0   
    [[ "$INSTALLEDVERSION" == "3.0.0" ]] && up_to_3.1.0
    [[ "$INSTALLEDVERSION" == "3.1.0" ]] && up_to_3.2.0
    true
 }
 set_postversion() {
    # Persist post-upgrade walk progress so an interrupted upgrade can resume the
    # remaining steps on the next soup run (see /etc/sopostversion handling).
    POSTVERSION="$1"
    echo "$POSTVERSION" > "$POSTVERSION_FILE"
 }
 postupgrade_changes() {
    # This function is to add any new pillar items if needed.
    echo "Running post upgrade processes."
    [[ "$POSTVERSION" =~ ^2\.4\.21[0-9]+$ ]] && post_to_3.0.0
    [[ "$POSTVERSION" == "3.0.0" ]] && post_to_3.1.0
    [[ "$POSTVERSION" == "3.1.0" ]] && post_to_3.2.0
    # All applicable post-upgrade steps completed; clear the resume marker.
    rm -f "$POSTVERSION_FILE"
    true
 }
@@ -584,7 +473,7 @@ post_to_3.0.0() {
  # convert yes/no in suricata pillars to true/false
  convert_suricata_yes_no
-  set_postversion 3.0.0
+  POSTVERSION=3.0.0
 }
 ### 3.0.0 End ###
@@ -644,23 +533,6 @@ elasticfleet_set_agent_logging_level_warn() {
    done <<< "$policies_to_update"
 }
 update_logstash_pipeline_name() {
    local original_pipeline_name="$1"
    local new_pipeline_name="$2"
    echo "Checking for conflicting logstash defined_pipelines pillar value."
    local LOGSTASH_FILE=/opt/so/saltstack/local/pillar/logstash/soc_logstash.sls
    local MINIONDIR=/opt/so/saltstack/local/pillar/minions
    for pillar_file in "$LOGSTASH_FILE" "$MINIONDIR"/*.sls; do
        [[ -f "$pillar_file" ]] || continue
        if grep -q "$original_pipeline_name$" "$pillar_file"; then
            echo "Found conflicting defined_pipeline pillar value in $pillar_file. Updating to use the new logstash pipeline name."
            sed -i "s#$original_pipeline_name\$#$new_pipeline_name#g" "$pillar_file"
            chown socore:socore "$pillar_file"
        fi
    done
 }
 check_transform_health_and_reauthorize() {
    . /usr/sbin/so-elastic-fleet-common
@@ -684,23 +556,14 @@ check_transform_health_and_reauthorize() {
    # - unhealthy (any non-green health status)
    # - metadata has run_as_kibana_system: false (this fix is specific to transforms started prior to Kibana 9.3.3)
    # - are not orphaned (integration is not somehow missing/corrupt/uninstalled)
    local tmp_transforms tmp_stats tmp_installed
    tmp_transforms=$(mktemp)
    tmp_stats=$(mktemp)
    tmp_installed=$(mktemp)
    echo "$transforms_doc" > "$tmp_transforms"
    echo "$stats_doc"      > "$tmp_stats"
    echo "$installed_doc"  > "$tmp_installed"
    local unhealthy_transforms
    unhealthy_transforms=$(jq -c -n \
-        --slurpfile t "$tmp_transforms" \
+        --argjson t "$transforms_doc" \
-        --slurpfile s "$tmp_stats" \
+        --argjson s "$stats_doc" \
-        --slurpfile i "$tmp_installed" '
+        --argjson i "$installed_doc" '
-        ($i[0].items | map({key: .name, value: .version}) | from_entries) as $pkg_ver
+        ($i.items | map({key: .name, value: .version}) | from_entries) as $pkg_ver
-        | ($s[0].transforms | map({key: .id, value: .health.status}) | from_entries) as $health
+        | ($s.transforms | map({key: .id, value: .health.status}) | from_entries) as $health
-        | [ $t[0].transforms[]
+        | [ $t.transforms[]
            | select(._meta.run_as_kibana_system == false)
            | select(($health[.id] // "unknown") != "green")
            | {id, pkg: ._meta.package.name, ver: ($pkg_ver[._meta.package.name])}
@@ -741,8 +604,6 @@ check_transform_health_and_reauthorize() {
        (( total_failures += $(jq 'map(select(.success != true)) | length' <<< "$resp" 2>/dev/null) ))
    done <<< "$unhealthy_transforms"
    rm -f "$tmp_transforms" "$tmp_stats" "$tmp_installed"
    if [[ "$total_failures" -gt 0 ]]; then
        echo "Some transform(s) failed to reauthorize."
    fi
@@ -762,21 +623,6 @@ ensure_postgres_local_pillar() {
  chown -R socore:socore "$dir"
 }
 ensure_salt_local_pillar() {
  # The salt.auto_apply settings are a new SOC settings
  # module, so the new pillar/top.sls references salt.soc_salt / salt.adv_salt
  # unconditionally. Managers upgrading from before this change have no
  # /opt/so/saltstack/local/pillar/salt/ (make_some_dirs only runs at install
  # time), so the stubs must be created here before salt-master restarts against
  # the new top.sls.
  echo "Ensuring salt local pillar stubs exist."
  local dir=/opt/so/saltstack/local/pillar/salt
  mkdir -p "$dir"
  [[ -f "$dir/soc_salt.sls" ]] || touch "$dir/soc_salt.sls"
  [[ -f "$dir/adv_salt.sls" ]] || touch "$dir/adv_salt.sls"
  chown -R socore:socore "$dir"
 }
 ensure_postgres_secret() {
  # On a fresh install, generate_passwords + secrets_pillar seed
  # secrets:postgres_pass in /opt/so/saltstack/local/pillar/secrets.sls. That
@@ -798,39 +644,14 @@ ensure_postgres_secret() {
  chown socore:socore "$secrets_file"
 }
 rename_strelka_scan_lnk() {
  echo "Renaming strelka pillar ScanLNK to ScanLnk."
  local STRELKA_FILE=/opt/so/saltstack/local/pillar/strelka/soc_strelka.sls
  local MINIONDIR=/opt/so/saltstack/local/pillar/minions
  local OLD_KEY=strelka.backend.config.backend.scanners.ScanLNK
  local NEW_KEY=strelka.backend.config.backend.scanners.ScanLnk
  local TMP_VALUE_FILE
  TMP_VALUE_FILE=$(mktemp)
  for pillar_file in "$STRELKA_FILE" "$MINIONDIR"/*.sls; do
    [[ -f "$pillar_file" ]] || continue
    # Skip if ScanLNK doesn't exist
    so-yaml.py get "$pillar_file" "$OLD_KEY" > "$TMP_VALUE_FILE" 2>/dev/null || continue
    echo "Found 'ScanLNK' key in $pillar_file. Renaming to 'ScanLnk'."
    so-yaml.py add "$pillar_file" "$NEW_KEY" "file:$TMP_VALUE_FILE"
    so-yaml.py remove "$pillar_file" "$OLD_KEY"
  done
  rm -f "$TMP_VALUE_FILE"
 }
 fix_logstash_0013_lumberjack_pipeline_name() {
    update_logstash_pipeline_name "so/0013_input_lumberjack_fleet.conf" "so/0013_input_lumberjack_fleet.conf.jinja"
 }
 up_to_3.1.0() {
  ensure_postgres_local_pillar
  ensure_postgres_secret
  determine_elastic_agent_upgrade
  elasticsearch_backup_index_templates
  # Clear existing component template state file.
  rm -f /opt/so/state/esfleet_component_templates.json
-  rename_strelka_scan_lnk
+
  fix_logstash_0013_lumberjack_pipeline_name
  INSTALLEDVERSION=3.1.0
 }
@@ -862,141 +683,11 @@ post_to_3.1.0() {
  # Check for unhealthy / unauthorized integration transform jobs and attempt reauthorizations
  check_transform_health_and_reauthorize || true
-  set_postversion 3.1.0
+  POSTVERSION=3.1.0
 }
 ### 3.1.0 End ###
 ### 3.2.0 Scripts ###
 recollate_postgres() {
  echo ""
  echo "Recollating PostgreSQL databases. The following output may contain warnings about a version mismatch, followed by a note indicating that the collation version has been changed."
  for db in postgres securityonion so_telegraf; do
    docker exec so-postgres psql -U postgres $db -c "reindex database $db"
    docker exec so-postgres psql -U postgres $db -c "alter database $db refresh collation version"
  done
  echo "Recollating PostgreSQL databases complete."
  echo ""
 }
 scrub_postgres_log_passwords() {
  # Purge plaintext passwords a pre-3.2 postgres could log on DDL errors.
  local log=/opt/so/log/postgres/postgres.log
  [[ -f "$log" ]] || return 0
  if ! grep -qai "PASSWORD '" "$log" 2>/dev/null; then
    echo "No leaked passwords found in $log."
    return 0
  fi
  echo "Removing leaked password statements from $log."
  local tmp
  tmp=$(mktemp)
  # Rewrite in place (cat >) to keep the inode postgres is writing to.
  grep -avi "PASSWORD '" "$log" > "$tmp" 2>/dev/null || true
  cat "$tmp" > "$log"
  rm -f "$tmp"
 }
 # Existing grids should keep ILM unless an admin explicitly opts in to DLM.
 pin_elasticsearch_data_retention_method() {
  local elasticsearch_file=/opt/so/saltstack/local/pillar/elasticsearch/soc_elasticsearch.sls
  mkdir -p "$(dirname "$elasticsearch_file")"
  [[ -f "$elasticsearch_file" ]] || touch "$elasticsearch_file"
  if so-yaml.py get -r "$elasticsearch_file" elasticsearch.data_retention_method >/dev/null 2>&1; then
    echo "elasticsearch.data_retention_method already set; leaving as-is."
    return 0
  fi
  echo "Pinning existing grid to ILM data retention."
  so-yaml.py add "$elasticsearch_file" elasticsearch.data_retention_method ILM
  chown socore:socore "$elasticsearch_file"
 }
 # Addes auto_expand_replicas setting to .kibana_streams index template
 #
 # In Kibana 9.3.3 the auto_expand_replicas setting was not added to the .kibana_streams index template. Causing single node deployments to be stuck in yellow state (unable to assign replica). Here we update the template in place using the so_kibana system user (system managed index template) to include the auto_expand_replicas setting
 #
 # Reference: https://github.com/elastic/kibana/issues/263048
 kibana_backport_streams_index_template() {
    local current_template updated_template
    set +e
    if ! current_template=$(so-elasticsearch-query "_index_template/.kibana_streams" --retry 3 --retry-delay 5 --fail); then
        echo "Index template .kibana_streams does not exist, skipping backport."
        return 0
    fi
    set -e
    updated_template=$(jq '.index_templates[0].index_template | .template.settings += {"index.auto_expand_replicas": "0-1"} | del(.created_date_millis, .modified_date_millis)' <<< "$current_template")
    if ! kibana_user_pass=$(/usr/sbin/so-yaml.py get -r /opt/so/saltstack/local/pillar/elasticsearch/auth.sls elasticsearch.auth.users.so_kibana_user.pass); then
        echo "Unable to retrieve so_kibana_user password, skipping .kibana_streams index template backport."
        return 0
    fi
    if ! so-elasticsearch-query "_index_template/.kibana_streams" -XPUT -d "$updated_template" -u "so_kibana:$kibana_user_pass" --retry 3 --retry-delay 5 --fail; then
        echo "Unable to automatically update .kibana_streams index template"
        return 0
    fi
 }
 # Runs kafka-features.sh upgrade --release-version $1
 #   Upgrades Kafka KRaft cluster metadata
 update_kafka_metadata() {
    metadata_version="$1"
    global_pillar="/opt/so/saltstack/local/pillar/global/soc_global.sls"
    if PIPELINE=$(so-yaml.py get -r "$global_pillar" global.pipeline 2> /dev/null) && [[ "$PIPELINE" == "KAFKA" ]]; then
        kafka_nodes_raw=$(salt-call pillar.get kafka:nodes --out=json)
        if kafka_nodes=$(jq -er '.local | select(type == "object" and length > 0)' <<< "$kafka_nodes_raw"); then
            bootstrap_servers=$(jq -r '[to_entries[] | select(.value.role | contains("broker")) | "\(.value.ip):9092"] | join(",")' <<< "$kafka_nodes")
            echo "Upgrading Kafka KRaft cluster version"
            so-kafka-cli kafka-features.sh --bootstrap-server "$bootstrap_servers" --command-config /opt/kafka/config/kraft/client.properties upgrade --release-version "$metadata_version" 2>/dev/null || true
            return 0
        else
            FINAL_MESSAGE_QUEUE+=("WARNING: Unable to automatically perform Kafka KRaft cluster metadata update. This step can be performed manually using the following command (replacing \$BROKER_IP with the ip of atleast 1 available Kafka broker):")
            FINAL_MESSAGE_QUEUE+=("    - so-kafka-cli kafka-features.sh --bootstrap-server \$BROKER_IP:9092 --command-config /opt/kafka/config/kraft/client.properties upgrade --release-version $metadata_version")
        fi
    else
        echo "Nothing to do!"
    fi
 }
 up_to_3.2.0() {
  ensure_salt_local_pillar
  # download 9.3.7 elastic agent packages
  determine_elastic_agent_upgrade
  fix_logstash_0013_lumberjack_pipeline_name
  pin_elasticsearch_data_retention_method
  INSTALLEDVERSION=3.2.0
 }
 post_to_3.2.0() {
  # Recollate due to image OS rebase
  recollate_postgres
  # SOC database bootstrap is handled by the postgres.enabled highstate.
  scrub_postgres_log_passwords
  # Generate 9.3.7 elastic agent installers
  echo "Regenerating Elastic Agent Installers"
  /sbin/so-elastic-agent-gen-installers
  kibana_backport_streams_index_template
  update_kafka_metadata "4.3"
  set_postversion 3.2.0
 }
 ### 3.2.0 End ###
 repo_sync() {
  echo "Sync the local repo."
@@ -1105,19 +796,13 @@ update_airgap_rules() {
  rsync -a $UPDATE_DIR/agrules/securityonion-resources/* /nsm/securityonion-resources/
 }
-update_airgap_repos() {
+update_airgap_repo() {
  # Update the files in the repo
-  echo "Syncing new updates to /nsm/repo & /nsm/kernelrepo"
+  echo "Syncing new updates to /nsm/repo"
-  # Airgap soup copies new files into the local repo, but doesn't remove old packages. Retaining the ability to rollback package updates
+  rsync -a $AGREPO/* /nsm/repo/
-  rsync -a "$AGREPO"/ /nsm/repo/
+  echo "Creating repo"
  rsync -a "$AGUEKREPO"/ /nsm/kernelrepo/
  dnf -y install yum-utils createrepo_c
  echo "Running createrepo for /nsm/repo"
  createrepo /nsm/repo
  echo "Running createrepo for /nsm/kernelrepo"
  createrepo /nsm/kernelrepo
 }
 update_salt_mine() {
@@ -1144,20 +829,8 @@ upgrade_check() {
  fi
  [[ -f /etc/sohotfix ]] && CURRENTHOTFIX=$(cat /etc/sohotfix)
  if [ "$INSTALLEDVERSION" == "$NEWVERSION" ]; then
    # A leftover post-version marker means a previous upgrade to this version
    # advanced /etc/soversion (the highstate stamps it from the pillar) but did not
    # finish its post-upgrade steps. Resume the upgrade instead of reporting "latest".
    if [ -f "$POSTVERSION_FILE" ] && [ "$(cat "$POSTVERSION_FILE")" != "$NEWVERSION" ]; then
      echo "A previous upgrade to $NEWVERSION did not complete its post-upgrade steps; resuming."
      is_hotfix=false
      return 0
    fi
    echo "Checking to see if there are hotfixes needed"
    if [ "$HOTFIXVERSION" == "$CURRENTHOTFIX" ]; then
      # Reaching here means we are at the target version and NOT resuming (the resume
      # check above returned otherwise). Clear any stale resume marker so a completed
      # upgrade is never mistaken for a partial one and re-run on a later invocation.
      rm -f "$POSTVERSION_FILE"
      echo "You are already running the latest version of Security Onion."
      exit 0
    else
@@ -1236,7 +909,7 @@ upgrade_salt() {
 verify_latest_update_script() {
  get_soup_script_hashes
-  if [[ "$CURRENTSOUP" == "$GITSOUP" && "$CURRENTCMN" == "$GITCMN" && "$CURRENTIMGCMN" == "$GITIMGCMN" && "$CURRENTSOFIREWALL" == "$GITSOFIREWALL" && "$CURRENTSOYAML" == "$GITSOYAML" ]]; then
+  if [[ "$CURRENTSOUP" == "$GITSOUP" && "$CURRENTCMN" == "$GITCMN" && "$CURRENTIMGCMN" == "$GITIMGCMN" && "$CURRENTSOFIREWALL" == "$GITSOFIREWALL" ]]; then
    echo "This version of the soup script is up to date. Proceeding."
  else
    echo "You are not running the latest soup version. Updating soup and its components. This might take multiple runs to complete."
@@ -1245,7 +918,7 @@ verify_latest_update_script() {
    # Verify that soup scripts updated as expected
    get_soup_script_hashes
-    if [[ "$CURRENTSOUP" == "$GITSOUP" && "$CURRENTCMN" == "$GITCMN" && "$CURRENTIMGCMN" == "$GITIMGCMN" && "$CURRENTSOFIREWALL" == "$GITSOFIREWALL" && "$CURRENTSOYAML" == "$GITSOYAML" ]]; then
+    if [[ "$CURRENTSOUP" == "$GITSOUP" && "$CURRENTCMN" == "$GITCMN" && "$CURRENTIMGCMN" == "$GITIMGCMN" && "$CURRENTSOFIREWALL" == "$GITSOFIREWALL" ]]; then
      echo "Succesfully updated soup scripts."
    else
      echo "There was a problem updating soup scripts. Trying to rerun script update."
@@ -1266,11 +939,7 @@ verify_es_version_compatibility() {
    local is_active_intermediate_upgrade=1
    # supported upgrade paths for SO-ES versions
    declare -A es_upgrade_map=(
-        ["8.18.4"]="8.18.6 8.18.8 9.0.8"
+        ["9.0.8"]="9.3.3"
 	    ["8.18.6"]="8.18.8 9.0.8"
 	    ["8.18.8"]="9.0.8"
        ["9.0.8"]="9.3.3 9.3.7"
        ["9.3.3"]="9.3.7"
    )
    # Elasticsearch MUST upgrade through these versions
@@ -1293,171 +962,6 @@ verify_es_version_compatibility() {
        exit 160
    fi
    compatible_es_versions="$target_es_version"
    for current_version in "${!es_upgrade_map[@]}"; do
        # shellcheck disable=SC2076
        if [[ " ${es_upgrade_map[$current_version]} " =~ " $target_es_version " ]]; then
            compatible_es_versions+=" $current_version"
        fi
    done
    # Check if the given ES version can directly upgrade to the target ES version. Used to assist with catching lagging nodes during the upgrade process
    es_version_can_upgrade_to_target() {
        local current_version="$1"
        # shellcheck disable=SC2076
        if [[ -n "$current_version" && " $compatible_es_versions " =~ " $current_version " ]]; then
            return 0
        fi
        return 1
    }
    # Gather Elasticsearch cluster version info and verify that each node in the cluster is running a version compatible with the target ES version.
    verify_searchnodes_es_target_compatibility() {
        local retries=20
        local retry_count=0
        local delay=180
        local expected_es_nodes searchnode_minions attempt
        local searchnode_discovery_success=false
        SEARCHNODE_ES_VERSIONS=""
        for attempt in {1..3}; do
            if searchnode_minions=$(set -o pipefail; salt-key --out=json --list=accepted 2> /dev/null | jq -r '.minions[]? | select(endswith("searchnode"))'); then
                searchnode_discovery_success=true
                break
            fi
            echo "Failed to retrieve grid searchnodes via salt-key... Retrying in 30 seconds. Attempt $attempt of 3."
            sleep 30
        done
        if [[ "$searchnode_discovery_success" != "true" ]]; then
            echo "Failed to retrieve grid searchnodes via salt-key."
            return 1
        fi
        # Always add node running soup to expected es nodes
        expected_es_nodes="${MINIONID%_*}"
        while IFS= read -r searchnode_minion; do
            [[ -z "$searchnode_minion" ]] && continue
            expected_es_nodes+=$'\n'"${searchnode_minion%_searchnode}"
        done <<< "$searchnode_minions"
        while [[ $retry_count -lt $retries ]]; do
            SEARCHNODE_ES_VERSIONS=$(so-elasticsearch-query _nodes/_all/version --retry 5 --retry-delay 10 --fail 2>&1)
            local exit_status=$?
            if [[ $exit_status -ne 0 ]]; then
                echo "Failed to retrieve Elasticsearch versions from searchnodes... Retrying in $delay seconds. Attempt $((retry_count + 1)) of $retries."
                ((retry_count++))
                sleep $delay
                continue
            fi
            local all_searchnodes_compatible=true
            while IFS=$'\t' read -r node current_version; do
                [[ -z "$node" ]] && continue
                if ! es_version_can_upgrade_to_target "$current_version"; then
                    echo "Searchnode $node is running Elasticsearch $current_version, which is not directly upgradable to Elasticsearch $target_es_version."
                    all_searchnodes_compatible=false
                fi
            done < <(echo "$SEARCHNODE_ES_VERSIONS" | jq -r '.nodes | to_entries[] | [.value.name, .value.version] | @tsv')
            while IFS= read -r expected_es_node; do
                [[ -z "$expected_es_node" ]] && continue
                if ! echo "$SEARCHNODE_ES_VERSIONS" | jq -e --arg node "$expected_es_node" '.nodes | to_entries | any(.value.name == $node)' > /dev/null; then
                    echo "Searchnode $expected_es_node did not report an Elasticsearch version. It may be offline or still upgrading."
                    all_searchnodes_compatible=false
                fi
            done <<< "$expected_es_nodes"
            if [[ "$all_searchnodes_compatible" == true ]]; then
                echo "All Searchnodes are upgradable to Elasticsearch $target_es_version."
                return 0
            fi
            echo "One or more Searchnodes cannot upgrade directly to Elasticsearch $target_es_version. Rechecking in $delay seconds. Attempt $((retry_count + 1)) of $retries."
            ((retry_count++))
            sleep $delay
        done
        return 1
    }
    # Gather heavynode version info and verify that each node is running a version compatible with the target ES version.
    verify_heavynodes_es_target_compatibility() {
        local heavynode_minions attempt
        local retries=20
        local retry_count=0
        local delay=180
        local heavynode_discovery_success=false
        HEAVYNODE_ES_VERSIONS=""
        for attempt in {1..3}; do
            if heavynode_minions=$(set -o pipefail; salt-key --out=json --list=accepted 2> /dev/null | jq -r '.minions[]? | select(endswith("heavynode"))'); then
                heavynode_discovery_success=true
                break
            fi
            echo "Failed to retrieve grid heavynodes via salt-key... Retrying in 30 seconds. Attempt $attempt of 3."
            sleep 30
        done
        if [[ "$heavynode_discovery_success" != "true" ]]; then
            echo "Failed to retrieve grid heavynodes via salt-key."
            return 1
        fi
        if [[ -z "$heavynode_minions" ]]; then
            echo "No heavynodes detected. Skipping heavynode Elasticsearch version compatibility check."
            return 0
        fi
        while [[ $retry_count -lt $retries ]]; do
            HEAVYNODE_ES_VERSIONS=$(salt -C 'G@role:so-heavynode' cmd.run 'set -o pipefail; so-elasticsearch-query / --retry 5 --retry-delay 10 | jq -er ".version.number"' shell=/bin/bash --out=json 2> /dev/null)
            local exit_status=$?
            if [[ $exit_status -ne 0 ]]; then
                echo "Failed to retrieve Elasticsearch version from one or more heavynodes... Retrying in $delay seconds. Attempt $((retry_count + 1)) of $retries."
                ((retry_count++))
                sleep $delay
                continue
            fi
            local all_heavynodes_compatible=true
            while IFS=$'\t' read -r node current_version; do
                [[ -z "$node" ]] && continue
                if ! es_version_can_upgrade_to_target "$current_version"; then
                    echo "Heavynode $node is running Elasticsearch $current_version, which is not directly upgradable to Elasticsearch $target_es_version."
                    all_heavynodes_compatible=false
                fi
            done < <(echo "$HEAVYNODE_ES_VERSIONS" | jq -r 'to_entries[] | [.key, .value] | @tsv')
            while IFS= read -r heavynode_minion; do
                [[ -z "$heavynode_minion" ]] && continue
                if ! echo "$HEAVYNODE_ES_VERSIONS" | jq -se --arg minion "$heavynode_minion" 'add | has($minion)' > /dev/null; then
                    echo "Heavynode $heavynode_minion did not report an Elasticsearch version. It may be offline or still upgrading."
                    all_heavynodes_compatible=false
                fi
            done <<< "$heavynode_minions"
            if [[ "$all_heavynodes_compatible" == true ]]; then
                echo -e "\nAll heavynodes can upgrade to Elasticsearch $target_es_version."
                return 0
            fi
            echo "One or more heavynodes cannot upgrade directly to Elasticsearch $target_es_version. Rechecking in $delay seconds. Attempt $((retry_count + 1)) of $retries."
            ((retry_count++))
            sleep $delay
        done
        return 1
    }
    if [[ ! -f "$es_verification_script" ]]; then
        create_intermediate_upgrade_verification_script "$es_verification_script"
    fi
    for statefile in "${es_required_version_statefile_base}"-*; do
        [[ -f $statefile ]] || continue
@@ -1476,6 +980,10 @@ verify_es_version_compatibility() {
            continue
        fi
        if [[ ! -f "$es_verification_script" ]]; then
            create_intermediate_upgrade_verification_script "$es_verification_script"
        fi
        echo -e "\n##############################################################################################################################\n"
        echo "A previously required intermediate Elasticsearch upgrade was detected. Verifying that all Searchnodes/Heavynodes have successfully upgraded Elasticsearch to $es_required_version_statefile_value before proceeding with soup to avoid potential data loss! This command can take up to an hour to complete."
        if ! timeout --foreground 4000 bash "$es_verification_script" "$es_required_version_statefile_value" "$statefile"; then
@@ -1497,26 +1005,6 @@ verify_es_version_compatibility() {
    # shellcheck disable=SC2076 # Do not want a regex here eg usage " 8.18.8 9.0.8 " =~ " 9.0.8 "
    if [[ " ${es_upgrade_map[$es_version]} " =~ " $target_es_version " || "$es_version" == "$target_es_version" ]]; then
        if ! verify_searchnodes_es_target_compatibility || ! verify_heavynodes_es_target_compatibility; then
            echo -e "\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n"
            echo "One or more Searchnode(s)/Heavynode(s) cannot upgrade directly to Elasticsearch $target_es_version. This can happen with soups that include Elasticsearch upgrades being run in quick succession. Typically, this will resolve itself as the grid synchronizes. Please allow time for all Searchnodes/Heavynodes to have upgraded Elasticsearch to a compatible version with $target_es_version before running soup again to avoid potential data loss!"
            if [[ -n "$HEAVYNODE_ES_VERSIONS" ]]; then
                echo "Current heavynode Elasticsearch versions:"
                echo "$HEAVYNODE_ES_VERSIONS" | jq '.'
            fi
            if [[ -n "$SEARCHNODE_ES_VERSIONS" ]]; then
                echo "Current searchnode Elasticsearch versions:"
                echo "$SEARCHNODE_ES_VERSIONS" | jq '.nodes | to_entries | map({(.value.name): .value.version}) | sort | add'
            fi
            echo -e "\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n"
            exit 161
        fi
        # supported upgrade
        return 0
    else
@@ -1556,13 +1044,18 @@ verify_es_version_compatibility() {
 }
 wait_for_salt_minion_with_restart() {
-    /usr/sbin/so-salt-minion-wait
+    local minion="$1"
    local max_wait="${2:-60}"
    local interval="${3:-3}"
    local logfile="$4"
    wait_for_salt_minion "$minion" "$max_wait" "$interval" "$logfile"
    local result=$?
    if [[ $result -ne 0 ]]; then
        echo "$(date '+%a %d %b %Y %H:%M:%S.%6N') - salt-minion not ready, attempting restart..."
        systemctl_func "restart" "salt-minion"
-        /usr/sbin/so-salt-minion-wait
+        wait_for_salt_minion "$minion" "$max_wait" "$interval" "$logfile"
        result=$?
    fi
@@ -1848,15 +1341,6 @@ main() {
  set_minionid
  MINION_ROLE=$(lookup_role)
  echo "Found that Security Onion $INSTALLEDVERSION is currently installed."
  # /etc/soversion is stamped to the target version before the upgrade fully
  # completes, so a lingering resume marker means this grid is only partially
  # upgraded even though the line above shows the target version. Make that explicit
  # so it is not mistaken for a finished upgrade.
  if [ -f "$POSTVERSION_FILE" ] && [ "$(cat "$POSTVERSION_FILE")" != "$INSTALLEDVERSION" ]; then
    echo ""
    echo "NOTE: A previous upgrade to $INSTALLEDVERSION did not finish. This grid is"
    echo "      partially upgraded and this soup run will resume and complete it."
  fi
  echo ""
  check_minimum_version
@@ -1878,25 +1362,19 @@ main() {
  echo "Verifying we have the latest soup script."
  verify_latest_update_script
  echo "Verifying Elasticsearch version compatibility before upgrading."
  verify_es_version_compatibility
  echo "Let's see if we need to update Security Onion."
  upgrade_check
  upgrade_space
  echo "Verifying Elasticsearch version compatibility across the grid before upgrading."
  verify_es_version_compatibility
  # Pre-flight health checks: confirm the grid is in a good state before we change
  # anything. These run before any modifications, so a failure exits cleanly and the
  # operator can fix the issue and re-run soup.
  check_cluster_health
  check_fleet_server
  echo "Checking for Salt Master and Minion updates."
  upgrade_check_salt
  set -e
  if [[ $is_airgap -eq 0 ]]; then
-    update_airgap_repos
+    update_airgap_repo
    dnf clean all
    check_os_updates
  elif [[ $OS == 'oracle' ]]; then
@@ -1907,27 +1385,19 @@ main() {
  fi
  if [ "$is_hotfix" == "true" ]; then
    SOUP_UPGRADE_STARTED=true
    echo "Applying $HOTFIXVERSION hotfix"
    # since we don't run the backup.config_backup state on import we wont snapshot previous version states and pillars
    if [[ ! "$MINION_ROLE" == "import" ]]; then
-        echo "Running so-config-backup script."
+      backup_old_states_pillars
        /sbin/so-config-backup
    fi
    copy_new_files
    create_local_directories "/opt/so/saltstack/default"
    apply_hotfix
    echo "Hotfix applied"
    update_version
    enable_highstate
    highstate
    # Record the hotfix only after the highstate succeeds. /etc/sohotfix is written
    # solely by soup (no salt state manages it), so deferring the write means a failed
    # hotfix highstate leaves the old hotfix value and re-running soup re-applies it,
    # rather than reporting "already latest". The soversion/pillar writes in
    # update_version are no-ops here since the version is unchanged for a hotfix.
    update_version
  else
    SOUP_UPGRADE_STARTED=true
    echo ""
    echo "Performing upgrade from Security Onion $INSTALLEDVERSION to Security Onion $NEWVERSION."
    echo ""
@@ -1974,8 +1444,8 @@ main() {
    # since we don't run the backup.config_backup state on import we wont snapshot previous version states and pillars
    if [[ ! "$MINION_ROLE" == "import" ]]; then
      echo ""
-      echo "Running so-config-backup script."
+      echo "Creating snapshots of default and local Salt states and pillars and saving to /nsm/backup/"
-      /sbin/so-config-backup
+      backup_old_states_pillars
    fi
    echo ""
@@ -1983,10 +1453,6 @@ main() {
    copy_new_files
    echo ""
    create_local_directories "/opt/so/saltstack/default"
    # Seed the resume marker before the highstate stamps /etc/soversion to the new
    # version, so an interrupted upgrade is detectable as "not finished" on re-run.
    # POSTVERSION still holds the pre-upgrade (or prior resume) version here.
    [ -f "$POSTVERSION_FILE" ] || echo "$POSTVERSION" > "$POSTVERSION_FILE"
    update_version
    echo ""
@@ -2016,9 +1482,9 @@ main() {
    enable_highstate
    echo ""
-    echo "Running a highstate at $(date +"%T.%6N"). This could take several minutes."
+    echo "Running a highstate. This could take several minutes."
    set +e
-    wait_for_salt_minion_with_restart || fail "Salt minion was not running or ready."
+    wait_for_salt_minion_with_restart "$MINIONID" "60" "3" "$SOUP_LOG" || fail "Salt minion was not running or ready."
    highstate
    set -e
@@ -2030,8 +1496,8 @@ main() {
    check_saltmaster_status
-    echo "Running a highstate at $(date +"%T.%6N") to complete the Security Onion upgrade on this manager. This could take several minutes."
+    echo "Running a highstate to complete the Security Onion upgrade on this manager. This could take several minutes."
-    wait_for_salt_minion_with_restart || fail "Salt minion was not running or ready."
+    wait_for_salt_minion_with_restart "$MINIONID" "60" "3" "$SOUP_LOG" || fail "Salt minion was not running or ready."
    # Stop long-running scripts to allow potentially updated scripts to load on the next execution.
    if pgrep salt-relay.sh > /dev/null 2>&1; then
@@ -2191,20 +1657,4 @@ EOF
  read -r input
 fi
-set -o errtrace
+main "$@" | tee -a $SOUP_LOG
 trap on_err ERR
 if [[ $SOUP_DEBUG == 1 ]]; then
  if [ -f $SOUP_DEBUG_LOG ]; then
    current_time=$(date +%Y%m%d.%H%M%S)
    mv $SOUP_DEBUG_LOG $SOUP_DEBUG_LOG.$INSTALLEDVERSION.$current_time
  fi
  exec {SOUP_XTRACE_FD}>>"$SOUP_DEBUG_LOG"
  export SOUP_XTRACE_FD
  BASH_XTRACEFD=$SOUP_XTRACE_FD
  PS4='+ [${BASH_SOURCE##*/}:${LINENO} ${FUNCNAME[0]:-main}()] | '
  set -x
  export SOUP_DEBUG
 fi
 main "$@" 2>&1 | tee -a $SOUP_LOG
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Mike Reeves	6bca92da4a	fix: stop pip's patchelf 'ERROR' line from polluting sosetup.log The cmd.run for psycopg2 install was already tolerating pip's non-zero exit with `\|\| true`, but pip's stderr — which contains the literal string "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'patchelf'" — was still being captured into salt's state-result dict. so-setup logs salt state output to /root/sosetup.log, and verify_setup() then greps for the substring "ERROR" to build /root/errors.log. The patchelf line then shows up at the end of every install as "WARNING: Errors detected during setup" even though the install is in fact green. Redirect pip's combined stdout/stderr to /opt/so/log/so_pillar/psycopg2_install.log so the noise lives in a dedicated, predictable triage location instead of leaking into salt's state result. The `unless: import psycopg2` check is still the actual readiness gate, so a real install failure (rather than just the patchelf RPATH-rewrite step that has no functional effect on the wheel) would still surface via the state being re-run on every apply and `import psycopg2` failing.	2026-05-05 10:38:57 -04:00
Mike Reeves	a7efabd90d	fix: tolerate pip's non-zero exit on psycopg2 patchelf step salt's pip.installed flagged so_pillar_psycopg2_in_salt_python as failed because pip exits non-zero when it can't find the patchelf binary to rewrite the psycopg2 wheel's RPATH after extraction. The wheel is fully installed and importable regardless — the patchelf step is a cosmetic post-install rewrite, not a build dependency. But salt's failure cascade then short-circuited so_pillar_initial_import and the so-yaml mode flip, leaving the install in dual-pillar mode instead of PG-canonical. Replaced with cmd.run that runs pip with `\|\| true` and uses an `import psycopg2` check as the actual readiness gate — same idea as how salt's own bootstrap does it. Also fixed the require: ref on so_pillar_initial_import (was `pip:`, needs to be `cmd:` for the new state type).	2026-05-04 22:08:31 -04:00
Mike Reeves	b25b221076	postsalt: move PG-canonical enable to AFTER the install highstate Supersedes the pre-install placement (right after secrets_pillar) from the previous commit, which was broken: salt's ext_pillar overlay shadowed disk pillar's elasticsearch subtree before so-pillar-import had populated PG, so elasticsearch.enabled.sls failed rendering on ELASTICSEARCHMERGED.auth.users.so_elastic_user.pass — that key lives in elasticsearch/auth.sls, which is on the importer's secrets allowlist and never makes it into so_pillar.pillar_entry. The install would then hang forever waiting for the elasticsearch container that the broken state never deployed. The new placement is right after the final state.highstate completes: 1. drop adv_postgres.sls flipping the flag to True 2. salt-call saltutil.refresh_pillar so the next state sees it 3. salt-call state.apply postgres.schema_pillar — deploys schema, ALTERs role login passwords, installs psycopg2 into salt's bundled python, runs so-pillar-import, writes /opt/so/conf/so-yaml/mode=postgres 4. salt-call state.apply salt.master — re-renders engines.conf with the pg_notify_pillar engine block, drops master.d ext_pillar config, watch_in restarts salt-master and ext_pillar takes over verify_setup runs after this so its final checks see PG-canonical mode in place. Same end state as the previous commit's intent, just without the bootstrap chicken-and-egg.	2026-05-04 21:02:08 -04:00
Mike Reeves	7b9ab2d9d1	postsalt: enable PG-canonical pillar mode by default during so-setup Drops a local pillar override (postgres.so_pillar.enabled = True) right after secrets_pillar so the install-time highstate brings up schema_pillar, ext_pillar_postgres, and the pg_notify_pillar engine without operator intervention. Without this the whole PG-canonical stack stays gated off on the default-False flag and the install lands in legacy disk-pillar mode — which defeats the point of being on the postsalt branch at all. The new enable_so_pillar_postgres() function in so-functions is idempotent (overwrites adv_postgres.sls with a fixed body) and the generated file is mode 0644 socore:socore so it merges into pillar under the existing local-pillar directory ownership convention. Rollback path: edit /opt/so/saltstack/local/pillar/postgres/adv_postgres.sls to set enabled: False, or delete the file. The schema and engine config states will tear themselves down on the next highstate via their existing else-branch absent states.	2026-05-04 19:56:14 -04:00
Mike Reeves	92a7bb3053	fix: get postsalt's PG-canonical pillar actually working end-to-end Five blockers turned up the first time the so_pillar schema was applied against a fresh standalone install. Fixing them in order: 1. 006_rls.sql ordering bug 006 GRANTed on so_pillar.change_queue and its sequence, but the table isn't created until 008_change_notify.sql. 006 errored mid-file with "relation so_pillar.change_queue does not exist", short-circuiting the rest of the pillar staging chain. Moved the three change_queue grants into 008 alongside the table creation so each file is self-contained. 2. so_pillar_* roles unable to log in 006 created the roles as NOLOGIN and set no password. Salt-master's ext_pillar (postgres) and the pg_notify_pillar engine both connect as so_pillar_master via TCP, so both came up with "password authentication failed for user so_pillar_master". Added a templated cmd.run step in schema_pillar.sls (so_pillar_role_login_passwords) that ALTERs all three roles WITH LOGIN PASSWORD pulling from secrets:pillar_master_pass — the same password ext_pillar_postgres.conf.jinja and the engines.conf pg_notify_pillar block render with. 3. Missing GRANT CONNECT ON DATABASE securityonion USAGE on the schema is granted in 006 but CONNECT on the database isn't. Engine + ext_pillar succeeded auth then died with "permission denied for database securityonion". Added the explicit GRANT CONNECT in 006. 4. psycopg2 missing from salt's bundled python /opt/saltstack/salt/bin/python3 doesn't ship psycopg by default, so when salt-master tries to load the pg_notify_pillar engine its `import psycopg2` fails inside salt's loader and the engine silently doesn't start (no error in the salt log — you only notice when nothing ever drains so_pillar.change_queue). Added a pip.installed state in schema_pillar.sls bound to that interpreter via bin_env. 5. engines.conf vs pg_notify_pillar_engine.conf list-replace Salt's master.d/.conf merge replaces top-level lists rather than concatenating them. The engine config used to live in its own master.d/pg_notify_pillar_engine.conf with `engines: [pg_notify_pillar]` alongside the legacy `engines.conf` carrying `engines: [checkmine, pillarWatch]`. Whichever loaded last won, so the engine never showed up in the loaded set even when the file existed. Fold the pg_notify_pillar declaration into engines.conf (now jinja-rendered, gated on postgres:so_pillar:enabled), drop the standalone state from pg_notify_pillar_engine.sls, and delete the now-orphaned conf jinja. End state validated against a live standalone-net install on the dev rig: salt-master ext_pillar reads from so_pillar. with no errors, the pg_notify_pillar engine LISTENs on so_pillar_change and drains the change_queue (134-row backlog → 0 within seconds), and a so-yaml replace on a pillar key flows disk → PG → ext_pillar → salt pillar.get with the new value visible after a saltutil.refresh_pillar.	2026-05-04 19:47:38 -04:00
Mike Reeves	155b5c5d66	fix: consistent allowed_states guard in postgres.schema_pillar Same `sls.split('.')[0]` pattern as ext_pillar_postgres + pg_notify_pillar_engine. For sls='postgres.schema_pillar' the split happened to evaluate 'postgres', which is in manager_states, so the guard worked accidentally — but it would break silently if anyone ever moved the file under a deeper SLS path. Switch to a literal `{% if 'postgres' in allowed_states %}` for the same intent- revealing pattern as the master.d guards.	2026-05-04 19:25:14 -04:00
Mike Reeves	f1746b0f59	fix: correct allowed_states guard in ext_pillar_postgres + pg_notify_pillar_engine Both SLS files used `sls.split('.')[0]` to derive what to look up in allowed_states. For these files (sls='salt.master.ext_pillar_postgres' and sls='salt.master.pg_notify_pillar_engine') that returns 'salt', which is never in any role's allowed_states list — only specific keys like 'salt.master', 'salt.minion', 'salt.cloud' are. The guard's else branch fired on every highstate, emitting two cosmetic ID: <sls>_state_not_allowed Function: test.fail_without_changes Comment: Failure! entries that polluted the so-setup error summary even on green installs. Both states drop config under /etc/salt/master.d/ and watch_in the salt-master service, so the natural intent is "only run when this node hosts the salt master". Switching the guard to a literal {% if 'salt.master' in allowed_states %} expresses that directly without string-parsing the SLS path, and matches the existing membership in manager_states (which is in turn included in every manager-bearing role: so-eval, so-manager, so-managerhype, so-managersearch, so-standalone, so-import).	2026-05-04 19:17:30 -04:00
Mike Reeves	2e411625c4	fix: subshell-scope umask 077 in so_pillar key generation The unscoped `umask 077` on postsalt's secrets_pillar path leaked into every subsequent file write by so-setup (and the salt-call processes it spawned) for the rest of the install. Every state-rendered config file under /opt/so/conf landed at mode 0600 instead of 0644, which broke any container that bind-mounts its config read-only and runs as a non-root user after the entrypoint's gosu drop. The first concrete casualty was the influxdb container, which exits with "failed to load config file: open /conf/config.yaml: permission denied" after init mode completes and re-execs as the influxdb user. The chmod 0400 immediately after the printf already enforces the intended file mode, so the umask was redundant for the key file itself; scoping it to a subshell preserves the defense-in-depth between the printf and the chmod without polluting the parent shell.	2026-05-04 18:02:58 -04:00
Mike Reeves	e43ad2ff74	Merge remote-tracking branch 'origin/feature/ensure-pyyaml' into postsalt	2026-05-04 16:37:42 -04:00
Mike Reeves	b39d259101	Merge remote-tracking branch 'origin/3/dev' into postsalt	2026-05-04 16:19:17 -04:00
Mike Reeves	3d11694d51	make so-yaml PG-canonical and add pillar-change reactor stack Two coupled changes that together let so_pillar.* be the canonical config store, with config edits driving service reloads automatically: so-yaml PG-canonical mode - Adds /opt/so/conf/so-yaml/mode (and SO_YAML_BACKEND env override) with three values: dual (legacy), postgres (PG-only for managed paths), disk (emergency rollback). Bootstrap files (secrets.sls, ca/init.sls, .nodes.sls, top.sls, ...) stay disk-only regardless via the existing SkipPath allowlist in so_yaml_postgres.locate. - loadYaml/writeYaml/purgeFile now route to so_pillar. in postgres mode: replace/add/get all read+write the database with no disk file ever appearing. PG failure is fatal in postgres mode (no silent fallback); dual mode preserves the prior best-effort mirror. - so_yaml_postgres gains read_yaml(path), is_pg_managed(path), and is_enabled() so so-yaml can answer "is this path PG-managed and is PG up" without reaching into private helpers. - schema_pillar.sls writes /opt/so/conf/so-yaml/mode = postgres after the importer succeeds, so flipping postgres:so_pillar:enabled flips so-yaml's behavior in lockstep with the schema being live. pg_notify-driven change fan-out - 008_change_notify.sql adds so_pillar.change_queue + an AFTER trigger on pillar_entry that enqueues the locator and pg_notifies 'so_pillar_change'. Queue is drained at-least-once so engine restarts don't lose events; pg_notify is just the wakeup signal. - New salt-master engine pg_notify_pillar.py LISTENs on the channel, drains the queue with FOR UPDATE SKIP LOCKED, debounces bursts, and fires 'so/pillar/changed' events grouped by (scope, role, minion). - Reactor so_pillar_changed.sls catches the tag and dispatches to orch.so_pillar_reload, which carries a DISPATCH map of pillar-path prefix -> (state sls, role grain set) so adding a new service to the auto-reload list is a one-line edit instead of a new reactor. - Engine + reactor wiring is gated on the same postgres:so_pillar:enabled flag as the schema and ext_pillar config so the whole stack flips on/off together. Tests: 21 new cases (112 total, all passing) covering mode resolution, PG-managed detection, and PG-canonical read/write/purge routing with the PG client stubbed.	2026-05-01 09:31:48 -04:00
Mike Reeves	23255f88e0	add so-yaml dual-write to so_pillar.* + purge verb Hooks every so-yaml.py write through a new so_yaml_postgres helper that mirrors disk YAML mutations into so_pillar.pillar_entry via docker exec psql. Disk remains canonical during the transition; PG mirror failures are logged only when a real write error occurs (skipped paths and postgres-unreachable cases stay silent so existing callers don't see new noise on stderr). Adds a `purge YAML_FILE` verb on so-yaml that deletes the file from disk and removes the matching pillar_entry rows. For minion files it also drops the so_pillar.minion row, which CASCADEs to pillar_entry + role_member. Designed for so-minion's delete path (replaces rm -f) so the audit log captures the deletion. setup/so-functions::generate_passwords + secrets_pillar generate secrets:pillar_master_pass and /opt/so/conf/postgres/so_pillar.key on fresh installs, and append the password to existing secrets.sls files on upgrade. - salt/manager/tools/sbin/so_yaml_postgres.py: locate(), write_yaml(), purge_yaml(), and a small CLI for diagnostics. Skips bootstrap and mine-driven paths via the same allowlist used by so-pillar-import. - salt/manager/tools/sbin/so-yaml.py: import the helper, hook writeYaml() to mirror after every disk write, add purgeFile() and the purge verb. - salt/manager/tools/sbin/so-yaml_test.py: 16 new tests covering the purge verb and the path-locator / write contract of so_yaml_postgres without contacting Postgres. All 91 tests pass. - setup/so-functions: generate_passwords adds PILLARMASTERPASS and SO_PILLAR_KEY; secrets_pillar writes pillar_master_pass and the pgcrypto master key file.	2026-04-30 17:09:58 -04:00
Mike Reeves	d30b52b327	add so-pillar-import — seeds so_pillar.* from on-disk pillar tree Idempotent importer that schema_pillar.sls runs once at end of postgres state on first install, and that so-minion can call per-minion on add / delete. UPSERTs into so_pillar.pillar_entry; the audit trigger handles versioning so re-runs without SLS edits produce no version bumps. Connects via docker exec so-postgres psql, so no DSN config is required at first-install time. Skips bootstrap files (secrets.sls, postgres/ auth.sls, etc.), mine-driven nodes.sls files, and any file containing Jinja templates — those stay disk-authoritative and ext_pillar_first: False means they render before the PG overlay. Auto-syncs to /usr/sbin via the existing manager_sbin file.recurse.	2026-04-30 16:34:05 -04:00
Mike Reeves	3fad895d6a	add so_pillar schema + ext_pillar wiring (postsalt foundation) Lays the database-backed pillar foundation for the postsalt branch. Salt continues to read on-disk SLS first; the new ext_pillar config overlays values from the so_pillar.* schema in so-postgres. - salt/postgres/files/schema/pillar/00{1..7}_*.sql: idempotent DDL for scope/role/role_member/minion/pillar_entry/pillar_entry_history/ drift_log, secret pgcrypto helpers, RLS, pg_cron retention. - salt/postgres/schema_pillar.sls: applies the SQL files inside the so-postgres container after it's healthy, configures the master_key GUC, and runs so-pillar-import once. Gated on postgres:so_pillar:enabled feature flag (default false). - salt/salt/master/ext_pillar_postgres.{sls,conf.jinja}: drops /etc/salt/master.d/ext_pillar_postgres.conf with list-form ext_pillar queries (global/role/minion/secrets) and ext_pillar_first: False so bootstrap pillars on disk render before the PG overlay. - salt/postgres/init.sls + salt/salt/master.sls: include the new states. Both new state branches are guarded so a default install with the flag off is a no-op.	2026-04-30 16:30:57 -04:00
		`@@ -1,2 +0,0 @@`
			`https://repo.securityonion.net/file/so-repo/prod/3/oracle/9-uek8`
			`https://repo-alt.securityonion.net/prod/3/oracle/9-uek8`