Move onionconfig writes out of so-yaml

Two coupled changes that together let so_pillar.* be the canonical
config store, with config edits driving service reloads automatically:

so-yaml PG-canonical mode
- Adds /opt/so/conf/so-yaml/mode (and SO_YAML_BACKEND env override) with
  three values: dual (legacy), postgres (PG-only for managed paths),
  disk (emergency rollback). Bootstrap files (secrets.sls, ca/init.sls,
  *.nodes.sls, top.sls, ...) stay disk-only regardless via the existing
  SkipPath allowlist in so_yaml_postgres.locate.
- loadYaml/writeYaml/purgeFile now route to so_pillar.* in postgres
  mode: replace/add/get all read+write the database with no disk file
  ever appearing. PG failure is fatal in postgres mode (no silent
  fallback); dual mode preserves the prior best-effort mirror.
- so_yaml_postgres gains read_yaml(path), is_pg_managed(path), and
  is_enabled() so so-yaml can answer "is this path PG-managed and is
  PG up" without reaching into private helpers.
- schema_pillar.sls writes /opt/so/conf/so-yaml/mode = postgres after
  the importer succeeds, so flipping postgres:so_pillar:enabled flips
  so-yaml's behavior in lockstep with the schema being live.

pg_notify-driven change fan-out
- 008_change_notify.sql adds so_pillar.change_queue + an AFTER trigger
  on pillar_entry that enqueues the locator and pg_notifies
  'so_pillar_change'. Queue is drained at-least-once so engine restarts
  don't lose events; pg_notify is just the wakeup signal.
- New salt-master engine pg_notify_pillar.py LISTENs on the channel,
  drains the queue with FOR UPDATE SKIP LOCKED, debounces bursts, and
  fires 'so/pillar/changed' events grouped by (scope, role, minion).
- Reactor so_pillar_changed.sls catches the tag and dispatches to
  orch.so_pillar_reload, which carries a DISPATCH map of pillar-path
  prefix -> (state sls, role grain set) so adding a new service to
  the auto-reload list is a one-line edit instead of a new reactor.
- Engine + reactor wiring is gated on the same postgres:so_pillar:enabled
  flag as the schema and ext_pillar config so the whole stack flips
  on/off together.

Tests: 21 new cases (112 total, all passing) covering mode resolution,
PG-managed detection, and PG-canonical read/write/purge routing with
the PG client stubbed.

VERSION

@@ -1 +1 @@
-3.0.0-bravo
+3.1.0

salt/allowed_states.map.jinja

@@ -35,6 +35,8 @@
     'kratos',
     'hydra',
     'elasticfleet',
+    'elasticfleet.manager',
+    'elasticsearch.cluster',
     'elastic-fleet-package-registry',
     'utility'
 ] %}
@@ -79,7 +81,7 @@
         ),
         'so-heavynode': (
             sensor_states +
-            ['elasticagent', 'elasticsearch', 'logstash', 'redis', 'nginx']
+            ['elasticagent', 'elasticsearch', 'elasticsearch.cluster', 'logstash', 'redis', 'nginx']
         ),
         'so-idh': (
             ['idh']

salt/common/soup_scripts.sls

@@ -48,6 +48,13 @@ copy_so-yaml_manager_tools_sbin:
     - force: True
     - preserve: True
 
+copy_so-config_manager_tools_sbin:
+  file.copy:
+    - name: /opt/so/saltstack/default/salt/manager/tools/sbin/so-config.py
+    - source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-config.py
+    - force: True
+    - preserve: True
+
 copy_so-repo-sync_manager_tools_sbin:
   file.copy:
     - name: /opt/so/saltstack/default/salt/manager/tools/sbin/so-repo-sync
@@ -97,6 +104,13 @@ copy_so-yaml_sbin:
     - force: True
     - preserve: True
 
+copy_so-config_sbin:
+  file.copy:
+    - name: /usr/sbin/so-config.py
+    - source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-config.py
+    - force: True
+    - preserve: True
+
 copy_so-repo-sync_sbin:
   file.copy:
     - name: /usr/sbin/so-repo-sync

salt/common/tools/sbin/so-image-common

@@ -188,8 +188,14 @@ update_docker_containers() {
         if [ -z "$HOSTNAME" ]; then
           HOSTNAME=$(hostname)
         fi
-        docker tag $CONTAINER_REGISTRY/$IMAGEREPO/$image $HOSTNAME:5000/$IMAGEREPO/$image >> "$LOG_FILE" 2>&1 
+        docker tag $CONTAINER_REGISTRY/$IMAGEREPO/$image $HOSTNAME:5000/$IMAGEREPO/$image >> "$LOG_FILE" 2>&1 || {
-        docker push $HOSTNAME:5000/$IMAGEREPO/$image >> "$LOG_FILE" 2>&1 
+          echo "Unable to tag $image" >> "$LOG_FILE" 2>&1 
+          exit 1
+        }
+        docker push $HOSTNAME:5000/$IMAGEREPO/$image >> "$LOG_FILE" 2>&1 || {
+          echo "Unable to push $image" >> "$LOG_FILE" 2>&1 
+          exit 1
+        }
       fi
     else
       echo "There is a problem downloading the $image image. Details: " >> "$LOG_FILE" 2>&1 

salt/common/tools/sbin/so-log-check

@@ -227,7 +227,7 @@ if [[ $EXCLUDE_KNOWN_ERRORS == 'Y' ]]; then
     EXCLUDED_ERRORS="$EXCLUDED_ERRORS|from NIC checksum offloading" # zeek reporter.log
     EXCLUDED_ERRORS="$EXCLUDED_ERRORS|marked for removal"           # docker container getting recycled
     EXCLUDED_ERRORS="$EXCLUDED_ERRORS|tcp 127.0.0.1:6791: bind: address already in use" # so-elastic-fleet agent restarting. Seen starting w/ 8.18.8 https://github.com/elastic/kibana/issues/201459
-    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|TransformTask\] \[logs-(tychon|aws_billing|microsoft_defender_endpoint).*user so_kibana lacks the required permissions \[logs-\1" # Known issue with 3 integrations using kibana_system role vs creating unique api creds with proper permissions.
+    EXCLUDED_ERRORS="$EXCLUDED_ERRORS|TransformTask\] \[logs-(tychon|aws_billing|microsoft_defender_endpoint|armis|o365_metrics|microsoft_sentinel|snyk).*user so_kibana lacks the required permissions \[(logs|metrics)-\1" # Known issue with integrations starting transform jobs that are explicitly not allowed to start as a system user. (installed as so_elastic / so_kibana)
     EXCLUDED_ERRORS="$EXCLUDED_ERRORS|manifest unknown"             # appears in so-dockerregistry log for so-tcpreplay following docker upgrade to 29.2.1-1
 fi
 

salt/common/tools/sbin_jinja/so-raid-status

@@ -9,7 +9,7 @@
 
 . /usr/sbin/so-common
 
-software_raid=("SOSMN" "SOSMN-DE02" "SOSSNNV" "SOSSNNV-DE02" "SOS10k-DE02" "SOS10KNV" "SOS10KNV-DE02" "SOS10KNV-DE02" "SOS2000-DE02" "SOS-GOFAST-LT-DE02" "SOS-GOFAST-MD-DE02" "SOS-GOFAST-HV-DE02")
+software_raid=("SOSMN" "SOSMN-DE02" "SOSSNNV" "SOSSNNV-DE02" "SOS10k-DE02" "SOS10KNV" "SOS10KNV-DE02" "SOS10KNV-DE02" "SOS2000-DE02" "SOS-GOFAST-LT-DE02" "SOS-GOFAST-MD-DE02" "SOS-GOFAST-HV-DE02" "HVGUEST")
 hardware_raid=("SOS1000" "SOS1000F" "SOSSN7200" "SOS5000" "SOS4000")
 
 {%- if salt['grains.get']('sosmodel', '') %}
@@ -87,6 +87,11 @@ check_boss_raid() {
 }
 
 check_software_raid() {
+  if [[ ! -f /proc/mdstat ]]; then
+    SWRAID=0
+    return
+  fi
+
   SWRC=$(grep "_" /proc/mdstat)
   if [[ -n $SWRC ]]; then
       # RAID is failed in some way
@@ -107,7 +112,9 @@ if [[ "$is_hwraid" == "true" ]]; then
 fi
 if [[ "$is_softwareraid" == "true" ]]; then
 	check_software_raid
+  if [ "$model" != "HVGUEST" ]; then
     check_boss_raid
+  fi
 fi
 
 sum=$(($SWRAID + $BOSSRAID + $HWRAID))

salt/elasticfleet/enabled.sls

@@ -17,65 +17,17 @@ include:
   - logstash.ssl
   - elasticfleet.config
   - elasticfleet.sostatus
+{%- if GLOBALS.role != "so-fleet" %}
+  - elasticfleet.manager
+{%- endif %}
 
-{% if grains.role not in ['so-fleet'] %}
+{% if GLOBALS.role != "so-fleet" %}
 # Wait for Elasticsearch to be ready - no reason to try running Elastic Fleet server if ES is not ready
 wait_for_elasticsearch_elasticfleet:
   cmd.run:
     - name: so-elasticsearch-wait
-{% endif %}
-
-# If enabled, automatically update Fleet Logstash Outputs
-{% if ELASTICFLEETMERGED.config.server.enable_auto_configuration and grains.role not in ['so-import', 'so-eval', 'so-fleet'] %}
-so-elastic-fleet-auto-configure-logstash-outputs:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-outputs-update
-    - retry:
-        attempts: 4
-        interval: 30
-
-{# Separate from above in order to catch elasticfleet-logstash.crt changes and force update to fleet output policy #}
-so-elastic-fleet-auto-configure-logstash-outputs-force:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-outputs-update --certs
-    - retry:
-        attempts: 4
-        interval: 30
-    - onchanges:
-        - x509: etc_elasticfleet_logstash_crt
-        - x509: elasticfleet_kafka_crt
-{% endif %}
-
-# If enabled, automatically update Fleet Server URLs & ES Connection
-{% if ELASTICFLEETMERGED.config.server.enable_auto_configuration and grains.role not in ['so-fleet'] %}
-so-elastic-fleet-auto-configure-server-urls:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-urls-update
-    - retry:
-        attempts: 4
-        interval: 30
-{% endif %}
-
-# Automatically update Fleet Server Elasticsearch URLs & Agent Artifact URLs
-{% if grains.role not in ['so-fleet'] %}
-so-elastic-fleet-auto-configure-elasticsearch-urls:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-es-url-update
-    - retry:
-        attempts: 4
-        interval: 30
-
-so-elastic-fleet-auto-configure-artifact-urls:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-artifacts-url-update
-    - retry:
-        attempts: 4
-        interval: 30
-
-{% endif %}
 
 # Sync Elastic Agent artifacts to Fleet Node
-{% if grains.role in ['so-fleet'] %}
 elasticagent_syncartifacts:
   file.recurse:
     - name: /nsm/elastic-fleet/artifacts/beats
@@ -149,57 +101,6 @@ so-elastic-fleet:
       - x509: etc_elasticfleet_crt
 {%   endif %}
 
-{%  if GLOBALS.role != "so-fleet" %}
-so-elastic-fleet-package-statefile:
-  file.managed:
-    - name: /opt/so/state/elastic_fleet_packages.txt
-    - contents: {{ELASTICFLEETMERGED.packages}}
-
-so-elastic-fleet-package-upgrade:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-package-upgrade
-    - retry:
-        attempts: 3
-        interval: 10
-    - onchanges:
-      - file: /opt/so/state/elastic_fleet_packages.txt
-
-so-elastic-fleet-integrations:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-integration-policy-load
-    - retry:
-        attempts: 3
-        interval: 10
-
-so-elastic-agent-grid-upgrade:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-agent-grid-upgrade
-    - retry:
-        attempts: 12
-        interval: 5
-
-so-elastic-fleet-integration-upgrade:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-integration-upgrade
-    - retry:
-        attempts: 3
-        interval: 10
-
-{# Optional integrations script doesn't need the retries like so-elastic-fleet-integration-upgrade which loads the default integrations #}
-so-elastic-fleet-addon-integrations:
-  cmd.run:
-    - name: /usr/sbin/so-elastic-fleet-optional-integrations-load
-
-{%   if ELASTICFLEETMERGED.config.defend_filters.enable_auto_configuration %}
-so-elastic-defend-manage-filters-file-watch:
-  cmd.run:
-    - name: python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters/ &>> /opt/so/log/elasticfleet/elastic-defend-manage-filters.log
-    - onchanges:
-      - file: elasticdefendcustom
-      - file: elasticdefenddisabled
-{%    endif %}
-{%  endif %}
-
 delete_so-elastic-fleet_so-status.disabled:
   file.uncomment:
     - name: /opt/so/conf/so-status/so-status.conf

salt/elasticfleet/manager.sls

@@ -0,0 +1,112 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls in allowed_states %}
+{%   from 'elasticfleet/map.jinja' import ELASTICFLEETMERGED %}
+
+include:
+  - elasticfleet.config
+
+# If enabled, automatically update Fleet Logstash Outputs
+{% if ELASTICFLEETMERGED.config.server.enable_auto_configuration and grains.role not in ['so-import', 'so-eval'] %}
+so-elastic-fleet-auto-configure-logstash-outputs:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-outputs-update
+    - retry:
+        attempts: 4
+        interval: 30
+
+{# Separate from above in order to catch elasticfleet-logstash.crt changes and force update to fleet output policy #}
+so-elastic-fleet-auto-configure-logstash-outputs-force:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-outputs-update --certs
+    - retry:
+        attempts: 4
+        interval: 30
+    - onchanges:
+        - x509: etc_elasticfleet_logstash_crt
+        - x509: elasticfleet_kafka_crt
+{% endif %}
+
+# If enabled, automatically update Fleet Server URLs & ES Connection
+so-elastic-fleet-auto-configure-server-urls:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-urls-update
+    - retry:
+        attempts: 4
+        interval: 30
+
+# Automatically update Fleet Server Elasticsearch URLs & Agent Artifact URLs
+so-elastic-fleet-auto-configure-elasticsearch-urls:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-es-url-update
+    - retry:
+        attempts: 4
+        interval: 30
+
+so-elastic-fleet-auto-configure-artifact-urls:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-artifacts-url-update
+    - retry:
+        attempts: 4
+        interval: 30
+
+so-elastic-fleet-package-statefile:
+  file.managed:
+    - name: /opt/so/state/elastic_fleet_packages.txt
+    - contents: {{ELASTICFLEETMERGED.packages}}
+
+so-elastic-fleet-package-upgrade:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-package-upgrade
+    - retry:
+        attempts: 3
+        interval: 10
+    - onchanges:
+      - file: /opt/so/state/elastic_fleet_packages.txt
+
+so-elastic-fleet-integrations:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-integration-policy-load
+    - retry:
+        attempts: 3
+        interval: 10
+
+so-elastic-agent-grid-upgrade:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-agent-grid-upgrade
+    - retry:
+        attempts: 12
+        interval: 5
+
+so-elastic-fleet-integration-upgrade:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-integration-upgrade
+    - retry:
+        attempts: 3
+        interval: 10
+
+{# Optional integrations script doesn't need the retries like so-elastic-fleet-integration-upgrade which loads the default integrations #}
+so-elastic-fleet-addon-integrations:
+  cmd.run:
+    - name: /usr/sbin/so-elastic-fleet-optional-integrations-load
+
+{% if ELASTICFLEETMERGED.config.defend_filters.enable_auto_configuration %}
+so-elastic-defend-manage-filters-file-watch:
+  cmd.run:
+    - name: python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters/ &>> /opt/so/log/elasticfleet/elastic-defend-manage-filters.log
+    - onchanges:
+      - file: elasticdefendcustom
+      - file: elasticdefenddisabled
+{% endif %}
+
+{% else %}
+
+{{sls}}_state_not_allowed:
+  test.fail_without_changes:
+    - name: {{sls}}_state_not_allowed
+
+{% endif %}

salt/elasticfleet/tools/sbin_jinja/so-elastic-agent-grid-upgrade

@@ -5,11 +5,12 @@
 # this file except in compliance with the Elastic License 2.0.
 
 . /usr/sbin/so-common
+. /usr/sbin/so-elastic-fleet-common
 {%- import_yaml 'elasticsearch/defaults.yaml' as ELASTICSEARCHDEFAULTS %}
 {%- import_yaml 'elasticfleet/defaults.yaml' as ELASTICFLEETDEFAULTS %}
 {# Optionally override Elasticsearch version for Elastic Agent patch releases #}
 {%- if ELASTICFLEETDEFAULTS.elasticfleet.patch_version is defined %}
-{%-   do ELASTICSEARCHDEFAULTS.update({'elasticsearch': {'version': ELASTICFLEETDEFAULTS.elasticfleet.patch_version}}) %}
+{%-   do ELASTICSEARCHDEFAULTS.elasticsearch.update({'version': ELASTICFLEETDEFAULTS.elasticfleet.patch_version}) %}
 {%- endif %}
 
 # Only run on Managers
@@ -19,11 +20,8 @@ if ! is_manager_node; then
 fi
 
 # Get current list of Grid Node Agents that need to be upgraded
-RAW_JSON=$(curl -K /opt/so/conf/elasticsearch/curl.config -L "http://localhost:5601/api/fleet/agents?perPage=20&page=1&kuery=NOT%20agent.version%3A%20{{ELASTICSEARCHDEFAULTS.elasticsearch.version}}%20AND%20policy_id%3A%20so-grid-nodes_%2A&showInactive=false&getStatusSummary=true" --retry 3 --retry-delay 30 --fail 2>/dev/null)
+if ! RAW_JSON=$(fleet_api "agents?perPage=20&page=1&kuery=NOT%20agent.version%3A%20{{ELASTICSEARCHDEFAULTS.elasticsearch.version | urlencode }}%20AND%20policy_id%3A%20so-grid-nodes_%2A&showInactive=false&getStatusSummary=true" -H 'kbn-xsrf: true' -H 'Content-Type: application/json'); then
 
-# Check to make sure that the server responded with good data - else, bail from script
-CHECKSUM=$(jq -r '.page' <<< "$RAW_JSON")
-if [ "$CHECKSUM" -ne 1 ]; then
     printf "Failed to query for current Grid Agents...\n"
     exit 1
 fi
@@ -36,10 +34,12 @@ if [ "$OUTDATED_LIST" != '[]' ]; then
    printf "Initiating upgrades for $AGENTNUMBERS Agents to Elastic {{ELASTICSEARCHDEFAULTS.elasticsearch.version}}...\n\n"
 
    # Generate updated JSON payload
-   JSON_STRING=$(jq -n --arg ELASTICVERSION {{ELASTICSEARCHDEFAULTS.elasticsearch.version}} --arg UPDATELIST $OUTDATED_LIST '{"version": $ELASTICVERSION,"agents": $UPDATELIST }')
+   JSON_STRING=$(jq -n --arg ELASTICVERSION "{{ELASTICSEARCHDEFAULTS.elasticsearch.version}}" --argjson UPDATELIST "$OUTDATED_LIST" '{"version": $ELASTICVERSION,"agents": $UPDATELIST }')
 
    # Update Node Agents
-   curl -K /opt/so/conf/elasticsearch/curl.config -L -X POST "http://localhost:5601/api/fleet/agents/bulk_upgrade" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d "$JSON_STRING"
+   if ! fleet_api "agents/bulk_upgrade" -XPOST -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d "$JSON_STRING"; then
+       printf "Failed to initiate Agent upgrades...\n"
+   fi
 else
     printf "No Agents need updates... Exiting\n\n"
     exit 0

salt/elasticfleet/tools/sbin_jinja/so-elastic-fleet-setup

@@ -232,6 +232,7 @@ printf '%s\n'\
     "      grid_enrollment_general: '$GRIDNODESENROLLMENTOKENGENERAL'"\
     "      grid_enrollment_heavy: '$GRIDNODESENROLLMENTOKENHEAVY'"\
     "" >> "$pillar_file"
+/usr/sbin/so-config.py import-file "$pillar_file" --note "so-elastic-fleet-setup"
 
 #Store Grid Nodes Enrollment token in Global pillar
 global_pillar_file=/opt/so/saltstack/local/pillar/global/soc_global.sls
@@ -239,6 +240,7 @@ printf '%s\n'\
     "  fleet_grid_enrollment_token_general: '$GRIDNODESENROLLMENTOKENGENERAL'"\
     "  fleet_grid_enrollment_token_heavy: '$GRIDNODESENROLLMENTOKENHEAVY'"\
     "" >> "$global_pillar_file"
+/usr/sbin/so-config.py import-file "$global_pillar_file" --note "so-elastic-fleet-setup"
 
 # Call Elastic-Fleet Salt State
 printf "\nApplying elasticfleet state"

salt/elasticsearch/cluster.sls

@@ -4,7 +4,7 @@
 # Elastic License 2.0.
 
 {% from 'allowed_states.map.jinja' import allowed_states %}
-{% if sls.split('.')[0] in allowed_states %}
+{% if sls in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {%   from 'elasticsearch/template.map.jinja' import ES_INDEX_SETTINGS, SO_MANAGED_INDICES %}

salt/elasticsearch/enabled.sls

@@ -17,7 +17,7 @@ include:
   - elasticsearch.ssl
   - elasticsearch.config
   - elasticsearch.sostatus
-{%- if GLOBALS.role != 'so-searchode' %}
+{%- if GLOBALS.role != "so-searchnode" %}
   - elasticsearch.cluster
 {%- endif%}
 
@@ -102,11 +102,6 @@ so-elasticsearch:
       - cmd: auth_users_roles_inode
       - cmd: auth_users_inode
 
-delete_so-elasticsearch_so-status.disabled:
-  file.uncomment:
-    - name: /opt/so/conf/so-status/so-status.conf
-    - regex: ^so-elasticsearch$
-
 wait_for_so-elasticsearch:
   http.wait_for_successful_query:
     - name: "https://localhost:9200/"
@@ -117,10 +112,14 @@ wait_for_so-elasticsearch:
     - status: 200
     - wait_for: 300
     - request_interval: 15
-    - backend: requests
     - require:
       - docker_container: so-elasticsearch
 
+delete_so-elasticsearch_so-status.disabled:
+  file.uncomment:
+    - name: /opt/so/conf/so-status/so-status.conf
+    - regex: ^so-elasticsearch$
+
 {% else %}
 
 {{sls}}_state_not_allowed:

salt/elasticsearch/tools/sbin/so-elasticsearch-templates-load

@@ -103,11 +103,13 @@ load_component_templates() {
     local pattern="${ELASTICSEARCH_TEMPLATES_DIR}/component/$2"
     local append_mappings="${3:-"false"}"
 
-    # current state of nullglob shell option
-    shopt -q nullglob && nullglob_set=1 || nullglob_set=0
-
-    shopt -s nullglob
     echo -e "\nLoading $printed_name component templates...\n"
+
+    if ! compgen -G "${pattern}/*.json" > /dev/null; then
+        echo "No $printed_name component templates found in ${pattern}, skipping."
+        return
+    fi
+
     for component in "$pattern"/*.json; do
         tmpl_name=$(basename "${component%.json}")
 
@@ -121,11 +123,6 @@ load_component_templates() {
             SO_LOAD_FAILURES_NAMES+=("$component")
         fi
     done
-
-    # restore nullglob shell option if needed
-    if [[ $nullglob_set -eq 1 ]]; then
-        shopt -u nullglob
-    fi
 }
 
 check_elasticsearch_responsive() {
@@ -136,7 +133,32 @@ check_elasticsearch_responsive() {
         fail "Elasticsearch is not responding. Please review Elasticsearch logs /opt/so/log/elasticsearch/securityonion.log for more details. Additionally, consider running so-elasticsearch-troubleshoot."
 }
 
-if [[ "$FORCE" == "true" || ! -f "$SO_STATEFILE_SUCCESS" ]]; then
+index_templates_exist() {
+    local templates_dir="$1"
+
+    if [[ ! -d "$templates_dir" ]]; then
+        return 1
+    fi
+
+    compgen -G "${templates_dir}/*.json" > /dev/null
+}
+
+should_load_addon_templates() {
+    if [[ "$IS_HEAVYNODE" == "true" ]]; then
+        return 1
+    fi
+
+    # Skip statefile checks when forcing template load
+    if [[ "$FORCE" != "true" ]]; then
+        if [[ ! -f "$SO_STATEFILE_SUCCESS" || -f "$ADDON_STATEFILE_SUCCESS" ]]; then
+            return 1
+        fi
+    fi
+
+    index_templates_exist "$ADDON_TEMPLATES_DIR"
+}
+
+if [[ "$FORCE" == "true" || ! -f "$SO_STATEFILE_SUCCESS" ]] && index_templates_exist "$SO_TEMPLATES_DIR"; then
     check_elasticsearch_responsive
 
     if [[ "$IS_HEAVYNODE" == "false" ]]; then
@@ -201,13 +223,14 @@ if [[ "$FORCE" == "true" || ! -f "$SO_STATEFILE_SUCCESS" ]]; then
             fail "Failed to load all Security Onion core templates successfully."
         fi
     fi
-else
+elif ! index_templates_exist "$SO_TEMPLATES_DIR"; then
-
+    echo "No Security Onion core index templates found in ${SO_TEMPLATES_DIR}, skipping."
+elif [[ -f "$SO_STATEFILE_SUCCESS" ]]; then
     echo "Security Onion core templates already loaded"
 fi
 
 # Start loading addon templates
-if [[ (-d "$ADDON_TEMPLATES_DIR" && -f "$SO_STATEFILE_SUCCESS" && "$IS_HEAVYNODE" == "false" && ! -f "$ADDON_STATEFILE_SUCCESS") || (-d "$ADDON_TEMPLATES_DIR" && "$IS_HEAVYNODE" == "false" && "$FORCE" == "true") ]]; then
+if should_load_addon_templates; then
 
     check_elasticsearch_responsive
 

salt/global/soc_global.yaml

@@ -59,5 +59,4 @@ global:
     description: Allows use of Endgame with Security Onion. This feature requires a license from Endgame.
     global: True
     advanced: True
-    helpLink: influxdb
 

salt/kafka/disabled.sls

@@ -20,8 +20,11 @@ so-kafka_so-status.disabled:
 ensure_default_pipeline:
   cmd.run:
     - name: |
-        /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.enabled False;
+        set -e
+        /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.enabled False
+        /usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls replace kafka.enabled False --note "kafka.disabled"
         /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/global/soc_global.sls global.pipeline REDIS
+        /usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/global/soc_global.sls replace global.pipeline REDIS --note "kafka.disabled"
 {% endif %}
 
 {# If Kafka has never been manually enabled, the 'Kafka' user does not exist. In this case certs for Kafka should not exist since they'll be owned by uid 960 #}

salt/kibana/defaults.yaml

@@ -22,7 +22,7 @@ kibana:
           - default
           - file
     migrations:
-      discardCorruptObjects: "8.18.8"
+      discardCorruptObjects: "9.3.3"
     telemetry:
       enabled: False
     xpack:

salt/kratos/soc_kratos.yaml

@@ -3,8 +3,8 @@ kratos:
     description: Enables or disables the Kratos authentication system. WARNING - Disabling this process will cause the grid to malfunction. Re-enabling this setting will require manual effort via SSH.
     forcedType: bool
     advanced: True
+    readonly: True
     helpLink: kratos
-
   oidc:
     enabled:
       description: Set to True to enable OIDC / Single Sign-On (SSO) to SOC. Requires a valid Security Onion license key.

salt/manager/tools/sbin/so-config.py

@@ -0,0 +1,448 @@
+#!/usr/bin/env python3
+
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+"""
+so-config.py writes SOC/onionconfig settings to Postgres.
+
+so-yaml.py remains a YAML file editor. Call this tool when a pillar-backed
+setting also needs to be reflected in the onionconfig database.
+"""
+
+import argparse
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+
+import yaml
+
+
+PILLAR_ROOT = Path(os.environ.get("SO_CONFIG_PILLAR_ROOT", "/opt/so/saltstack/local/pillar"))
+DOCKER_CONTAINER = os.environ.get("SO_CONFIG_PG_CONTAINER", "so-postgres")
+PG_DATABASE = os.environ.get("SO_CONFIG_PG_DATABASE", "securityonion")
+PG_USER = os.environ.get("SO_CONFIG_PG_USER", "postgres")
+DEFAULT_USER_ID = os.environ.get("SO_CONFIG_USER_ID", "so-config")
+
+EXCLUDE_BASENAMES = {
+    "secrets.sls",
+    "auth.sls",
+    "top.sls",
+}
+EXCLUDE_PATH_FRAGMENTS = (
+    "/elasticsearch/nodes.sls",
+    "/redis/nodes.sls",
+    "/kafka/nodes.sls",
+    "/hypervisor/nodes.sls",
+    "/logstash/nodes.sls",
+    "/node_data/ips.sls",
+    "/postgres/auth.sls",
+    "/elasticsearch/auth.sls",
+    "/kibana/secrets.sls",
+)
+
+
+class SkipPath(Exception):
+    pass
+
+
+def pg_str(value):
+    if value is None:
+        return "NULL"
+    return "'" + str(value).replace("'", "''") + "'"
+
+
+def pg_jsonb(value):
+    return pg_str(json.dumps(value)) + "::jsonb"
+
+
+def docker_psql(sql):
+    proc = subprocess.run(
+        ["docker", "exec", "-i", DOCKER_CONTAINER,
+         "psql", "-U", PG_USER, "-d", PG_DATABASE,
+         "-tA", "-q", "-v", "ON_ERROR_STOP=1"],
+        input=sql.encode(),
+        capture_output=True,
+        check=False,
+        timeout=60,
+    )
+    if proc.returncode != 0:
+        sys.stderr.write(proc.stderr.decode(errors="replace"))
+        raise RuntimeError(f"docker exec psql failed with rc={proc.returncode}")
+    return proc.stdout.decode(errors="replace")
+
+
+def schema_ready():
+    sql = """
+SELECT to_regclass('public.settings') IS NOT NULL
+   AND to_regclass('public.audit_settings') IS NOT NULL;
+"""
+    return docker_psql(sql).strip() == "t"
+
+
+def cmd_wait_schema(args):
+    import time
+
+    deadline = time.time() + args.timeout
+    while time.time() <= deadline:
+        if schema_ready():
+            return 0
+        time.sleep(args.interval)
+    print("so-config: onionconfig schema is not ready", file=sys.stderr)
+    return 1
+
+
+def upsert_setting(setting_id, value, *, node_id="", duplicated_from_id=None,
+                   user_id=DEFAULT_USER_ID, note=None):
+    note = note or "so-config upsert"
+    sql = f"""
+BEGIN;
+WITH old_row AS (
+    SELECT value
+      FROM settings
+     WHERE setting_id = {pg_str(setting_id)}
+       AND node_id = {pg_str(node_id)}
+     FOR UPDATE
+),
+upserted AS (
+    INSERT INTO settings (setting_id, value, duplicated_from_id, node_id)
+    VALUES ({pg_str(setting_id)}, {pg_jsonb(value)}, {pg_str(duplicated_from_id)}, {pg_str(node_id)})
+    ON CONFLICT (setting_id, node_id) DO UPDATE
+       SET value = EXCLUDED.value,
+           duplicated_from_id = EXCLUDED.duplicated_from_id
+    RETURNING value
+)
+INSERT INTO audit_settings (setting_id, node_id, user_id, old_value, new_value, note)
+SELECT {pg_str(setting_id)},
+       {pg_str(node_id)},
+       {pg_str(user_id)},
+       (SELECT value FROM old_row),
+       (SELECT value FROM upserted),
+       {pg_str(note)}
+ WHERE NOT EXISTS (SELECT 1 FROM old_row)
+    OR (SELECT value FROM old_row) IS DISTINCT FROM (SELECT value FROM upserted);
+COMMIT;
+"""
+    docker_psql(sql)
+
+
+def delete_setting(setting_id, *, node_id="", user_id=DEFAULT_USER_ID, note=None):
+    note = note or "so-config delete"
+    sql = f"""
+BEGIN;
+WITH deleted AS (
+    DELETE FROM settings
+     WHERE setting_id = {pg_str(setting_id)}
+       AND node_id = {pg_str(node_id)}
+     RETURNING value
+)
+INSERT INTO audit_settings (setting_id, node_id, user_id, old_value, new_value, note)
+SELECT {pg_str(setting_id)}, {pg_str(node_id)}, {pg_str(user_id)}, value, NULL::jsonb, {pg_str(note)}
+  FROM deleted;
+COMMIT;
+"""
+    docker_psql(sql)
+
+
+def delete_setting_prefix(setting_id, *, node_id="", user_id=DEFAULT_USER_ID, note=None):
+    if not setting_id:
+        raise ValueError("setting_id prefix cannot be empty")
+    note = note or "so-config delete-prefix"
+    sql = f"""
+BEGIN;
+WITH deleted AS (
+    DELETE FROM settings
+     WHERE node_id = {pg_str(node_id)}
+       AND (
+             setting_id = {pg_str(setting_id)}
+          OR substring(setting_id from 1 for char_length({pg_str(setting_id)}) + 1) = {pg_str(setting_id + ".")}
+       )
+     RETURNING setting_id, value
+)
+INSERT INTO audit_settings (setting_id, node_id, user_id, old_value, new_value, note)
+SELECT setting_id, {pg_str(node_id)}, {pg_str(user_id)}, value, NULL::jsonb, {pg_str(note)}
+  FROM deleted;
+COMMIT;
+"""
+    docker_psql(sql)
+
+
+def purge_node(node_id, *, user_id=DEFAULT_USER_ID, note=None):
+    note = note or "so-config purge-node"
+    sql = f"""
+BEGIN;
+WITH deleted AS (
+    DELETE FROM settings
+     WHERE node_id = {pg_str(node_id)}
+     RETURNING setting_id, value
+)
+INSERT INTO audit_settings (setting_id, node_id, user_id, old_value, new_value, note)
+SELECT setting_id, {pg_str(node_id)}, {pg_str(user_id)}, value, NULL::jsonb, {pg_str(note)}
+  FROM deleted;
+COMMIT;
+"""
+    docker_psql(sql)
+
+
+def parse_value(value, value_file=None):
+    if value_file:
+        with open(value_file, "r") as fh:
+            value = fh.read()
+    parsed = yaml.safe_load(value)
+    if parsed is None and value == "":
+        return ""
+    return parsed
+
+
+def parse_yaml_file(path):
+    with open(path, "rb") as fh:
+        raw = fh.read()
+    if b"{%" in raw or b"{{" in raw:
+        raise SkipPath(f"{path}: Jinja-templated files stay disk-only")
+    if not raw.strip():
+        return {}
+    parsed = yaml.safe_load(raw)
+    return parsed if parsed is not None else {}
+
+
+def flatten(prefix, value):
+    if isinstance(value, dict):
+        for key, child in value.items():
+            child_id = f"{prefix}.{key}" if prefix else str(key)
+            yield from flatten(child_id, child)
+    else:
+        yield prefix, value
+
+
+def classify_pillar_path(path):
+    norm = Path(path).resolve()
+    norm_str = str(norm)
+
+    if norm.name in EXCLUDE_BASENAMES:
+        raise SkipPath(f"{path}: excluded basename")
+    for fragment in EXCLUDE_PATH_FRAGMENTS:
+        if fragment in norm_str:
+            raise SkipPath(f"{path}: excluded path fragment {fragment}")
+    if norm.suffix != ".sls":
+        raise SkipPath(f"{path}: not an .sls file")
+
+    parent = norm.parent.name
+    stem = norm.stem
+
+    if parent == "minions":
+        if stem.startswith("adv_"):
+            return {"kind": "advanced", "setting_id": "advanced", "node_id": stem[4:]}
+        return {"kind": "normal", "node_id": stem}
+
+    section = parent
+    if stem == f"soc_{section}":
+        return {"kind": "normal", "node_id": ""}
+    if stem == f"adv_{section}":
+        return {"kind": "advanced", "setting_id": f"{section}.advanced", "node_id": ""}
+
+    raise SkipPath(f"{path}: not a SOC-managed pillar file")
+
+
+def import_pillar_file(path, *, user_id=DEFAULT_USER_ID, note=None):
+    meta = classify_pillar_path(path)
+    note = note or f"so-config import-file {path}"
+
+    if meta["kind"] == "advanced":
+        with open(path, "r") as fh:
+            upsert_setting(meta["setting_id"], fh.read(), node_id=meta["node_id"],
+                           user_id=user_id, note=note)
+        return 1
+
+    data = parse_yaml_file(path)
+    if not isinstance(data, dict):
+        raise SkipPath(f"{path}: top-level YAML is not a map")
+
+    count = 0
+    for setting_id, value in flatten("", data):
+        upsert_setting(setting_id, value, node_id=meta["node_id"],
+                       user_id=user_id, note=note)
+        count += 1
+    return count
+
+
+def iter_pillar_files(root):
+    root = Path(root)
+    if not root.is_dir():
+        return
+    for path in sorted(root.rglob("*.sls")):
+        if path.is_file():
+            yield path
+
+
+def cmd_set(args):
+    upsert_setting(args.setting_id, parse_value(args.value, args.value_file),
+                   node_id=args.node_id,
+                   duplicated_from_id=args.duplicated_from_id,
+                   user_id=args.user_id,
+                   note=args.note)
+    return 0
+
+
+def cmd_delete(args):
+    delete_setting(args.setting_id, node_id=args.node_id,
+                   user_id=args.user_id, note=args.note)
+    return 0
+
+
+def cmd_delete_prefix(args):
+    delete_setting_prefix(args.setting_id, node_id=args.node_id,
+                          user_id=args.user_id, note=args.note)
+    return 0
+
+
+def cmd_purge_node(args):
+    purge_node(args.node_id, user_id=args.user_id, note=args.note)
+    return 0
+
+
+def cmd_import_file(args):
+    count = import_pillar_file(args.path, user_id=args.user_id, note=args.note)
+    print(f"imported {count} settings from {args.path}")
+    return 0
+
+
+def cmd_import_minion(args):
+    count = 0
+    for name in (f"{args.node_id}.sls", f"adv_{args.node_id}.sls"):
+        path = PILLAR_ROOT / "minions" / name
+        if path.exists():
+            count += import_pillar_file(path, user_id=args.user_id, note=args.note)
+    print(f"imported {count} settings for node {args.node_id}")
+    return 0
+
+
+def cmd_import_all(args):
+    count = 0
+    skipped = 0
+    for path in iter_pillar_files(args.root):
+        try:
+            count += import_pillar_file(path, user_id=args.user_id, note=args.note)
+        except SkipPath as exc:
+            skipped += 1
+            if args.verbose:
+                print(f"skip: {exc}", file=sys.stderr)
+    print(f"imported {count} settings, skipped {skipped} files")
+    if args.state_file:
+        with open(args.state_file, "w") as fh:
+            fh.write("ok\n")
+    return 0
+
+
+def cmd_sync_yaml_mutation(args):
+    meta = classify_pillar_path(args.path)
+    note = args.note or f"so-config sync-yaml-mutation {args.operation} {args.path}"
+
+    if meta["kind"] == "advanced":
+        import_pillar_file(args.path, user_id=args.user_id, note=note)
+        return 0
+
+    if args.operation in ("add", "replace"):
+        upsert_setting(args.key, parse_value(args.value, args.value_file),
+                       node_id=meta["node_id"],
+                       user_id=args.user_id,
+                       note=note)
+    elif args.operation == "remove":
+        delete_setting_prefix(args.key, node_id=meta["node_id"],
+                              user_id=args.user_id, note=note)
+    else:
+        raise ValueError(f"unsupported operation: {args.operation}")
+    return 0
+
+
+def build_parser():
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p = sub.add_parser("wait-schema", help="wait for SOC-created onionconfig tables")
+    p.add_argument("--timeout", type=int, default=120)
+    p.add_argument("--interval", type=int, default=2)
+    p.set_defaults(func=cmd_wait_schema)
+
+    p = sub.add_parser("set", help="upsert one setting")
+    p.add_argument("setting_id")
+    p.add_argument("value", nargs="?", default="")
+    p.add_argument("--value-file")
+    p.add_argument("--node-id", default="")
+    p.add_argument("--duplicated-from-id")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_set)
+
+    p = sub.add_parser("delete", help="delete one setting")
+    p.add_argument("setting_id")
+    p.add_argument("--node-id", default="")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_delete)
+
+    p = sub.add_parser("delete-prefix", help="delete one setting and all child settings")
+    p.add_argument("setting_id")
+    p.add_argument("--node-id", default="")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_delete_prefix)
+
+    p = sub.add_parser("purge-node", help="delete all settings for one node")
+    p.add_argument("node_id")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_purge_node)
+
+    p = sub.add_parser("import-file", help="import one SOC-managed pillar file")
+    p.add_argument("path")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_import_file)
+
+    p = sub.add_parser("import-minion", help="import one minion's pillar files")
+    p.add_argument("node_id")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_import_minion)
+
+    p = sub.add_parser("import-all", help="import all SOC-managed local pillar files")
+    p.add_argument("--root", default=str(PILLAR_ROOT))
+    p.add_argument("--state-file")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note", default="so-config initial import")
+    p.add_argument("--verbose", action="store_true")
+    p.set_defaults(func=cmd_import_all)
+
+    p = sub.add_parser("sync-yaml-mutation",
+                       help="mirror one so-yaml add/replace/remove mutation to onionconfig")
+    p.add_argument("path")
+    p.add_argument("operation", choices=("add", "replace", "remove"))
+    p.add_argument("key")
+    p.add_argument("value", nargs="?", default="")
+    p.add_argument("--value-file")
+    p.add_argument("--user-id", default=DEFAULT_USER_ID)
+    p.add_argument("--note")
+    p.set_defaults(func=cmd_sync_yaml_mutation)
+
+    return parser
+
+
+def main(argv):
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    try:
+        return args.func(args)
+    except SkipPath as exc:
+        print(f"skip: {exc}", file=sys.stderr)
+        return 2
+    except Exception as exc:
+        print(f"so-config: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

salt/manager/tools/sbin/so-config_test.py

@@ -0,0 +1,178 @@
+import importlib
+import os
+import tempfile
+import unittest
+from unittest.mock import patch
+
+
+soconfig = importlib.import_module("so-config")
+
+
+class TestSoConfigPathMapping(unittest.TestCase):
+
+    def test_classify_global_soc(self):
+        meta = soconfig.classify_pillar_path(
+            "/opt/so/saltstack/local/pillar/soc/soc_soc.sls")
+        self.assertEqual(meta["kind"], "normal")
+        self.assertEqual(meta["node_id"], "")
+
+    def test_classify_global_advanced(self):
+        meta = soconfig.classify_pillar_path(
+            "/opt/so/saltstack/local/pillar/soc/adv_soc.sls")
+        self.assertEqual(meta["kind"], "advanced")
+        self.assertEqual(meta["setting_id"], "soc.advanced")
+        self.assertEqual(meta["node_id"], "")
+
+    def test_classify_minion(self):
+        meta = soconfig.classify_pillar_path(
+            "/opt/so/saltstack/local/pillar/minions/h1_sensor.sls")
+        self.assertEqual(meta["kind"], "normal")
+        self.assertEqual(meta["node_id"], "h1_sensor")
+
+    def test_classify_minion_advanced(self):
+        meta = soconfig.classify_pillar_path(
+            "/opt/so/saltstack/local/pillar/minions/adv_h1_sensor.sls")
+        self.assertEqual(meta["kind"], "advanced")
+        self.assertEqual(meta["setting_id"], "advanced")
+        self.assertEqual(meta["node_id"], "h1_sensor")
+
+    def test_classify_skips_bootstrap(self):
+        with self.assertRaises(soconfig.SkipPath):
+            soconfig.classify_pillar_path(
+                "/opt/so/saltstack/local/pillar/secrets.sls")
+
+
+class TestSoConfigImport(unittest.TestCase):
+
+    def test_flatten_keeps_lists_as_values(self):
+        flattened = dict(soconfig.flatten("", {
+            "host": {"mainip": "10.0.0.1"},
+            "suricata": {"pcap": {"enabled": True}},
+            "items": ["a", "b"],
+        }))
+        self.assertEqual(flattened["host.mainip"], "10.0.0.1")
+        self.assertEqual(flattened["suricata.pcap.enabled"], True)
+        self.assertEqual(flattened["items"], ["a", "b"])
+
+    def test_import_file_upserts_flattened_settings(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "h1_sensor.sls")
+            minions = os.path.join(tmp, "minions")
+            os.mkdir(minions)
+            path = os.path.join(minions, "h1_sensor.sls")
+            with open(path, "w") as fh:
+                fh.write("host:\n  mainip: 10.0.0.1\nsuricata:\n  enabled: true\n")
+
+            calls = []
+            with patch.object(soconfig, "upsert_setting",
+                              side_effect=lambda *args, **kwargs: calls.append((args, kwargs))):
+                count = soconfig.import_pillar_file(path)
+
+        self.assertEqual(count, 2)
+        self.assertIn((("host.mainip", "10.0.0.1"), {"node_id": "h1_sensor", "user_id": "so-config", "note": f"so-config import-file {path}"}), calls)
+        self.assertIn((("suricata.enabled", True), {"node_id": "h1_sensor", "user_id": "so-config", "note": f"so-config import-file {path}"}), calls)
+
+    def test_import_advanced_file_upserts_raw_content(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            minions = os.path.join(tmp, "minions")
+            os.mkdir(minions)
+            path = os.path.join(minions, "adv_h1_sensor.sls")
+            with open(path, "w") as fh:
+                fh.write("custom:\n  raw: true\n")
+
+            calls = []
+            with patch.object(soconfig, "upsert_setting",
+                              side_effect=lambda *args, **kwargs: calls.append((args, kwargs))):
+                count = soconfig.import_pillar_file(path)
+
+        self.assertEqual(count, 1)
+        self.assertEqual(calls[0][0], ("advanced", "custom:\n  raw: true\n"))
+        self.assertEqual(calls[0][1]["node_id"], "h1_sensor")
+
+
+class TestSoConfigSql(unittest.TestCase):
+
+    def test_schema_ready_checks_soc_tables(self):
+        captured = {}
+        with patch.object(soconfig, "docker_psql",
+                          side_effect=lambda sql: captured.update({"sql": sql}) or "t\n"):
+            ready = soconfig.schema_ready()
+
+        self.assertTrue(ready)
+        self.assertIn("to_regclass('public.settings')", captured["sql"])
+        self.assertIn("to_regclass('public.audit_settings')", captured["sql"])
+
+    def test_set_writes_settings_and_audit(self):
+        captured = {}
+        with patch.object(soconfig, "docker_psql",
+                          side_effect=lambda sql: captured.setdefault("sql", sql)):
+            soconfig.upsert_setting("host.mainip", "10.0.0.1",
+                                    node_id="h1_sensor", user_id="tester", note="unit")
+
+        self.assertIn("INSERT INTO settings", captured["sql"])
+        self.assertIn("INSERT INTO audit_settings", captured["sql"])
+        self.assertIn("'host.mainip'", captured["sql"])
+        self.assertIn("'h1_sensor'", captured["sql"])
+        self.assertIn("'tester'", captured["sql"])
+
+    def test_purge_node_audits_deleted_rows(self):
+        captured = {}
+        with patch.object(soconfig, "docker_psql",
+                          side_effect=lambda sql: captured.setdefault("sql", sql)):
+            soconfig.purge_node("h1_sensor", user_id="tester", note="unit")
+
+        self.assertIn("DELETE FROM settings", captured["sql"])
+        self.assertIn("WHERE node_id = 'h1_sensor'", captured["sql"])
+        self.assertIn("INSERT INTO audit_settings", captured["sql"])
+
+    def test_delete_prefix_removes_children_and_audits(self):
+        captured = {}
+        with patch.object(soconfig, "docker_psql",
+                          side_effect=lambda sql: captured.setdefault("sql", sql)):
+            soconfig.delete_setting_prefix("elasticfleet", node_id="h1_sensor",
+                                           user_id="tester", note="unit")
+
+        self.assertIn("DELETE FROM settings", captured["sql"])
+        self.assertIn("setting_id = 'elasticfleet'", captured["sql"])
+        self.assertIn("'elasticfleet.'", captured["sql"])
+        self.assertIn("INSERT INTO audit_settings", captured["sql"])
+
+    def test_sync_yaml_replace_uses_path_node_id(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            minions = os.path.join(tmp, "minions")
+            os.mkdir(minions)
+            path = os.path.join(minions, "h1_sensor.sls")
+            open(path, "w").close()
+
+            calls = []
+            args = soconfig.build_parser().parse_args([
+                "sync-yaml-mutation", path, "replace", "suricata.enabled", "true"
+            ])
+            with patch.object(soconfig, "upsert_setting",
+                              side_effect=lambda *a, **kw: calls.append((a, kw))):
+                soconfig.cmd_sync_yaml_mutation(args)
+
+        self.assertEqual(calls[0][0], ("suricata.enabled", True))
+        self.assertEqual(calls[0][1]["node_id"], "h1_sensor")
+
+    def test_sync_yaml_remove_deletes_prefix(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            minions = os.path.join(tmp, "minions")
+            os.mkdir(minions)
+            path = os.path.join(minions, "h1_sensor.sls")
+            open(path, "w").close()
+
+            calls = []
+            args = soconfig.build_parser().parse_args([
+                "sync-yaml-mutation", path, "remove", "elasticfleet"
+            ])
+            with patch.object(soconfig, "delete_setting_prefix",
+                              side_effect=lambda *a, **kw: calls.append((a, kw))):
+                soconfig.cmd_sync_yaml_mutation(args)
+
+        self.assertEqual(calls[0][0], ("elasticfleet",))
+        self.assertEqual(calls[0][1]["node_id"], "h1_sensor")
+
+
+if __name__ == "__main__":
+    unittest.main()

salt/manager/tools/sbin/so-minion

@@ -314,6 +314,24 @@ EOSQL
 	fi
 }
 
+function sync_minion_config_to_db() {
+	log "INFO" "Syncing minion config to onionconfig for $MINION_ID"
+	/usr/sbin/so-config.py import-minion "$MINION_ID" --note "so-minion $OPERATION"
+	if [ $? -ne 0 ]; then
+		log "ERROR" "Failed to sync minion config to onionconfig for $MINION_ID"
+		return 1
+	fi
+}
+
+function purge_minion_config_from_db() {
+	log "INFO" "Purging minion config from onionconfig for $MINION_ID"
+	/usr/sbin/so-config.py purge-node "$MINION_ID" --note "so-minion delete"
+	if [ $? -ne 0 ]; then
+		log "ERROR" "Failed to purge minion config from onionconfig for $MINION_ID"
+		return 1
+	fi
+}
+
 # Create the minion file
 function ensure_socore_ownership() {
 	log "INFO" "Setting socore ownership on minion files"
@@ -1088,6 +1106,10 @@ case "$OPERATION" in
 			log "ERROR" "Failed to setup minion files for $MINION_ID"
 			exit 1
 		}
+		sync_minion_config_to_db || {
+			log "ERROR" "Failed to sync minion config to onionconfig for $MINION_ID"
+			exit 1
+		}
 		updateMineAndApplyStates || {
 			log "ERROR" "Failed to update mine and apply states for $MINION_ID"
 			exit 1
@@ -1108,12 +1130,20 @@ case "$OPERATION" in
 			log "ERROR" "Failed to setup VM minion files for $MINION_ID"
 			exit 1
 		}
+		sync_minion_config_to_db || {
+			log "ERROR" "Failed to sync VM minion config to onionconfig for $MINION_ID"
+			exit 1
+		}
 		log "INFO" "Successfully added VM minion $MINION_ID"
 		;;
 
 	"delete")
 		log "INFO" "Removing minion $MINION_ID"
 		remove_postgres_telegraf_from_minion
+		purge_minion_config_from_db || {
+			log "ERROR" "Failed to purge minion config from onionconfig for $MINION_ID"
+			exit 1
+		}
 		deleteMinionFiles || {
 			log "ERROR" "Failed to delete minion files for $MINION_ID"
 			exit 1

salt/manager/tools/sbin/so-yaml.py

@@ -25,6 +25,7 @@ def showUsage(args):
     print('    get [-r]         - Displays (to stdout) the value stored in the given key. Requires KEY arg. Use -r for raw output without YAML formatting.', file=sys.stderr)
     print('    remove           - Removes a yaml key, if it exists. Requires KEY arg.', file=sys.stderr)
     print('    replace          - Replaces (or adds) a new key and set its value. Requires KEY and VALUE args.', file=sys.stderr)
+    print('    purge            - Delete the YAML file from disk (no KEY arg).', file=sys.stderr)
     print('    help             - Prints this usage information.', file=sys.stderr)
     print('', file=sys.stderr)
     print('  Where:', file=sys.stderr)
@@ -53,7 +54,20 @@ def loadYaml(filename):
 
 def writeYaml(filename, content):
     file = open(filename, "w")
-    return yaml.safe_dump(content, file)
+    result = yaml.safe_dump(content, file)
+    file.close()
+    return result
+
+
+def purgeFile(filename):
+    """Delete a YAML file from disk. Idempotent; missing files are success."""
+    if os.path.exists(filename):
+        try:
+            os.remove(filename)
+        except Exception as e:
+            print(f"Failed to remove {filename}: {e}", file=sys.stderr)
+            return 1
+    return 0
 
 
 def appendItem(content, key, listItem):
@@ -371,6 +385,15 @@ def get(args):
     return 0
 
 
+def purge(args):
+    """purge YAML_FILE - delete the file from disk."""
+    if len(args) != 1:
+        print('Missing filename arg', file=sys.stderr)
+        showUsage(None)
+        return 1
+    return purgeFile(args[0])
+
+
 def main():
     args = sys.argv[1:]
 
@@ -388,6 +411,7 @@ def main():
         "get": get,
         "remove": remove,
         "replace": replace,
+        "purge": purge,
     }
 
     code = 1

salt/manager/tools/sbin/so-yaml_test.py

@@ -991,3 +991,31 @@ class TestLoadYaml(unittest.TestCase):
                     soyaml.loadYaml("/tmp/so-yaml_test-unreadable.yaml")
                     sysmock.assert_called_with(1)
                     self.assertIn("Error reading file", mock_stderr.getvalue())
+
+
+class TestPurge(unittest.TestCase):
+
+    def test_purge_missing_arg(self):
+        # showUsage calls sys.exit(1); patch it like the other tests do.
+        with patch('sys.exit', new=MagicMock()):
+            with patch('sys.stderr', new=StringIO()) as mock_stderr:
+                rc = soyaml.purge([])
+                self.assertEqual(rc, 1)
+                self.assertIn("Missing filename", mock_stderr.getvalue())
+
+    def test_purge_existing_file(self):
+        filename = "/tmp/so-yaml_test_purge.yaml"
+        with open(filename, "w") as f:
+            f.write("key: value\n")
+        rc = soyaml.purge([filename])
+        self.assertEqual(rc, 0)
+        import os as _os
+        self.assertFalse(_os.path.exists(filename))
+
+    def test_purge_missing_file_idempotent(self):
+        filename = "/tmp/so-yaml_test_purge_missing.yaml"
+        import os as _os
+        if _os.path.exists(filename):
+            _os.remove(filename)
+        rc = soyaml.purge([filename])
+        self.assertEqual(rc, 0)

salt/manager/tools/sbin/soup

@@ -24,6 +24,14 @@ BACKUPTOPFILE=/opt/so/saltstack/default/salt/top.sls.backup
 SALTUPGRADED=false
 SALT_CLOUD_INSTALLED=false
 SALT_CLOUD_CONFIGURED=false
+# Check if salt-cloud is installed
+if rpm -q salt-cloud &>/dev/null; then
+  SALT_CLOUD_INSTALLED=true
+fi
+# Check if salt-cloud is configured
+if [[ -f /etc/salt/cloud.profiles.d/socloud.conf ]]; then
+  SALT_CLOUD_CONFIGURED=true
+fi
 # used to display messages to the user at the end of soup
 declare -a FINAL_MESSAGE_QUEUE=()
 
@@ -526,6 +534,10 @@ up_to_3.1.0() {
 
 post_to_3.1.0() {
   /usr/sbin/so-kibana-space-defaults
+  # ensure manager has new version of socloud.conf
+  if [[ $SALT_CLOUD_CONFIGURED == true ]]; then
+    salt-call state.apply salt.cloud.config concurrent=True
+  fi
 
   # Backfill the Telegraf creds pillar for every accepted minion. so-telegraf-cred
   # add is idempotent — it no-ops when an entry already exists — so this is safe
@@ -714,15 +726,6 @@ upgrade_check_salt() {
 upgrade_salt() {
   echo "Performing upgrade of Salt from $INSTALLEDSALTVERSION to $NEWSALTVERSION."
   echo ""
-  # Check if salt-cloud is installed
-  if rpm -q salt-cloud &>/dev/null; then
-    SALT_CLOUD_INSTALLED=true
-  fi
-  # Check if salt-cloud is configured
-  if [[ -f /etc/salt/cloud.profiles.d/socloud.conf ]]; then
-    SALT_CLOUD_CONFIGURED=true
-  fi
-
   echo "Removing yum versionlock for Salt."
   echo ""
   yum versionlock delete "salt"

salt/manager/tools/sbin_jinja/so-elastic-fleet-reset

@@ -33,8 +33,11 @@ so-elastic-fleet-stop --force
 
 status "Deleting Fleet Data from Pillars..."
 so-yaml.py remove /opt/so/saltstack/local/pillar/minions/{{ GLOBALS.minion_id }}.sls elasticfleet
+/usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/minions/{{ GLOBALS.minion_id }}.sls remove elasticfleet --note "so-elastic-fleet-reset"
 so-yaml.py remove /opt/so/saltstack/local/pillar/global/soc_global.sls global.fleet_grid_enrollment_token_general
+/usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/global/soc_global.sls remove global.fleet_grid_enrollment_token_general --note "so-elastic-fleet-reset"
 so-yaml.py remove /opt/so/saltstack/local/pillar/global/soc_global.sls global.fleet_grid_enrollment_token_heavy
+/usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/global/soc_global.sls remove global.fleet_grid_enrollment_token_heavy --note "so-elastic-fleet-reset"
 
 status "Restarting Kibana..."
 so-kibana-restart --force

salt/postgres/schema_pillar.sls

@@ -0,0 +1,20 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls.split('.')[0] in allowed_states %}
+
+# Deprecated: the old so_pillar schema has been replaced by SOC-owned
+# onionconfig tables. SOC creates its schema on first startup.
+postgres_schema_pillar_deprecated:
+  test.nop
+
+{% else %}
+
+{{sls}}_state_not_allowed:
+  test.fail_without_changes:
+    - name: {{sls}}_state_not_allowed
+
+{% endif %}

salt/reactor/sominion_setup.sls

@@ -6,39 +6,74 @@
 # Elastic License 2.0.
 
 import logging
-from subprocess import call
+import os
-import yaml
+import re
+import shlex
+import subprocess
 
 log = logging.getLogger(__name__)
 
+SO_MINION = '/usr/sbin/so-minion'
+
+_NODETYPE_RE = re.compile(r'^[A-Z][A-Z0-9_]{0,31}$')
+_MINIONID_RE = re.compile(r'^[A-Za-z0-9._-]{1,253}$')
+_HOSTPART_RE = re.compile(r'^[A-Za-z0-9._-]{1,253}$')
+_IPV4_RE = re.compile(
+    r'^(?:(?:25[0-5]|2[0-4]\d|[01]?\d?\d)\.){3}'
+    r'(?:25[0-5]|2[0-4]\d|[01]?\d?\d)$'
+)
+_HEAP_RE = re.compile(r'^\d{1,6}[kKmMgG]?$')
+
+
+def _check(name, value, pattern):
+  s = str(value)
+  if not pattern.match(s):
+    raise ValueError("sominion_setup_reactor: refusing unsafe %s=%r" % (name, value))
+  return s
+
+
 def run():
   log.info('sominion_setup_reactor: Running')
   minionid = data['id']
   DATA = data['data']
-  hv_name = DATA['HYPERVISOR_HOST']
   log.info('sominion_setup_reactor: DATA: %s' % DATA)
 
-  # Build the base command
+  nodetype = _check('NODETYPE', DATA['NODETYPE'], _NODETYPE_RE)
-  cmd = "NODETYPE=" + DATA['NODETYPE'] + " /usr/sbin/so-minion -o=addVM -m=" + minionid + " -n=" + DATA['MNIC'] + " -i=" + DATA['MAINIP'] + " -c=" + str(DATA['CPUCORES']) + " -d='" + DATA['NODE_DESCRIPTION'] + "'"
+
+  argv = [
+    SO_MINION,
+    '-o=addVM',
+    '-m=' + _check('minionid', minionid,        _MINIONID_RE),
+    '-n=' + _check('MNIC',     DATA['MNIC'],    _HOSTPART_RE),
+    '-i=' + _check('MAINIP',   DATA['MAINIP'],  _IPV4_RE),
+    '-c=' + str(int(DATA['CPUCORES'])),
+    '-d=' + str(DATA['NODE_DESCRIPTION']),
+  ]
 
-  # Add optional arguments only if they exist in DATA
   if 'CORECOUNT' in DATA:
-    cmd += " -C=" + str(DATA['CORECOUNT'])
+    argv.append('-C=' + str(int(DATA['CORECOUNT'])))
 
   if 'INTERFACE' in DATA:
-    cmd += " -a=" + DATA['INTERFACE']
+    argv.append('-a=' + _check('INTERFACE', DATA['INTERFACE'], _HOSTPART_RE))
 
   if 'ES_HEAP_SIZE' in DATA:
-    cmd += " -e=" + DATA['ES_HEAP_SIZE']
+    argv.append('-e=' + _check('ES_HEAP_SIZE', DATA['ES_HEAP_SIZE'], _HEAP_RE))
 
   if 'LS_HEAP_SIZE' in DATA:
-    cmd += " -l=" + DATA['LS_HEAP_SIZE']
+    argv.append('-l=' + _check('LS_HEAP_SIZE', DATA['LS_HEAP_SIZE'], _HEAP_RE))
 
   if 'LSHOSTNAME' in DATA:
-    cmd += " -L=" + DATA['LSHOSTNAME']
+    argv.append('-L=' + _check('LSHOSTNAME', DATA['LSHOSTNAME'], _HOSTPART_RE))
 
-  log.info('sominion_setup_reactor: Command: %s' % cmd)
+  env = os.environ.copy()
-  rc = call(cmd, shell=True)
+  env['NODETYPE'] = nodetype
+
+  log.info(
+    'sominion_setup_reactor: argv: %s (NODETYPE=%s)',
+    ' '.join(shlex.quote(a) for a in argv),
+    shlex.quote(nodetype),
+  )
+  rc = subprocess.call(argv, shell=False, env=env)
 
   log.info('sominion_setup_reactor: rc: %s' % rc)
 

salt/salt/cloud/cloud.profiles.d/socloud.conf.jinja

@@ -27,6 +27,7 @@ sool9_{{host}}:
     log_file: /opt/so/log/salt/minion
   grains:
     hypervisor_host: {{host ~ "_" ~ role}}
+    sosmodel: HVGUEST
   preflight_cmds:
     - |
       {%- set hostnames = [MANAGERHOSTNAME] %}

salt/salt/files/engines.conf

@@ -17,7 +17,7 @@ engines:
                 to:
                   'KAFKA':
                   - cmd.run:
-                      cmd: /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.enabled True
+                      cmd: /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.enabled True && /usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls replace kafka.enabled True --note "pillarWatch global.pipeline"
                   - cmd.run:
                       cmd: salt -C 'G@role:so-standalone or G@role:so-manager or G@role:so-managersearch or G@role:so-receiver or G@role:so-searchnode' saltutil.kill_all_jobs
                   - cmd.run:
@@ -28,7 +28,7 @@ engines:
                 to:
                   'REDIS':
                   - cmd.run:
-                      cmd: /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.enabled False
+                      cmd: /usr/sbin/so-yaml.py replace /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.enabled False && /usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls replace kafka.enabled False --note "pillarWatch global.pipeline"
                   - cmd.run:
                       cmd: salt -C 'G@role:so-standalone or G@role:so-manager or G@role:so-managersearch or G@role:so-receiver or G@role:so-searchnode' saltutil.kill_all_jobs
                   - cmd.run:
@@ -66,5 +66,5 @@ engines:
                   - cmd.run:
                       cmd: salt -C 'G@role:so-standalone or G@role:so-manager or G@role:so-managersearch or G@role:so-receiver' state.apply kafka.disabled,kafka.reset
                   - cmd.run:
-                      cmd: /usr/sbin/so-yaml.py remove /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.reset
+                      cmd: /usr/sbin/so-yaml.py remove /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls kafka.reset && /usr/sbin/so-config.py sync-yaml-mutation /opt/so/saltstack/local/pillar/kafka/soc_kafka.sls remove kafka.reset --note "pillarWatch kafka.reset"
       interval: 10

salt/salt/master.sls

@@ -14,6 +14,8 @@
 
 include:
   - salt.minion
+  - salt.master.ext_pillar_postgres
+  - salt.master.pg_notify_pillar_engine
 {%   if 'vrt' in salt['pillar.get']('features', []) %}
   - salt.cloud
   - salt.cloud.reactor_config_hypervisor

salt/salt/master/ext_pillar_postgres.sls

@@ -0,0 +1,24 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+# Deprecated. SOC/onionconfig owns the settings database now; this state only
+# removes the old so_pillar ext_pillar config if it was previously deployed.
+
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls.split('.')[0] in allowed_states %}
+
+ext_pillar_postgres_config_absent:
+  file.absent:
+    - name: /etc/salt/master.d/ext_pillar_postgres.conf
+    - watch_in:
+      - service: salt_master_service
+
+{% else %}
+
+{{sls}}_state_not_allowed:
+  test.fail_without_changes:
+    - name: {{sls}}_state_not_allowed
+
+{% endif %}

salt/salt/master/pg_notify_pillar_engine.sls

@@ -0,0 +1,37 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+# Deprecated. SOC/onionconfig owns the settings database now; this state only
+# removes the old so_pillar notify engine and reactor config if previously
+# deployed.
+
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls.split('.')[0] in allowed_states %}
+
+pg_notify_pillar_engine_module_absent:
+  file.absent:
+    - name: /etc/salt/engines/pg_notify_pillar.py
+    - watch_in:
+      - service: salt_master_service
+
+pg_notify_pillar_engine_config_absent:
+  file.absent:
+    - name: /etc/salt/master.d/pg_notify_pillar_engine.conf
+    - watch_in:
+      - service: salt_master_service
+
+pg_notify_pillar_reactor_config_absent:
+  file.absent:
+    - name: /etc/salt/master.d/so_pillar_reactor.conf
+    - watch_in:
+      - service: salt_master_service
+
+{% else %}
+
+{{sls}}_state_not_allowed:
+  test.fail_without_changes:
+    - name: {{sls}}_state_not_allowed
+
+{% endif %}

salt/soc/enabled.sls

@@ -100,6 +100,29 @@ so-soc:
       - file: socusersroles
       - file: socclientsroles
 
+onionconfig_initial_import:
+  cmd.run:
+    - name: |
+        set -e
+        SOCONFIG=/usr/sbin/so-config.py
+        if [ ! -x "$SOCONFIG" ]; then
+          SOCONFIG=/opt/so/saltstack/default/salt/manager/tools/sbin/so-config.py
+        fi
+        for i in $(seq 1 60); do
+          if docker exec so-postgres pg_isready -h 127.0.0.1 -U postgres -q >/dev/null 2>&1 \
+             && curl -fsS --connect-timeout 2 http://{{ DOCKERMERGED.containers['so-soc'].ip }}:9822/ >/dev/null 2>&1; then
+            "$SOCONFIG" wait-schema --timeout 120
+            "$SOCONFIG" import-all --state-file /opt/so/state/onionconfig_initial_import.done
+            exit 0
+          fi
+          sleep 2
+        done
+        echo "so-soc or so-postgres did not become ready within 120s" >&2
+        exit 1
+    - unless: test -f /opt/so/state/onionconfig_initial_import.done
+    - require:
+      - docker_container: so-soc
+
 delete_so-soc_so-status.disabled:
   file.uncomment:
     - name: /opt/so/conf/so-status/so-status.conf

salt/soc/soc_soc.yaml

@@ -3,6 +3,7 @@ soc:
     description: Enables or disables SOC. WARNING - Disabling this setting is unsupported and will cause the grid to malfunction. Re-enabling this setting is a manual effort via SSH.
     forcedType: bool
     advanced: True
+    readonly: True
   telemetryEnabled:
     title: SOC Telemetry
     description: When this setting is enabled and the grid is not in airgap mode, SOC will provide feature usage data to the Security Onion development team via Google Analytics. This data helps Security Onion developers determine which product features are being used and can also provide insight into improving the user interface. When changing this setting, wait for the grid to fully synchronize and then perform a hard browser refresh on SOC, to force the browser cache to update and reflect the new setting.
@@ -890,12 +891,16 @@ soc:
             suricata:
               description: The template used when creating a new Suricata detection. [publicId] will be replaced with an unused Public Id.
               multiline: True
+              forcedType: string
             strelka:
               description: The template used when creating a new Strelka detection.
               multiline: True
+              forcedType: string
             elastalert:
               description: The template used when creating a new ElastAlert detection. [publicId] will be replaced with an unused Public Id.
               multiline: True
+              forcedType: string
+
         grid:
           maxUploadSize:
             description: The maximum number of bytes for an uploaded PCAP import file.

setup/so-functions

@@ -202,10 +202,10 @@ check_service_status() {
 	systemctl status $service_name > /dev/null 2>&1
 	local status=$?
 	if [ $status -gt 0 ]; then
-		info "  $service_name is not running" 
+		info "$service_name is not running" 
 		return 1;
 	else
-		info "  $service_name is running"
+		info "$service_name is running"
 		return 0;
 	fi
 
@@ -1057,6 +1057,11 @@ generate_passwords(){
   POSTGRESPASS=$(get_random_value)
   SOCSRVKEY=$(get_random_value 64)
   IMPORTPASS=$(get_random_value)
+  # postsalt: salt-master connects to so_pillar.* as so_pillar_master, and the
+  # so-postgres container needs a symmetric key for pgcrypto-encrypted secrets.
+  # Both are generated here so they survive reinstall like the other secrets.
+  PILLARMASTERPASS=$(get_random_value)
+  SO_PILLAR_KEY=$(get_random_value 64)
 }
 
 generate_interface_vars() {
@@ -1549,13 +1554,8 @@ clear_previous_setup_results() {
 reinstall_init() {
 	info "Putting system in state to run setup again"
 
-	if [[ $install_type =~ ^(MANAGER|EVAL|MANAGERSEARCH|MANAGERHYPE|STANDALONE|FLEET|IMPORT)$ ]]; then
+	# Always include both services. check_service_status skips units that aren't present.
 	local salt_services=( "salt-master" "salt-minion" )
-	else
-		local salt_services=( "salt-minion" )
-	fi
-
-	local service_retry_count=20
 
 	{
 		# remove all of root's cronjobs
@@ -1571,31 +1571,51 @@ reinstall_init() {
 
 		salt-call state.apply ca.remove -linfo --local --file-root=../salt
 
-		# Kill any salt processes (safely)
+		# Stop salt services and force-kill any lingering salt processes (including orphans
+		# from an earlier reinstall attempt where the unit file is gone but processes survive)
+		# so dnf remove salt can run cleanly
 		for service in "${salt_services[@]}"; do
-			# Stop the service in the background so we can exit after a certain amount of time
 			if check_service_status "$service"; then
-				systemctl stop "$service" &
+				info "Stopping $service via systemctl"
+				systemctl stop "$service"
 			fi
-			local pid=$!
+		done
 
-			local count=0
+		# Unconditionally force-kill any remaining salt binaries — these may be orphaned
-			while check_service_status "$service"; do
+		# from a prior aborted reinstall (no unit file, so systemctl can't see them).
-				if [[ $count -gt $service_retry_count ]]; then
+		for salt_bin in salt-master salt-minion salt-call salt-cloud; do
-					echo "Could not stop $service after 1 minute, exiting setup."
+			if pgrep -f "/usr/bin/${salt_bin}" > /dev/null 2>&1; then
-
+				info "Force-killing lingering $salt_bin processes"
-					# Stop the systemctl process trying to kill the service, show user a message, then exit setup
+				pkill -9 -ef "/usr/bin/${salt_bin}" 2>/dev/null
-					kill -9 $pid
-					fail_setup
 			fi
+		done
+		# Catch stray `salt` CLI children from saltutil.kill_all_jobs / state.apply invocations
+		pkill -9 -ef "/usr/bin/python3 /bin/salt" 2>/dev/null
 
-				sleep 5
+		# Give the kernel a moment to reap the killed processes before dnf removes the binaries
-				((count++))
+		local kill_wait=0
-			done
+		while pgrep -f "/usr/bin/salt-" > /dev/null 2>&1; do
+			if [[ $kill_wait -gt 10 ]]; then
+				info "Salt processes still present after SIGKILL + 10s wait; proceeding anyway"
+				pgrep -af "/usr/bin/salt-" | while read -r line; do info "  lingering: $line"; done
+				break
+			fi
+			sleep 1
+			((kill_wait++))
 		done
 
+		# Clear the 'failed' state SIGKILL left on the units before removing the package
+		systemctl reset-failed salt-master.service salt-minion.service 2>/dev/null || true
+
 		# Remove all salt configs
-		rm -rf /etc/salt/engines/* /etc/salt/grains /etc/salt/master /etc/salt/master.d/* /etc/salt/minion /etc/salt/minion.d/* /etc/salt/pki/* /etc/salt/proxy /etc/salt/proxy.d/* /var/cache/salt/
+		dnf -y remove salt
+		rm -rf /etc/salt/ /var/cache/salt/
+
+		# Drop systemd's in-memory references to the now-removed units
+		systemctl daemon-reload
+
+		# Uninstall local Elastic Agent, if installed
+		elastic-agent uninstall -f
 
 		if command -v docker &> /dev/null; then
 			# Stop and remove all so-* containers so files can be changed with more safety
@@ -1619,10 +1639,7 @@ reinstall_init() {
 		backup_dir /nsm/hydra "$date_string"
 		backup_dir /nsm/influxdb "$date_string"
 
-		# Uninstall local Elastic Agent, if installed
+	} 2>&1 | tee -a "$setup_log"
-		elastic-agent uninstall -f
-
-	} >> "$setup_log" 2>&1
 
 	info "System reinstall init has been completed."
 }
@@ -1841,7 +1858,34 @@ secrets_pillar(){
 		"secrets:"\
 		"  import_pass: $IMPORTPASS"\
 		"  influx_pass: $INFLUXPASS"\
+		"  pillar_master_pass: $PILLARMASTERPASS"\
 		"  postgres_pass: $POSTGRESPASS" > $local_salt_dir/pillar/secrets.sls
+  elif ! grep -q '^[[:space:]]*pillar_master_pass:' $local_salt_dir/pillar/secrets.sls; then
+	# Existing install pre-postsalt — append the new key without disturbing
+	# the values already on disk. Keys we already wrote stay; only the new
+	# pillar_master_pass is added.
+	info "Appending pillar_master_pass to existing Secrets Pillar"
+	if [ -z "$PILLARMASTERPASS" ]; then
+	  PILLARMASTERPASS=$(get_random_value)
+	fi
+	printf '  pillar_master_pass: %s\n' "$PILLARMASTERPASS" >> $local_salt_dir/pillar/secrets.sls
+  fi
+
+  # postsalt: write the so_pillar pgcrypto master key to a 0400 file owned by
+  # root. The key itself is never read by Salt — schema_pillar.sls loads it
+  # into the so-postgres container via ALTER ROLE so_pillar_secret_owner SET
+  # so_pillar.master_key = '<key>'; the file just lets the value survive
+  # container restarts.
+  if [ ! -f /opt/so/conf/postgres/so_pillar.key ]; then
+	info "Generating so_pillar pgcrypto master key"
+	mkdir -p /opt/so/conf/postgres
+	if [ -z "$SO_PILLAR_KEY" ]; then
+	  SO_PILLAR_KEY=$(get_random_value 64)
+	fi
+	umask 077
+	printf '%s' "$SO_PILLAR_KEY" > /opt/so/conf/postgres/so_pillar.key
+	chmod 0400 /opt/so/conf/postgres/so_pillar.key
+	chown root:root /opt/so/conf/postgres/so_pillar.key
   fi
 }
 

setup/so-setup

@@ -219,7 +219,7 @@ if [ -n "$test_profile" ]; then
 	WEBUSER=onionuser@somewhere.invalid
 	WEBPASSWD1=0n10nus3r
 	WEBPASSWD2=0n10nus3r
-	NODE_DESCRIPTION="${HOSTNAME} - ${install_type} - ${MAINIP}"
+	NODE_DESCRIPTION="${HOSTNAME} - ${install_type} - ${MSRVIP_OFFSET}"
 
 	update_sudoers_for_testing
 fi