Merge remote-tracking branch 'origin/3/dev' into zeekrestart

zeekctl's post-terminate archives the final logs in the background and returns
immediately unless StopWait is set, so the container exits and takes the archiving
with it, stranding unarchived logs in /nsm/zeek/spool/tmp on every restart. Docker's
default 10s grace is also too tight for the entrypoint's SIGTERM trap; overrunning it
means SIGKILL and crash directories on the next start.

Both are needed. StopWait alone gives the stop more work to do inside the same 10s
window, which was measured ending in SIGKILL with logs stranded in the spool.

disabled.sls used docker rm -f, which never delivers SIGTERM, so stop the container
before removing it.

Reported in discussion #16174.

DOWNLOAD_AND_VERIFY_ISO.md

@@ -1,17 +1,17 @@
-### 3.3.0-20260911 ISO image released on 2026/09/11
+### 3.3.0-20260908 ISO image released on 2026/09/08
 
 
 ### Download and Verify
 
-3.3.0-20260911 ISO image:  
+3.3.0-20260908 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
+https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260908.iso
  
-MD5: 12B18433D3A2198A185892FF79CF638F  
+MD5: 5A2C42D0083F2D7B4DC2178C30EBC05F  
-SHA1: 2B3C2E1FA7A78ED1F956E7EDCC12E32593C14EEE  
+SHA1: 505220A8A3315AFEE601C13772996018425CCD29  
-SHA256: 0938C73B76CE30EC9E4394D312C79EA7CAC721B6818541697279A6221F7D870D  
+SHA256: 6EB8401296A1D051FEC558C520D2D4AB1912A72D351A2426FBF8C87FEE2BA844  
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260908.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/main/KEYS  
@@ -25,22 +25,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260908.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260908.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-3.3.0-20260911.iso.sig securityonion-3.3.0-20260911.iso
+gpg --verify securityonion-3.3.0-20260908.iso.sig securityonion-3.3.0-20260908.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Fri 11 Sep 2026 11:23:56 AM EDT using RSA key ID FE507013
+gpg: Signature made Tue 08 Sep 2026 10:07:12 AM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

salt/manager/tools/sbin/soup

@@ -120,9 +120,6 @@ check_err() {
       161)
         echo 'Required intermediate Elasticsearch upgrade not complete'
       ;;
-      162)
-        echo 'One or more Elastic Agent nodes do not support the x86-64-v3 CPU instruction set'
-      ;;
       170)
         echo "Intermediate upgrade completed successfully to $next_step_so_version, but next soup to Security Onion $originally_requested_so_version could not be started automatically."
         echo "Start soup again manually to continue the upgrade to Security Onion $originally_requested_so_version."
@@ -350,83 +347,6 @@ check_cluster_health() {
   exit 0
 }
 
-no_soup_for_you() {
-  echo ""
-  echo "No soup for you!"
-  exit 162
-}
-
-check_cpu_compatibility() {
-  # Roles running a container built from the so-elastic-agent image; mirrors the
-  # elasticagent and elasticfleet entries in salt/reactor/pillar_push_map.yaml.
-  local cpu_target='G@role:so-heavynode or G@role:so-eval or G@role:so-fleet or G@role:so-import or G@role:so-manager or G@role:so-managerhype or G@role:so-managersearch or G@role:so-standalone'
-  local expected_nodes cpu_results node result confirm
-  local -a unsupported=() offline=()
-
-  echo "Checking that Elastic Agent nodes support the x86-64-v3 CPU instruction set now required by Elastic."
-
-  if [[ "$SKIP_CPU_CHECK" == "true" ]]; then
-    printf "\nSkipping the x86-64-v3 CPU check because --skip-cpu-check was specified.\n\n"
-    return
-  fi
-
-  # Nodes that never answer are absent from the results, so diff against who should have.
-  expected_nodes=$(salt -C "$cpu_target" --preview-target --out=json 2>/dev/null | jq -r '.[]?') || true
-  if [[ -z "$expected_nodes" ]]; then
-    printf "\nCould not determine which nodes run the Elastic Agent, so the x86-64-v3 CPU check cannot run.\n"
-    no_soup_for_you
-  fi
-
-  cpu_results=$(salt -t 30 -C "$cpu_target" cmd.run "/lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v3" --out=json 2>/dev/null) || true
-
-  while IFS= read -r node; do
-    [[ -z "$node" ]] && continue
-    result=$(jq -r --arg node "$node" '.[$node] // empty' <<< "$cpu_results" 2>/dev/null)
-    if [[ -z "$result" || "$result" == *"did not return"* ]]; then
-      offline+=("$node")
-    elif [[ "$result" != *"x86-64-v3 (supported"* ]]; then
-      # glibc appends "(supported, searched)" only when supported; the open paren keeps
-      # this from matching a future "(unsupported".
-      unsupported+=("$node")
-    fi
-  done <<< "$expected_nodes"
-
-  if [[ ${#unsupported[@]} -eq 0 && ${#offline[@]} -eq 0 ]]; then
-    printf "\nAll Elastic Agent nodes support x86-64-v3. We can proceed with SOUP.\n\n"
-    return
-  fi
-
-  echo ""
-  if [[ ${#unsupported[@]} -gt 0 ]]; then
-    echo "The following node(s) do NOT support the x86-64-v3 CPU instruction set:"
-    printf '  %s\n' "${unsupported[@]}"
-    echo ""
-    echo "Upstream Elastic now builds its binaries for x86-64-v3, so these nodes can no"
-    echo "longer run Elastic. Upgrading them WILL BREAK them."
-    echo ""
-  fi
-  if [[ ${#offline[@]} -gt 0 ]]; then
-    echo "The following node(s) did not respond and could not be checked:"
-    printf '  %s\n' "${offline[@]}"
-    echo ""
-    echo "These nodes are offline, so we cannot confirm they support x86-64-v3, which"
-    echo "upstream Elastic now requires."
-    echo ""
-  fi
-
-  if [[ -n $UNATTENDED ]]; then
-    echo "Unattended mode cannot prompt for an override. Re-run soup interactively, or pass --skip-cpu-check to bypass this check."
-    no_soup_for_you
-  fi
-
-  read -rp "Type 'override' to continue anyway, or press Enter to exit: " confirm
-  if [[ "${confirm,,}" == "override" ]]; then
-    printf "\nOverride accepted. Continuing at your own risk.\n\n"
-  else
-    no_soup_for_you
-  fi
-}
-
 check_fleet_server() {
   echo "Checking that Elastic Fleet Server is responding."
   # Modeled on the wait_for_so-elastic-fleet state check in elasticfleet/enabled.sls,
@@ -2057,9 +1977,6 @@ main() {
 
   echo "Let's see if we need to update Security Onion."
   upgrade_check
-
-  check_cpu_compatibility
-
   upgrade_space
 
   echo "Verifying Elasticsearch version compatibility across the grid before upgrading."
@@ -2338,17 +2255,6 @@ fi
   echo "### soup has been served at $(date) ###"
 }
 
-SKIP_CPU_CHECK=false
-declare -a SOUP_ARGS=()
-for arg in "$@"; do
-  if [[ "$arg" == "--skip-cpu-check" ]]; then
-    SKIP_CPU_CHECK=true
-  else
-    SOUP_ARGS+=("$arg")
-  fi
-done
-set -- "${SOUP_ARGS[@]}"
-
 while getopts ":b:f:y" opt; do
   case ${opt} in
     b )
@@ -2372,7 +2278,7 @@ while getopts ":b:f:y" opt; do
       ISOLOC="$OPTARG"
     ;;
     \? )
-      echo "Usage: soup [-b] [-y] [-f <iso location>] [--skip-cpu-check]"
+      echo "Usage: soup [-b] [-y] [-f <iso location>]"
       exit 1
     ;;
     : )

salt/soc/defaults.yaml

@@ -1537,7 +1537,7 @@ soc:
             Orchestrator: sonnet@SOAI
             Investigator: gemma@SOAI
             DetectionEngineer: gemma@SOAI
-          useMemory: false
+          useMemory: true
           useMemoryScanner: false
           dontScanBefore: ""
           memoryScanIntervalSeconds: 300

salt/zeek/defaults.yaml

@@ -18,6 +18,7 @@ zeek:
       StatsLogEnable: 0
       StatsLogExpireInterval: 0
       StatusCmdShowAll: 0
+      StopWait: 1
       CrashExpireInterval: 0
       SitePolicyScripts: local.zeek
       LogDir: /nsm/zeek/logs

salt/zeek/disabled.sls

@@ -9,9 +9,19 @@
 include:
   - zeek.sostatus
   
+# Stop first so the entrypoint's SIGTERM trap can archive the final logs; docker_container.absent
+# with force is a 'docker rm -f', which never delivers SIGTERM. force stays so the state still
+# converges if the stop overruns.
+so-zeek_stopped:
+  docker_container.stopped:
+    - name: so-zeek
+    - error_on_absent: False
+
 so-zeek:
   docker_container.absent:
     - force: True
+    - require:
+      - docker_container: so-zeek_stopped
 
 so-zeek_so-status.disabled:
   file.comment:

salt/zeek/enabled.sls

@@ -19,6 +19,10 @@ so-zeek:
     - restart_policy: unless-stopped
     - start: True
     - privileged: True
+    # Docker's default 10s grace is not enough for the entrypoint's SIGTERM trap to run
+    # 'zeekctl stop' and let StopWait archive the final logs. Overrunning it means SIGKILL,
+    # which strands those logs in spool/tmp and marks every node crashed on the next start.
+    - stop_timeout: 180
     {% if DOCKERMERGED.containers['so-zeek'].ulimits %}
     - ulimits:
     {%   for ULIMIT in DOCKERMERGED.containers['so-zeek'].ulimits %}

salt/zeek/soc_zeek.yaml

@@ -99,6 +99,18 @@ zeek:
         regexFailureMessage: You must enter a whole number of days, or 0 to keep crash directories forever.
         helpLink: zeek
         advanced: True
+      StopWait:
+        description: >-
+          Set to 1 to make "zeekctl stop" wait for the final logs to be archived instead of
+          letting that finish in the background. Security Onion stops Zeek by stopping its
+          container, so anything still running in the background is killed when the container
+          exits - without this, the last logs of each run are stranded unarchived in
+          /nsm/zeek/spool/tmp and never reach Elasticsearch. It is read only for that reason.
+        regex: ^[01]$
+        regexFailureMessage: You must enter 0 or 1.
+        helpLink: zeek
+        advanced: True
+        readonly: True
       MinDiskSpace:
         description: >-
           Percentage of free disk space below which ZeekControl reports a warning, or 0 to disable the check