Compare commits

..
Author SHA1 Message Date
Mike Reeves 866a046853 Annotate the soauth network leaves instead of the networks map
SOC reported "Malformed config setting (docker.networks.soauth.range): Setting
name 'networks' conflicts with another similarly named setting" and refused to
render the config tree.

soc_docker.yaml annotated docker.networks itself, and every key in that block
was a scalar, so FlattenAnnotations registered docker.networks as a setting.
defaults.yaml holds a nested map there, so FlattenPillar separately registered
docker.networks.soauth.range, .gateway and .manager_only, and
HydrateAnnotations unions the two sets. The config tree gives each id segment
either a value or children, never both, so addToNode pushed docker.networks as
a childless leaf and then threw when docker.networks.soauth.range tried to
descend through it.

The annotations move down to the three leaves that actually carry values, so
docker.networks is a plain branch. This matches docker.containers, which is
nested the same way and has never carried annotations of its own.

docker.ulimits and the per-container networks list are unaffected: their pillar
values are lists rather than maps, so they stay single settings.
2026-09-18 13:40:05 -04:00
68 changed files with 209 additions and 706 deletions
+11 -11
View File
@@ -1,17 +1,17 @@
### 3.3.0-20260911 ISO image released on 2026/09/11
### 3.3.0-20260908 ISO image released on 2026/09/08
### Download and Verify
3.3.0-20260911 ISO image:
https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
3.3.0-20260908 ISO image:
https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260908.iso
MD5: 12B18433D3A2198A185892FF79CF638F
SHA1: 2B3C2E1FA7A78ED1F956E7EDCC12E32593C14EEE
SHA256: 0938C73B76CE30EC9E4394D312C79EA7CAC721B6818541697279A6221F7D870D
MD5: 5A2C42D0083F2D7B4DC2178C30EBC05F
SHA1: 505220A8A3315AFEE601C13772996018425CCD29
SHA256: 6EB8401296A1D051FEC558C520D2D4AB1912A72D351A2426FBF8C87FEE2BA844
Signature for ISO image:
https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260908.iso.sig
Signing key:
https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/main/KEYS
@@ -25,22 +25,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/
Download the signature file for the ISO:
```
wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260908.iso.sig
```
Download the ISO image:
```
wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260908.iso
```
Verify the downloaded ISO image using the signature file:
```
gpg --verify securityonion-3.3.0-20260911.iso.sig securityonion-3.3.0-20260911.iso
gpg --verify securityonion-3.3.0-20260908.iso.sig securityonion-3.3.0-20260908.iso
```
The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
```
gpg: Signature made Fri 11 Sep 2026 11:23:56 AM EDT using RSA key ID FE507013
gpg: Signature made Tue 08 Sep 2026 10:07:12 AM EDT using RSA key ID FE507013
gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
+4 -17
View File
@@ -117,25 +117,14 @@ elastic_curl_config:
{% endif %}
# A non-root owner here can chmod the directory and replace any script in it, including
# the root-owned ones. 555 is the mode the filesystem RPM ships; root ignores it anyway.
usr_sbin_perms:
file.directory:
- name: /usr/sbin
- user: root
- group: root
- mode: 555
common_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://common/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
- show_changes: False
- require:
- file: usr_sbin_perms
{% if GLOBALS.role == 'so-heavynode' %}
- exclude_pat:
- so-pcap-import
@@ -170,8 +159,8 @@ common_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://common/tools/sbin_jinja
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
- template: jinja
- show_changes: False
@@ -184,8 +173,6 @@ so-status_script:
file.managed:
- name: /usr/sbin/so-status
- source: salt://common/tools/sbin/so-status
- user: root
- group: root
- mode: 755
{% if GLOBALS.is_sensor %}
+14 -42
View File
@@ -18,61 +18,47 @@ copy_so-common_common_tools_sbin:
- name: /opt/so/saltstack/default/salt/common/tools/sbin/so-common
- source: {{UPDATE_DIR}}/salt/common/tools/sbin/so-common
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-image-common_common_tools_sbin:
file.copy:
- name: /opt/so/saltstack/default/salt/common/tools/sbin/so-image-common
- source: {{UPDATE_DIR}}/salt/common/tools/sbin/so-image-common
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_soup_manager_tools_sbin:
file.copy:
- name: /opt/so/saltstack/default/salt/manager/tools/sbin/soup
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/soup
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-firewall_manager_tools_sbin:
file.copy:
- name: /opt/so/saltstack/default/salt/manager/tools/sbin/so-firewall
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-firewall
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-yaml_manager_tools_sbin:
file.copy:
- name: /opt/so/saltstack/default/salt/manager/tools/sbin/so-yaml.py
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-yaml.py
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-repo-sync_manager_tools_sbin:
file.copy:
- name: /opt/so/saltstack/default/salt/manager/tools/sbin/so-repo-sync
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-repo-sync
- user: root
- group: root
- mode: 755
- preserve: True
copy_bootstrap-salt_manager_tools_sbin:
file.copy:
- name: /opt/so/saltstack/default/salt/salt/scripts/bootstrap-salt.sh
- source: {{UPDATE_DIR}}/salt/salt/scripts/bootstrap-salt.sh
- user: root
- group: root
- mode: 644
- preserve: True
# This section is used to put the new script in place so that it can be called during soup.
# It is faster than calling the states that normally manage them to put them in place.
@@ -81,60 +67,46 @@ copy_so-common_sbin:
- name: /usr/sbin/so-common
- source: {{UPDATE_DIR}}/salt/common/tools/sbin/so-common
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-image-common_sbin:
file.copy:
- name: /usr/sbin/so-image-common
- source: {{UPDATE_DIR}}/salt/common/tools/sbin/so-image-common
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_soup_sbin:
file.copy:
- name: /usr/sbin/soup
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/soup
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-firewall_sbin:
file.copy:
- name: /usr/sbin/so-firewall
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-firewall
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-yaml_sbin:
file.copy:
- name: /usr/sbin/so-yaml.py
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-yaml.py
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_so-repo-sync_sbin:
file.copy:
- name: /usr/sbin/so-repo-sync
- source: {{UPDATE_DIR}}/salt/manager/tools/sbin/so-repo-sync
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
copy_bootstrap-salt_sbin:
file.copy:
- name: /usr/sbin/bootstrap-salt.sh
- source: {{UPDATE_DIR}}/salt/salt/scripts/bootstrap-salt.sh
- force: True
- user: root
- group: root
- mode: 755
- preserve: True
+1 -2
View File
@@ -240,8 +240,7 @@ copy_new_files() {
cd $UPDATE_DIR
rsync -a salt $DEFAULT_SALT_DIR/ --delete "${EXCLUDE_ARGS[@]}"
rsync -a pillar $DEFAULT_SALT_DIR/ --delete "${EXCLUDE_ARGS[@]}"
# Root-executed code; SOC only needs to read it. Local dirs stay socore-owned.
chown -R root:root $DEFAULT_SALT_DIR/
chown -R socore:socore $DEFAULT_SALT_DIR/
cd /tmp
}
+47 -78
View File
@@ -8,37 +8,21 @@
# Elastic License 2.0.
SENSOR_DIR="${SENSOR_DIR:-/nsm}"
SENSOR_DIR='/nsm'
CRIT_DISK_USAGE=90
LOG="${LOG:-/opt/so/log/sensor_clean.log}"
LOCK="${LOCK:-/var/tmp/so-sensor-clean.lock}"
MAX_PASSES=100
CUR_USAGE=$(df -P $SENSOR_DIR | tail -1 | awk '{print $5}' | tr -d %)
LOG="/opt/so/log/sensor_clean.log"
TODAY=$(date -u "+%Y-%m-%d")
ZEEK_LOGS="$SENSOR_DIR/zeek/logs"
STRELKA_FILES="$SENSOR_DIR/strelka/processed"
SURICATA_LOGS="$SENSOR_DIR/suricata"
PCAPS="$SENSOR_DIR/pcapout"
log() {
echo "$(date) - $*" >>"$LOG"
}
disk_usage() {
df -P "$SENSOR_DIR" | tail -1 | awk '{print $5}' | tr -d %
}
disk_avail() {
df -P "$SENSOR_DIR" | tail -1 | awk '{print $4}'
}
# sets REMOVED=1 if anything was actually deleted
clean() {
## find the oldest Zeek logs directory
OLDEST_DIR=$(ls "$ZEEK_LOGS" 2>/dev/null | grep -v "current" | grep -v "stats" | grep -v "packetloss" | grep -v "zeek_clean" | sort | head -n 1)
if [ -n "$OLDEST_DIR" ]; then
log "Removing directory: $ZEEK_LOGS/$OLDEST_DIR"
rm -rf "$ZEEK_LOGS/$OLDEST_DIR"
REMOVED=1
OLDEST_DIR=$(ls /nsm/zeek/logs/ | grep -v "current" | grep -v "stats" | grep -v "packetloss" | grep -v "zeek_clean" | sort | head -n 1)
if [ -z "$OLDEST_DIR" -o "$OLDEST_DIR" == ".." -o "$OLDEST_DIR" == "." ]; then
echo "$(date) - No old Zeek logs available to clean up in /nsm/zeek/logs/" >>$LOG
#exit 0
else
echo "$(date) - Removing directory: /nsm/zeek/logs/$OLDEST_DIR" >>$LOG
rm -rf /nsm/zeek/logs/"$OLDEST_DIR"
fi
## Remarking for now, as we are moving extracted files to /nsm/strelka/processed
@@ -59,73 +43,58 @@ clean() {
#fi
## Clean up Zeek extracted files processed by Strelka
OLDEST_STRELKA=$(find "$STRELKA_FILES" -type f -printf '%T+ %p\n' 2>/dev/null | sort -n | head -n 1)
if [ -n "$OLDEST_STRELKA" ]; then
STRELKA_FILES='/nsm/strelka/processed'
OLDEST_STRELKA=$(find $STRELKA_FILES -type f -printf '%T+ %p\n' | sort -n | head -n 1)
if [ -z "$OLDEST_STRELKA" -o "$OLDEST_STRELKA" == ".." -o "$OLDEST_STRELKA" == "." ]; then
echo "$(date) - No old files available to clean up in $STRELKA_FILES" >>$LOG
else
OLDEST_STRELKA_DATE=$(echo $OLDEST_STRELKA | awk '{print $1}' | cut -d+ -f1)
log "Removing extracted files for $OLDEST_STRELKA_DATE"
REMOVED=1
find "$STRELKA_FILES" -type f -printf '%T+ %p\n' 2>/dev/null | grep $OLDEST_STRELKA_DATE | awk '{print $2}' | while read FILE; do
log "Removing file: $FILE"
OLDEST_STRELKA_FILE=$(echo $OLDEST_STRELKA | awk '{print $2}')
echo "$(date) - Removing extracted files for $OLDEST_STRELKA_DATE" >>$LOG
find $STRELKA_FILES -type f -printf '%T+ %p\n' | grep $OLDEST_STRELKA_DATE | awk '{print $2}' | while read FILE; do
echo "$(date) - Removing file: $FILE" >>$LOG
rm -f "$FILE"
done
fi
## Clean up Suricata log files
OLDEST_SURICATA=$(find "$SURICATA_LOGS" -type f -printf '%T+ %p\n' 2>/dev/null | sort -n | head -n 1)
if [ -n "$OLDEST_SURICATA" ]; then
SURICATA_LOGS='/nsm/suricata'
OLDEST_SURICATA=$(find $SURICATA_LOGS -type f -printf '%T+ %p\n' | sort -n | head -n 1)
if [[ -z "$OLDEST_SURICATA" ]] || [[ "$OLDEST_SURICATA" == ".." ]] || [[ "$OLDEST_SURICATA" == "." ]]; then
echo "$(date) - No old files available to clean up in $SURICATA_LOGS" >>$LOG
else
OLDEST_SURICATA_DATE=$(echo $OLDEST_SURICATA | awk '{print $1}' | cut -d+ -f1)
log "Removing logs for $OLDEST_SURICATA_DATE"
REMOVED=1
find "$SURICATA_LOGS" -type f -printf '%T+ %p\n' 2>/dev/null | grep $OLDEST_SURICATA_DATE | awk '{print $2}' | while read FILE; do
log "Removing file: $FILE"
OLDEST_SURICATA_FILE=$(echo $OLDEST_SURICATA | awk '{print $2}')
echo "$(date) - Removing logs for $OLDEST_SURICATA_DATE" >>$LOG
find $SURICATA_LOGS -type f -printf '%T+ %p\n' | grep $OLDEST_SURICATA_DATE | awk '{print $2}' | while read FILE; do
echo "$(date) - Removing file: $FILE" >>$LOG
rm -f "$FILE"
done
fi
## Clean up extracted pcaps
OLDEST_PCAP=$(find "$PCAPS" -type f -printf '%T+ %p\n' 2>/dev/null | sort -n | head -n 1)
if [ -n "$OLDEST_PCAP" ]; then
PCAPS='/nsm/pcapout'
OLDEST_PCAP=$(find $PCAPS -type f -printf '%T+ %p\n' | sort -n | head -n 1)
if [ -z "$OLDEST_PCAP" -o "$OLDEST_PCAP" == ".." -o "$OLDEST_PCAP" == "." ]; then
echo "$(date) - No old files available to clean up in $PCAPS" >>$LOG
else
OLDEST_PCAP_DATE=$(echo $OLDEST_PCAP | awk '{print $1}' | cut -d+ -f1)
log "Removing extracted files for $OLDEST_PCAP_DATE"
REMOVED=1
find "$PCAPS" -type f -printf '%T+ %p\n' 2>/dev/null | grep $OLDEST_PCAP_DATE | awk '{print $2}' | while read FILE; do
log "Removing file: $FILE"
OLDEST_PCAP_FILE=$(echo $OLDEST_PCAP | awk '{print $2}')
echo "$(date) - Removing extracted files for $OLDEST_PCAP_DATE" >>$LOG
find $PCAPS -type f -printf '%T+ %p\n' | grep $OLDEST_PCAP_DATE | awk '{print $2}' | while read FILE; do
echo "$(date) - Removing file: $FILE" >>$LOG
rm -f "$FILE"
done
fi
}
# Only one instance at a time; the lock is the fd, so it releases on any exit
exec 9>"$LOCK" || exit 1
if ! flock -n 9; then
log "another so-sensor-clean is already running (lock $LOCK held); exiting"
exit 0
# Check to see if we are already running
NUM_RUNNING=$(pgrep -cf "/bin/bash /usr/sbin/so-sensor-clean")
[ "$NUM_RUNNING" -gt 1 ] && echo "$(date) - $NUM_RUNNING sensor clean script processes running...exiting." >>$LOG && exit 0
if [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; then
while [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; do
clean
CUR_USAGE=$(df -P $SENSOR_DIR | tail -1 | awk '{print $5}' | tr -d %)
done
fi
CUR_USAGE=$(disk_usage)
[ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ] || exit 0
log "$SENSOR_DIR at ${CUR_USAGE}% (threshold ${CRIT_DISK_USAGE}%); starting cleanup"
PASS=0
while [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; do
PASS=$((PASS + 1))
if [ "$PASS" -gt "$MAX_PASSES" ]; then
log "stopping after $MAX_PASSES passes; $SENSOR_DIR still at ${CUR_USAGE}%"
break
fi
REMOVED=0
BEFORE=$(disk_avail)
clean
CUR_USAGE=$(disk_usage)
if [ "$REMOVED" -eq 0 ]; then
log "nothing left to remove in $ZEEK_LOGS, $STRELKA_FILES, $SURICATA_LOGS, $PCAPS; $SENSOR_DIR still at ${CUR_USAGE}% - space is consumed outside of NSM cleanup scope"
break
fi
if [ "$(disk_avail)" -le "$BEFORE" ]; then
log "pass $PASS freed no space; $SENSOR_DIR still at ${CUR_USAGE}% - stopping until next run"
break
fi
done
+4 -11
View File
@@ -8,20 +8,13 @@ docker:
helpLink: docker
advanced: True
networks:
sobridge:
description: |
The default docker network, carrying most containers. Its range and gateway are taken
from the docker.range and docker.gateway settings above rather than set here.
helpLink: docker
readonly: True
advanced: True
global: True
soauth:
range:
description: |
IP range for the soauth docker network, an isolated network for the authentication
services, so that the Kratos and Hydra admin APIs are only reachable from the
containers placed on it.
IP range for the soauth docker network. soauth is an isolated network for the
authentication services, so that the Kratos and Hydra admin APIs are only reachable
from the containers placed on it. Most containers instead use sobridge, which takes
its range and gateway from the docker.range and docker.gateway settings above.
helpLink: docker
readonly: True
advanced: True
+2 -2
View File
@@ -33,8 +33,8 @@ elastalert_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://elastalert/tools/sbin
- user: root
- group: root
- user: 933
- group: 939
- file_mode: 755
#elastalert_sbin_jinja:
+2 -2
View File
@@ -39,8 +39,8 @@ elasticagent_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://elasticagent/tools/sbin_jinja
- user: root
- group: root
- user: 949
- group: 939
- file_mode: 755
- template: jinja
+6 -6
View File
@@ -31,8 +31,8 @@ elasticfleet_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://elasticfleet/tools/sbin
- user: root
- group: root
- user: 947
- group: 939
- file_mode: 755
- show_changes: False
@@ -40,8 +40,8 @@ elasticfleet_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://elasticfleet/tools/sbin_jinja
- user: root
- group: root
- user: 947
- group: 939
- file_mode: 755
- template: jinja
- exclude_pat:
@@ -81,8 +81,8 @@ eapackageupgrade:
file.managed:
- name: /usr/sbin/so-elastic-fleet-package-upgrade
- source: salt://elasticfleet/tools/sbin_jinja/so-elastic-fleet-package-upgrade
- user: root
- group: root
- user: 947
- group: 939
- mode: 755
- template: jinja
+2 -2
View File
@@ -14,8 +14,8 @@ so-elastic-agent-install:
file.managed:
- name: /usr/sbin/so-elastic-agent-install
- source: salt://elasticfleet/tools/sbin/so-elastic-agent-install
- user: root
- group: root
- user: 947
- group: 939
- mode: 755
- show_changes: False
@@ -30,56 +30,6 @@ fleet_api() {
curl -sK /opt/so/conf/elasticsearch/curl.config -L "localhost:5601/api/fleet/${QUERYPATH}" "$@" --retry 3 --retry-delay 10 --fail 2>/dev/null
}
elastic_fleet_require_agent_policy() {
local AGENT_POLICY=$1
local POLICY_JSON
if ! POLICY_JSON=$(fleet_api "agent_policies/$AGENT_POLICY") || [ -z "$POLICY_JSON" ]; then
echo "Error: Agent policy '$AGENT_POLICY' was not found or is not visible in the current Kibana space." >&2
return 1
fi
if ! jq -e '.item.package_policies | type == "array"' <<<"$POLICY_JSON" >/dev/null 2>&1; then
echo "Error: Agent policy '$AGENT_POLICY' was not found or is not visible in the current Kibana space." >&2
return 1
fi
echo "$POLICY_JSON"
}
# Print the single active enrollment token for POLICY_ID.
# Exit 1: retryable (API failure, invalid response, no active token)
# Exit 2: multiple active tokens - Shouldn't get into this state without manual intervention
elastic_fleet_active_enrollment_token() {
local POLICY_ID=$1
local RESP TOKEN_COUNT API_KEY
if ! RESP=$(fleet_api "enrollment_api_keys?perPage=100" -H 'kbn-xsrf: true' -H 'Content-Type: application/json'); then
echo "Error: Failed to retrieve enrollment tokens for agent policy '$POLICY_ID'." >&2
return 1
fi
if ! jq -e '.list' <<<"$RESP" >/dev/null 2>&1; then
echo "Error: Invalid enrollment token response for agent policy '$POLICY_ID'." >&2
return 1
fi
TOKEN_COUNT=$(jq --arg pid "$POLICY_ID" '[.list[] | select(.policy_id == $pid and .active == true)] | length' <<<"$RESP")
if [ "${TOKEN_COUNT:-0}" -eq 0 ]; then
echo "Error: No active enrollment token found for agent policy '$POLICY_ID'." >&2
return 1
fi
if [ "$TOKEN_COUNT" -gt 1 ]; then
echo "Error: Found $TOKEN_COUNT active enrollment tokens for agent policy '$POLICY_ID'; expected exactly one." >&2
return 2
fi
API_KEY=$(jq -r --arg pid "$POLICY_ID" '.list[] | select(.policy_id == $pid and .active == true) | .api_key' <<<"$RESP")
echo "$API_KEY"
}
# Max number of concurrent Fleet write jobs (create/update). Override via env if needed.
MAX_FLEET_JOBS=${MAX_FLEET_JOBS:-10}
@@ -112,7 +62,15 @@ elastic_fleet_load_integrations_dir() {
i=0
# Fetch the agent policy a single time; we look up integration ids locally below.
if ! POLICY_JSON=$(elastic_fleet_require_agent_policy "$AGENT_POLICY"); then
if ! POLICY_JSON=$(fleet_api "agent_policies/$AGENT_POLICY"); then
echo "Error: Failed to retrieve agent policy '$AGENT_POLICY'."
rm -f "$FAIL_FILE"
rm -rf "$OUT_DIR"
return 1
fi
if ! jq -e '.item.package_policies' <<<"$POLICY_JSON" >/dev/null 2>&1; then
echo "Error: Invalid agent policy response for '$AGENT_POLICY'."
rm -f "$FAIL_FILE"
rm -rf "$OUT_DIR"
return 1
@@ -166,15 +124,9 @@ elastic_fleet_integration_check() {
JSON_STRING=$2
NAME=$(jq -r .name "$JSON_STRING")
INTEGRATION_ID=""
NAME=$(jq -r .name $JSON_STRING)
local POLICY_JSON
if ! POLICY_JSON=$(elastic_fleet_require_agent_policy "$AGENT_POLICY"); then
return 1
fi
INTEGRATION_ID=$(jq -r --arg n "$NAME" '.item.package_policies[]? | select(.name==$n) | .id' <<<"$POLICY_JSON")
INTEGRATION_ID=$(/usr/sbin/so-elastic-fleet-agent-policy-view "$AGENT_POLICY" | jq -r '.item.package_policies[] | select(.name=="'"$NAME"'") | .id')
}
@@ -196,16 +148,7 @@ elastic_fleet_integration_remove() {
NAME=$2
local POLICY_JSON
if ! POLICY_JSON=$(elastic_fleet_require_agent_policy "$AGENT_POLICY"); then
return 1
fi
INTEGRATION_ID=$(jq -r --arg n "$NAME" '.item.package_policies[]? | select(.name==$n) | .id' <<<"$POLICY_JSON")
if [ -z "$INTEGRATION_ID" ]; then
echo "Error: Integration '$NAME' was not found in agent policy '$AGENT_POLICY'." >&2
return 1
fi
INTEGRATION_ID=$(/usr/sbin/so-elastic-fleet-agent-policy-view "$AGENT_POLICY" | jq -r '.item.package_policies[] | select(.name=="'"$NAME"'") | .id')
JSON_STRING=$( jq -n \
--arg INTEGRATIONID "$INTEGRATION_ID" \
@@ -13,10 +13,7 @@ ERROR=false
for INTEGRATION in /opt/so/conf/elastic-fleet/integrations/elastic-defend/*.json
do
printf "\n\nInitial Endpoints Policy - Loading $INTEGRATION\n"
if ! elastic_fleet_integration_check "endpoints-initial" "$INTEGRATION"; then
ERROR=true
continue
fi
elastic_fleet_integration_check "endpoints-initial" "$INTEGRATION"
if [ -n "$INTEGRATION_ID" ]; then
printf "\n\nIntegration $NAME exists - Upgrading integration policy\n"
if ! elastic_fleet_integration_policy_upgrade "$INTEGRATION_ID"; then
@@ -7,35 +7,20 @@
. /usr/sbin/so-elastic-fleet-common
# Get all the fleet policies
if ! json_output=$(fleet_api "agent_policies" -H 'kbn-xsrf: true'); then
echo "Error: Failed to retrieve Fleet agent policies." >&2
exit 1
fi
if ! jq -e '.items' <<<"$json_output" >/dev/null 2>&1; then
echo "Error: Invalid Fleet agent policies response." >&2
exit 1
fi
json_output=$(curl -s -K /opt/so/conf/elasticsearch/curl.config -L -X GET "localhost:5601/api/fleet/agent_policies" -H 'kbn-xsrf: true')
# Extract the IDs that start with "FleetServer_"
POLICY=$(jq -r '.items[] | select(.id | startswith("FleetServer_")) | .id' <<<"$json_output")
POLICY=$(echo "$json_output" | jq -r '.items[] | select(.id | startswith("FleetServer_")) | .id')
# Iterate over each ID in the POLICY variable
for POLICYNAME in $POLICY; do
printf "\nUpdating Policy: $POLICYNAME\n"
if ! POLICY_JSON=$(elastic_fleet_require_agent_policy "$POLICYNAME"); then
exit 1
fi
INTEGRATION_ID=$(jq -r '.item.package_policies[]? | select(.package.name == "fleet_server") | .id' <<<"$POLICY_JSON")
if [ -z "$INTEGRATION_ID" ]; then
echo "Error: fleet_server integration was not found in agent policy '$POLICYNAME'." >&2
exit 1
fi
# First get the Integration ID
INTEGRATION_ID=$(/usr/sbin/so-elastic-fleet-agent-policy-view "$POLICYNAME" | jq -r '.item.package_policies[] | select(.package.name == "fleet_server") | .id')
# Modify the default integration policy to update the policy_id and an with the correct naming
UPDATED_INTEGRATION_POLICY=$(jq --arg policy_id "$POLICYNAME" --arg name "fleet_server-$POLICYNAME" '
UPDATED_INTEGRATION_POLICY=$(jq --arg policy_id "$POLICYNAME" --arg name "fleet_server-$POLICYNAME" '
.policy_id = $policy_id |
.name = $name' /opt/so/conf/elastic-fleet/integrations/fleet-server/fleet-server.json)
@@ -22,19 +22,12 @@ NUM_RUNNING=$(pgrep -cf "/bin/bash /sbin/so-elastic-agent-gen-installers")
for i in {1..30}
do
ENROLLMENTOKEN=$(elastic_fleet_active_enrollment_token "endpoints-initial")
TOKEN_RC=$?
if [ "$TOKEN_RC" -eq 2 ]; then
exit 1
fi
ENROLLMENTOKEN=$(curl -K /opt/so/conf/elasticsearch/curl.config -L "localhost:5601/api/fleet/enrollment_api_keys?perPage=100" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' | jq .list | jq -r -c '.[] | select(.policy_id | contains("endpoints-initial")) | .api_key')
FLEETHOST=$(curl -K /opt/so/conf/elasticsearch/curl.config 'http://localhost:5601/api/fleet/fleet_server_hosts/grid-default' | jq -r '.item.host_urls[]' | paste -sd ',')
if [[ -n "$FLEETHOST" ]] && [[ -n "$ENROLLMENTOKEN" ]]; then
break
fi
sleep 10
if [[ $FLEETHOST ]] && [[ $ENROLLMENTOKEN ]]; then break; else sleep 10; fi
done
if [[ -z "$FLEETHOST" ]] || [[ -z "$ENROLLMENTOKEN" ]]; then
if [[ -z $FLEETHOST ]] || [[ -z $ENROLLMENTOKEN ]]; then
printf "\nFleet Host URL, Enrollment Token or Elastic Version empty - exiting..."
printf "\nFleet Host: $FLEETHOST, Enrollment Token: $ENROLLMENTOKEN\n"
exit 1
@@ -74,25 +67,19 @@ for GOOS in "${GOTARGETOS[@]}"; do
GOARCH="amd64"
if [[ $GOOS == 'darwin/arm64' ]]; then GOOS="darwin" && GOARCH="arm64"; fi
printf "\n\n### Generating $GOOS/$GOARCH Installer...\n"
if ! docker run -e CGO_ENABLED=0 -e GOOS=$GOOS -e GOARCH=$GOARCH \
docker run -e CGO_ENABLED=0 -e GOOS=$GOOS -e GOARCH=$GOARCH \
--mount type=bind,source=/etc/pki/tls/certs/,target=/workspace/files/cert/ \
--mount type=bind,source=/nsm/elastic-agent-workspace/,target=/workspace/files/elastic-agent/ \
--mount type=bind,source=/opt/so/saltstack/local/salt/elasticfleet/files/,target=/output/ \
{{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent-builder:{{ GLOBALS.so_version }} go build -ldflags "-X main.fleetHostURLsList=$FLEETHOST -X main.enrollmentToken=$ENROLLMENTOKEN" -o /output/so-elastic-agent_${GOOS}_${GOARCH}; then
printf "\n### ERROR: Failed to generate $GOOS/$GOARCH installer. Exiting...\n"
exit 1
fi
{{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent-builder:{{ GLOBALS.so_version }} go build -ldflags "-X main.fleetHostURLsList=$FLEETHOST -X main.enrollmentToken=$ENROLLMENTOKEN" -o /output/so-elastic-agent_${GOOS}_${GOARCH}
printf "\n### $GOOS/$GOARCH Installer Generated...\n"
done
printf "\n\n### Generating MSI...\n"
cp /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64 /opt/so/saltstack/local/salt/elasticfleet/files/so-elastic-agent_windows_amd64.exe
if ! docker run \
docker run \
--mount type=bind,source=/opt/so/saltstack/local/salt/elasticfleet/files/,target=/output/ -w /output \
{{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent-builder:{{ GLOBALS.so_version }} wixl -o so-elastic-agent_windows_amd64_msi --arch x64 /workspace/so-elastic-agent.wxs; then
printf "\n### ERROR: Failed to generate MSI. Exiting...\n"
exit 1
fi
{{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elastic-agent-builder:{{ GLOBALS.so_version }} wixl -o so-elastic-agent_windows_amd64_msi --arch x64 /workspace/so-elastic-agent.wxs
printf "\n### MSI Generated...\n"
# Verify installers were created
@@ -202,9 +202,26 @@ fi
### Finalization ###
# Query for Enrollment Tokens for default policies
ENDPOINTSENROLLMENTOKEN=$(elastic_fleet_active_enrollment_token "endpoints-initial") || exit 1
GRIDNODESENROLLMENTOKENGENERAL=$(elastic_fleet_active_enrollment_token "so-grid-nodes_general") || exit 1
GRIDNODESENROLLMENTOKENHEAVY=$(elastic_fleet_active_enrollment_token "so-grid-nodes_heavy") || exit 1
if ENDPOINTSENROLLMENTOKEN_RAW=$(fleet_api "enrollment_api_keys" -H 'kbn-xsrf: true' -H 'Content-Type: application/json'); then
ENDPOINTSENROLLMENTOKEN=$(echo "$ENDPOINTSENROLLMENTOKEN_RAW" | jq .list | jq -r -c '.[] | select(.policy_id | contains("endpoints-initial")) | .api_key')
else
echo -e "\nFailed to query for Endpoints enrollment token"
exit 1
fi
if GRIDNODESENROLLMENTOKENGENERAL_RAW=$(fleet_api "enrollment_api_keys" -H 'kbn-xsrf: true' -H 'Content-Type: application/json'); then
GRIDNODESENROLLMENTOKENGENERAL=$(echo "$GRIDNODESENROLLMENTOKENGENERAL_RAW" | jq .list | jq -r -c '.[] | select(.policy_id | contains("so-grid-nodes_general")) | .api_key')
else
echo -e "\nFailed to query for Grid nodes - General enrollment token"
exit 1
fi
if GRIDNODESENROLLMENTOKENHEAVY_RAW=$(fleet_api "enrollment_api_keys" -H 'kbn-xsrf: true' -H 'Content-Type: application/json'); then
GRIDNODESENROLLMENTOKENHEAVY=$(echo "$GRIDNODESENROLLMENTOKENHEAVY_RAW" | jq .list | jq -r -c '.[] | select(.policy_id | contains("so-grid-nodes_heavy")) | .api_key')
else
echo -e "\nFailed to query for Grid nodes - Heavy enrollment token"
exit 1
fi
# Store needed data in minion pillar
pillar_file=/opt/so/saltstack/local/pillar/minions/{{ GLOBALS.minion_id }}.sls
+10 -10
View File
@@ -37,8 +37,8 @@ elasticsearch_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://elasticsearch/tools/sbin
- user: root
- group: root
- user: 930
- group: 939
- file_mode: 755
- exclude_pat:
- so-elasticsearch-pipelines # exclude this because we need to watch it for changes, we sync it in another state
@@ -49,8 +49,8 @@ so-elasticsearch-system-indices-patch-script:
file.managed:
- name: /usr/sbin/so-elasticsearch-system-indices-patch
- source: salt://elasticsearch/tools/sbin/so-elasticsearch-system-indices-patch
- user: root
- group: root
- user: 930
- group: 939
- mode: 755
- show_changes: False
@@ -58,8 +58,8 @@ elasticsearch_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://elasticsearch/tools/sbin_jinja
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
- template: jinja
- exclude_pat:
@@ -72,8 +72,8 @@ so-elasticsearch-ilm-policy-load-script:
file.managed:
- name: /usr/sbin/so-elasticsearch-ilm-policy-load
- source: salt://elasticsearch/tools/sbin_jinja/so-elasticsearch-ilm-policy-load
- user: root
- group: root
- user: 930
- group: 939
- mode: 754
- template: jinja
- defaults:
@@ -84,8 +84,8 @@ so-elasticsearch-pipelines-script:
file.managed:
- name: /usr/sbin/so-elasticsearch-pipelines
- source: salt://elasticsearch/tools/sbin/so-elasticsearch-pipelines
- user: root
- group: root
- user: 930
- group: 939
- mode: 754
- show_changes: False
@@ -5,8 +5,7 @@
{ "rename": { "field": "message2.proto", "target_field": "network.transport", "ignore_missing": true } },
{ "rename": { "field": "message2.app_proto", "target_field": "network.protocol", "ignore_missing": true } },
{ "rename": { "field": "message2.fileinfo.filename", "target_field": "file.name", "ignore_missing": true } },
{ "rename": { "field": "message2.fileinfo.gaps", "target_field": "suricata.fileinfo.gaps", "ignore_missing": true } },
{ "set": { "if": "ctx.suricata?.fileinfo?.gaps == false", "field": "file.bytes.missing", "value": 0 } },
{ "rename": { "field": "message2.fileinfo.gaps", "target_field": "file.bytes.missing", "ignore_missing": true } },
{ "rename": { "field": "message2.fileinfo.magic", "target_field": "file.mime_type", "ignore_missing": true } },
{ "rename": { "field": "message2.fileinfo.md5", "target_field": "hash.md5", "ignore_missing": true } },
{ "rename": { "field": "message2.fileinfo.sha1", "target_field": "hash.sha1", "ignore_missing": true } },
-4
View File
@@ -21,16 +21,12 @@ hypervisor_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://hypervisor/tools/sbin
- user: root
- group: root
- file_mode: 744
hypervisor_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://hypervisor/tools/sbin_jinja
- user: root
- group: root
- template: jinja
- file_mode: 744
+2 -2
View File
@@ -86,8 +86,8 @@ idh_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://idh/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#idh_sbin_jinja:
+2 -2
View File
@@ -41,8 +41,8 @@ influxdb_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://influxdb/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#influxdb_sbin_jinja:
+4 -4
View File
@@ -30,16 +30,16 @@ kafka_sbin_tools:
file.recurse:
- name: /usr/sbin
- source: salt://kafka/tools/sbin
- user: root
- group: root
- user: 960
- group: 960
- file_mode: 755
kafka_sbin_jinja_tools:
file.recurse:
- name: /usr/sbin
- source: salt://kafka/tools/sbin_jinja
- user: root
- group: root
- user: 960
- group: 960
- file_mode: 755
- template: jinja
- defaults:
+4 -4
View File
@@ -36,16 +36,16 @@ kibana_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://kibana/tools/sbin
- user: root
- group: root
- user: 932
- group: 939
- file_mode: 755
kibana_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://kibana/tools/sbin_jinja
- user: root
- group: root
- user: 932
- group: 939
- file_mode: 755
- template: jinja
- defaults:
-2
View File
@@ -6,8 +6,6 @@ so-fix-salt-ldap_script:
file.managed:
- name: /usr/sbin/so-fix-salt-ldap.py
- source: salt://libvirt/64962/scripts/so-fix-salt-ldap.py
- user: root
- group: root
- mode: 744
fix-salt-ldap:
+2 -2
View File
@@ -40,8 +40,8 @@ logstash_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://logstash/tools/sbin
- user: root
- group: root
- user: 931
- group: 939
- file_mode: 755
#logstash_sbin_jinja:
+7 -11
View File
@@ -113,8 +113,8 @@ manager_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://manager/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
- exclude_pat:
- "*_test.py"
@@ -124,8 +124,8 @@ manager_sbin_jinja:
file.recurse:
- name: /usr/sbin/
- source: salt://manager/tools/sbin_jinja/
- user: root
- group: root
- user: socore
- group: socore
- file_mode: 755
- template: jinja
- show_changes: False
@@ -190,15 +190,11 @@ so_fleetagent_monitor:
- month: '*'
- dayweek: '*'
# This tree is the source of every root-executed script (/usr/sbin, reactors, _runners,
# engines, salt-relay.sh). SOC mounts /opt/so/saltstack rw as uid 939 but only writes
# under local/. Do not add dir_mode/file_mode here -- SOC reads default/ and 750/640
# would break its config load.
root_own_saltstack_default:
socore_own_saltstack_default:
file.directory:
- name: /opt/so/saltstack/default
- user: root
- group: root
- user: socore
- group: socore
- recurse:
- user
- group
+1 -1
View File
@@ -126,7 +126,7 @@ function fail() {
}
function hydraCurl() {
docker exec "$hydraContainer" curl "$@"
docker exec -i "$hydraContainer" curl "$@"
}
function require() {
+6 -84
View File
@@ -121,14 +121,8 @@ for i in "$@"; do
esac
done
if [[ -n "$MINION_ID" && ! "$MINION_ID" =~ ^[A-Za-z0-9._-]{1,253}$ ]]; then
echo "Invalid minion id: $MINION_ID"
log "ERROR" "Invalid minion id: $MINION_ID"
exit 1
fi
readonly PILLARFILE=/opt/so/saltstack/local/pillar/minions/$MINION_ID.sls
readonly ADVPILLARFILE=/opt/so/saltstack/local/pillar/minions/adv_$MINION_ID.sls
PILLARFILE=/opt/so/saltstack/local/pillar/minions/$MINION_ID.sls
ADVPILLARFILE=/opt/so/saltstack/local/pillar/minions/adv_$MINION_ID.sls
function getinstallinfo() {
log "INFO" "Getting install info for minion $MINION_ID"
@@ -139,23 +133,10 @@ function getinstallinfo() {
return 1
fi
# install.txt is controlled by the minion; only accept known keys and never eval or export them
local line key
while IFS= read -r line; do
[[ "$line" == *=* ]] || continue
key=${line%%=*}
case "$key" in
MAINIP|MNIC|NODE_DESCRIPTION|ES_HEAP_SIZE|PATCHSCHEDULENAME|INTERFACE|NODETYPE|CORECOUNT|LSHOSTNAME|LSHEAP|CPUCORES|IDH_MGTRESTRICT|IDH_SERVICES)
printf -v "$key" '%s' "${line#*=}"
;;
*)
log "WARN" "Ignoring unexpected install var from $MINION_ID: ${key:0:64}"
;;
esac
done <<< "$INSTALLVARS"
if [[ "$NODE_DESCRIPTION" == \'*\' ]]; then
NODE_DESCRIPTION=${NODE_DESCRIPTION:1:-1}
while read -r var; do export "$var"; done <<< "$INSTALLVARS"
if [ $? -ne 0 ]; then
log "ERROR" "Failed to source install variables"
return 1
fi
log "INFO" "Fetched install info for $MINION_ID (node type: ${NODETYPE:-unset})"
@@ -195,12 +176,6 @@ function pcapspace() {
fi
fi
# Must be checked before arithmetic expansion, which evaluates array subscripts
if [[ ! "$SPACESIZE" =~ ^[0-9]+$ ]]; then
log "ERROR" "Invalid disk size for $MINION_ID: ${SPACESIZE:0:64}"
return 1
fi
local s=$(( $SPACESIZE / 1000000 ))
local s1=$(( $s / 4 * $PCAP_PERCENTAGE ))
@@ -1075,57 +1050,6 @@ function updateMineAndApplyStates() {
fi
}
# Values end up in a Jinja-rendered pillar and in bash, and may come from the minion
function validate_minion_vars() {
local error_msg=""
# Inline rather than valid_ip4: so-common is not installed yet when setup runs -o=setup
local octet='(25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])'
local ip4_re="^($octet\.){3}$octet$"
case "$NODETYPE" in
EVAL|STANDALONE|MANAGER|MANAGERSEARCH|MANAGERHYPE|IMPORT)
# Manager pillars also rewrite the CA pillar, so never accept them from a remote node
[[ "$OPERATION" == "setup" ]] || error_msg="Node type $NODETYPE can only be configured during setup"
;;
FLEET|IDH|HEAVYNODE|SENSOR|SEARCHNODE|RECEIVER|HYPERVISOR|DESKTOP)
;;
*)
error_msg="Invalid node type: ${NODETYPE:0:64}"
;;
esac
if [[ -z "$error_msg" ]]; then
if [[ ! "$MAINIP" =~ $ip4_re ]]; then
error_msg="Invalid MAINIP: ${MAINIP:0:64}"
elif [[ ! "$MNIC" =~ ^[A-Za-z0-9._-]*$ ]]; then
error_msg="Invalid MNIC: ${MNIC:0:64}"
elif [[ ! "$INTERFACE" =~ ^[A-Za-z0-9._-]*$ ]]; then
error_msg="Invalid INTERFACE: ${INTERFACE:0:64}"
elif [[ ! "$LSHOSTNAME" =~ ^[A-Za-z0-9._-]*$ ]]; then
error_msg="Invalid LSHOSTNAME: ${LSHOSTNAME:0:64}"
elif [[ ! "$ES_HEAP_SIZE" =~ ^([0-9]+[kKmMgG]?)?$ ]]; then
error_msg="Invalid ES_HEAP_SIZE: ${ES_HEAP_SIZE:0:64}"
elif [[ ! "$LSHEAP" =~ ^([0-9]+[kKmMgG]?)?$ ]]; then
error_msg="Invalid LSHEAP: ${LSHEAP:0:64}"
elif [[ ! "$CORECOUNT" =~ ^[0-9]*$ ]]; then
error_msg="Invalid CORECOUNT: ${CORECOUNT:0:64}"
elif [[ ! "$CPUCORES" =~ ^[0-9]*$ ]]; then
error_msg="Invalid CPUCORES: ${CPUCORES:0:64}"
elif [[ ! "$IDH_MGTRESTRICT" =~ ^(True|False)?$ ]]; then
error_msg="Invalid IDH_MGTRESTRICT: ${IDH_MGTRESTRICT:0:64}"
fi
fi
if [[ -n "$error_msg" ]]; then
log "ERROR" "$error_msg"
echo "$error_msg"
return 1
fi
# Free text; removing braces is enough to prevent any Jinja delimiter
NODE_DESCRIPTION=${NODE_DESCRIPTION//[\{\}[:cntrl:]]/}
}
function setupMinionFiles() {
log "INFO" "Setting up minion files for $MINION_ID (pillar: $PILLARFILE)"
@@ -1137,8 +1061,6 @@ function setupMinionFiles() {
return 1
fi
validate_minion_vars || return 1
# Create the base minion files
create_minion_files || return 1
+2 -2
View File
@@ -124,8 +124,8 @@ copy_new_files() {
rsync -a salt $default_salt_dir/
rsync -a pillar $default_salt_dir/
chown -R root:root $default_salt_dir/salt
chown -R root:root $default_salt_dir/pillar
chown -R socore:socore $default_salt_dir/salt
chown -R socore:socore $default_salt_dir/pillar
chmod 755 $default_salt_dir/pillar/firewall/addfirewall.sh
rm -rf /tmp/sogh
+1 -1
View File
@@ -156,7 +156,7 @@ function fail() {
}
function kratosCurl() {
docker exec "$kratosContainer" curl "$@"
docker exec -i "$kratosContainer" curl "$@"
}
function require() {
+1 -95
View File
@@ -121,9 +121,6 @@ check_err() {
161)
echo 'Required intermediate Elasticsearch upgrade not complete'
;;
162)
echo 'One or more Elastic Agent nodes do not support the x86-64-v3 CPU instruction set'
;;
170)
echo "Intermediate upgrade completed successfully to $next_step_so_version, but next soup to Security Onion $originally_requested_so_version could not be started automatically."
echo "Start soup again manually to continue the upgrade to Security Onion $originally_requested_so_version."
@@ -351,83 +348,6 @@ check_cluster_health() {
exit 0
}
no_soup_for_you() {
echo ""
echo "No soup for you!"
exit 162
}
check_cpu_compatibility() {
# Roles running a container built from the so-elastic-agent image; mirrors the
# elasticagent and elasticfleet entries in salt/reactor/pillar_push_map.yaml.
local cpu_target='G@role:so-heavynode or G@role:so-eval or G@role:so-fleet or G@role:so-import or G@role:so-manager or G@role:so-managerhype or G@role:so-managersearch or G@role:so-standalone'
local expected_nodes cpu_results node result confirm
local -a unsupported=() offline=()
echo "Checking that Elastic Agent nodes support the x86-64-v3 CPU instruction set now required by Elastic."
if [[ "$SKIP_CPU_CHECK" == "true" ]]; then
printf "\nSkipping the x86-64-v3 CPU check because --skip-cpu-check was specified.\n\n"
return
fi
# Nodes that never answer are absent from the results, so diff against who should have.
expected_nodes=$(salt -C "$cpu_target" --preview-target --out=json 2>/dev/null | jq -r '.[]?') || true
if [[ -z "$expected_nodes" ]]; then
printf "\nCould not determine which nodes run the Elastic Agent, so the x86-64-v3 CPU check cannot run.\n"
no_soup_for_you
fi
cpu_results=$(salt -t 30 -C "$cpu_target" cmd.run "/lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v3" --out=json 2>/dev/null) || true
while IFS= read -r node; do
[[ -z "$node" ]] && continue
result=$(jq -r --arg node "$node" '.[$node] // empty' <<< "$cpu_results" 2>/dev/null)
if [[ -z "$result" || "$result" == *"did not return"* ]]; then
offline+=("$node")
elif [[ "$result" != *"x86-64-v3 (supported"* ]]; then
# glibc appends "(supported, searched)" only when supported; the open paren keeps
# this from matching a future "(unsupported".
unsupported+=("$node")
fi
done <<< "$expected_nodes"
if [[ ${#unsupported[@]} -eq 0 && ${#offline[@]} -eq 0 ]]; then
printf "\nAll Elastic Agent nodes support x86-64-v3. We can proceed with SOUP.\n\n"
return
fi
echo ""
if [[ ${#unsupported[@]} -gt 0 ]]; then
echo "The following node(s) do NOT support the x86-64-v3 CPU instruction set:"
printf ' %s\n' "${unsupported[@]}"
echo ""
echo "Upstream Elastic now builds its binaries for x86-64-v3, so these nodes can no"
echo "longer run Elastic. Upgrading them WILL BREAK them."
echo ""
fi
if [[ ${#offline[@]} -gt 0 ]]; then
echo "The following node(s) did not respond and could not be checked:"
printf ' %s\n' "${offline[@]}"
echo ""
echo "These nodes are offline, so we cannot confirm they support x86-64-v3, which"
echo "upstream Elastic now requires."
echo ""
fi
if [[ -n $UNATTENDED ]]; then
echo "Unattended mode cannot prompt for an override. Re-run soup interactively, or pass --skip-cpu-check to bypass this check."
no_soup_for_you
fi
read -rp "Type 'override' to continue anyway, or press Enter to exit: " confirm
if [[ "${confirm,,}" == "override" ]]; then
printf "\nOverride accepted. Continuing at your own risk.\n\n"
else
no_soup_for_you
fi
}
check_fleet_server() {
echo "Checking that Elastic Fleet Server is responding."
# Modeled on the wait_for_so-elastic-fleet state check in elasticfleet/enabled.sls,
@@ -2136,9 +2056,6 @@ main() {
echo "Let's see if we need to update Security Onion."
upgrade_check
check_cpu_compatibility
upgrade_space
echo "Verifying Elasticsearch version compatibility across the grid before upgrading."
@@ -2417,17 +2334,6 @@ fi
echo "### soup has been served at $(date) ###"
}
SKIP_CPU_CHECK=false
declare -a SOUP_ARGS=()
for arg in "$@"; do
if [[ "$arg" == "--skip-cpu-check" ]]; then
SKIP_CPU_CHECK=true
else
SOUP_ARGS+=("$arg")
fi
done
set -- "${SOUP_ARGS[@]}"
while getopts ":b:f:y" opt; do
case ${opt} in
b )
@@ -2451,7 +2357,7 @@ while getopts ":b:f:y" opt; do
ISOLOC="$OPTARG"
;;
\? )
echo "Usage: soup [-b] [-y] [-f <iso location>] [--skip-cpu-check]"
echo "Usage: soup [-b] [-y] [-f <iso location>]"
exit 1
;;
: )
+2 -2
View File
@@ -57,8 +57,8 @@ nginx_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://nginx/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#nginx_sbin_jinja:
+1 -1
View File
@@ -183,7 +183,7 @@ http {
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.2 TLSv1.3;
location ~* (^/login|^/login/.*|^/js/.*|^/css/.*|^/images/.*|^/pages/.*|^/docs/.*) {
location ~* (^/login/.*|^/js/.*|^/css/.*|^/images/.*|^/pages/.*|^/docs/.*) {
proxy_pass http://{{ GLOBALS.manager }}:9822;
proxy_read_timeout 90;
proxy_connect_timeout 90;
+4 -4
View File
@@ -50,16 +50,16 @@ redis_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://redis/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
redis_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://redis/tools/sbin_jinja
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
- template: jinja
+2 -4
View File
@@ -3,8 +3,6 @@ salt_bootstrap:
file.managed:
- name: /usr/sbin/bootstrap-salt.sh
- source: salt://salt/scripts/bootstrap-salt.sh
- user: root
- group: root
- mode: 755
- show_changes: False
@@ -12,6 +10,6 @@ salt_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://salt/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
-2
View File
@@ -35,8 +35,6 @@ combine_bond_script:
file.managed:
- name: /usr/sbin/so-combine-bond
- source: salt://sensor/tools/sbin_jinja/so-combine-bond
- user: root
- group: root
- mode: 755
- template: jinja
- defaults:
+2 -2
View File
@@ -64,8 +64,8 @@ sensoroni_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://sensoroni/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#sensoroni_sbin_jinja:
@@ -1,3 +1,2 @@
requests>=2.34.0
whoisit>=4.0.5
anyio>=4.15.1
requests>=2.31.0
whoisit>=2.7.0
+2 -2
View File
@@ -171,8 +171,8 @@ soc_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://soc/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#soc_sbin_jinja:
+1 -68
View File
@@ -1466,7 +1466,6 @@ soc:
- core
- emerging_threats_addon
useEsql: false
esqlCaseInsensitive: true
elastic:
hostUrl:
remoteHostUrls: []
@@ -1494,9 +1493,6 @@ soc:
org: Security Onion
bucket: telegraf/so_short_term
verifyCert: false
notification:
dismissedPruneDays: 30
enabled: false
playbook:
autoUpdateEnabled: true
playbookImportFrequencySeconds: 86400
@@ -1542,7 +1538,7 @@ soc:
Orchestrator: sonnet@SOAI
Investigator: gemma@SOAI
DetectionEngineer: gemma@SOAI
useMemory: false
useMemory: true
useMemoryScanner: false
dontScanBefore: ""
memoryScanIntervalSeconds: 300
@@ -1561,69 +1557,6 @@ soc:
reconcilePersona: ""
toolUseTurnAttempts: 12
toolUseTurnDelayMs: 175
tools:
filterEventFields:
- "@timestamp"
- "client.name"
- "destination.ip"
- "destination.port"
- "destination.geo.country_name"
- "dns.query.name"
- "dns.query_name"
- "event.action"
- "event.category"
- "event.module"
- "event.dataset"
- "event.outcome"
- "event.severity"
- "event.severity_label"
- "event.type"
- "event_data.agent.name"
- "event_data.host.os.name"
- "file.mime_type"
- "file.name"
- "hash.md5"
- "hash.sha1"
- "host.mac"
- "host.name"
- "host.os.name"
- "http.method"
- "http.useragent"
- "http.virtual_host"
- "log.id.uid"
- "network.community_id"
- "network.protocol"
- "network.transport"
- "notice.message"
- "observer.name"
- "process.name"
- "process.executable"
- "process.entity_id"
- "process.command_line"
- "process.Ext.ancestry"
- "process.parent.entity_id"
- "process.parent.command_line"
- "rule.category"
- "rule.name"
- "rule.uuid"
- "software.name"
- "software.type"
- "software.version.unparsed"
- "source.ip"
- "source.port"
- "source.geo.country_name"
- "ssh.cypher_algorithm"
- "ssh.client"
- "ssh.server"
- "ssl.cipher"
- "ssl.server_name"
- "ssl.version"
- "system.auth.sudo.command"
- "user.name"
- "user.domain"
- "user.effective.name"
- "weird.name"
- "tags"
onionconfig:
saltstackDir: /opt/so/saltstack
bypassEnabled: false
+2 -2
View File
@@ -18,8 +18,8 @@ hypervisor_annotation:
- name: /opt/so/saltstack/default/salt/hypervisor/soc_hypervisor.yaml
- source: salt://soc/dyanno/hypervisor/soc_hypervisor.yaml.jinja
- template: jinja
- user: root
- group: root
- user: socore
- group: socore
- defaults:
HYPERVISORS: {{ HYPERVISORS }}
baseDomainStatus: {{ salt['pillar.get']('baseDomain:status', 'Initialized') }}
-25
View File
@@ -1,31 +1,6 @@
name: Security Onion Baseline Pipeline
priority: 90
transformations:
# ES|QL scalar == returns null on multivalued fields; the
# backend reads this key and emits MV_INTERSECTS instead.
- id: declare_multivalue_fields
type: set_state
key: multivalue_fields
val:
- event.type
- event.action
- event.category
- tags
- process.args
- related.ip
- dns.resolved_ip
- id: esql_default_index
type: set_state
key: index
val: .ds-logs-*
- id: esql_source_metadata
type: set_state
key: metadata
val: "_id, _index, _source"
- id: esql_source_keep
type: set_state
key: keep
val: "_id, _index, _source"
- id: baseline_field_name_mapping
type: field_name_mapping
mapping:
-37
View File
@@ -155,14 +155,6 @@ soc:
description: Path to custom markdown templates for PDF report generation. All markdown files in this directory will be available as custom reports in the SOC Reports interface.
global: True
advanced: True
schedules:
title: Schedules
description: Schedules that are shared across the Security Onion product. Modify via one of the SOC Schedules view.
readonlyUi: True
global: True
forcedType: string
syntax: json
storage: db
subgrids:
title: Subordinate Grids
description: |
@@ -404,11 +396,6 @@ soc:
global: True
advanced: True
forcedType: bool
esqlCaseInsensitive:
description: "Match string values case-insensitively when converting Sigma rules. Applies to ES|QL only"
global: True
advanced: True
forcedType: bool
elastic:
index:
description: Comma-separated list of indices or index patterns (wildcard "*" supported) that SOC will search for records.
@@ -489,24 +476,6 @@ soc:
global: True
advanced: True
forcedType: bool
notification:
destinations:
title: Notification Destinations
description: JSON list of notifications. Modify via the SOC Notifications view.
readonlyUi: True
global: True
forcedType: string
syntax: json
storage: db
dismissedPruneDays:
title: Dismissed Retention Days
description: The number of days to retain dismissed notifications. When a notification is dismissed, it will be pruned after this many days. Only one user need dismiss a notification for it to be pruned.
forcedType: int
global: True
enabled:
description: Enables or disables the SOC notification module.
forcedType: bool
global: True
postgres:
host:
description: Hostname or IP address of the PostgreSQL server used by SOC. Defaults to the manager hostname.
@@ -791,7 +760,6 @@ soc:
- gemini
- openai_responses
- openai_chat
- openai_embeddings
- field: apiUrl
label: API URL
required: False
@@ -948,11 +916,6 @@ soc:
description: The number of times to retry extracting memories from a session if errors occur.
global: True
advanced: True
tools:
filterEventFields:
description: A whitelist of fields to return when OnionAI uses the query_events tool. All other fields are removed. One field per line.
global: True
multiline: True
client:
assistant:
enabled:
+2 -2
View File
@@ -51,8 +51,8 @@ strelka_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://strelka/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
{% else %}
+4 -4
View File
@@ -76,16 +76,16 @@ suricata_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://suricata/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
suricata_sbin_jinja:
file.recurse:
- name: /usr/sbin
- source: salt://suricata/tools/sbin_jinja
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
- template: jinja
+2 -2
View File
@@ -65,8 +65,8 @@ telegraf_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://telegraf/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#telegraf_sbin_jinja:
+2 -2
View File
@@ -101,8 +101,8 @@ zeek_sbin:
file.recurse:
- name: /usr/sbin
- source: salt://zeek/tools/sbin
- user: root
- group: root
- user: 939
- group: 939
- file_mode: 755
#zeek_sbin_jinja:
-1
View File
@@ -18,7 +18,6 @@ zeek:
StatsLogEnable: 0
StatsLogExpireInterval: 0
StatusCmdShowAll: 0
StopWait: 1
CrashExpireInterval: 0
SitePolicyScripts: local.zeek
LogDir: /nsm/zeek/logs
-10
View File
@@ -9,19 +9,9 @@
include:
- zeek.sostatus
# Stop first so the entrypoint's SIGTERM trap can archive the final logs; docker_container.absent
# with force is a 'docker rm -f', which never delivers SIGTERM. force stays so the state still
# converges if the stop overruns.
so-zeek_stopped:
docker_container.stopped:
- name: so-zeek
- error_on_absent: False
so-zeek:
docker_container.absent:
- force: True
- require:
- docker_container: so-zeek_stopped
so-zeek_so-status.disabled:
file.comment:
-4
View File
@@ -19,10 +19,6 @@ so-zeek:
- restart_policy: unless-stopped
- start: True
- privileged: True
# Docker's default 10s grace is not enough for the entrypoint's SIGTERM trap to run
# 'zeekctl stop' and let StopWait archive the final logs. Overrunning it means SIGKILL,
# which strands those logs in spool/tmp and marks every node crashed on the next start.
- stop_timeout: 180
{% if DOCKERMERGED.containers['so-zeek'].ulimits %}
- ulimits:
{% for ULIMIT in DOCKERMERGED.containers['so-zeek'].ulimits %}
-12
View File
@@ -99,18 +99,6 @@ zeek:
regexFailureMessage: You must enter a whole number of days, or 0 to keep crash directories forever.
helpLink: zeek
advanced: True
StopWait:
description: >-
Set to 1 to make "zeekctl stop" wait for the final logs to be archived instead of
letting that finish in the background. Security Onion stops Zeek by stopping its
container, so anything still running in the background is killed when the container
exits - without this, the last logs of each run are stranded unarchived in
/nsm/zeek/spool/tmp and never reach Elasticsearch. It is read only for that reason.
regex: ^[01]$
regexFailureMessage: You must enter 0 or 1.
helpLink: zeek
advanced: True
readonly: True
MinDiskSpace:
description: >-
Percentage of free disk space below which ZeekControl reports a warning, or 0 to disable the check
-2
View File
@@ -2125,8 +2125,6 @@ setup_salt_master_dirs() {
info "Chown the salt dirs on the manager for socore"
logCmd "chown -R socore:socore /opt/so"
# The default tree is root-executed code; SOC reads it but never writes it.
logCmd "chown -R root:root $default_salt_dir"
}
set_progress_str() {
Binary file not shown.