Merge remote-tracking branch 'remotes/origin/dev' into feature/users

Fix FleetDM - disable stats

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Contributing to Security Onion
+
+### Questions, suggestions, and general comments
+* Security Onion uses GitHub's [Discussions](https://github.com/Security-Onion-Solutions/securityonion/discussions) to provide a forum where the community and developers can interact as well as ask and answer questions.
+
+### Reporting a bug
+* The primary place to report unexpected behavior or possible bugs is the repo's [Discussions forum](https://github.com/Security-Onion-Solutions/securityonion/discussions).
+
+*  **If you are familiar with the current version of Security Onion and are confident you've discovered a bug**, first ensure there is not already an issue present by searching the open [issues](https://github.com/Security-Onion-Solutions/securityonion/issues). If there is, a thumbs up :+1: is a great way to show this bug is affecting you too.
+
+* If an issue doesn't exist, [open a new one](https://github.com/Security-Onion-Solutions/securityonion/issues/new), following the directions in the issue template. This means including:
+  * **System information** and how Security Onion was installed
+  * **Log files** relevant to the bug report
+  * **Reproduction steps** 
+
+### Contributing code
+
+* **All commits must be signed** with a valid key that has been added to your GitHub account. Each commit should have the "**Verified**" tag when viewed on GitHub as shown below:
+  
+  <img src="./assets/images/verified-commit-1.png" width="450">
+
+* If an issue does not already exist for the bug or feature for which you are submitting a pull request, [create one](https://github.com/Security-Onion-Solutions/securityonion/issues/new) with the relevant prefix. (**`FIX:`** for bug fixes, **`FEATURE:`** for new features.)
+
+* Link the PR to the related issue, either using [keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) in the PR description, or [manually](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue).
+
+* **Pull requests should be opened against the `dev` branch of this repo**, and should clearly describe the problem and solution.
+
+* Be sure you have tested your changes and are confident they will not break other parts of the product.
+
+* See this document's [code styling and conventions section](#code-style-and-conventions) below to be sure your PR fits our code requirements prior to submitting.
+
+
+
+### Code style and conventions
+* **Keep code [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)**. For example, Bash code used by multiple scripts will likely best be added to <span style="white-space: nowrap;">[`so-common`](salt/common/tools/sbin/so-common)</span>.
+
+* All new Bash code should pass [ShellCheck](https://www.shellcheck.net/) analysis. Where errors can be *safely* [ignored](https://github.com/koalaman/shellcheck/wiki/Ignore), the relevant disable directive should be accompanied by a brief explanation as to why the error is being ignored.
+
+* **Ensure all YAML (this includes Salt states and pillars) is properly formatted**. The spec for YAML v1.2 can be found [here](https://yaml.org/spec/1.2/spec.html), however there are numerous online resources with simpler descriptions of its formatting rules. 

README.md

@@ -1,14 +1,14 @@
-## Security Onion 2.3.51
+## Security Onion 2.3.80
 
-Security Onion 2.3.51 is here!
+Security Onion 2.3.80 is here!
 
 ## Screenshots
 
 Alerts
-![Alerts](https://raw.githubusercontent.com/security-onion-solutions/securityonion/master/screenshots/alerts-1.png)
+![Alerts](./assets/images/screenshots/alerts-1.png)
 
 Hunt
-![Hunt](https://raw.githubusercontent.com/security-onion-solutions/securityonion/master/screenshots/hunt-1.png)
+![Hunt](./assets/images/screenshots/hunt-1.png)
 
 ### Release Notes
 

SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x.x   | :white_check_mark: |
+| 16.04.x | :x:                |
+
+Security Onion 16.04 has reached End Of Life and is no longer supported.
+
+## Reporting a Vulnerability
+
+If you have any security concerns regarding Security Onion or believe you have uncovered a vulnerability, please follow these steps:
+
+- send an email to security@securityonion.net
+- include a description of the issue and steps to reproduce
+- please use plain text format (no Word documents or PDF files)
+- please do not disclose publicly until we have had sufficient time to resolve the issue
+
+This security address should be used only for undisclosed vulnerabilities. Dealing with fixed issues or general questions on how to use Security Onion should be handled via the normal support channels.

VERIFY_ISO.md

@@ -1,17 +1,18 @@
-### 2.3.51 ISO image built on 2021/04/27
+### 2.3.80 ISO image built on 2021/09/27
+
 
 
 ### Download and Verify
 
-2.3.51 ISO image:  
+2.3.80 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-2.3.51.iso
+https://download.securityonion.net/file/securityonion/securityonion-2.3.80.iso
 
-MD5: 7CFB525BEFC0A9F2ED148F5831E387FA  
+MD5: 24F38563860416F4A8ABE18746913E14  
-SHA1: 8CC34FCCC36822B309B8168AA706B3D1EC7F3BFD  
+SHA1: F923C005F54EA2A17AB225ADA0DA46042707AAD9  
-SHA256: 9892C2546C9AE5A48015160F379B070F0BE30C89693B97F3F1E1592DDCE1DEE0 
+SHA256: 8E95D10AF664D9A406C168EC421D943CB23F0D0C1813C6C2DBA9B4E131984018 
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.51.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.80.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS  
@@ -25,22 +26,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/ma
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.51.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.80.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-2.3.51.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-2.3.80.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-2.3.51.iso.sig securityonion-2.3.51.iso
+gpg --verify securityonion-2.3.80.iso.sig securityonion-2.3.80.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Thu 20 May 2021 07:49:57 AM EDT using RSA key ID FE507013
+gpg: Signature made Mon 27 Sep 2021 08:55:01 AM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

VERSION

@@ -1 +1 @@
-2.3.51
+2.3.90

files/firewall/hostgroups.local.yaml

@@ -16,6 +16,10 @@ firewall:
       ips:
         delete:
         insert:
+    endgame:
+      ips:
+        delete:
+        insert:
     fleet:
       ips:
         delete:

files/salt/master/master

@@ -67,3 +67,7 @@ peer:
 reactor:
   - 'so/fleet':
     - salt://reactor/fleet.sls
+  - 'salt/beacon/*/watch_sqlite_db//opt/so/conf/kratos/db/sqlite.db':
+    - salt://reactor/kratos.sls
+
+

pillar/elasticsearch/eval.sls

@@ -1,7 +1,7 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-common-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/elasticsearch/manager.sls

@@ -1,7 +1,8 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-common-template.json.jinja
+    - so/so-endgame-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/elasticsearch/search.sls

@@ -1,7 +1,8 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-common-template.json.jinja
+    - so/so-endgame-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/logstash/init.sls

@@ -1,6 +1,7 @@
 logstash:
   docker_options:
     port_bindings:
+      - 0.0.0.0:3765:3765
       - 0.0.0.0:5044:5044
       - 0.0.0.0:5644:5644
       - 0.0.0.0:6050:6050

pillar/logstash/manager.sls

@@ -5,5 +5,6 @@ logstash:
       config:
         - so/0009_input_beats.conf      
         - so/0010_input_hhbeats.conf
+        - so/0011_input_endgame.conf
         - so/9999_output_redis.conf.jinja
         

pillar/logstash/search.sls

@@ -7,8 +7,11 @@ logstash:
         - so/9000_output_zeek.conf.jinja
         - so/9002_output_import.conf.jinja
         - so/9034_output_syslog.conf.jinja
+        - so/9050_output_filebeatmodules.conf.jinja
         - so/9100_output_osquery.conf.jinja  
         - so/9400_output_suricata.conf.jinja
         - so/9500_output_beats.conf.jinja
         - so/9600_output_ossec.conf.jinja
         - so/9700_output_strelka.conf.jinja
+        - so/9800_output_logscan.conf.jinja
+        - so/9900_output_endgame.conf.jinja

pillar/top.sls

@@ -2,6 +2,7 @@ base:
   '*':
     - patch.needs_restarting
     - logrotate
+    - users
 
   '*_eval or *_helixsensor or *_heavynode or *_sensor or *_standalone or *_import':
     - match: compound
@@ -22,6 +23,12 @@ base:
   '*_manager or *_managersearch':
     - match: compound
     - data.*
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - secrets
     - global
     - minions.{{ grains.id }}
@@ -38,6 +45,12 @@ base:
     - secrets
     - healthcheck.eval
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - global
     - minions.{{ grains.id }}
 
@@ -46,6 +59,12 @@ base:
     - logstash.manager
     - logstash.search
     - elasticsearch.search
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - data.*
     - zeeklogs
     - secrets
@@ -59,6 +78,7 @@ base:
 
   '*_heavynode':
     - zeeklogs
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
 
@@ -80,6 +100,7 @@ base:
     - logstash
     - logstash.search
     - elasticsearch.search
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
     - data.nodestab
@@ -88,5 +109,11 @@ base:
     - zeeklogs
     - secrets
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - global
     - minions.{{ grains.id }}

pillar/users/init.sls

@@ -0,0 +1,2 @@
+# users pillar goes in /opt/so/saltstack/local/pillar/users/init.sls
+# the users directory may need to be created under /opt/so/saltstack/local/pillar

pillar/users/pillar.example

@@ -0,0 +1,19 @@
+users:
+  sclapton:
+    # required fields
+    status: present
+    # node_access determines which node types the user can access.
+    # this can either be by grains.role or by final part of the minion id after the _
+    node_access:
+      - standalone
+      - searchnode
+    # optional fields
+    fullname: Stevie Claptoon
+    uid: 1001
+    gid: 1001
+    homephone: does not have a phone
+    groups:
+      - mygroup1
+      - mygroup2
+      - wheel # give sudo access
+      

pillar/users/pillar.usage

@@ -0,0 +1,20 @@
+users:
+  sclapton:
+    # required fields
+    status: <present | absent>
+    # node_access determines which node types the user can access.
+    # this can either be by grains.role or by final part of the minion id after the _
+    node_access:
+      - standalone
+      - searchnode
+    # optional fields
+    fullname: <string>
+    uid: <integer>
+    gid: <integer>
+    roomnumber: <string>
+    workphone: <string>
+    homephone: <string>
+    groups:
+      - <string>
+      - <string>
+      - wheel # give sudo access

pillar/zeek/init.sls

@@ -52,5 +52,4 @@ zeek:
       - frameworks/signatures/detect-windows-shells
     redef:
       - LogAscii::use_json = T;
-      - LogAscii::json_timestamps = JSON::TS_ISO8601;
       - CaptureLoss::watch_interval = 5 mins;

salt/allowed_states.map.jinja

@@ -35,6 +35,7 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
@@ -45,7 +46,8 @@
           'schedule',
           'soctopus',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-heavynode': [
           'ca',
@@ -99,6 +101,7 @@
           'manager',
           'nginx',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
@@ -108,7 +111,8 @@
           'zeek',
           'schedule',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-manager': [
           'salt.master',
@@ -121,13 +125,15 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
           'utility',
           'schedule',
           'soctopus',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-managersearch': [
           'salt.master',
@@ -139,6 +145,7 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'manager',
           'idstools',
@@ -146,7 +153,8 @@
           'utility',
           'schedule',
           'soctopus',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-node': [
           'ca',
@@ -168,6 +176,7 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
@@ -178,7 +187,8 @@
           'schedule',
           'soctopus',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-sensor': [
           'ca',
@@ -233,11 +243,16 @@
     {% do allowed_states.append('elasticsearch') %}
   {% endif %}
 
-  {% if KIBANA and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+  {% if ELASTICSEARCH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
-    {% do allowed_states.append('kibana') %}
+    {% do allowed_states.append('elasticsearch.auth') %}
   {% endif %}
 
-  {% if CURATOR and grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+  {% if KIBANA and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    {% do allowed_states.append('kibana') %}
+    {% do allowed_states.append('kibana.secrets') %}
+  {% endif %}
+
+  {% if grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-manager'] %}
     {% do allowed_states.append('curator') %}
   {% endif %}
 

salt/ca/init.sls

@@ -24,8 +24,9 @@ pki_private_key:
       - x509: /etc/pki/ca.crt
     {%- endif %}
 
-/etc/pki/ca.crt:
+pki_public_ca_crt:
   x509.certificate_managed:
+    - name: /etc/pki/ca.crt
     - signing_private_key: /etc/pki/ca.key
     - CN: {{ manager }}
     - C: US

salt/common/files/log-rotate.conf

@@ -22,6 +22,7 @@
 /opt/so/log/salt/so-salt-minion-check
 /opt/so/log/salt/minion
 /opt/so/log/salt/master
+/opt/so/log/logscan/*.log
 {
     {{ logrotate_conf | indent(width=4) }}
 }

salt/common/init.sls

@@ -2,12 +2,18 @@
 {% if sls in allowed_states %}
 
 {% set role = grains.id.split('_') | last %}
+{% from 'elasticsearch/auth.map.jinja' import ELASTICAUTH with context %}
 
 # Remove variables.txt from /tmp - This is temp
 rmvariablesfile:
   file.absent:
     - name: /tmp/variables.txt
 
+dockergroup:
+  group.present:
+    - name: docker
+    - gid: 920
+
 # Add socore Group
 socoregroup:
   group.present:
@@ -95,22 +101,29 @@ commonpkgs:
       - netcat
       - python3-mysqldb
       - sqlite3
-      - argon2
       - libssl-dev
       - python3-dateutil
       - python3-m2crypto
       - python3-mysqldb
       - python3-packaging
+      - python3-lxml
       - git
       - vim
 
 heldpackages:
   pkg.installed:
     - pkgs:
+    {% if grains['oscodename'] == 'bionic' %}
       - containerd.io: 1.4.4-1
       - docker-ce: 5:20.10.5~3-0~ubuntu-bionic
       - docker-ce-cli: 5:20.10.5~3-0~ubuntu-bionic
       - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-bionic
+    {% elif grains['oscodename'] == 'focal' %}
+      - containerd.io: 1.4.9-1
+      - docker-ce: 5:20.10.8~3-0~ubuntu-focal
+      - docker-ce-cli: 5:20.10.5~3-0~ubuntu-focal
+      - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-focal
+    {% endif %}
     - hold: True
     - update_holds: True
 
@@ -128,7 +141,6 @@ commonpkgs:
       - net-tools
       - curl
       - sqlite
-      - argon2
       - mariadb-devel
       - nmap-ncat
       - python3
@@ -137,6 +149,7 @@ commonpkgs:
       - python36-m2crypto
       - python36-mysql
       - python36-packaging
+      - python36-lxml
       - yum-utils
       - device-mapper-persistent-data
       - lvm2
@@ -169,6 +182,14 @@ alwaysupdated:
 Etc/UTC:
   timezone.system
 
+elastic_curl_config:
+  file.managed:
+    - name: /opt/so/conf/elasticsearch/curl.config
+    - source: salt://elasticsearch/curl.config
+    - mode: 600
+    - show_changes: False
+    - makedirs: True
+
 # Sync some Utilities
 utilsyncscripts:
   file.recurse:
@@ -178,6 +199,10 @@ utilsyncscripts:
     - file_mode: 755
     - template: jinja
     - source: salt://common/tools/sbin
+    - defaults:
+        ELASTICCURL: 'curl'
+    - context:
+        ELASTICCURL: {{ ELASTICAUTH.elasticcurl }}
 
 {% if role in ['eval', 'standalone', 'sensor', 'heavynode'] %}
 # Add sensor cleanup
@@ -315,6 +340,16 @@ dockerreserveports:
     - name: /etc/sysctl.d/99-reserved-ports.conf
 
 {% if salt['grains.get']('sosmodel', '') %}
+  {% if grains['os'] == 'CentOS' %}     
+# Install Raid tools
+raidpkgs:
+  pkg.installed:
+    - skip_suggestions: True
+    - pkgs:
+      - securityonion-raidtools
+      - securityonion-megactl
+  {% endif %}
+
 # Install raid check cron
 /usr/sbin/so-raid-status > /dev/null 2>&1:
   cron.present:

salt/common/tools/sbin/so-airgap-hotfixapply

@@ -1,64 +0,0 @@
-#!/bin/bash
-
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-. /usr/sbin/so-common
-
-UPDATE_DIR=/tmp/sohotfixapply
-
-if [ -z "$1" ]; then
-  echo "No tarball given. Please provide the filename so I can run the hotfix"
-  echo "so-airgap-hotfixapply /path/to/sohotfix.tar" 
-  exit 1
-else
-  if [ ! -f "$1" ]; then
-    echo "Unable to find $1. Make sure your path is correct and retry."
-    exit 1
-  else
-    echo "Determining if we need to apply this hotfix"
-    rm -rf $UPDATE_DIR
-    mkdir -p $UPDATE_DIR
-    tar xvf $1 -C $UPDATE_DIR
-
-    # Compare some versions
-    NEWVERSION=$(cat $UPDATE_DIR/VERSION)
-    HOTFIXVERSION=$(cat $UPDATE_DIR/HOTFIX)
-    CURRENTHOTFIX=$(cat /etc/sohotfix)
-    INSTALLEDVERSION=$(cat /etc/soversion)
-
-    if [ "$INSTALLEDVERSION" == "$NEWVERSION" ]; then
-      echo "Checking to see if there are hotfixes needed"
-      if [ "$HOTFIXVERSION" == "$CURRENTHOTFIX" ]; then
-        echo "You are already running the latest version of Security Onion."
-        rm -rf $UPDATE_DIR
-        exit 1
-      else
-        echo "We need to apply a hotfix"
-        copy_new_files
-        echo $HOTFIXVERSION > /etc/sohotfix
-        salt-call state.highstate -l info queue=True
-        echo "The Hotfix $HOTFIXVERSION has been applied"
-        # Clean up 
-        rm -rf $UPDATE_DIR
-        exit 0
-      fi
-    else
-      echo "This hotfix is not compatible with your current version. Download the latest ISO and run soup"
-      rm -rf $UPDATE_DIR
-    fi
-
-  fi
-fi

salt/common/tools/sbin/so-allow

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env python3
 
 # Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
@@ -15,152 +15,199 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-. /usr/sbin/so-common
+import ipaddress
+import textwrap
+import os
+import subprocess
+import sys
+import argparse
+import re
+from lxml import etree as ET
+from xml.dom import minidom
+from datetime import datetime as dt
+from datetime import timezone as tz
 
-local_salt_dir=/opt/so/saltstack/local
-
-SKIP=0
-
-function usage {
-
-cat << EOF
-
-Usage: $0 [-abefhoprsw] [ -i IP ]
-
-This program allows you to add a firewall rule to allow connections from a new IP address or CIDR range.
-
-If you run this program with no arguments, it will present a menu for you to choose your options.
-
-If you want to automate and skip the menu, you can pass the desired options as command line arguments.
-
-EXAMPLES
-
-To add 10.1.2.3 to the analyst role:
-so-allow -a -i 10.1.2.3
-
-To add 10.1.2.0/24 to the osquery role:
-so-allow -o -i 10.1.2.0/24
-
-EOF
 
+LOCAL_SALT_DIR='/opt/so/saltstack/local'
+WAZUH_CONF='/nsm/wazuh/etc/ossec.conf'
+VALID_ROLES = {
+  'a': { 'role': 'analyst','desc': 'Analyst - 80/tcp, 443/tcp' },
+  'b': { 'role': 'beats_endpoint', 'desc': 'Logstash Beat - 5044/tcp' },
+  'e': { 'role': 'elasticsearch_rest', 'desc': 'Elasticsearch REST API - 9200/tcp' },
+  'f': { 'role': 'strelka_frontend', 'desc': 'Strelka frontend - 57314/tcp' },
+  'o': { 'role': 'osquery_endpoint', 'desc': 'Osquery endpoint - 8090/tcp' },
+  's': { 'role': 'syslog', 'desc': 'Syslog device - 514/tcp/udp' },
+  'w': { 'role': 'wazuh_agent', 'desc': 'Wazuh agent - 1514/tcp/udp' },
+  'p': { 'role': 'wazuh_api', 'desc': 'Wazuh API - 55000/tcp' },
+  'r': { 'role': 'wazuh_authd', 'desc': 'Wazuh registration service - 1515/tcp' }
 }
 
-while getopts "ahfesprbowi:" OPTION
-do
-    case $OPTION in
-        h)
-            usage
-            exit 0
-            ;;
-        a)
-            FULLROLE="analyst"
-            SKIP=1
-            ;;
-        b)
-            FULLROLE="beats_endpoint"
-            SKIP=1
-            ;;
-	e)
-            FULLROLE="elasticsearch_rest"
-            SKIP=1
-            ;;
-        f)
-	    FULLROLE="strelka_frontend"
-            SKIP=1
-            ;;
-        i)  IP=$OPTARG
-            ;;
-        o)
-            FULLROLE="osquery_endpoint"
-            SKIP=1
-            ;;
-        w)
-            FULLROLE="wazuh_agent"
-            SKIP=1
-            ;;
-        s)
-            FULLROLE="syslog"
-            SKIP=1
-            ;;
-        p)
-            FULLROLE="wazuh_api"
-            SKIP=1
-            ;;
-        r)
-            FULLROLE="wazuh_authd"
-            SKIP=1
-            ;;
-        *)
-            usage
-            exit 0
-            ;;
-    esac
-done
 
-if [ "$SKIP" -eq 0 ]; then
+def validate_ip_cidr(ip_cidr: str) -> bool:
+  try:
+    ipaddress.ip_address(ip_cidr)
+  except ValueError:
+    try:
+      ipaddress.ip_network(ip_cidr)
+    except ValueError:
+      return False
+  return True
 
-    echo "This program allows you to add a firewall rule to allow connections from a new IP address."
-    echo ""
-    echo "Choose the role for the IP or Range you would like to add"
-    echo ""
-    echo "[a] - Analyst - ports 80/tcp and 443/tcp"
-    echo "[b] - Logstash Beat - port 5044/tcp"
-    echo "[e] - Elasticsearch REST API - port 9200/tcp"
-    echo "[f] - Strelka frontend - port 57314/tcp"
-    echo "[o] - Osquery endpoint - port 8090/tcp"
-    echo "[s] - Syslog device - 514/tcp/udp"
-    echo "[w] - Wazuh agent - port 1514/tcp/udp"
-    echo "[p] - Wazuh API - port 55000/tcp"
-    echo "[r] - Wazuh registration service - 1515/tcp"
-    echo ""
-    echo "Please enter your selection:"
-    read -r ROLE
-    echo "Enter a single ip address or range to allow (example: 10.10.10.10 or 10.10.0.0/16):"
-    read -r IP
 
-    if [ "$ROLE" == "a" ]; then
+def role_prompt() -> str:
-        FULLROLE=analyst
+  print()
-    elif [ "$ROLE" == "b" ]; then
+  print('Choose the role for the IP or Range you would like to allow')
-        FULLROLE=beats_endpoint
+  print()
-    elif [ "$ROLE" == "e" ]; then
+  for role in VALID_ROLES:
-        FULLROLE=elasticsearch_rest
+    print(f'[{role}] - {VALID_ROLES[role]["desc"]}')
-    elif [ "$ROLE" == "f" ]; then
+  print()
-        FULLROLE=strelka_frontend
+  role = input('Please enter your selection: ')
-    elif [ "$ROLE" == "o" ]; then
+  if role in VALID_ROLES.keys():
-        FULLROLE=osquery_endpoint
+    return VALID_ROLES[role]['role']
-    elif [ "$ROLE" == "w" ]; then
+  else:
-        FULLROLE=wazuh_agent
+    print(f'Invalid role \'{role}\', please try again.', file=sys.stderr)
-    elif [ "$ROLE" == "s" ]; then
+    sys.exit(1)
-        FULLROLE=syslog
-    elif [ "$ROLE" == "p" ]; then
-        FULLROLE=wazuh_api
-    elif [ "$ROLE" == "r" ]; then
-        FULLROLE=wazuh_authd
-    else
-        echo "I don't recognize that role"
-        exit 1
-    fi
   
-fi
 
-echo "Adding $IP to the $FULLROLE role. This can take a few seconds"
+def ip_prompt() -> str:
-/usr/sbin/so-firewall includehost $FULLROLE $IP
+  ip = input('Enter a single ip address or range to allow (ex: 10.10.10.10 or 10.10.0.0/16): ')
-salt-call state.apply firewall queue=True
+  if validate_ip_cidr(ip):
+    return ip
+  else:
+    print(f'Invalid IP address or CIDR block \'{ip}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+
+
+def wazuh_enabled() -> bool:
+  for file in os.listdir(f'{LOCAL_SALT_DIR}/pillar'):
+    with open(file, 'r') as pillar:
+      if 'wazuh: 1' in pillar.read():
+        return True
+  return False
+
+
+def root_to_str(root: ET.ElementTree) -> str:
+    xml_str = ET.tostring(root, encoding='unicode', method='xml').replace('\n', '')
+    xml_str = re.sub(r'(?:(?<=>) *)', '', xml_str)
+    xml_str = re.sub(r'       -', '', xml_str)
+    xml_str = re.sub(r'      -->', ' -->', xml_str)
+    dom = minidom.parseString(xml_str)
+    return dom.toprettyxml(indent="  ")
+
+
+def add_wl(ip):
+    parser = ET.XMLParser(remove_blank_text=True)
+    with open(WAZUH_CONF, 'rb') as wazuh_conf:
+        tree = ET.parse(wazuh_conf, parser)
+    root = tree.getroot()
+
+    source_comment = ET.Comment(f'Address {ip} added by /usr/sbin/so-allow on {dt.utcnow().replace(tzinfo=tz.utc).strftime("%a %b %e %H:%M:%S %Z %Y")}')
+    new_global = ET.Element("global")
+    new_wl = ET.SubElement(new_global, 'white_list')
+    new_wl.text = ip
+
+    root.append(source_comment)
+    root.append(new_global)
+
+    with open(WAZUH_CONF, 'w') as add_out:
+        add_out.write(root_to_str(root))
+
+
+def apply(role: str, ip: str) -> int:
+  firewall_cmd = ['so-firewall', 'includehost', role, ip]
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'firewall', 'queue=True']
+  restart_wazuh_cmd = ['so-wazuh-restart']
+  print(f'Adding {ip} to the {role} role. This can take a few seconds...')
+  cmd = subprocess.run(firewall_cmd)
+  if cmd.returncode == 0:
+    cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  else:
+    return cmd.returncode
+  if cmd.returncode == 0:
+    if wazuh_enabled and role=='analyst':
+      try:
+        add_wl(ip)
+        print(f'Added whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+      except Exception as e:
+        print(f'Failed to add whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+        print(e)
+        return 1
+      print('Restarting OSSEC Server...')
+      cmd = subprocess.run(restart_wazuh_cmd)
+    else:
+      return cmd.returncode
+  else:
+    print(f'Commmand \'{" ".join(salt_cmd)}\' failed.', file=sys.stderr)
+    return cmd.returncode
+  if cmd.returncode != 0:
+    print('Failed to restart OSSEC server.')
+  return cmd.returncode
+
+
+def main():
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=textwrap.dedent(f'''\
+        additional information:
+                      To use this script in interactive mode call it with no arguments
+        '''
+  ))
+
+  group = main_parser.add_argument_group(title='roles')
+  group.add_argument('-a', dest='roles', action='append_const', const=VALID_ROLES['a']['role'], help="Analyst - 80/tcp, 443/tcp")
+  group.add_argument('-b', dest='roles', action='append_const', const=VALID_ROLES['b']['role'], help="Logstash Beat - 5044/tcp")
+  group.add_argument('-e', dest='roles', action='append_const', const=VALID_ROLES['e']['role'], help="Elasticsearch REST API - 9200/tcp")
+  group.add_argument('-f', dest='roles', action='append_const', const=VALID_ROLES['f']['role'], help="Strelka frontend - 57314/tcp")
+  group.add_argument('-o', dest='roles', action='append_const', const=VALID_ROLES['o']['role'], help="Osquery endpoint - 8090/tcp")
+  group.add_argument('-s', dest='roles', action='append_const', const=VALID_ROLES['s']['role'], help="Syslog device - 514/tcp/udp")
+  group.add_argument('-w', dest='roles', action='append_const', const=VALID_ROLES['w']['role'], help="Wazuh agent - 1514/tcp/udp")
+  group.add_argument('-p', dest='roles', action='append_const', const=VALID_ROLES['p']['role'], help="Wazuh API - 55000/tcp")
+  group.add_argument('-r', dest='roles', action='append_const', const=VALID_ROLES['r']['role'], help="Wazuh registration service - 1515/tcp")
+  
+  ip_g = main_parser.add_argument_group(title='allow')
+  ip_g.add_argument('-i', help="IP or CIDR block to disallow connections from, requires at least one role argument", metavar='', dest='ip')
+
+  args = main_parser.parse_args(sys.argv[1:])
+
+  if args.roles is None:
+    role = role_prompt()
+    ip = ip_prompt()
+    try:
+      return_code = apply(role, ip)
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+    sys.exit(return_code)
+  elif args.roles is not None and args.ip is None:
+    if os.environ.get('IP') is None:
+      main_parser.print_help()
+      sys.exit(1)
+    else:
+      args.ip = os.environ['IP']
+  
+  if validate_ip_cidr(args.ip):
+    try:
+      for role in args.roles:
+        return_code = apply(role, args.ip)
+        if return_code > 0:
+          break
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+  else:
+    print(f'Invalid IP address or CIDR block \'{args.ip}\', please try again.', file=sys.stderr)
+    return_code = 1
+    
+  sys.exit(return_code)
+
+
+if __name__ == '__main__':
+  try:
+    main()
+  except KeyboardInterrupt:
+    sys.exit(1)
 
-# Check if Wazuh enabled
-if grep -q -R "wazuh: 1" $local_salt_dir/pillar/*; then
-    # If analyst, add to Wazuh AR whitelist
-    if [ "$FULLROLE" == "analyst" ]; then
-        WAZUH_MGR_CFG="/nsm/wazuh/etc/ossec.conf"
-        if ! grep -q "<white_list>$IP</white_list>" $WAZUH_MGR_CFG ; then
-            DATE=$(date)
-            sed -i 's/<\/ossec_config>//' $WAZUH_MGR_CFG
-            sed -i '/^$/N;/^\n$/D' $WAZUH_MGR_CFG
-            echo -e "<!--Address $IP added by /usr/sbin/so-allow on \"$DATE\"-->\n  <global>\n    <white_list>$IP</white_list>\n  </global>\n</ossec_config>" >> $WAZUH_MGR_CFG
-            echo "Added whitelist entry for $IP in $WAZUH_MGR_CFG."
-        echo
-            echo "Restarting OSSEC Server..."
-            /usr/sbin/so-wazuh-restart
-        fi
-    fi
-fi

salt/common/tools/sbin/so-checkin

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-salt-call state.highstate
+salt-call state.highstate -l info 

salt/common/tools/sbin/so-common

@@ -88,19 +88,6 @@ add_interface_bond0() {
 	fi
 }
 
-check_airgap() {
-  # See if this is an airgap install
-  AIRGAP=$(cat /opt/so/saltstack/local/pillar/global.sls | grep airgap: | awk '{print $2}')
-  if [[ "$AIRGAP" == "True" ]]; then
-      is_airgap=0
-      UPDATE_DIR=/tmp/soagupdate/SecurityOnion
-      AGDOCKER=/tmp/soagupdate/docker
-      AGREPO=/tmp/soagupdate/Packages
-  else 
-      is_airgap=1
-  fi
-}
-
 check_container() {
 	docker ps | grep "$1:" > /dev/null 2>&1
 	return $?
@@ -112,6 +99,15 @@ check_password() {
 	return $?
 }
 
+check_password_and_exit() {
+	local password=$1
+	if ! check_password "$password"; then
+		echo "Password is invalid. Do not include single quotes, double quotes, dollar signs, and backslashes in the password."
+		exit 2
+	fi
+	return 0
+}
+
 check_elastic_license() {
 
 	[ -n "$TESTING" ] && return
@@ -153,7 +149,7 @@ Do you agree to the terms of the Elastic License?
 If so, type AGREE to accept the Elastic License and continue.  Otherwise, press Enter to exit this program without making any changes.
 EOM
 
-AGREED=$(whiptail --title "Security Onion Setup" --inputbox \
+	AGREED=$(whiptail --title "$whiptail_title" --inputbox \
 	"$message" 20 75 3>&1 1>&2 2>&3)
 
 	if [ "${AGREED^^}" = 'AGREE' ]; then
@@ -252,6 +248,7 @@ lookup_salt_value() {
 	key=$1
 	group=$2
 	kind=$3
+	output=${4:-newline_values_only}
 
 	if [ -z "$kind" ]; then
 		kind=pillar
@@ -261,7 +258,7 @@ lookup_salt_value() {
 		group=${group}:
 	fi
 
-	salt-call --no-color  ${kind}.get ${group}${key} --out=newline_values_only
+	salt-call --no-color  ${kind}.get ${group}${key} --out=${output}
 }
 
 lookup_pillar() {
@@ -289,7 +286,7 @@ lookup_role() {
 
 require_manager() {
 	if is_manager_node; then
-		echo "This is a manager, We can proceed."
+		echo "This is a manager, so we can proceed."
 	else
 		echo "Please run this command on the manager; the manager controls the grid."
 		exit 1
@@ -302,6 +299,7 @@ retry() {
 	cmd=$3
 	expectedOutput=$4
 	attempt=0
+	local exitcode=0
 	while [[ $attempt -lt $maxAttempts ]]; do			
 		attempt=$((attempt+1))
 		echo "Executing command with retry support: $cmd"
@@ -321,7 +319,29 @@ retry() {
 		sleep $sleepDelay
 	done
 	echo "Command continues to fail; giving up."
-	return 1
+	return $exitcode
+}
+
+run_check_net_err() {
+	local cmd=$1
+	local err_msg=${2:-"Unknown error occured, please check /root/$WHATWOULDYOUSAYYAHDOHERE.log for details."} # Really need to rename that variable
+	local no_retry=$3
+
+	local exit_code
+	if [[ -z $no_retry ]]; then
+		retry 5 60 "$cmd"
+		exit_code=$?
+	else
+		eval "$cmd"
+		exit_code=$?
+	fi
+	
+	if [[ $exit_code -ne 0 ]]; then
+		ERR_HANDLED=true
+		[[ -z $no_retry ]] || echo "Command failed with error $exit_code"
+		echo "$err_msg"
+		exit $exit_code
+	fi
 }
 
 set_os() {
@@ -361,18 +381,29 @@ set_version() {
 	fi
 }
 
+has_uppercase() {
+	local string=$1
+
+	echo "$string" | grep -qP '[A-Z]' \
+		&& return 0 \
+		|| return 1
+}
+
 valid_cidr() {
 	# Verify there is a backslash in the string
 	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
 
-	local cidr
+	valid_ip4_cidr_mask "$1" && return 0 || return 1
-	local ip
 	
-	cidr=$(echo "$1" | sed 's/.*\///')
+	local cidr="$1"
-	ip=$(echo "$1" | sed 's/\/.*//' )
+	local ip
+	ip=$(echo "$cidr" | sed 's/\/.*//' )
 	
 	if valid_ip4 "$ip"; then
-		[[ $cidr =~ ([0-9]|[1-2][0-9]|3[0-2]) ]] && return 0 || return 1
+		local ip1 ip2 ip3 ip4 N
+		IFS="./" read -r ip1 ip2 ip3 ip4 N <<< "$cidr"
+		ip_total=$((ip1 * 256 ** 3 + ip2 * 256 ** 2 + ip3 * 256 + ip4))
+		[[ $((ip_total % 2**(32-N))) == 0 ]] && return 0 || return 1
 	else
 		return 1
 	fi
@@ -422,6 +453,23 @@ valid_ip4() {
 	echo "$ip" | grep -qP '^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$' && return 0 || return 1
 }
 
+valid_ip4_cidr_mask() {
+	# Verify there is a backslash in the string
+	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
+
+	local cidr
+	local ip
+
+	cidr=$(echo "$1" | sed 's/.*\///')
+	ip=$(echo "$1" | sed 's/\/.*//' )
+
+	if valid_ip4 "$ip"; then
+		[[ $cidr =~ ^([0-9]|[1-2][0-9]|3[0-2])$ ]] && return 0 || return 1
+	else
+		return 1
+	fi
+}
+
 valid_int() {
 	local num=$1
 	local min=${2:-1}
@@ -486,13 +534,14 @@ wait_for_web_response() {
 	url=$1
 	expected=$2
 	maxAttempts=${3:-300}
+	curlcmd=${4:-curl}
 	logfile=/root/wait_for_web_response.log
 	truncate -s 0 "$logfile"
 	attempt=0
 	while [[ $attempt -lt $maxAttempts ]]; do
 		attempt=$((attempt+1))
 		echo "Waiting for value '$expected' at '$url' ($attempt/$maxAttempts)"
-		result=$(curl -ks -L $url)
+		result=$($curlcmd -ks -L $url)
 		exitcode=$?
 
 		echo "--------------------------------------------------" >> $logfile

salt/common/tools/sbin/so-config-backup

@@ -35,6 +35,7 @@ if [ ! -f $BACKUPFILE ]; then
   {%- endfor %}
   tar -rf $BACKUPFILE /etc/pki
   tar -rf $BACKUPFILE /etc/salt
+  tar -rf $BACKUPFILE /opt/so/conf/kratos
 
 fi
 

salt/common/tools/sbin/so-deny

@@ -0,0 +1,213 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import ipaddress
+import textwrap
+import os
+import subprocess
+import sys
+import argparse
+import re
+from lxml import etree as ET
+from xml.dom import minidom
+
+
+LOCAL_SALT_DIR='/opt/so/saltstack/local'
+WAZUH_CONF='/nsm/wazuh/etc/ossec.conf'
+VALID_ROLES = {
+  'a': { 'role': 'analyst','desc': 'Analyst - 80/tcp, 443/tcp' },
+  'b': { 'role': 'beats_endpoint', 'desc': 'Logstash Beat - 5044/tcp' },
+  'e': { 'role': 'elasticsearch_rest', 'desc': 'Elasticsearch REST API - 9200/tcp' },
+  'f': { 'role': 'strelka_frontend', 'desc': 'Strelka frontend - 57314/tcp' },
+  'o': { 'role': 'osquery_endpoint', 'desc': 'Osquery endpoint - 8090/tcp' },
+  's': { 'role': 'syslog', 'desc': 'Syslog device - 514/tcp/udp' },
+  'w': { 'role': 'wazuh_agent', 'desc': 'Wazuh agent - 1514/tcp/udp' },
+  'p': { 'role': 'wazuh_api', 'desc': 'Wazuh API - 55000/tcp' },
+  'r': { 'role': 'wazuh_authd', 'desc': 'Wazuh registration service - 1515/tcp' }
+}
+
+
+def validate_ip_cidr(ip_cidr: str) -> bool:
+  try:
+    ipaddress.ip_address(ip_cidr)
+  except ValueError:
+    try:
+      ipaddress.ip_network(ip_cidr)
+    except ValueError:
+      return False
+  return True
+
+
+def role_prompt() -> str:
+  print()
+  print('Choose the role for the IP or Range you would like to deny')
+  print()
+  for role in VALID_ROLES:
+    print(f'[{role}] - {VALID_ROLES[role]["desc"]}')
+  print()
+  role = input('Please enter your selection: ')
+  if role in VALID_ROLES.keys():
+    return VALID_ROLES[role]['role']
+  else:
+    print(f'Invalid role \'{role}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+  
+
+def ip_prompt() -> str:
+  ip = input('Enter a single ip address or range to deny (ex: 10.10.10.10 or 10.10.0.0/16): ')
+  if validate_ip_cidr(ip):
+    return ip
+  else:
+    print(f'Invalid IP address or CIDR block \'{ip}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+
+
+def wazuh_enabled() -> bool:
+  for file in os.listdir(f'{LOCAL_SALT_DIR}/pillar'):
+    with open(file, 'r') as pillar:
+      if 'wazuh: 1' in pillar.read():
+        return True
+  return False
+
+
+def root_to_str(root: ET.ElementTree) -> str:
+    xml_str = ET.tostring(root, encoding='unicode', method='xml').replace('\n', '')
+    xml_str = re.sub(r'(?:(?<=>) *)', '', xml_str)
+
+    # Remove specific substrings to better format comments on intial parse/write
+    xml_str = re.sub(r'       -', '', xml_str)
+    xml_str = re.sub(r'      -->', ' -->', xml_str)
+    
+    dom = minidom.parseString(xml_str)
+    return dom.toprettyxml(indent="  ")
+
+
+def rem_wl(ip):
+    parser = ET.XMLParser(remove_blank_text=True)
+    with open(WAZUH_CONF, 'rb') as wazuh_conf:
+        tree = ET.parse(wazuh_conf, parser)
+    root = tree.getroot()
+
+    global_elems = root.findall(f"global/white_list[. = '{ip}']/..")
+    if len(global_elems) > 0:
+      for g_elem in global_elems:
+          ge_index = list(root).index(g_elem)
+          if ge_index > 0 and root[list(root).index(g_elem) - 1].tag == ET.Comment:
+              root.remove(root[ge_index - 1])
+          root.remove(g_elem)
+
+      with open(WAZUH_CONF, 'w') as out:
+          out.write(root_to_str(root))
+
+
+def apply(role: str, ip: str) -> int:
+  firewall_cmd = ['so-firewall', 'excludehost', role, ip]
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'firewall', 'queue=True']
+  restart_wazuh_cmd = ['so-wazuh-restart']
+  print(f'Removing {ip} from the {role} role. This can take a few seconds...')
+  cmd = subprocess.run(firewall_cmd)
+  if cmd.returncode == 0:
+    cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  else:
+    return cmd.returncode
+  if cmd.returncode == 0:
+    if wazuh_enabled and role=='analyst':
+      try:
+        rem_wl(ip)
+        print(f'Removed whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+      except Exception as e:
+        print(f'Failed to remove whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+        print(e)
+        return 1
+      print('Restarting OSSEC Server...')
+      cmd = subprocess.run(restart_wazuh_cmd)
+    else:
+      return cmd.returncode
+  else:
+    print(f'Commmand \'{" ".join(salt_cmd)}\' failed.', file=sys.stderr)
+    return cmd.returncode
+  if cmd.returncode != 0:
+    print('Failed to restart OSSEC server.')
+  return cmd.returncode
+
+
+def main():
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=textwrap.dedent(f'''\
+        additional information:
+                      To use this script in interactive mode call it with no arguments
+        '''
+  ))
+
+  group = main_parser.add_argument_group(title='roles')
+  group.add_argument('-a', dest='roles', action='append_const', const=VALID_ROLES['a']['role'], help="Analyst - 80/tcp, 443/tcp")
+  group.add_argument('-b', dest='roles', action='append_const', const=VALID_ROLES['b']['role'], help="Logstash Beat - 5044/tcp")
+  group.add_argument('-e', dest='roles', action='append_const', const=VALID_ROLES['e']['role'], help="Elasticsearch REST API - 9200/tcp")
+  group.add_argument('-f', dest='roles', action='append_const', const=VALID_ROLES['f']['role'], help="Strelka frontend - 57314/tcp")
+  group.add_argument('-o', dest='roles', action='append_const', const=VALID_ROLES['o']['role'], help="Osquery endpoint - 8090/tcp")
+  group.add_argument('-s', dest='roles', action='append_const', const=VALID_ROLES['s']['role'], help="Syslog device - 514/tcp/udp")
+  group.add_argument('-w', dest='roles', action='append_const', const=VALID_ROLES['w']['role'], help="Wazuh agent - 1514/tcp/udp")
+  group.add_argument('-p', dest='roles', action='append_const', const=VALID_ROLES['p']['role'], help="Wazuh API - 55000/tcp")
+  group.add_argument('-r', dest='roles', action='append_const', const=VALID_ROLES['r']['role'], help="Wazuh registration service - 1515/tcp")
+  
+  ip_g = main_parser.add_argument_group(title='allow')
+  ip_g.add_argument('-i', help="IP or CIDR block to disallow connections from, requires at least one role argument", metavar='', dest='ip')
+
+  args = main_parser.parse_args(sys.argv[1:])
+
+  if args.roles is None:
+    role = role_prompt()
+    ip = ip_prompt()
+    try:
+      return_code = apply(role, ip)
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+    sys.exit(return_code)
+  elif args.roles is not None and args.ip is None:
+    if os.environ.get('IP') is None:
+      main_parser.print_help()
+      sys.exit(1)
+    else:
+      args.ip = os.environ['IP']
+  
+  if validate_ip_cidr(args.ip):
+    try:
+      for role in args.roles:
+        return_code = apply(role, args.ip)
+        if return_code > 0:
+          break
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+  else:
+    print(f'Invalid IP address or CIDR block \'{args.ip}\', please try again.', file=sys.stderr)
+    return_code = 1
+    
+  sys.exit(return_code)
+
+
+if __name__ == '__main__':
+  try:
+    main()
+  except KeyboardInterrupt:
+    sys.exit(1)

salt/common/tools/sbin/so-docker-prune

@@ -32,19 +32,25 @@ def get_image_version(string) -> str:
   ver = string.split(':')[-1]
   if ver == 'latest':
     # Version doesn't like "latest", so use a high semver
-    return '999999.9.9'
+    return '99999.9.9'
   else:
     try:
       Version(ver)
     except InvalidVersion:
-      # Strip the last substring following a hyphen for automated branches
+      # Also return a very high semver for any version 
-      ver = '-'.join(ver.split('-')[:-1])  
+      # with a dash in it since it will likely be a dev version of some kind
+      if '-' in ver:
+        return '999999.9.9'
     return ver
 
 
 def main(quiet):
   client = docker.from_env()
 
+  # Prune old/stopped containers
+  if not quiet: print('Pruning old containers')
+  client.containers.prune()
+
   image_list = client.images.list(filters={ 'dangling': False })
 
   # Map list of image objects to flattened list of tags (format: "name:version")
@@ -72,9 +78,16 @@ def main(quiet):
         for group in grouped_t_list[2:]:
           for tag in group:
             if not quiet: print(f'Removing image {tag}')
-            client.images.remove(tag)
+            try:
-    except InvalidVersion as e:
+              client.images.remove(tag, force=True)
-      print(f'so-{get_so_image_basename(t_list[0])}: {e.args[0]}', file=sys.stderr)
+            except docker.errors.ClientError as e:
+              print(f'Could not remove image {tag}, continuing...')
+    except (docker.errors.APIError, InvalidVersion) as e:
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+    except Exception as e:
+      print('Unhandled exception occurred:')
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
       exit(1)
 
   if no_prunable and not quiet:

salt/common/tools/sbin/so-elastalert-create

@@ -145,9 +145,9 @@ EOF
    rulename=$(echo ${raw_rulename,,} | sed 's/ /_/g')
 
 cat << EOF >> "$rulename.yaml"
-# Elasticsearch Host
+# Elasticsearch Host Override (optional)
-es_host: elasticsearch
+# es_host: elasticsearch
-es_port: 9200
+# es_port: 9200
 
 # (Required)
 # Rule name, must be unique

salt/common/tools/sbin/so-elastalert-test

@@ -70,7 +70,7 @@ do
 done
 
 docker_exec(){
-	CMD="docker exec -it so-elastalert elastalert-test-rule /opt/elastalert/rules/$RULE_NAME --config /opt/config/elastalert_config.yaml $OPTIONS"
+	CMD="docker exec -it so-elastalert elastalert-test-rule /opt/elastalert/rules/$RULE_NAME --config /opt/elastalert/config.yaml $OPTIONS"
 	if [ "${RESULTS_TO_LOG,,}" = "y" ] ; then
 		$CMD > "$FILE_SAVE_LOCATION"
 	else

salt/common/tools/sbin/so-elastic-auth

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if [ -f "/usr/sbin/so-common" ]; then
+  . /usr/sbin/so-common
+fi
+
+ES_AUTH_PILLAR=${ELASTIC_AUTH_PILLAR:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+ES_USERS_FILE=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+
+authEnable=$1
+
+if ! grep -q "enabled: " "$ES_AUTH_PILLAR"; then
+  echo "Elastic auth pillar file is invalid. Unable to proceed."
+  exit 1
+fi
+
+function restart() {
+  if [[ -z "$ELASTIC_AUTH_SKIP_HIGHSTATE" ]]; then
+    echo "Elasticsearch on all affected minions will now be stopped and then restarted..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' cmd.run so-elastic-stop queue=True
+    echo "Applying highstate to all affected minions..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.highstate queue=True
+  fi
+}
+
+if [[ "$authEnable" == "true" ]]; then
+  if grep -q "enabled: False" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: False/enabled: True/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now enabled."
+    if grep -q "argon" "$ES_USERS_FILE"; then
+      echo ""
+      echo "IMPORTANT: The following users will need to change their password, after logging into SOC, in order to access Kibana:"
+      grep argon "$ES_USERS_FILE" | cut -d ":" -f 1
+    fi
+  else
+    echo "Auth is already enabled."
+  fi
+elif [[ "$authEnable" == "false" ]]; then
+  if grep -q "enabled: True" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: True/enabled: False/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now disabled."
+  else
+    echo "Auth is already disabled."
+  fi
+else
+  echo "Usage: $0 <true|false>"
+  echo ""
+  echo "Toggles Elastic authentication. Elasticsearch will be restarted on each affected minion."
+  echo ""
+fi

salt/common/tools/sbin/so-elastic-auth-password-reset

@@ -0,0 +1,155 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+source $(dirname $0)/so-common
+require_manager
+
+user=$1
+elasticUsersFile=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+elasticAuthPillarFile=${ELASTIC_AUTH_PILLAR_FILE:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+
+if [[ $# -ne 1 ]]; then
+  echo "Usage: $0 <user>"
+  echo ""
+  echo " where <user> is one of the following:"
+  echo ""
+  echo "         all: Reset the password for the so_elastic, so_kibana, so_logstash, so_beats, and so_monitor users"
+  echo "  so_elastic: Reset the password for the so_elastic user"
+  echo "   so_kibana: Reset the password for the so_kibana user"
+  echo " so_logstash: Reset the password for the so_logstash user"
+  echo "    so_beats: Reset the password for the so_beats user"
+  echo "  so_monitor: Reset the password for the so_monitor user"
+  echo ""
+  exit 1
+fi
+
+# function to create a lock so that the so-user sync cronjob can't run while this is running
+function lock() {
+  # Obtain file descriptor lock
+  exec 99>/var/tmp/so-user.lock || fail "Unable to create lock descriptor; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  flock -w 10 99 || fail "Another process is using so-user; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  trap 'rm -f /var/tmp/so-user.lock' EXIT
+}
+
+function unlock() {
+  rm -f /var/tmp/so-user.lock
+}
+
+function fail() {
+  msg=$1
+  echo "$1"
+  exit 1
+}
+
+function removeSingleUserPass() {
+  local user=$1
+  sed -i '/user: '"${user}"'/{N;/pass: /d}' "${elasticAuthPillarFile}"
+}
+
+function removeAllUserPass() {
+  local userList=("so_elastic" "so_kibana" "so_logstash" "so_beats" "so_monitor")
+
+  for u in ${userList[@]}; do
+    removeSingleUserPass "$u"
+  done
+}
+
+function removeElasticUsersFile() {
+  rm -f "$elasticUsersFile"
+}
+
+function createElasticAuthPillar() {
+  salt-call state.apply elasticsearch.auth queue=True
+}
+
+# this will disable highstate to prevent a highstate from starting while the script is running
+# will also disable salt.minion-state-apply-test allow so-salt-minion-check cronjob to restart salt-minion service incase
+function disableSaltStates() {
+  printf "\nDisabling salt.minion-state-apply-test and highstate from running.\n\n"
+  salt-call state.disable salt.minion-state-apply-test
+  salt-call state.disable highstate
+}
+
+function enableSaltStates() {
+  printf "\nEnabling salt.minion-state-apply-test and highstate.\n\n"
+  salt-call state.enable salt.minion-state-apply-test
+  salt-call state.enable highstate
+}
+
+function killAllSaltJobs() {
+  printf "\nKilling all running salt jobs.\n\n"
+  salt-call saltutil.kill_all_jobs
+}
+
+function soUserSync() {
+  # apply this state to update /opt/so/saltstack/local/salt/elasticsearch/curl.config on the manager
+  salt-call state.sls_id elastic_curl_config_distributed manager queue=True
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' saltutil.kill_all_jobs
+  # apply this state to get the curl.config
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.sls_id elastic_curl_config common queue=True
+  $(dirname $0)/so-user sync
+  printf "\nApplying logstash state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply logstash queue=True
+  printf "\nApplying filebeat state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode or G@role:so-sensor or G@role:so-fleet' state.apply filebeat queue=True
+  printf "\nApplying kibana state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch' state.apply kibana queue=True
+  printf "\nApplying curator state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply curator queue=True
+}
+
+function highstateManager() {
+  killAllSaltJobs
+  printf "\nRunning highstate on the manager to finalize password reset.\n\n"
+  salt-call state.highstate -linfo queue=True
+}
+
+case "${user}" in
+
+  so_elastic | so_kibana | so_logstash | so_beats | so_monitor)
+    lock
+    killAllSaltJobs
+    disableSaltStates
+    removeSingleUserPass "$user"
+    createElasticAuthPillar
+    removeElasticUsersFile
+    unlock
+    soUserSync
+    enableSaltStates
+    highstateManager
+    ;;
+
+  all)
+    lock
+    killAllSaltJobs
+    disableSaltStates
+    removeAllUserPass
+    createElasticAuthPillar
+    removeElasticUsersFile
+    unlock
+    soUserSync
+    enableSaltStates
+    highstateManager
+    ;;
+
+  *)
+    fail "Unsupported user: $user"
+    ;;
+
+esac
+
+exit 0

salt/common/tools/sbin/so-elastic-clear

@@ -50,7 +50,7 @@ done
 if [ $SKIP -ne 1 ]; then
         # List indices
         echo 
-        curl -k -L https://{{ NODEIP }}:9200/_cat/indices?v
+        {{ ELASTICCURL }} -k -L https://{{ NODEIP }}:9200/_cat/indices?v
         echo
         # Inform user we are about to delete all data
         echo
@@ -89,10 +89,10 @@ fi
 # Delete data
 echo "Deleting data..."
 
-INDXS=$(curl -s -XGET -k -L https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
+INDXS=$({{ ELASTICCURL }} -s -XGET -k -L https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
 for INDX in ${INDXS}
 do
-    curl -XDELETE -k -L https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
+    {{ ELASTICCURL }} -XDELETE -k -L https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
 done
 
 #Start Logstash/Filebeat

salt/common/tools/sbin/so-elasticsearch-indices-list

@@ -18,4 +18,4 @@
 
 . /usr/sbin/so-common
 
-curl -s -k -L https://{{ NODEIP }}:9200/_cat/indices?pretty
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/indices?pretty

salt/common/tools/sbin/so-elasticsearch-indices-rw

@@ -21,5 +21,5 @@ THEHIVEESPORT=9400
 
 echo "Removing read only attributes for indices..."
 echo
-curl -s -k -XPUT -H "Content-Type: application/json" -L https://$IP:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
+{{ ELASTICCURL }} -s -k -XPUT -H "Content-Type: application/json" -L https://$IP:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
-curl -XPUT -H "Content-Type: application/json" -L http://$IP:9400/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
+{{ ELASTICCURL }} -XPUT -H "Content-Type: application/json" -L http://$IP:9400/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;

salt/common/tools/sbin/so-elasticsearch-pipeline-stats

@@ -19,7 +19,7 @@
 . /usr/sbin/so-common
 
 if [ "$1" == "" ]; then
-        curl -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
 else
-        curl -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
 fi

salt/common/tools/sbin/so-elasticsearch-pipeline-view

@@ -19,7 +19,7 @@
 . /usr/sbin/so-common
 
 if [ "$1" == "" ]; then
-        curl -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq .
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq .
 else
-        curl -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq .
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq .[]
 fi

salt/common/tools/sbin/so-elasticsearch-pipelines-list

@@ -17,7 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    curl -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
 else
-    curl -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-query

@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+. /usr/sbin/so-common
+
+if [[ $# -lt 1 ]]; then
+	echo "Submit a cURL request to the local Security Onion Elasticsearch host."
+	echo ""
+	echo "Usage: $0 <PATH> [ARGS,...]"
+	echo ""
+  echo "Where "
+  echo "   PATH represents the elastic function being requested."
+  echo "   ARGS is used to specify additional, optional curl parameters."
+  echo ""
+  echo "Examples:"
+  echo "   $0 /"
+  echo "   $0 '*:so-*/_search' -d '{\"query\": {\"match_all\": {}},\"size\": 1}' | jq"
+  exit 1
+fi
+
+QUERYPATH=$1
+shift
+
+{{ ELASTICCURL }} -s -k -L -H "Content-Type: application/json" "https://localhost:9200/${QUERYPATH}" "$@"

salt/common/tools/sbin/so-elasticsearch-roles-load

@@ -0,0 +1,57 @@
+#!/bin/bash
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
+default_conf_dir=/opt/so/conf
+ELASTICSEARCH_HOST="{{ MYIP }}"
+ELASTICSEARCH_PORT=9200
+
+# Define a default directory to load roles from
+ELASTICSEARCH_ROLES="$default_conf_dir/elasticsearch/roles/"
+
+# Wait for ElasticSearch to initialize
+echo -n "Waiting for ElasticSearch..."
+COUNT=0
+ELASTICSEARCH_CONNECTED="no"
+while [[ "$COUNT" -le 240 ]]; do
+    {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+	if [ $? -eq 0 ]; then
+		ELASTICSEARCH_CONNECTED="yes"
+		echo "connected!"
+		break
+	else
+		((COUNT+=1))
+		sleep 1
+		echo -n "."
+	fi
+done
+if [ "$ELASTICSEARCH_CONNECTED" == "no" ]; then
+	echo
+	echo -e "Connection attempt timed out.  Unable to connect to ElasticSearch.  \nPlease try: \n  -checking log(s) in /var/log/elasticsearch/\n  -running 'sudo docker ps' \n  -running 'sudo so-elastic-restart'"
+	echo
+fi
+
+cd ${ELASTICSEARCH_ROLES}
+
+echo "Loading templates..."
+for role in *; do
+	name=$(echo "$role" | cut -d. -f1)
+	so-elasticsearch-query _security/role/$name -XPUT -d @"$role"
+done
+
+cd - >/dev/null

salt/common/tools/sbin/so-elasticsearch-shards-list

@@ -18,4 +18,4 @@
 
 . /usr/sbin/so-common
 
-curl -s -k -L https://{{ NODEIP }}:9200/_cat/shards?pretty
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/shards?pretty

salt/common/tools/sbin/so-elasticsearch-template-remove

@@ -18,4 +18,4 @@
 
 . /usr/sbin/so-common
 
-curl -s -k -L -XDELETE https://{{ NODEIP }}:9200/_template/$1
+{{ ELASTICCURL }} -s -k -L -XDELETE https://{{ NODEIP }}:9200/_template/$1

salt/common/tools/sbin/so-elasticsearch-template-view

@@ -19,7 +19,7 @@
 . /usr/sbin/so-common
 
 if [ "$1" == "" ]; then
-        curl -s -k -L https://{{ NODEIP }}:9200/_template/* | jq .
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/* | jq .
 else
-        curl -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq .
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq .
 fi

salt/common/tools/sbin/so-elasticsearch-templates-list

@@ -17,7 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    curl -s -k -L https://{{ NODEIP }}:9200/_template/* | jq 'keys'
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/* | jq 'keys'
 else
-    curl -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-templates-load

@@ -1,8 +1,5 @@
-{%- set mainint = salt['pillar.get']('host:mainint') %}
-{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
-
 #!/bin/bash
-# Copyright 2014,2015,2016,2017,2018,2019 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,6 +14,9 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
 default_conf_dir=/opt/so/conf
 ELASTICSEARCH_HOST="{{ MYIP }}"
 ELASTICSEARCH_PORT=9200
@@ -30,7 +30,7 @@ echo -n "Waiting for ElasticSearch..."
 COUNT=0
 ELASTICSEARCH_CONNECTED="no"
 while [[ "$COUNT" -le 240 ]]; do
-    curl -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+    {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
 	if [ $? -eq 0 ]; then
 		ELASTICSEARCH_CONNECTED="yes"
 		echo "connected!"
@@ -51,7 +51,7 @@ cd ${ELASTICSEARCH_TEMPLATES}
 
 
 echo "Loading templates..."
-for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; curl -k ${ELASTICSEARCH_AUTH} -s -XPUT -L https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
+for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; {{ ELASTICCURL }} -k ${ELASTICSEARCH_AUTH} -s -XPUT -L https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
 echo
 
 cd - >/dev/null

salt/common/tools/sbin/so-elasticsearch-wait

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+. /usr/sbin/so-common
+
+wait_for_web_response "https://localhost:9200/_cat/indices/.kibana*" "green open" 300 "{{ ELASTICCURL }}"

salt/common/tools/sbin/so-filebeat-module-setup

@@ -0,0 +1,67 @@
+#!/bin/bash
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
+default_conf_dir=/opt/so/conf
+ELASTICSEARCH_HOST="{{ MYIP }}"
+ELASTICSEARCH_PORT=9200
+#ELASTICSEARCH_AUTH=""
+
+# Define a default directory to load pipelines from
+FB_MODULE_YML="/usr/share/filebeat/module-setup.yml"
+
+
+# Wait for ElasticSearch to initialize
+echo -n "Waiting for ElasticSearch..."
+COUNT=0
+ELASTICSEARCH_CONNECTED="no"
+while [[ "$COUNT" -le 240 ]]; do
+        {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+        if [ $? -eq 0 ]; then
+                ELASTICSEARCH_CONNECTED="yes"
+                echo "connected!"
+                break
+        else
+                ((COUNT+=1))
+                sleep 1
+                echo -n "."
+        fi
+done
+if [ "$ELASTICSEARCH_CONNECTED" == "no" ]; then
+        echo
+        echo -e "Connection attempt timed out.  Unable to connect to ElasticSearch.  \nPlease try: \n  -checking log(s) in /var/log/elasticsearch/\n  -running 'sudo docker ps' \n  -running 'sudo so-elastic-restart'"
+        echo
+fi
+echo "Testing to see if the pipelines are already applied"
+ESVER=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT" |jq .version.number |tr -d \")
+PIPELINES=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"/_ingest/pipeline/filebeat-$ESVER-suricata-eve-pipeline | jq . | wc -c)
+
+if [[ "$PIPELINES" -lt 5 ]]; then
+  echo "Setting up ingest pipeline(s)"
+
+  for MODULE in activemq apache auditd aws azure barracuda bluecoat cef checkpoint cisco coredns crowdstrike cyberark cylance elasticsearch envoyproxy f5 fortinet gcp google_workspace googlecloud gsuite haproxy ibmmq icinga iis imperva infoblox iptables juniper kafka kibana logstash microsoft mongodb mssql mysql nats netscout nginx o365 okta osquery panw postgresql rabbitmq radware redis santa snort snyk sonicwall sophos squid suricata system threatintel tomcat traefik zeek zscaler
+  do
+    echo "Loading $MODULE"
+    docker exec -i so-filebeat filebeat setup modules -pipelines -modules $MODULE -c $FB_MODULE_YML
+    sleep 2
+  done
+else
+  exit 0
+fi
+
+

salt/common/tools/sbin/so-firewall

@@ -35,6 +35,7 @@ def showUsage(options, args):
   print('')
   print('  General commands:')
   print('    help           - Prints this usage information.')
+  print('    apply          - Apply the firewall state.')
   print('')
   print('  Host commands:')
   print('    listhostgroups - Lists the known host groups.')
@@ -66,11 +67,11 @@ def checkDefaultPortsOption(options):
 
 def checkApplyOption(options):
   if "--apply" in options:
-    return apply()
+    return apply(None, None)
 
 def loadYaml(filename):
   file = open(filename, "r")
-  return yaml.load(file.read())
+  return yaml.safe_load(file.read())
 
 def writeYaml(filename, content):
   file = open(filename, "w")
@@ -328,7 +329,7 @@ def removehost(options, args):
     code = checkApplyOption(options)
   return code
 
-def apply():
+def apply(options, args):
   proc = subprocess.run(['salt-call', 'state.apply', 'firewall', 'queue=True'])
   return proc.returncode
 
@@ -356,7 +357,8 @@ def main():
     "addport": addport,
     "removeport": removeport,
     "addhostgroup": addhostgroup,
-    "addportgroup": addportgroup
+    "addportgroup": addportgroup,
+    "apply": apply
   }
 
   code=1

salt/common/tools/sbin/so-fleet-setup

@@ -2,11 +2,16 @@
 
 #so-fleet-setup $FleetEmail $FleetPassword 
 
+. /usr/sbin/so-common
+
 if [[ $# -ne 2 ]] ; then
       echo "Username or Password was not set - exiting now."
       exit 1
 fi
 
+USER_EMAIL=$1
+USER_PW=$2
+
 # Checking to see if required containers are started...
 if [ ! "$(docker ps -q -f name=so-fleet)" ]; then
         echo "Starting Docker Containers..."
@@ -17,8 +22,16 @@ fi
 
 docker exec so-fleet fleetctl config set --address https://127.0.0.1:8080 --tls-skip-verify --url-prefix /fleet
 docker exec so-fleet bash -c 'while [[ "$(curl -s -o /dev/null --insecure -w ''%{http_code}'' https://127.0.0.1:8080/fleet)" != "301" ]]; do sleep 5; done'
-docker exec so-fleet fleetctl setup --email $1 --password $2
 
+# Create Security Onion Fleet Service Account + Setup Fleet
+FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
+FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+docker exec so-fleet fleetctl setup --email $FLEET_SA_EMAIL --password $FLEET_SA_PW --name SO_ServiceAccount --org-name SO
+
+# Create User Account
+echo "$USER_PW" | so-fleet-user-add "$USER_EMAIL"
+
+# Import Packs & Configs
 docker exec so-fleet fleetctl apply -f /packs/palantir/Fleet/Endpoints/MacOS/osquery.yaml
 docker exec so-fleet fleetctl apply -f /packs/palantir/Fleet/Endpoints/Windows/osquery.yaml
 docker exec so-fleet fleetctl apply -f /packs/so/so-default.yml

salt/common/tools/sbin/so-fleet-user-add

@@ -18,7 +18,7 @@
 . /usr/sbin/so-common
 
 usage() {
-    echo "Usage: $0 <new-user-name>"
+    echo "Usage: $0 <new-user-email>"
     echo ""
     echo "Adds a new user to Fleet. The new password will be read from STDIN."
     exit 1
@@ -28,37 +28,42 @@ if [ $# -ne 1 ]; then
   usage
 fi
 
-USER=$1
 
-MYSQL_PASS=$(lookup_pillar_secret mysql)
+USER_EMAIL=$1
-FLEET_IP=$(lookup_pillar fleet_ip)
+FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
-FLEET_USER=$USER
+FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+MYSQL_PW=$(lookup_pillar_secret mysql)
 
 # Read password for new user from stdin
 test -t 0
 if [[ $? == 0 ]]; then
   echo "Enter new password:"
 fi
-read -rs FLEET_PASS
+read -rs USER_PASS
 
-if ! check_password "$FLEET_PASS"; then
+check_password_and_exit "$USER_PASS"
-  echo "Password is invalid. Please exclude single quotes, double quotes and backslashes from the password."
+
-  exit 2
+# Config fleetctl & login with the SO Service Account
-fi
+CONFIG_OUTPUT=$(docker exec so-fleet fleetctl config set --address https://127.0.0.1:8080 --tls-skip-verify --url-prefix /fleet  2>&1 )
+SALOGIN_OUTPUT=$(docker exec so-fleet fleetctl login --email $FLEET_SA_EMAIL --password $FLEET_SA_PW 2>&1)
 
-FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_PASS'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
 if [[ $? -ne 0 ]]; then
-	echo "Failed to generate Fleet password hash"
+    echo "Unable to add user to Fleet; Fleet Service account login failed"
+    echo "$SALOGIN_OUTPUT"
     exit 2
 fi
 
-MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+# Create New User
-    "INSERT INTO users (password,salt,username,email,admin,enabled) VALUES ('$FLEET_HASH','','$FLEET_USER','$FLEET_USER',1,1)" 2>&1)
+CREATE_OUTPUT=$(docker exec so-fleet fleetctl user create --email $USER_EMAIL --name $USER_EMAIL --password $USER_PASS --global-role admin 2>&1)
 
 if [[ $? -eq 0 ]]; then
     echo "Successfully added user to Fleet"
 else
     echo "Unable to add user to Fleet; user might already exist"
-    echo "$MYSQL_OUTPUT"
+    echo "$CREATE_OUTPUT"
     exit 2
 fi
+
+# Disable forced password reset
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PW fleet -e \
+"UPDATE users SET admin_forced_password_reset = 0 WHERE email = '$USER_EMAIL'" 2>&1)

salt/common/tools/sbin/so-fleet-user-delete

@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-email>"
+    echo ""
+    echo "Deletes a user in Fleet"
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER_EMAIL=$1
+FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
+FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+
+# Config fleetctl & login with the SO Service Account
+CONFIG_OUTPUT=$(docker exec so-fleet fleetctl config set --address https://127.0.0.1:8080 --tls-skip-verify --url-prefix /fleet  2>&1 )
+SALOGIN_OUTPUT=$(docker exec so-fleet fleetctl login --email $FLEET_SA_EMAIL --password $FLEET_SA_PW 2>&1)
+
+if [[ $? -ne 0 ]]; then
+    echo "Unable to delete user from Fleet; Fleet Service account login failed"
+    echo "$SALOGIN_OUTPUT"
+    exit 2
+fi
+
+# Delete User
+DELETE_OUTPUT=$(docker exec so-fleet fleetctl user delete --email $USER_EMAIL 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully deleted user from Fleet"
+else
+    echo "Unable to delete user from Fleet"
+    echo "$DELETE_OUTPUT"
+    exit 2
+fi
+
+

salt/common/tools/sbin/so-fleet-user-update

@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name>"
+    echo ""
+    echo "Update password for an existing Fleet user. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+MYSQL_PASS=$(lookup_pillar_secret mysql)
+FLEET_IP=$(lookup_pillar fleet_ip)
+FLEET_USER=$USER
+
+# test existence of user
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "SELECT count(1) FROM users WHERE email='$FLEET_USER'" 2>/dev/null | tail -1)
+if [[ $? -ne 0 ]] || [[ $MYSQL_OUTPUT -ne 1 ]] ; then
+    echo "Test for email [${FLEET_USER}] failed"
+    echo "    expect 1 hit in users database, return $MYSQL_OUTPUT hit(s)."
+    echo "Unable to update Fleet user password."
+    exit 2
+fi
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -rs FLEET_PASS
+
+if ! check_password "$FLEET_PASS"; then
+  echo "Password is invalid. Please exclude single quotes, double quotes, dollar signs, and backslashes from the password."
+  exit 2
+fi
+
+FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_PASS'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
+if [[ $? -ne 0 ]]; then
+	echo "Failed to generate Fleet password hash"
+	exit 2
+fi
+
+
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "UPDATE users SET password='$FLEET_HASH', salt='' where email='$FLEET_USER'" 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully updated Fleet user password"
+else
+    echo "Unable to update Fleet user password"
+    echo "$MYSQL_OUTPUT"
+    exit 2
+fi

salt/common/tools/sbin/so-grafana-dashboard-folder-delete

@@ -0,0 +1,17 @@
+# this script is used to delete the default Grafana dashboard folders that existed prior to Grafana dashboard and Salt management changes in 2.3.70
+
+folders=$(curl -X GET http://admin:{{salt['pillar.get']('secrets:grafana_admin')}}@localhost:3000/api/folders | jq -r '.[] | @base64')
+delfolder=("Manager" "Manager Search" "Sensor Nodes" "Search Nodes" "Standalone" "Eval Mode")
+
+for row in $folders; do
+   title=$(echo ${row} | base64 --decode | jq -r '.title')
+   uid=$(echo ${row} | base64 --decode | jq -r '.uid')
+
+  if [[ " ${delfolder[@]} " =~ " ${title} " ]]; then
+   curl -X DELETE http://admin:{{salt['pillar.get']('secrets:grafana_admin')}}@localhost:3000/api/folders/$uid
+  fi
+done
+
+echo "so-grafana-dashboard-folder-delete has been run to delete default Grafana dashboard folders that existed prior to 2.3.70" > /opt/so/state/so-grafana-dashboard-folder-delete-complete
+
+exit 0

salt/common/tools/sbin/so-image-common

@@ -17,7 +17,9 @@
 
 # NOTE: This script depends on so-common
 IMAGEREPO=security-onion-solutions
+STATUS_CONF='/opt/so/conf/so-status/so-status.conf'
 
+# shellcheck disable=SC2120
 container_list() {
   MANAGERCHECK=$1
   
@@ -128,13 +130,18 @@ update_docker_containers() {
   mkdir -p $SIGNPATH >> "$LOG_FILE" 2>&1 
 
   # Let's make sure we have the public key
-  retry 50 10 "curl -sSL https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS -o $SIGNPATH/KEYS" >> "$LOG_FILE" 2>&1
+  run_check_net_err \
+  "curl --retry 5 --retry-delay 60 -sSL https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS -o $SIGNPATH/KEYS" \
+  "Could not pull signature key file, please ensure connectivity to https://raw.gihubusercontent.com" \
+  noretry >> "$LOG_FILE" 2>&1
   result=$?
   if [[ $result -eq 0 ]]; then
     cat $SIGNPATH/KEYS | gpg --import - >> "$LOG_FILE" 2>&1
-  else
+  fi
-    echo "Failed to pull signature key file: $result"
+
-    exit 1
+  # If downloading for soup, check if any optional images need to be pulled
+  if [[ $CURLTYPE == 'soup' ]]; then
+    grep -q "so-logscan" "$STATUS_CONF" && TRUSTED_CONTAINERS+=("so-logscan")
   fi
   
   # Download the containers from the interwebs
@@ -148,14 +155,15 @@ update_docker_containers() {
 
     # Pull down the trusted docker image
     local image=$i:$VERSION$IMAGE_TAG_SUFFIX
-    retry 50 10 "docker pull $CONTAINER_REGISTRY/$IMAGEREPO/$image" >> "$LOG_FILE" 2>&1 
+    run_check_net_err \
+    "docker pull $CONTAINER_REGISTRY/$IMAGEREPO/$image" \
+    "Could not pull $image, please ensure connectivity to $CONTAINER_REGISTRY" >> "$LOG_FILE" 2>&1 
     
     # Get signature
-    retry 50 10 "curl -A '$CURLTYPE/$CURRENTVERSION/$OS/$(uname -r)' https://sigs.securityonion.net/$VERSION/$i:$VERSION$IMAGE_TAG_SUFFIX.sig --output $SIGNPATH/$image.sig" >> "$LOG_FILE" 2>&1
+    run_check_net_err \
-    if [[ $? -ne 0 ]]; then
+    "curl --retry 5 --retry-delay 60 -A '$CURLTYPE/$CURRENTVERSION/$OS/$(uname -r)' https://sigs.securityonion.net/$VERSION/$i:$VERSION$IMAGE_TAG_SUFFIX.sig --output $SIGNPATH/$image.sig" \
-      echo "Unable to pull signature file for $image" >> "$LOG_FILE" 2>&1 
+    "Could not pull signature file for $image, please ensure connectivity to https://sigs.securityonion.net " \
-      exit 1
+    noretry >> "$LOG_FILE" 2>&1
-    fi
     # Dump our hash values
     DOCKERINSPECT=$(docker inspect $CONTAINER_REGISTRY/$IMAGEREPO/$image)
        

salt/common/tools/sbin/so-image-pull

@@ -0,0 +1,58 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+. /usr/sbin/so-image-common
+
+usage() {
+	read -r -d '' message <<- EOM
+		usage: so-image-pull [-h] IMAGE [IMAGE ...]
+
+		positional arguments:
+			IMAGE         One or more 'so-' prefixed images to download and verify.
+
+		optional arguments:
+			-h, --help    Show this help message and exit.
+	EOM
+	echo "$message"
+	exit 1
+}
+
+for arg; do
+	shift
+	[[ "$arg" = "--quiet" || "$arg" = "-q" ]] && quiet=true && continue
+	set -- "$@" "$arg"
+done
+
+if [[ $# -eq 0 || $# -gt 1 ]] || [[ $1 == '-h' || $1 == '--help' ]]; then
+	usage
+fi
+
+TRUSTED_CONTAINERS=("$@")
+set_version
+
+for image in "${TRUSTED_CONTAINERS[@]}"; do
+	if ! docker images | grep "$image" | grep ":5000" | grep -q "$VERSION"; then
+		if [[ $quiet == true ]]; then
+			update_docker_containers "$image" "" "" "/dev/null"
+		else
+			update_docker_containers "$image" "" "" "" 
+		fi
+	else
+		echo "$image:$VERSION image exists."
+	fi
+done

salt/common/tools/sbin/so-import-evtx

@@ -0,0 +1,176 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set MANAGER = salt['grains.get']('master') %}
+{%- set VERSION = salt['pillar.get']('global:soversion') %}
+{%- set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
+{%- set MANAGERIP = salt['pillar.get']('global:managerip') -%}
+{%- set URLBASE = salt['pillar.get']('global:url_base') %}
+{% set ES_USER = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') %}
+{% set ES_PW = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') %}
+
+INDEX_DATE=$(date +'%Y.%m.%d')
+RUNID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1)
+LOG_FILE=/nsm/import/evtx-import.log
+
+. /usr/sbin/so-common
+
+function usage {
+  cat << EOF
+Usage: $0 <evtx-file-1> [evtx-file-2] [evtx-file-*]
+
+Imports one or more evtx files into Security Onion. The evtx files will be analyzed and made available for review in the Security Onion toolset.
+EOF
+}
+
+
+function evtx2es() {
+  EVTX=$1
+  HASH=$2
+  
+  ES_PW=$(lookup_pillar "auth:users:so_elastic_user:pass" "elasticsearch")
+  ES_USER=$(lookup_pillar "auth:users:so_elastic_user:user" "elasticsearch")
+
+  docker run --rm \
+    -v "$EVTX:/tmp/$RUNID.evtx" \
+    --entrypoint evtx2es \
+    {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} \
+    --host {{ MANAGERIP }} --scheme https \
+    --index so-beats-$INDEX_DATE --pipeline import.wel \
+    --login $ES_USER --pwd $ES_PW \
+    "/tmp/$RUNID.evtx" >> $LOG_FILE 2>&1
+
+  docker run --rm \
+    -v "$EVTX:/tmp/import.evtx" \
+    -v "/nsm/import/evtx-end_newest:/tmp/newest" \
+    -v "/nsm/import/evtx-start_oldest:/tmp/oldest" \
+    --entrypoint '/evtx_calc_timestamps.sh' \
+    {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }}
+}
+
+# if no parameters supplied, display usage
+if [ $# -eq 0 ]; then
+  usage
+  exit 1
+fi
+
+# ensure this is a Manager node
+require_manager
+
+# verify that all parameters are files
+for i in "$@"; do
+  if ! [ -f "$i" ]; then
+    usage
+    echo "\"$i\" is not a valid file!"
+    exit 2
+  fi
+done
+
+# track if we have any valid or invalid evtx
+INVALID_EVTXS="no"
+VALID_EVTXS="no"
+
+# track oldest start and newest end so that we can generate the Kibana search hyperlink at the end
+START_OLDEST="2050-12-31"
+END_NEWEST="1971-01-01"
+
+touch /nsm/import/evtx-start_oldest
+touch /nsm/import/evtx-end_newest
+
+echo $START_OLDEST > /nsm/import/evtx-start_oldest
+echo $END_NEWEST > /nsm/import/evtx-end_newest
+
+# paths must be quoted in case they include spaces
+for EVTX in "$@"; do
+  EVTX=$(/usr/bin/realpath "$EVTX")
+  echo "Processing Import: ${EVTX}"
+
+  # generate a unique hash to assist with dedupe checks
+  HASH=$(md5sum "${EVTX}" | awk '{ print $1 }')
+  HASH_DIR=/nsm/import/${HASH}
+  echo "- assigning unique identifier to import: $HASH"
+
+  if [ -d $HASH_DIR ]; then
+    echo "- this EVTX has already been imported; skipping"
+    INVALID_EVTXS="yes"
+  else
+    VALID_EVTXS="yes"
+
+    EVTX_DIR=$HASH_DIR/evtx
+    mkdir -p $EVTX_DIR
+
+    # import evtx and write them to import ingest pipeline
+    echo "- importing logs to Elasticsearch..."
+    evtx2es "${EVTX}" $HASH
+
+    # compare $START to $START_OLDEST
+    START=$(cat /nsm/import/evtx-start_oldest)
+    START_COMPARE=$(date -d $START +%s)
+    START_OLDEST_COMPARE=$(date -d $START_OLDEST +%s)
+    if [ $START_COMPARE -lt $START_OLDEST_COMPARE ]; then
+      START_OLDEST=$START
+    fi  
+
+    # compare $ENDNEXT to $END_NEWEST
+    END=$(cat /nsm/import/evtx-end_newest)
+    ENDNEXT=`date +%Y-%m-%d --date="$END 1 day"`
+    ENDNEXT_COMPARE=$(date -d $ENDNEXT +%s)
+    END_NEWEST_COMPARE=$(date -d $END_NEWEST +%s)
+    if [ $ENDNEXT_COMPARE -gt $END_NEWEST_COMPARE ]; then
+      END_NEWEST=$ENDNEXT
+    fi  
+
+    cp -f "${EVTX}" "${EVTX_DIR}"/data.evtx
+    chmod 644 "${EVTX_DIR}"/data.evtx
+
+  fi # end of valid evtx
+
+  echo
+
+done # end of for-loop processing evtx files
+
+# remove temp files
+echo "Cleaning up:"
+for TEMP_EVTX in ${TEMP_EVTXS[@]}; do
+  echo "- removing temporary evtx $TEMP_EVTX"
+  rm -f $TEMP_EVTX
+done
+
+# output final messages
+if [ "$INVALID_EVTXS" = "yes" ]; then
+  echo
+  echo "Please note!  One or more evtx was invalid!  You can scroll up to see which ones were invalid."
+fi
+
+START_OLDEST_FORMATTED=`date +%Y-%m-%d --date="$START_OLDEST"`
+START_OLDEST_SLASH=$(echo $START_OLDEST_FORMATTED | sed -e 's/-/%2F/g')
+END_NEWEST_SLASH=$(echo $END_NEWEST | sed -e 's/-/%2F/g')
+
+if [ "$VALID_EVTXS" = "yes" ]; then
+cat << EOF
+
+Import complete!
+
+You can use the following hyperlink to view data in the time range of your import.  You can triple-click to quickly highlight the entire hyperlink and you can then copy it into your browser:
+https://{{ URLBASE }}/#/hunt?q=import.id:${RUNID}%20%7C%20groupby%20event.module%20event.dataset&t=${START_OLDEST_SLASH}%2000%3A00%3A00%20AM%20-%20${END_NEWEST_SLASH}%2000%3A00%3A00%20AM&z=UTC
+
+or you can manually set your Time Range to be (in UTC):
+From: $START_OLDEST_FORMATTED    To: $END_NEWEST
+
+Please note that it may take 30 seconds or more for events to appear in Hunt.
+EOF
+fi

salt/common/tools/sbin/so-import-pcap

@@ -132,6 +132,8 @@ for PCAP in "$@"; do
     PCAP_FIXED=`mktemp /tmp/so-import-pcap-XXXXXXXXXX.pcap`
     echo "- attempting to recover corrupted PCAP file"
     pcapfix "${PCAP}" "${PCAP_FIXED}"
+    # Make fixed file world readable since the Suricata docker container will runas a non-root user
+    chmod a+r "${PCAP_FIXED}"
     PCAP="${PCAP_FIXED}"
     TEMP_PCAPS+=(${PCAP_FIXED})
   fi

salt/common/tools/sbin/so-index-list

@@ -15,4 +15,4 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-curl -X GET -k -L "https://localhost:9200/_cat/indices?v&s=index"
+{{ ELASTICCURL }} -X GET -k -L "https://localhost:9200/_cat/indices?v&s=index"

salt/common/tools/sbin/so-influxdb-clean

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+wdurregex="^[0-9]+w$"
+ddurregex="^[0-9]+d$"
+
+echo -e "\nThis script is used to reduce the size of InfluxDB by removing old data and retaining only the duration specified."
+echo "The duration will need to be specified as an integer followed by the duration unit without a space."
+echo -e "\nFor example, to purge all data but retain the past 12 weeks, specify 12w for the duration."
+echo "The duration units are as follows:"
+echo "  w  - week(s)"
+echo "  d  - day(s)"
+
+while true; do
+  echo ""
+  read -p 'Enter the duration of past data that you would like to retain: ' duration
+  duration=$(echo $duration | tr '[:upper:]' '[:lower:]')
+
+  if [[ "$duration" =~ $wdurregex ]] || [[ "$duration" =~ $ddurregex ]]; then
+    break
+  fi
+
+  echo -e "\nInvalid duration."
+done
+
+echo -e "\nInfluxDB will now be cleaned and leave only the past $duration worth of data."
+read -r -p "Are you sure you want to continue? [y/N] " yorn
+if [[ "$yorn" =~ ^([yY][eE][sS]|[yY])$ ]]; then
+  echo -e "\nCleaning InfluxDb and saving only the past $duration. This may could take several minutes depending on how much data needs to be cleaned."
+    if docker exec -t so-influxdb /bin/bash -c "influx -ssl -unsafeSsl -database telegraf -execute \"DELETE FROM /.*/ WHERE \"time\" >= '2020-01-01T00:00:00.0000000Z' AND \"time\" <= now() - $duration\""; then
+      echo -e "\nInfluxDb clean complete."
+    else
+      echo -e "\nSomething went wrong with cleaning InfluxDB. Please verify that the so-influxdb Docker container is running, and check the log at /opt/so/log/influxdb/influxdb.log for any details."
+    fi
+else
+  echo -e "\nExiting as requested."
+fi

salt/common/tools/sbin/so-influxdb-downsample

@@ -0,0 +1,63 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+{%- set role = grains.id.split('_') | last %}
+{%- if role in ['manager', 'managersearch', 'eval', 'standalone'] %}
+  {%- import_yaml 'influxdb/defaults.yaml' as default_settings %}
+  {%- set influxdb = salt['grains.filter_by'](default_settings, default='influxdb', merge=salt['pillar.get']('influxdb', {})) %}
+
+. /usr/sbin/so-common
+
+echo -e "\nThis script is used to reduce the size of InfluxDB by downsampling old data into the so_long_term retention policy."
+
+echo -e "\nInfluxDB will now be downsampled. This could take a few hours depending on how large the database is and hardware resources available."
+read -r -p "Are you sure you want to continue? [y/N] " yorn
+if [[ "$yorn" =~ ^([yY][eE][sS]|[yY])$ ]]; then
+  echo -e "\nDownsampling InfluxDb started at `date`. This may take several hours depending on how much data needs to be downsampled."
+
+  {% for dest_rp in influxdb.downsample.keys() -%}
+    {% for measurement in influxdb.downsample[dest_rp].get('measurements', []) -%}
+
+  day=0
+  startdate=`date`
+  while docker exec -t so-influxdb /bin/bash -c "influx -ssl -unsafeSsl -database telegraf -execute \"SELECT mean(*) INTO \"so_long_term\".\"{{measurement}}\" FROM \"autogen\".\"{{measurement}}\" WHERE \"time\" >= '2020-07-21T00:00:00.0000000Z' + ${day}d AND \"time\" <= '2020-07-21T00:00:00.0000000Z' + $((day+1))d GROUP BY time(5m),*\""; do
+    # why 2020-07-21? 
+    migrationdate=`date -d "2020-07-21 + ${day} days" +"%y-%m-%d"`
+
+    echo "Downsampling of measurement: {{measurement}} from $migrationdate started at $startdate and completed at `date`."
+
+    newdaytomigrate=$(date -d "$migrationdate + 1 days" +"%s")
+    today=$(date +"%s")
+    if [ $newdaytomigrate -ge $today ]; then
+      break
+    else
+      ((day=day+1))
+      startdate=`date`
+      echo -e "\nDownsampling the next day's worth of data for measurement: {{measurement}}."
+    fi
+  done
+
+    {% endfor -%}
+  {% endfor -%}
+
+  echo -e "\nInfluxDb data downsampling complete."
+
+else
+  echo -e "\nExiting as requested."
+fi
+{%- else %}
+echo -e "\nThis script can only be run on a node running InfluxDB."
+{%- endif %}

salt/common/tools/sbin/so-influxdb-drop-autogen

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+echo -e "\nThis script is used to reduce the size of InfluxDB by dropping the autogen retention policy."
+echo "If you want to retain historical data prior to 2.3.60, then this should only be run after you have downsampled your data using so-influxdb-downsample."
+
+echo -e "\nThe autogen retention policy will now be dropped from InfluxDB."
+read -r -p "Are you sure you want to continue? [y/N] " yorn
+if [[ "$yorn" =~ ^([yY][eE][sS]|[yY])$ ]]; then
+  echo -e "\nDropping autogen retention policy."
+    if docker exec -t so-influxdb influx -format json -ssl -unsafeSsl -execute "drop retention policy autogen on telegraf"; then
+      echo -e "\nAutogen retention policy dropped from InfluxDb."
+    else
+      echo -e "\nSomething went wrong dropping then autogen retention policy from InfluxDB. Please verify that the so-influxdb Docker container is running, and check the log at /opt/so/log/influxdb/influxdb.log for any details."
+    fi
+else
+  echo -e "\nExiting as requested."
+fi

salt/common/tools/sbin/so-ip-update

@@ -8,9 +8,9 @@ fi
 
 echo "This tool will update a manager's IP address to the new IP assigned to the management network interface."
 
-echo
+echo ""
 echo "WARNING: This tool is still undergoing testing, use at your own risk!"
-echo
+echo ""
 
 if [ -z "$OLD_IP" ]; then
 	OLD_IP=$(lookup_pillar "managerip")
@@ -39,9 +39,9 @@ fi
 
 echo "About to change old IP $OLD_IP to new IP $NEW_IP."
 
-echo
+echo ""
 read -n 1 -p "Would you like to continue? (y/N) " CONTINUE
-echo
+echo ""
 
 if [ "$CONTINUE" == "y" ]; then
 	for file in $(grep -rlI $OLD_IP /opt/so/saltstack /etc); do
@@ -49,6 +49,11 @@ if [ "$CONTINUE" == "y" ]; then
 		sed -i "s|$OLD_IP|$NEW_IP|g" $file
 	done
 
+	echo "Granting MySQL root user permissions on $NEW_IP"
+	docker exec -i so-mysql mysql --user=root --password=$(lookup_pillar_secret 'mysql') -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'$NEW_IP' IDENTIFIED BY '$(lookup_pillar_secret 'mysql')' WITH GRANT OPTION;" &> /dev/null
+	echo "Removing MySQL root user from $OLD_IP"
+	docker exec -i so-mysql mysql --user=root --password=$(lookup_pillar_secret 'mysql') -e "DROP USER 'root'@'$OLD_IP';" &> /dev/null
+
 	echo "The IP has been changed from $OLD_IP to $NEW_IP."
 
 	echo

salt/common/tools/sbin/so-kibana-config-export

@@ -23,7 +23,9 @@
 KIBANA_HOST={{ MANAGER }}
 KSO_PORT=5601
 OUTFILE="saved_objects.ndjson"
-curl -s -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST -L $KIBANA_HOST:$KSO_PORT/api/saved_objects/_export -d '{ "type": [ "index-pattern", "config", "visualization", "dashboard", "search" ], "excludeExportDetails": false }' > $OUTFILE
+
+SESSIONCOOKIE=$({{ ELASTICCURL }} -c - -X GET http://$KIBANA_HOST:$KSO_PORT/ | grep sid | awk '{print $7}')
+{{ ELASTICCURL }} -b "sid=$SESSIONCOOKIE" -s -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST -L $KIBANA_HOST:$KSO_PORT/api/saved_objects/_export -d '{ "type": [ "index-pattern", "config", "visualization", "dashboard", "search" ], "excludeExportDetails": false }' > $OUTFILE
 
 # Clean up using PLACEHOLDER
 sed -i "s/$KIBANA_HOST/PLACEHOLDER/g" $OUTFILE

salt/common/tools/sbin/so-kibana-savedobjects-defaults

@@ -1,5 +1,5 @@
 #!/bin/bash
-#
+
 # Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
@@ -17,42 +17,14 @@
 
 . /usr/sbin/so-common
 
-usage() {
+echo $banner
-    echo "Usage: $0 <user-name>"
+echo "Running kibana.so_savedobjects_defaults Salt state to restore default saved objects."
-    echo ""
+printf "This could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
-    echo "Enables or disables a user in Fleet"
+echo $banner
-    exit 1
-}
 
-if [ $# -ne 2 ]; then
+    if [ "$1" = "--force" ]; then
-  usage
+        printf "\nForce-stopping all Salt jobs before proceeding\n\n"
+        salt-call saltutil.kill_all_jobs
     fi
 
-USER=$1
+salt-call state.apply kibana.so_savedobjects_defaults -linfo queue=True
-
-MYSQL_PASS=$(lookup_pillar_secret mysql)
-FLEET_IP=$(lookup_pillar fleet_ip)
-FLEET_USER=$USER
-
-case "${2^^}" in
-    FALSE | NO | 0)
-        FLEET_STATUS=0
-        ;;
-    TRUE | YES | 1)
-        FLEET_STATUS=1
-        ;;
-    *)
-        usage
-        ;;
-esac
-
-MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
-    "UPDATE users SET enabled=$FLEET_STATUS WHERE username='$FLEET_USER'" 2>&1)
-
-if [[ $? -eq 0 ]]; then
-    echo "Successfully updated user in Fleet"
-else
-    echo "Failed to update user in Fleet"
-    echo $resp
-    exit 2
-fi

salt/common/tools/sbin/so-kibana-space-defaults

@@ -1,13 +1,17 @@
 . /usr/sbin/so-common
-
+{% set HIGHLANDER = salt['pillar.get']('global:highlander', False) %}
-wait_for_web_response "http://localhost:5601/app/kibana" "Elastic"
+wait_for_web_response "http://localhost:5601/app/kibana" "Elastic" 300 "{{ ELASTICCURL }}"
 ## This hackery will be removed if using Elastic Auth ##
 
 # Let's snag a cookie from Kibana
-THECOOKIE=$(curl -c - -X GET http://localhost:5601/ | grep sid | awk '{print $7}')
+SESSIONCOOKIE=$({{ ELASTICCURL }} -c - -X GET http://localhost:5601/ | grep sid | awk '{print $7}')
 
 # Disable certain Features from showing up in the Kibana UI
 echo
 echo "Setting up default Space:"
-curl -b "sid=$THECOOKIE" -L -X PUT "localhost:5601/api/spaces/space/default" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d' {"id":"default","name":"Default","disabledFeatures":["ml","enterpriseSearch","siem","logs","infrastructure","apm","uptime","monitoring","stackAlerts","actions","fleet"]} ' >> /opt/so/log/kibana/misc.log
+{% if HIGHLANDER %}
+{{ ELASTICCURL }} -b "sid=$SESSIONCOOKIE" -L -X PUT "localhost:5601/api/spaces/space/default" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d' {"id":"default","name":"Default","disabledFeatures":["enterpriseSearch"]} ' >> /opt/so/log/kibana/misc.log
+{% else %}
+{{ ELASTICCURL }} -b "sid=$SESSIONCOOKIE" -L -X PUT "localhost:5601/api/spaces/space/default" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d' {"id":"default","name":"Default","disabledFeatures":["ml","enterpriseSearch","siem","logs","infrastructure","apm","uptime","monitoring","stackAlerts","actions","fleet"]} ' >> /opt/so/log/kibana/misc.log
+{% endif %}
 echo

salt/common/tools/sbin/so-learn

@@ -0,0 +1,303 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from itertools import chain
+from typing import List
+
+import signal
+import sys
+import os
+import re
+import subprocess
+import argparse
+import textwrap
+import yaml
+import multiprocessing
+import docker
+import pty
+
+minion_pillar_dir = '/opt/so/saltstack/local/pillar/minions'
+so_status_conf = '/opt/so/conf/so-status/so-status.conf'
+proc: subprocess.CompletedProcess = None
+
+# Temp store of modules, will likely be broken out into salt
+def get_learn_modules():
+  return {
+    'logscan': { 'cpu_period': get_cpu_period(fraction=0.25), 'enabled': False, 'description': 'Scan log files against pre-trained models to alert on anomalies.' }
+  }
+
+
+def get_cpu_period(fraction: float):
+  multiplier = 10000
+  
+  num_cores = multiprocessing.cpu_count()
+  if num_cores <= 2:
+    fraction = 1.
+  
+  num_used_cores = int(num_cores * fraction)
+  cpu_period = num_used_cores * multiplier
+  return cpu_period
+
+
+def sigint_handler(*_):
+  print('Exiting gracefully on Ctrl-C')
+  if proc is not None: proc.send_signal(signal.SIGINT)
+  sys.exit(1)
+
+
+def find_minion_pillar() -> str:
+  regex = '^.*_(manager|managersearch|standalone|import|eval)\.sls$'
+  
+  result = []
+  for root, _, files in os.walk(minion_pillar_dir):
+    for f_minion_id in files:
+      if re.search(regex, f_minion_id):
+        result.append(os.path.join(root, f_minion_id))
+  
+  if len(result) == 0:
+    print('Could not find manager-type pillar (eval, standalone, manager, managersearch, import). Are you running this script on the manager?', file=sys.stderr)
+    sys.exit(3)
+  elif len(result) > 1:
+    res_str = ', '.join(f'\"{result}\"')
+    print('(This should not happen, the system is in an error state if you see this message.)\n', file=sys.stderr)
+    print('More than one manager-type pillar exists, minion id\'s listed below:', file=sys.stderr)
+    print(f'  {res_str}', file=sys.stderr)
+    sys.exit(3)
+  else:
+    return result[0]
+
+
+def read_pillar(pillar: str):
+  try:
+    with open(pillar, 'r') as pillar_file:
+      loaded_yaml = yaml.safe_load(pillar_file.read())
+      if loaded_yaml is None:
+        print(f'Could not parse {pillar}', file=sys.stderr)
+        sys.exit(3)
+      return loaded_yaml
+  except:
+    print(f'Could not open {pillar}', file=sys.stderr)
+    sys.exit(3)
+
+
+def write_pillar(pillar: str, content: dict):
+  try:
+    with open(pillar, 'w') as pillar_file:
+      yaml.dump(content, pillar_file, default_flow_style=False)
+  except:
+    print(f'Could not open {pillar}', file=sys.stderr)
+    sys.exit(3)
+
+
+def mod_so_status(action: str, item: str):
+  with open(so_status_conf, 'a+') as conf:
+    conf.seek(0)
+    containers = conf.readlines()
+    
+    if f'so-{item}\n' in containers:
+      if action == 'remove': containers.remove(f'so-{item}\n')
+      if action == 'add': pass
+    else:
+      if action == 'remove': pass
+      if action == 'add': containers.append(f'so-{item}\n')
+      
+    [containers.remove(c_name) for c_name in containers if c_name == '\n'] # remove extra newlines
+    
+    conf.seek(0)
+    conf.truncate(0)
+    conf.writelines(containers)
+
+
+def create_pillar_if_not_exist(pillar:str, content: dict):
+  pillar_dict = content
+
+  if pillar_dict.get('learn', {}).get('modules') is None:
+    pillar_dict['learn'] = {}
+    pillar_dict['learn']['modules'] = get_learn_modules()
+    content.update()
+    write_pillar(pillar, content)
+  
+  return content
+
+
+def salt_call(module: str):
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', f'learn.{module}', 'queue=True']
+
+  print(f'  Applying salt state for {module} module...')
+  proc = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+  return_code = proc.returncode
+  if return_code != 0:
+    print(f'  [ERROR] Failed to apply salt state for {module} module.')
+
+  return return_code
+
+
+def pull_image(module: str):
+  container_basename = f'so-{module}'
+
+  client = docker.from_env()
+  image_list = client.images.list(filters={ 'dangling': False })  
+  tag_list = list(chain.from_iterable(list(map(lambda x: x.attrs.get('RepoTags'), image_list))))
+  basename_match = list(filter(lambda x: f'{container_basename}' in x, tag_list))
+  local_registry_match = list(filter(lambda x: ':5000' in x, basename_match))
+
+  if len(local_registry_match) == 0:
+    print(f'Pulling and verifying missing image for {module} (may take several minutes) ...')
+    pull_command = ['so-image-pull', '--quiet', container_basename]
+  
+    proc = subprocess.run(pull_command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+    return_code = proc.returncode
+    if return_code != 0:
+      print(f'[ERROR] Failed to pull image so-{module}, skipping state.')
+  else:
+    return_code = 0
+  return return_code
+
+
+def apply(module_list: List):
+  return_code = 0
+  for module in module_list:
+    salt_ret = salt_call(module)
+    # Only update return_code if the command returned a non-zero return
+    if salt_ret != 0:
+      return_code = salt_ret
+    
+  return return_code
+
+
+def check_apply(args: dict):
+  if args.apply:
+    print('Configuration updated. Applying changes:')
+    return apply(args.modules)
+  else:
+    message = 'Configuration updated. Would you like to apply your changes now? (y/N) '
+    answer = input(message)
+    while answer.lower() not in [ 'y', 'n', '' ]:
+      answer = input(message)
+    if answer.lower() in [ 'n', '' ]:
+      return 0
+    else:
+      print('Applying changes:')
+      return apply(args.modules)
+
+
+def enable_disable_modules(args, enable: bool):
+  pillar_modules = args.pillar_dict.get('learn', {}).get('modules')
+  pillar_mod_names = args.pillar_dict.get('learn', {}).get('modules').keys()
+  
+  action_str = 'add' if enable else 'remove'
+
+  if 'all' in args.modules:
+    for module, details in pillar_modules.items():
+      details['enabled'] = enable
+      mod_so_status(action_str, module)
+      if enable: pull_image(module)
+    args.pillar_dict.update()
+    write_pillar(args.pillar, args.pillar_dict)
+  else:
+    write_needed = False
+    for module in args.modules:
+      if module in pillar_mod_names:
+        if pillar_modules[module]['enabled'] == enable:
+          state_str = 'enabled' if enable else 'disabled'
+          print(f'{module} module already {state_str}.', file=sys.stderr)
+        else:
+          if enable and pull_image(module) != 0:
+            continue
+          pillar_modules[module]['enabled'] = enable
+          mod_so_status(action_str, module)
+          write_needed = True
+    if write_needed:
+      args.pillar_dict.update()
+      write_pillar(args.pillar, args.pillar_dict)
+  
+  cmd_ret = check_apply(args)
+  return cmd_ret
+
+
+def enable_modules(args):
+  enable_disable_modules(args, enable=True)
+
+
+def disable_modules(args):
+  enable_disable_modules(args, enable=False)
+
+
+def list_modules(*_):
+  print('Available ML modules:')
+  for module, details in get_learn_modules().items():
+    print(f'  - { module } : {details["description"]}')
+  return 0
+
+
+def main():
+  beta_str = 'BETA - SUBJECT TO CHANGE\n'
+
+  apply_help='After ACTION the chosen modules, apply any necessary salt states.'
+  enable_apply_help = apply_help.replace('ACTION', 'enabling')
+  disable_apply_help = apply_help.replace('ACTION', 'disabling')
+
+  signal.signal(signal.SIGINT, sigint_handler)
+
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
+
+  subcommand_desc = textwrap.dedent(
+    """\
+      enable              Enable one or more ML modules.
+      disable             Disable one or more ML modules.
+      list                List all available ML modules.
+    """
+  )
+
+  subparsers = main_parser.add_subparsers(title='commands', description=subcommand_desc, metavar='', dest='command')
+
+  module_help_str = 'One or more ML modules, which can be listed using \'so-learn list\'. Use the keyword \'all\' to apply the action to all available modules.'
+
+  enable = subparsers.add_parser('enable')
+  enable.set_defaults(func=enable_modules)
+  enable.add_argument('modules', metavar='ML_MODULE', nargs='+', help=module_help_str)
+  enable.add_argument('--apply', action='store_const', const=True, required=False, help=enable_apply_help)
+
+  disable = subparsers.add_parser('disable')
+  disable.set_defaults(func=disable_modules)
+  disable.add_argument('modules', metavar='ML_MODULE', nargs='+', help=module_help_str)
+  disable.add_argument('--apply', action='store_const', const=True, required=False, help=disable_apply_help)
+
+  list = subparsers.add_parser('list')
+  list.set_defaults(func=list_modules)
+
+  args = main_parser.parse_args(sys.argv[1:])
+  args.pillar = find_minion_pillar()
+  args.pillar_dict = create_pillar_if_not_exist(args.pillar, read_pillar(args.pillar))
+
+  if hasattr(args, 'func'):
+    exit_code = args.func(args)
+  else:
+    if args.command is None:
+      print(beta_str)
+      main_parser.print_help()
+    sys.exit(0)
+
+  sys.exit(exit_code)
+
+
+if __name__ == '__main__':
+  main()

salt/common/tools/sbin/so-pcap-export

@@ -1,5 +1,5 @@
 #!/bin/bash
-
+#
 # Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
@@ -15,19 +15,12 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-# Get the latest code
+if [ $# -lt 2 ]; then
-rm -rf /tmp/sohotfix
+  echo "Usage: $0 <steno-query> Output-Filename"
-mkdir -p /tmp/sohotfix
-cd /tmp/sohotfix
-git clone https://github.com/Security-Onion-Solutions/securityonion
-if [ ! -d "/tmp/sohotfix/securityonion" ]; then
-  echo "I was unable to get the latest code. Check your internet and try again."
   exit 1
-else
-  echo "Looks like we have the code lets create the tarball."
-  cd /tmp/sohotfix/securityonion
-  tar cvf /tmp/sohotfix/sohotfix.tar HOTFIX VERSION salt pillar
-  echo ""
-  echo "Copy /tmp/sohotfix/sohotfix.tar to portable media and then copy it to your airgap manager."
-  exit 0
 fi
+
+docker exec -it so-sensoroni scripts/stenoquery.sh "$1" -w /nsm/pcapout/$2.pcap
+
+echo ""
+echo "If successful, the output was written to: /nsm/pcapout/$2.pcap"

salt/common/tools/sbin/so-playbook-import

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+ENABLEPLAY=${1:-False}
+
+docker exec so-soctopus /usr/local/bin/python -c "import playbook; print(playbook.play_import($ENABLEPLAY))"

salt/common/tools/sbin/so-playbook-reset

@@ -22,5 +22,5 @@ salt-call state.apply playbook.db_init,playbook,playbook.automation_user_create
 /usr/sbin/so-soctopus-restart
 
 echo "Importing Plays - this will take some time...."
-wait 5
+sleep 5
 /usr/sbin/so-playbook-ruleupdate

salt/common/tools/sbin/so-raid-status

@@ -17,67 +17,101 @@
 
 . /usr/sbin/so-common
 
-#check_boss_raid() {
+appliance_check() {
-#    BOSSBIN=/opt/boss/mvcli
+  {%- if salt['grains.get']('sosmodel', '') %}
-#    BOSSRC=$($BOSSBIN info -o vd | grep functional)
+  APPLIANCE=1
-#
+  {%- if grains['sosmodel'] in ['SO2AMI01', 'SO2GCI01', 'SO2AZI01'] %}
-#    if [[ $BOSSRC ]]; then
+  exit 0
-#        # Raid is good
+  {%- endif %}
-#        BOSSRAID=0
+  DUDEYOUGOTADELL=$(dmidecode |grep Dell)
-#    else
+  if [[ -n $DUDEYOUGOTADELL ]]; then
-#        BOSSRAID=1
+  APPTYPE=dell
-#    fi
-#}
-
-check_lsi_raid() {
-    # For use for LSI on Ubuntu
-    #MEGA=/opt/MegaRAID/MegeCli/MegaCli64
-    #LSIRC=$($MEGA -LDInfo -Lall -aALL | grep Optimal)
-    # Open Source Centos
-    MEGA=/opt/mega/megasasctl
-    LSIRC=$($MEGA | grep optimal)
-
-    if [[ $LSIRC ]]; then
-        # Raid is good
-        LSIRAID=0
   else
-        LSIRAID=1
+  APPTYPE=sm
+  fi
+  mkdir -p /opt/so/log/raid
+
+  {%- else %}
+  echo "This is not an appliance"
+  exit 0
+  {%- endif %}
+}
+
+check_nsm_raid() {
+  PERCCLI=$(/opt/raidtools/perccli/perccli64 /c0/v0 show|grep RAID|grep Optl)
+  MEGACTL=$(/opt/raidtools/megasasctl |grep optimal)
+
+  if [[ $APPLIANCE == '1' ]]; then
+    if [[ -n $PERCCLI ]]; then
+      HWRAID=0
+    elif [[ -n $MEGACTL ]]; then
+      HWRAID=0
+    else
+      HWRAID=1
+    fi
+
   fi
 
 }
 
+check_boss_raid() {
+  MVCLI=$(/usr/local/bin/mvcli info -o vd |grep status |grep functional)
+
+  if [[ -n $DUDEYOUGOTADELL ]]; then
+    if [[ -n $MVCLI ]]; then
+      BOSSRAID=0
+    else
+      BOSSRAID=1
+    fi
+  fi
+}
+
 check_software_raid() {
+  if [[ -n $DUDEYOUGOTADELL ]]; then
     SWRC=$(grep "_" /proc/mdstat)
 
-    if [[ $SWRC ]]; then
+    if [[ -n $SWRC ]]; then
         # RAID is failed in some way
         SWRAID=1
     else
         SWRAID=0
     fi
+  fi
 }
 
 # This script checks raid status if you use SO appliances
 
 # See if this is an appliance
 
+appliance_check
+check_nsm_raid
+check_boss_raid
 {%- if salt['grains.get']('sosmodel', '') %}
-mkdir -p /opt/so/log/raid
 {%- if grains['sosmodel'] in ['SOSMN', 'SOSSNNV'] %}
-#check_boss_raid
 check_software_raid
-#echo "osraid=$BOSSRAID nsmraid=$SWRAID" > /opt/so/log/raid/status.log
-echo "osraid=1 nsmraid=$SWRAID" > /opt/so/log/raid/status.log
-  {%- elif grains['sosmodel'] in ['SOS1000F', 'SOS1000', 'SOSSN7200', 'SOS10K', 'SOS4000'] %}
-#check_boss_raid
-check_lsi_raid
-#echo "osraid=$BOSSRAID nsmraid=$LSIRAID" > /opt/so/log/raid/status.log
-echo "osraid=1 nsmraid=$LSIRAID" > /opt/so/log/raid/status.log
-  {%- else %}
-exit 0
 {%- endif %}
-{%- else %}
-exit 0
 {%- endif %}
 
+if [[ -n $SWRAID ]]; then
+  if [[ $SWRAID == '0' && $BOSSRAID == '0' ]]; then
+    RAIDSTATUS=0
+  else
+    RAIDSTATUS=1
+  fi
+elif [[ -n $DUDEYOUGOTADELL ]]; then
+  if [[ $BOSSRAID == '0' && $HWRAID == '0' ]]; then
+    RAIDSTATUS=0
+  else
+    RAIDSTATUS=1
+  fi
+elif [[ "$APPTYPE" == 'sm' ]]; then
+  if [[ -n "$HWRAID" ]]; then
+    RAIDSTATUS=0
+  else
+    RAIDSTATUS=1
+  fi
+fi
+
+echo "nsmraid=$RAIDSTATUS" > /opt/so/log/raid/status.log
+
 

salt/common/tools/sbin/so-redis-count

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-docker exec -it so-redis redis-cli llen logstash:unparsed
+docker exec so-redis redis-cli llen logstash:unparsed

salt/common/tools/sbin/so-rule

@@ -405,7 +405,7 @@ def main():
   enabled_list.set_defaults(func=list_enabled_rules)
   
 
-  search_term_help='A quoted regex search term (ex: "\$EXTERNAL_NET")'
+  search_term_help='A properly escaped regex search term (ex: "\\\$EXTERNAL_NET")'
   replace_term_help='The text to replace the search term with'
 
   # Modify actions

salt/common/tools/sbin/so-rule-update

@@ -1,13 +1,10 @@
 #!/bin/bash
-got_root() {
 
-  # Make sure you are root
+. /usr/sbin/so-common
-  if [ "$(id -u)" -ne 0 ]; then
-          echo "This script must be run using sudo!"
-          exit 1
-  fi
 
-}
+argstr=""
+for arg in "$@"; do
+    argstr="${argstr} \"${arg}\""
+done
 
-got_root
+docker exec so-idstools /bin/bash -c "cd /opt/so/idstools/etc && idstools-rulecat --force ${argstr}"
-docker exec so-idstools /bin/bash -c "cd /opt/so/idstools/etc && idstools-rulecat $1"

salt/common/tools/sbin/so-salt-minion-check

@@ -92,6 +92,10 @@ if [ $CURRENT_TIME -ge $((SYSTEM_START_TIME+$UPTIME_REQ))  ]; then
     log "last highstate completed at `date -d @$LAST_HIGHSTATE_END`" I
     log "checking if any jobs are running" I
     logCmd "salt-call --local saltutil.running" I
+    log "ensure salt.minion-state-apply-test is enabled" I
+    logCmd "salt-call state.enable salt.minion-state-apply-test" I
+    log "ensure highstate is enabled" I
+    logCmd "salt-call state.enable highstate" I
     log "killing all salt-minion processes" I
     logCmd "pkill -9 -ef /usr/bin/salt-minion" I
     log "starting salt-minion service" I

salt/common/tools/sbin/so-suricata-testrule

@@ -23,6 +23,11 @@ TESTPCAP=$2
 
 . /usr/sbin/so-common
 
+if [ $# -lt 2 ]; then
+  echo "Usage: $0 <CustomRule> <TargetPCAP>"
+  exit 1
+fi
+
 echo ""
 echo "==============="
 echo "Running all.rules and $TESTRULE against the following pcap: $TESTPCAP" 

salt/common/tools/sbin/so-tcpreplay

@@ -31,7 +31,7 @@ if [[ $# -lt 1 ]]; then
 	echo "Usage: $0 <pcap-sample(s)>"
 	echo 
 	echo "All PCAPs must be placed in the /opt/so/samples directory unless replaying"
-	echo "a sample pcap that is included in the so-tcpreplay image. Those PCAP sampes"
+	echo "a sample pcap that is included in the so-tcpreplay image. Those PCAP samples"
 	echo "are located in the /opt/samples directory inside of the image."
 	echo 
 	echo "Customer provided PCAP example:"

salt/common/tools/sbin/so-thehive-user-add

@@ -41,10 +41,7 @@ if [[ $? == 0 ]]; then
 fi
 read -rs THEHIVE_PASS
 
-if ! check_password "$THEHIVE_PASS"; then
+check_password_and_exit "$THEHIVE_PASS"
-  echo "Password is invalid. Please exclude single quotes, double quotes and backslashes from the password."
-  exit 2
-fi
 
 # Create new user in TheHive
 resp=$(curl -sk -XPOST -H "Authorization: Bearer $THEHIVE_KEY" -H "Content-Type: application/json" -L "https://$THEHVIE_API_URL/user" -d "{\"login\" : \"$THEHIVE_USER\",\"name\" : \"$THEHIVE_USER\",\"roles\" : [\"read\",\"alert\",\"write\",\"admin\"],\"preferences\" : \"{}\",\"password\" : \"$THEHIVE_PASS\"}")

salt/common/tools/sbin/so-thehive-user-update

@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name>"
+    echo ""
+    echo "Update password for an existing TheHive user. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+THEHIVE_KEY=$(lookup_pillar hivekey)
+THEHVIE_API_URL="$(lookup_pillar url_base)/thehive/api"
+THEHIVE_USER=$USER
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -rs THEHIVE_PASS
+
+if ! check_password "$THEHIVE_PASS"; then
+  echo "Password is invalid. Please exclude single quotes, double quotes, dollar signs, and backslashes from the password."
+  exit 2
+fi
+
+# Change password for user in TheHive
+resp=$(curl -sk -XPOST -H "Authorization: Bearer $THEHIVE_KEY" -H "Content-Type: application/json" -L "https://$THEHVIE_API_URL/user/${THEHIVE_USER}/password/set" -d "{\"password\" : \"$THEHIVE_PASS\"}")
+if [[ -z "$resp" ]]; then
+    echo "Successfully updated TheHive user password"
+else
+    echo "Unable to update TheHive user password"
+    echo $resp
+    exit 2
+fi

salt/common/tools/sbin/so-user

@@ -18,11 +18,17 @@
 
 source $(dirname $0)/so-common
 
-if [[ $# -lt 1 || $# -gt 2 ]]; then
+DEFAULT_ROLE=analyst
-  echo "Usage: $0 <list|add|update|enable|disable|validate|valemail|valpass> [email]"
+
+if [[ $# -lt 1 || $# -gt 3 ]]; then
+  echo "Usage: $0 <operation> [email] [role]"
+  echo ""
+  echo " where <operation> is one of the following:"
   echo ""
   echo "     list: Lists all user email addresses currently defined in the identity system"
-  echo "      add: Adds a new user to the identity system; requires 'email' parameter"
+  echo "      add: Adds a new user to the identity system; requires 'email' parameter, while 'role' parameter is optional and defaults to $DEFAULT_ROLE"
+  echo "  addrole: Grants a role to an existing user; requires 'email' and 'role' parameters"
+  echo "  delrole: Removes a role from an existing user; requires 'email' and 'role' parameters"
   echo "   update: Updates a user's password; requires 'email' parameter"
   echo "   enable: Enables a user; requires 'email' parameter"
   echo "  disable: Disables a user; requires 'email' parameter"
@@ -36,13 +42,25 @@ fi
 
 operation=$1
 email=$2
+role=$3
 
 kratosUrl=${KRATOS_URL:-http://127.0.0.1:4434}
 databasePath=${KRATOS_DB_PATH:-/opt/so/conf/kratos/db/db.sqlite}
-argon2Iterations=${ARGON2_ITERATIONS:-3}
+bcryptRounds=${BCRYPT_ROUNDS:-12}
-argon2Memory=${ARGON2_MEMORY:-14}
+elasticUsersFile=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
-argon2Parallelism=${ARGON2_PARALLELISM:-2}
+elasticRolesFile=${ELASTIC_ROLES_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users_roles}
-argon2HashSize=${ARGON2_HASH_SIZE:-32}
+socRolesFile=${SOC_ROLES_FILE:-/opt/so/conf/soc/soc_users_roles}
+esUID=${ELASTIC_UID:-930}
+esGID=${ELASTIC_GID:-930}
+soUID=${SOCORE_UID:-939}
+soGID=${SOCORE_GID:-939}
+
+function lock() {
+  # Obtain file descriptor lock
+  exec 99>/var/tmp/so-user.lock || fail "Unable to create lock descriptor; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  flock -w 10 99 || fail "Another process is using so-user; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  trap 'rm -f /var/tmp/so-user.lock' EXIT
+}
 
 function fail() {
   msg=$1
@@ -58,7 +76,7 @@ function require() {
 
 # Verify this environment is capable of running this script
 function verifyEnvironment() {
-  require "argon2"
+  require "htpasswd"
   require "jq"
   require "curl"
   require "openssl"
@@ -72,7 +90,7 @@ function findIdByEmail() {
   email=$1
 
   response=$(curl -Ss -L ${kratosUrl}/identities)
-  identityId=$(echo "${response}" | jq ".[] | select(.verifiable_addresses[0].value == \"$email\") | .id")
+  identityId=$(echo "${response}" | jq -r ".[] | select(.verifiable_addresses[0].value == \"$email\") | .id")
   echo $identityId
 }
 
@@ -81,20 +99,36 @@ function validatePassword() {
 
   len=$(expr length "$password")
   if [[ $len -lt 6 ]]; then
-    echo "Password does not meet the minimum requirements"
+    fail "Password does not meet the minimum requirements"
-    exit 2
   fi
+  if [[ $len -gt 72 ]]; then
+    fail "Password is too long (max: 72)"
+  fi
+  check_password_and_exit "$password"
 }
 
 function validateEmail() {
   email=$1
   # (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
   if [[ ! "$email" =~ ^[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}$ ]]; then
-    echo "Email address is invalid"
+    fail "Email address is invalid"
-    exit 3
+  fi
+
+  if [[ "$email" =~ [A-Z] ]]; then
+    fail "Email addresses cannot contain uppercase letters"
   fi
 }
 
+function hashPassword() {
+  password=$1
+
+  passwordHash=$(echo "${password}" | htpasswd -niBC $bcryptRounds SOUSER)
+  passwordHash=$(echo "$passwordHash" | cut -c 11-)
+  passwordHash="\$2a${passwordHash}" # still waiting for https://github.com/elastic/elasticsearch/issues/51132
+  echo "$passwordHash"
+}
+
+
 function updatePassword() {
   identityId=$1
   
@@ -109,26 +143,229 @@ function updatePassword() {
     validatePassword "$password"
   fi
 
-  if [[ -n $identityId ]]; then
+  if [[ -n "$identityId" ]]; then
     # Generate password hash
-    salt=$(openssl rand -hex 8)
+    passwordHash=$(hashPassword "$password")
-    passwordHash=$(echo "${password}" | argon2 ${salt} -id -t $argon2Iterations -m $argon2Memory -p $argon2Parallelism -l $argon2HashSize -e)
-
     # Update DB with new hash
-    echo "update identity_credentials set config=CAST('{\"hashed_password\":\"${passwordHash}\"}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
+    echo "update identity_credentials set config=CAST('{\"hashed_password\":\"$passwordHash\"}' as BLOB) where identity_id='${identityId}';" | sqlite3 "$databasePath"
     [[ $? != 0 ]] && fail "Unable to update password"
   fi
 }
 
+function createFile() {
+  filename=$1
+  uid=$2
+  gid=$3
+
+  mkdir -p $(dirname "$filename")
+  truncate -s 0 "$filename"
+  chmod 600 "$filename"
+  chown "${uid}:${gid}" "$filename"
+}
+
+function ensureRoleFileExists() {
+  if [[ ! -f "$socRolesFile" || ! -s "$socRolesFile" ]]; then
+    # Generate the new users file
+    rolesTmpFile="${socRolesFile}.tmp"
+    createFile "$rolesTmpFile" "$soUID" "$soGID"
+
+    if [[ -f "$databasePath" ]]; then
+      echo "Migrating roles to new file: $socRolesFile"
+
+      echo "select 'superuser:' || id from identities;" | sqlite3 "$databasePath" \
+        >> "$rolesTmpFile"
+      [[ $? != 0 ]] && fail "Unable to read identities from database"
+
+      echo "The following users have all been migrated with the super user role:"
+      cat "${rolesTmpFile}"
+    else
+      echo "Database file does not exist yet, installation is likely not yet complete."
+    fi
+
+    if [[ -d "$socRolesFile" ]]; then
+      echo "Removing invalid roles directory created by Docker"
+      rm -fr "$socRolesFile"
+    fi
+    mv "${rolesTmpFile}" "${socRolesFile}"
+  fi
+}
+
+function syncElasticSystemUser() {
+  json=$1
+  userid=$2
+  usersFile=$3
+
+  user=$(echo "$json" | jq -r ".local.users.$userid.user")
+  pass=$(echo "$json" | jq -r ".local.users.$userid.pass")
+  
+  [[ -z "$user" || -z "$pass" ]] && fail "Elastic auth credentials for system user '$userid' are missing"
+  hash=$(hashPassword "$pass")
+
+  echo "${user}:${hash}" >> "$usersFile"
+}
+
+function syncElasticSystemRole() {
+  json=$1
+  userid=$2
+  role=$3
+  rolesFile=$4
+
+  user=$(echo "$json" | jq -r ".local.users.$userid.user")
+  
+  [[ -z "$user" ]] && fail "Elastic auth credentials for system user '$userid' are missing"
+
+  echo "${role}:${user}" >> "$rolesFile"
+}
+
+function syncElastic() {
+  echo "Syncing users and roles between SOC and Elastic..."
+
+  usersTmpFile="${elasticUsersFile}.tmp"
+  createFile "${usersTmpFile}" "$esUID" "$esGID"
+  rolesTmpFile="${elasticRolesFile}.tmp"
+  createFile "${rolesTmpFile}" "$esUID" "$esGID"
+
+  authPillarJson=$(lookup_salt_value "auth" "elasticsearch" "pillar" "json")
+
+  syncElasticSystemUser "$authPillarJson" "so_elastic_user"  "$usersTmpFile"
+  syncElasticSystemUser "$authPillarJson" "so_kibana_user"   "$usersTmpFile"
+  syncElasticSystemUser "$authPillarJson" "so_logstash_user" "$usersTmpFile"
+  syncElasticSystemUser "$authPillarJson" "so_beats_user"    "$usersTmpFile"
+  syncElasticSystemUser "$authPillarJson" "so_monitor_user"  "$usersTmpFile"
+
+  syncElasticSystemRole "$authPillarJson" "so_elastic_user"  "superuser" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_kibana_user"   "superuser" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_logstash_user" "superuser" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_beats_user"    "superuser" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_monitor_user"  "remote_monitoring_collector" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_monitor_user"  "remote_monitoring_agent" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_monitor_user"  "monitoring_user" "$rolesTmpFile"
+
+  if [[ -f "$databasePath" && -f "$socRolesFile" ]]; then
+    # Append the SOC users 
+    echo "select '{\"user\":\"' || ici.identifier || '\", \"data\":' || ic.config || '}'" \
+      "from identity_credential_identifiers ici, identity_credentials ic, identities i " \
+      "where " \
+      "      ici.identity_credential_id=ic.id " \
+      "  and ic.identity_id=i.id " \
+      "  and instr(ic.config, 'hashed_password') " \
+      "  and i.state == 'active' " \
+      "order by ici.identifier;" | \
+      sqlite3 "$databasePath" | \
+      jq -r '.user + ":" + .data.hashed_password' \
+      >> "$usersTmpFile"
+    [[ $? != 0 ]] && fail "Unable to read credential hashes from database"
+
+    # Append the user roles
+    while IFS="" read -r rolePair || [ -n "$rolePair" ]; do
+      userId=$(echo "$rolePair" | cut -d: -f2)
+      role=$(echo "$rolePair" | cut -d: -f1)
+      echo "select '$role:' || ici.identifier " \
+        "from identity_credential_identifiers ici, identity_credentials ic " \
+        "where ici.identity_credential_id=ic.id and ic.identity_id = '$userId';" | \
+        sqlite3 "$databasePath" >> "$rolesTmpFile"
+    done < "$socRolesFile"
+
+  else
+    echo "Database file or soc roles file does not exist yet, skipping users export"
+  fi
+
+  if [[ -s "${usersTmpFile}" ]]; then
+    mv "${usersTmpFile}" "${elasticUsersFile}"
+    mv "${rolesTmpFile}" "${elasticRolesFile}"
+
+    if [[ -z "$SKIP_STATE_APPLY" ]]; then
+      echo "Elastic state will be re-applied to affected minions. This may take several minutes..."
+      echo "Applying elastic state to elastic minions at $(date)" >> /opt/so/log/soc/sync.log 2>&1
+      salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply elasticsearch queue=True >> /opt/so/log/soc/sync.log 2>&1
+    fi
+  else
+    echo "Newly generated users/roles files are incomplete; aborting."
+  fi
+}
+
+function syncAll() {
+  ensureRoleFileExists
+
+  # Check if a sync is needed. Sync is not needed if the following are true:
+  # - user database entries are all older than the elastic users file
+  # - soc roles file last modify date is older than the elastic roles file
+  if [[ -z "$FORCE_SYNC" && -f "$databasePath" && -f "$elasticUsersFile" ]]; then
+    usersFileAgeSecs=$(echo $(($(date +%s) - $(date +%s -r "$elasticUsersFile"))))
+    staleCount=$(echo "select count(*) from identity_credentials where updated_at >= Datetime('now', '-${usersFileAgeSecs} seconds');" \
+      | sqlite3 "$databasePath")
+    if [[ "$staleCount" == "0" && "$elasticRolesFile" -nt "$socRolesFile" ]]; then
+      return 1
+    fi
+  fi
+
+  syncElastic
+
+  return 0
+}
+
 function listUsers() {
   response=$(curl -Ss -L ${kratosUrl}/identities)
   [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
 
-  echo "${response}" | jq -r ".[] | .verifiable_addresses[0].value" | sort
+  users=$(echo "${response}" | jq -r ".[] | .verifiable_addresses[0].value" | sort)
+  for user in $users; do
+    roles=$(grep "$user" "$elasticRolesFile" | cut -d: -f1 | tr '\n' ' ')
+    echo "$user: $roles"
+  done
+}
+
+function addUserRole() {
+  email=$1
+  role=$2
+
+  adjustUserRole "$email" "$role" "add"
+}
+
+function deleteUserRole() {
+  email=$1
+  role=$2
+
+  adjustUserRole "$email" "$role" "del"
+}
+
+function adjustUserRole() {
+  email=$1
+  role=$2
+  op=$3
+
+  identityId=$(findIdByEmail "$email")
+  [[ ${identityId} == "" ]] && fail "User not found"
+
+  ensureRoleFileExists
+
+  filename="$socRolesFile"
+  hasRole=0
+  grep "$role:" "$socRolesFile" | grep -q "$identityId" && hasRole=1
+  if [[ "$op" == "add" ]]; then
+    if [[ "$hasRole" == "1" ]]; then
+      echo "User '$email' already has the role: $role"
+      return 1
+    else
+      echo "$role:$identityId" >> "$filename"
+    fi
+  elif [[ "$op" == "del" ]]; then
+    if [[ "$hasRole" -ne 1 ]]; then
+      fail "User '$email' does not have the role: $role"
+    else
+      sed "/^$role:$identityId\$/d" "$filename" > "$filename.tmp"
+      cat "$filename".tmp > "$filename"
+      rm -f "$filename".tmp
+    fi
+  else
+    fail "Unsupported role adjustment operation: $op"
+  fi
+  return 0
 }
 
 function createUser() {
   email=$1
+  role=$2
 
   now=$(date -u +%FT%TZ)
   addUserJson=$(cat <<EOF
@@ -142,16 +379,30 @@ EOF
   response=$(curl -Ss -L ${kratosUrl}/identities -d "$addUserJson")
   [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
 
-  identityId=$(echo "${response}" | jq ".id")
+  identityId=$(echo "${response}" | jq -r ".id")
-  if [[ ${identityId} == "null" ]]; then
+  if [[ "${identityId}" == "null" ]]; then
     code=$(echo "${response}" | jq ".error.code")
     [[ "${code}" == "409" ]] && fail "User already exists"
 
     reason=$(echo "${response}" | jq ".error.message")
     [[ $? == 0 ]] && fail "Unable to add user: ${reason}"
+  else
+    updatePassword "$identityId"
+    addUserRole "$email" "$role"
   fi
+}
 
-  updatePassword $identityId
+function migrateLockedUsers() {
+  # This is a migration function to convert locked users from prior to 2.3.90
+  # to inactive users using the newer Kratos functionality. This should only
+  # find locked users once.
+  lockedEmails=$(curl -s http://localhost:4434/identities | jq -r '.[] | select(.traits.status == "locked") | .traits.email')
+  if [[ -n "$lockedEmails" ]]; then
+    echo "Disabling locked users..."
+    for email in $lockedEmails; do
+      updateStatus "$email" locked
+    done
+  fi
 }
 
 function updateStatus() {
@@ -164,24 +415,18 @@ function updateStatus() {
   response=$(curl -Ss -L "${kratosUrl}/identities/$identityId")
   [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
 
-  oldConfig=$(echo "select config from identity_credentials where identity_id=${identityId};" | sqlite3 "$databasePath")
+  schemaId=$(echo "$response" | jq -r .schema_id)
+
+  # Capture traits and remove obsolete 'status' trait if exists
+  traitBlock=$(echo "$response" | jq -c .traits | sed -re 's/,?"status":".*?"//')
+
+  state="active"
   if [[ "$status" == "locked" ]]; then
-    config=$(echo $oldConfig | sed -e 's/hashed/locked/')
+    state="inactive"
-    echo "update identity_credentials set config=CAST('${config}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
-    [[ $? != 0 ]] && fail "Unable to lock credential record"
-
-    echo "delete from sessions where identity_id=${identityId};" | sqlite3 "$databasePath"
-    [[ $? != 0 ]] && fail "Unable to invalidate sessions"    
-  else
-    config=$(echo $oldConfig | sed -e 's/locked/hashed/')
-    echo "update identity_credentials set config=CAST('${config}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
-    [[ $? != 0 ]] && fail "Unable to unlock credential record"
   fi
-
+  body="{ \"schema_id\": \"$schemaId\", \"state\": \"$state\", \"traits\": $traitBlock }"
-  updatedJson=$(echo "$response" | jq ".traits.status = \"$status\" | del(.verifiable_addresses) | del(.id) | del(.schema_url)")
+  response=$(curl -fSsL -XPUT "${kratosUrl}/identities/$identityId" -d "$body")
-  response=$(curl -Ss -XPUT -L ${kratosUrl}/identities/$identityId -d "$updatedJson")
+  [[ $? != 0 ]] && fail "Unable to update user"
-  [[ $? != 0 ]] && fail "Unable to mark user as locked"
-
 }
 
 function updateUser() {
@@ -190,7 +435,7 @@ function updateUser() {
   identityId=$(findIdByEmail "$email")
   [[ ${identityId} == "" ]] && fail "User not found"
 
-  updatePassword $identityId 
+  updatePassword "$identityId"
 }
 
 function deleteUser() {
@@ -201,6 +446,11 @@ function deleteUser() {
 
   response=$(curl -Ss -XDELETE -L "${kratosUrl}/identities/$identityId")
   [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
+
+  rolesTmpFile="${socRolesFile}.tmp"
+  createFile "$rolesTmpFile" "$soUID" "$soGID"
+  grep -v "$identityId" "$socRolesFile" > "$rolesTmpFile"
+  mv "$rolesTmpFile" "$socRolesFile"
 }
 
 case "${operation}" in
@@ -208,12 +458,14 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     validateEmail "$email"
     updatePassword
-    createUser "$email"
+    createUser "$email" "${role:-$DEFAULT_ROLE}"
+    syncAll
     echo "Successfully added new user to SOC"
-    check_container thehive && echo $password | so-thehive-user-add "$email"
+    check_container thehive && echo "$password" | so-thehive-user-add "$email"
-    check_container fleet && echo $password | so-fleet-user-add "$email"
+    check_container fleet && echo "$password" | so-fleet-user-add "$email"
     ;;
 
   "list")
@@ -221,11 +473,38 @@ case "${operation}" in
     listUsers
     ;;
 
+  "addrole")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+    [[ "$role" == "" ]] && fail "Role must be provided"
+
+    lock
+    validateEmail "$email"
+    if addUserRole "$email" "$role"; then
+      syncElastic
+      echo "Successfully added role to user"
+    fi
+    ;;
+
+  "delrole")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+    [[ "$role" == "" ]] && fail "Role must be provided"
+
+    lock
+    validateEmail "$email"
+    deleteUserRole "$email" "$role"
+    syncElastic
+    echo "Successfully removed role from user"
+    ;;
+
   "update")
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     updateUser "$email"
+    syncAll
     echo "Successfully updated user"
     ;;
 
@@ -233,30 +512,41 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     updateStatus "$email" 'active'
+    syncAll
     echo "Successfully enabled user"
     check_container thehive && so-thehive-user-enable "$email" true
-    check_container fleet && so-fleet-user-enable "$email" true   
+    echo "Fleet user will need to be recreated manually with so-fleet-user-add"
     ;;
 
   "disable")
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     updateStatus "$email" 'locked'
+    syncAll
     echo "Successfully disabled user"
     check_container thehive && so-thehive-user-enable "$email" false
-    check_container fleet && so-fleet-user-enable "$email" false   
+    check_container fleet && so-fleet-user-delete "$email"   
     ;;    
 
   "delete")
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     deleteUser "$email"
+    syncAll
     echo "Successfully deleted user"
     check_container thehive && so-thehive-user-enable "$email" false
-    check_container fleet && so-fleet-user-enable "$email" false
+    check_container fleet && so-fleet-user-delete "$email"
+    ;;
+
+  "sync")
+    lock
+    syncAll
     ;;
 
   "validate")
@@ -275,6 +565,11 @@ case "${operation}" in
     echo "Password is acceptable"
     ;;
 
+  "migrate")
+    migrateLockedUsers
+    echo "User migration complete"
+    ;;
+
   *)
     fail "Unsupported operation: $operation"
     ;;

salt/common/tools/sbin/so-yara-update

@@ -1,5 +1,4 @@
 #!/bin/bash
-
 # Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
@@ -20,13 +19,8 @@ echo "Starting to check for yara rule updates at $(date)..."
 
 output_dir="/opt/so/saltstack/default/salt/strelka/rules"
 mkdir -p $output_dir
-
 repos="$output_dir/repos.txt"
-ignorefile="$output_dir/ignore.txt"
-
-deletecounter=0
 newcounter=0
-updatecounter=0
 
 {% if ISAIRGAP is sameas true %}
 
@@ -35,58 +29,21 @@ echo "Airgap mode enabled."
 clone_dir="/nsm/repo/rules/strelka"
 repo_name="signature-base"
 mkdir -p /opt/so/saltstack/default/salt/strelka/rules/signature-base
-
+# Ensure a copy of the license is available for the rules
 [ -f $clone_dir/LICENSE ] && cp $clone_dir/$repo_name/LICENSE $output_dir/$repo_name
 
 # Copy over rules
 for i in $(find $clone_dir/yara -name "*.yar*"); do
   rule_name=$(echo $i | awk -F '/' '{print $NF}')
-  repo_sum=$(sha256sum $i | awk '{print $1}')
+  echo "Adding rule: $rule_name..."
-
-  # Check rules against those in ignore list -- don't copy if ignored.
-  if ! grep -iq $rule_name $ignorefile; then
-    existing_rules=$(find $output_dir/$repo_name/ -name $rule_name | wc -l)
-
-    # For existing rules, check to see if they need to be updated, by comparing checksums
-    if [ $existing_rules -gt 0 ];then
-      local_sum=$(sha256sum $output_dir/$repo_name/$rule_name | awk '{print $1}')
-      if [ "$repo_sum" != "$local_sum" ]; then
-        echo "Checksums do not match!"
-        echo "Updating $rule_name..."
-        cp $i $output_dir/$repo_name;
-        ((updatecounter++))
-      fi
-    else
-      # If rule doesn't exist already, we'll add it
-      echo "Adding new rule: $rule_name..."
   cp $i $output_dir/$repo_name
   ((newcounter++))
-    fi
-  fi;
-done
-
-# Check to see if we have any old rules that need to be removed
-for i in $(find $output_dir/$repo_name -name "*.yar*" | awk -F '/' '{print $NF}'); do
-  is_repo_rule=$(find $clone_dir -name "$i" | wc -l)
-  if [ $is_repo_rule -eq 0 ]; then
-    echo "Could not find $i in source $repo_name repo...removing from $output_dir/$repo_name..."
-    rm $output_dir/$repo_name/$i
-    ((deletecounter++))
-  fi
 done
 
 echo "Done!"
 
 if [ "$newcounter" -gt 0 ];then
-  echo "$newcounter new rules added."
+  echo "$newcounter rules added."
-fi
-
-if [ "$updatecounter" -gt 0 ];then
-  echo "$updatecounter rules updated."
-fi
-
-if [ "$deletecounter" -gt 0 ];then
-  echo "$deletecounter rules removed because they were deprecated or don't exist in the source repo."
 fi
 
 {% else %}
@@ -99,50 +56,21 @@ if [ "$gh_status" == "200" ]  || [ "$gh_status" == "301" ]; then
     if ! $(echo "$repo" | grep -qE '^#'); then
       # Remove old repo if existing bc of previous error condition or unexpected disruption
       repo_name=`echo $repo | awk -F '/' '{print $NF}'`
-      [ -d $repo_name ] && rm -rf $repo_name
+      [ -d $output_dir/$repo_name ] && rm -rf $output_dir/$repo_name
 
       # Clone repo and make appropriate directories for rules
-
       git clone $repo $clone_dir/$repo_name
       echo "Analyzing rules from $clone_dir/$repo_name..."
       mkdir -p $output_dir/$repo_name
+      # Ensure a copy of the license is available for the rules
       [ -f $clone_dir/$repo_name/LICENSE ] && cp $clone_dir/$repo_name/LICENSE $output_dir/$repo_name
 
       # Copy over rules
       for i in $(find $clone_dir/$repo_name -name "*.yar*"); do
         rule_name=$(echo $i | awk -F '/' '{print $NF}')
-        repo_sum=$(sha256sum $i | awk '{print $1}')
+        echo "Adding rule: $rule_name..."
-
-        # Check rules against those in ignore list -- don't copy if ignored.
-        if ! grep -iq $rule_name $ignorefile; then
-          existing_rules=$(find $output_dir/$repo_name/ -name $rule_name | wc -l)
-
-          # For existing rules, check to see if they need to be updated, by comparing checksums
-          if [ $existing_rules -gt 0 ];then
-            local_sum=$(sha256sum $output_dir/$repo_name/$rule_name | awk '{print $1}')
-            if [ "$repo_sum" != "$local_sum" ]; then
-              echo "Checksums do not match!"
-              echo "Updating $rule_name..."
-              cp $i $output_dir/$repo_name;
-              ((updatecounter++))
-            fi
-          else
-            # If rule doesn't exist already, we'll add it
-            echo "Adding new rule: $rule_name..."
         cp $i $output_dir/$repo_name
         ((newcounter++))
-          fi
-        fi;
-        done
-
-        # Check to see if we have any old rules that need to be removed
-        for i in $(find $output_dir/$repo_name -name "*.yar*" | awk -F '/' '{print $NF}'); do
-          is_repo_rule=$(find $clone_dir/$repo_name -name "$i" | wc -l)
-          if [ $is_repo_rule -eq 0 ]; then
-            echo "Could not find $i in source $repo_name repo...removing from $output_dir/$repo_name..."
-            rm $output_dir/$repo_name/$i
-            ((deletecounter++))
-          fi
      done
      rm -rf $clone_dir/$repo_name
     fi
@@ -151,15 +79,7 @@ if [ "$gh_status" == "200" ]  || [ "$gh_status" == "301" ]; then
   echo "Done!"
   
   if [ "$newcounter" -gt 0 ];then
-    echo "$newcounter new rules added."
+    echo "$newcounter rules added."
-  fi
-
-  if [ "$updatecounter" -gt 0 ];then
-    echo "$updatecounter rules updated."
-  fi
-
-  if [ "$deletecounter" -gt 0 ];then
-    echo "$deletecounter rules removed because they were deprecated or don't exist in the source repo."
   fi
   
 else

salt/common/tools/sbin/so-zeek-logs

@@ -10,11 +10,10 @@ zeek_logs_enabled() {
 }
 
 whiptail_manager_adv_service_zeeklogs() {
-  BLOGS=$(whiptail --title "Security Onion Setup" --checklist "Please Select Logs to Send:" 24 78 12 \
+  BLOGS=$(whiptail --title "so-zeek-logs" --checklist "Please Select Logs to Send:" 24 78 12 \
   "conn" "Connection Logging" ON \
   "dce_rpc" "RPC Logs" ON \
   "dhcp" "DHCP Logs" ON \
-  "dhcpv6" "DHCP IPv6 Logs" ON \
   "dnp3" "DNP3 Logs" ON \
   "dns" "DNS Logs" ON \
   "dpd" "DPD Logs" ON \
@@ -25,25 +24,20 @@ whiptail_manager_adv_service_zeeklogs() {
   "irc" "IRC Chat Logs" ON \
   "kerberos" "Kerberos Logs" ON \
   "modbus" "MODBUS Logs" ON \
-  "mqtt" "MQTT Logs" ON \
   "notice" "Zeek Notice Logs" ON \
   "ntlm" "NTLM Logs" ON \
-  "openvpn" "OPENVPN Logs" ON \
   "pe" "PE Logs" ON \
   "radius" "Radius Logs" ON \
   "rfb" "RFB Logs" ON \
   "rdp" "RDP Logs" ON \
-  "signatures" "Signatures Logs" ON \
   "sip" "SIP Logs" ON \
   "smb_files" "SMB Files Logs" ON \
   "smb_mapping" "SMB Mapping Logs" ON \
   "smtp" "SMTP Logs" ON \
   "snmp" "SNMP Logs" ON \
-  "software" "Software Logs" ON \
   "ssh" "SSH Logs" ON \
   "ssl" "SSL Logs" ON \
   "syslog" "Syslog Logs" ON \
-  "telnet" "Telnet Logs" ON \
   "tunnel" "Tunnel Logs" ON \
   "weird" "Zeek Weird Logs" ON \
   "mysql" "MySQL Logs" ON \
@@ -61,10 +55,10 @@ whiptail_manager_adv_service_zeeklogs
 return_code=$?
 case $return_code in
   1)
-    whiptail --title "Security Onion Setup" --msgbox "Cancelling. No changes have been made." 8 75
+    whiptail --title "so-zeek-logs" --msgbox "Cancelling. No changes have been made." 8 75
     ;;
   255)
-    whiptail --title "Security Onion Setup" --msgbox "Whiptail error occured, exiting." 8 75
+    whiptail --title "so-zeek-logs" --msgbox "Whiptail error occured, exiting." 8 75
     ;;
   *)
     zeek_logs_enabled

salt/common/tools/sbin/so-zeek-stats

@@ -24,11 +24,11 @@ show_stats() {
   echo
   echo "Average throughput:"
   echo
-  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin runuser -l zeek -c '/opt/zeek/bin/zeekctl capstats'
+  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin /opt/zeek/bin/zeekctl capstats
   echo
   echo "Average packet loss:"
   echo
-  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin runuser -l zeek -c '/opt/zeek/bin/zeekctl netstats'
+  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin /opt/zeek/bin/zeekctl netstats
   echo
 }
 

salt/common/tools/sbin/soup

@@ -18,12 +18,83 @@
 . /usr/sbin/so-common
 
 UPDATE_DIR=/tmp/sogh/securityonion
+DEFAULT_SALT_DIR=/opt/so/saltstack/default
 INSTALLEDVERSION=$(cat /etc/soversion)
 POSTVERSION=$INSTALLEDVERSION
-INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk {'print $2'})
+INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk '{print $2}')
 BATCHSIZE=5
 SOUP_LOG=/root/soup.log
+INFLUXDB_MIGRATION_LOG=/opt/so/log/influxdb/soup_migration.log
 WHATWOULDYOUSAYYAHDOHERE=soup
+whiptail_title='Security Onion UPdater'
+NOTIFYCUSTOMELASTICCONFIG=false
+
+check_err() {
+  local exit_code=$1
+  local err_msg="Unhandled error occured, please check $SOUP_LOG for details."
+
+  [[ $ERR_HANDLED == true ]] && exit $exit_code
+  if [[ $exit_code -ne 0 ]]; then
+    printf '%s' "Soup failed with error $exit_code: "
+    case $exit_code in
+      2)
+        echo 'No such file or directory'
+      ;;
+      5)
+        echo 'Interrupted system call'
+      ;;
+      12)
+        echo 'Out of memory'
+      ;;
+      28)
+        echo 'No space left on device'
+        echo 'Likely ran out of space on disk, please review hardware requirements for Security Onion: https://docs.securityonion.net/en/2.3/hardware.html'
+      ;;
+      30)
+        echo 'Read-only file system'
+      ;;
+      35)
+        echo 'Resource temporarily unavailable'
+      ;;
+      64)
+        echo 'Machine is not on the network'
+      ;;
+      67)
+        echo 'Link has been severed'
+      ;;
+      100)
+        echo 'Network is down'
+      ;;
+      101)
+        echo 'Network is unreachable'
+      ;;
+      102)
+        echo 'Network reset'
+      ;;
+      110)
+        echo 'Connection timed out'
+      ;;
+      111)
+        echo 'Connection refused'
+      ;;
+      112)
+        echo 'Host is down'
+      ;;
+      113)
+        echo 'No route to host'
+      ;;
+      *)
+        echo 'Unhandled error'
+        echo "$err_msg"
+      ;;
+    esac
+    if [[ $exit_code -ge 64 && $exit_code -le 113 ]]; then
+      echo "$err_msg"
+    fi
+    exit $exit_code
+  fi
+
+}
 
 add_common() {
   cp $UPDATE_DIR/salt/common/tools/sbin/so-common $DEFAULT_SALT_DIR/salt/common/tools/sbin/
@@ -35,19 +106,21 @@ add_common() {
 
 airgap_mounted() {
   # Let's see if the ISO is already mounted. 
-  if [ -f /tmp/soagupdate/SecurityOnion/VERSION ]; then
+  if [[ -f /tmp/soagupdate/SecurityOnion/VERSION ]]; then
     echo "The ISO is already mounted"
   else
+    if [[ -z $ISOLOC ]]; then
+      echo "This is airgap. Ask for a location."
       echo ""
-    echo "Looks like we need access to the upgrade content"
+      cat << EOF
-    echo "" 
+In order for soup to proceed, the path to the downloaded Security Onion ISO file, or the path to the CD-ROM or equivalent device containing the ISO media must be provided. 
-    echo "If you just copied the .iso file over you can specify the path."
+For example, if you have copied the new Security Onion ISO file to your home directory, then the path might look like /home/myuser/securityonion-2.x.y.iso. 
-    echo "If you burned the ISO to a disk the standard way you can specify the device."
+Or, if you have burned the new ISO onto an optical disk then the path might look like /dev/cdrom.
-    echo "Example: /home/user/securityonion-2.X.0.iso"
+
-    echo "Example: /dev/sdx1"
+EOF
-    echo ""
+      read -rp 'Enter the path to the new Security Onion ISO content: ' ISOLOC
-    read -p 'Enter the location of the iso: ' ISOLOC
+    fi
-    if [ -f $ISOLOC ]; then
+    if [[ -f $ISOLOC ]]; then
       # Mounting the ISO image
       mkdir -p /tmp/soagupdate
       mount -t iso9660 -o loop $ISOLOC /tmp/soagupdate
@@ -59,10 +132,10 @@ airgap_mounted() {
       else
         echo "ISO has been mounted!"
       fi  
-    elif [ -f $ISOLOC/SecurityOnion/VERSION ]; then
+    elif [[ -f $ISOLOC/SecurityOnion/VERSION ]]; then
       ln -s $ISOLOC /tmp/soagupdate
       echo "Found the update content"
-    else 
+    elif [[ -b $ISOLOC ]]; then
       mkdir -p /tmp/soagupdate
       mount $ISOLOC /tmp/soagupdate
       if [ ! -f /tmp/soagupdate/SecurityOnion/VERSION ]; then
@@ -72,24 +145,28 @@ airgap_mounted() {
       else
         echo "Device has been mounted!"
       fi
+    else
+      echo "Could not find Security Onion ISO content at ${ISOLOC}" 
+      echo "Ensure the path you entered is correct, and that you verify the ISO that you downloaded."
+      exit 0
     fi
   fi
 }
 
 airgap_update_dockers() {
-  if [ $is_airgap -eq 0 ]; then
+  if [[ $is_airgap -eq 0 ]]; then
     # Let's copy the tarball
-    if [ ! -f $AGDOCKER/registry.tar ]; then
+    if [[ ! -f $AGDOCKER/registry.tar ]]; then
       echo "Unable to locate registry. Exiting"
-      exit 1
+      exit 0
     else
       echo "Stopping the registry docker"
       docker stop so-dockerregistry
       docker rm so-dockerregistry
       echo "Copying the new dockers over"
-      tar xvf $AGDOCKER/registry.tar -C /nsm/docker-registry/docker
+      tar xvf "$AGDOCKER/registry.tar" -C /nsm/docker-registry/docker
       echo "Add Registry back"
-      docker load -i $AGDOCKER/registry_image.tar
+      docker load -i "$AGDOCKER/registry_image.tar"
     fi
   fi
 }
@@ -100,6 +177,63 @@ update_registry() {
   salt-call state.apply registry queue=True
 }
 
+check_airgap() {
+  # See if this is an airgap install
+  AIRGAP=$(cat /opt/so/saltstack/local/pillar/global.sls | grep airgap: | awk '{print $2}')
+  if [[ "$AIRGAP" == "True" ]]; then
+      is_airgap=0
+      UPDATE_DIR=/tmp/soagupdate/SecurityOnion
+      AGDOCKER=/tmp/soagupdate/docker
+      AGREPO=/tmp/soagupdate/Packages
+  else 
+      is_airgap=1
+  fi
+}
+
+# {% raw %}
+
+check_local_mods() {
+  local salt_local=/opt/so/saltstack/local
+
+  local_mod_arr=()
+
+  while IFS= read -r -d '' local_file; do
+    stripped_path=${local_file#"$salt_local"}
+    default_file="${DEFAULT_SALT_DIR}${stripped_path}"
+    if [[ -f $default_file ]]; then
+      file_diff=$(diff "$default_file" "$local_file" )
+      if [[ $(echo "$file_diff" | grep -c "^<") -gt 0 ]]; then
+        local_mod_arr+=( "$local_file" )
+      fi
+    fi
+  done< <(find $salt_local -type f -print0)
+
+  if [[ ${#local_mod_arr} -gt 0 ]]; then
+    echo "Potentially breaking changes found in the following files (check ${DEFAULT_SALT_DIR} for original copy):"
+    for file_str in "${local_mod_arr[@]}"; do
+      echo "  $file_str"
+    done
+    echo ""
+    echo "To reference this list later, check $SOUP_LOG"
+    sleep 10
+  fi
+}
+
+# {% endraw %}
+
+check_pillar_items() {
+  local pillar_output=$(salt-call pillar.items --out=json)
+
+  cond=$(jq '.local | has("_errors")' <<< "$pillar_output")
+  if [[ "$cond" == "true" ]]; then
+    printf "\nThere is an issue rendering the manager's pillars. Please correct the issues in the sls files mentioned below before running SOUP again.\n\n"
+    jq '.local._errors[]' <<< "$pillar_output"
+    exit 0
+  else
+    printf "\nThe manager's pillars can be rendered. We can proceed with SOUP.\n\n"
+  fi
+}
+
 check_sudoers() {
   if grep -q "so-setup" /etc/sudoers; then
     echo "There is an entry for so-setup in the sudoers file, this can be safely deleted using \"visudo\"."
@@ -169,23 +303,31 @@ check_os_updates() {
     OSUPDATES=$(yum -q list updates | wc -l)
   fi
   if [[ "$OSUPDATES" -gt 0 ]]; then
-      echo $NEEDUPDATES
+      if [[ -z $UNATTENDED ]]; then
+        echo "$NEEDUPDATES"
         echo ""
-      read -p "Press U to update OS packages (recommended), C to continue without updates, or E to exit: " confirm
+        read -rp "Press U to update OS packages (recommended), C to continue without updates, or E to exit: " confirm
-
         if [[ "$confirm" == [cC] ]]; then
           echo "Continuing without updating packages"
         elif [[ "$confirm" == [uU] ]]; then
           echo "Applying Grid Updates"
-          salt \* -b 5 state.apply patch.os queue=True
+          update_flag=true
         else
           echo "Exiting soup"
           exit 0
         fi
+      else
+        update_flag=true
+      fi
   else
     echo "Looks like you have an updated OS"
   fi
 
+  if [[ $update_flag == true ]]; then
+    set +e
+    run_check_net_err "salt '*' -b 5 state.apply patch.os queue=True" 'Could not apply OS updates, please check your network connection.'
+    set -e
+  fi
 }
 
 clean_dockers() {
@@ -217,7 +359,7 @@ generate_and_clean_tarballs() {
   local new_version
   new_version=$(cat $UPDATE_DIR/VERSION)
   [ -d /opt/so/repo ] || mkdir -p /opt/so/repo
-  tar -czf "/opt/so/repo/$new_version.tar.gz" "$UPDATE_DIR"
+  tar -czf "/opt/so/repo/$new_version.tar.gz" -C "$UPDATE_DIR" .
   find "/opt/so/repo" -type f -not -name "$new_version.tar.gz" -exec rm -rf {} \;
 }
 
@@ -247,136 +389,82 @@ masterunlock() {
   fi
 }
 
-preupgrade_changes_2.3.50_repo() {
-    # We made repo changes in 2.3.50 and this prepares for that on upgrade
-    echo "Checking to see if 2.3.50 repo changes are needed."
-
-    [[ "$INSTALLEDVERSION" == 2.3.30 || "$INSTALLEDVERSION" == 2.3.40 ]] && up_2.3.3X_to_2.3.50_repo
-}
-
 preupgrade_changes() {
     # This function is to add any new pillar items if needed.
     echo "Checking to see if changes are needed."
 
-    [[ "$INSTALLEDVERSION" =~ rc.1 ]] && rc1_to_rc2
+    [[ "$INSTALLEDVERSION" == 2.3.0 || "$INSTALLEDVERSION" == 2.3.1 || "$INSTALLEDVERSION" == 2.3.2 || "$INSTALLEDVERSION" == 2.3.10 ]] && up_to_2.3.20
-    [[ "$INSTALLEDVERSION" =~ rc.2 ]] && rc2_to_rc3
+    [[ "$INSTALLEDVERSION" == 2.3.20 || "$INSTALLEDVERSION" == 2.3.21 ]] && up_to_2.3.30
-    [[ "$INSTALLEDVERSION" =~ rc.3 ]] && rc3_to_2.3.0
+    [[ "$INSTALLEDVERSION" == 2.3.30 || "$INSTALLEDVERSION" == 2.3.40 ]] && up_to_2.3.50
-    [[ "$INSTALLEDVERSION" == 2.3.0 || "$INSTALLEDVERSION" == 2.3.1 || "$INSTALLEDVERSION" == 2.3.2 || "$INSTALLEDVERSION" == 2.3.10 ]] && up_2.3.0_to_2.3.20
+    [[ "$INSTALLEDVERSION" == 2.3.50 || "$INSTALLEDVERSION" == 2.3.51 || "$INSTALLEDVERSION" == 2.3.52 || "$INSTALLEDVERSION" == 2.3.60 || "$INSTALLEDVERSION" == 2.3.61 || "$INSTALLEDVERSION" == 2.3.70 ]] && up_to_2.3.80
-    [[ "$INSTALLEDVERSION" == 2.3.20 || "$INSTALLEDVERSION" == 2.3.21 ]] && up_2.3.2X_to_2.3.30
+    [[ "$INSTALLEDVERSION" == 2.3.80 ]] && up_to_2.3.90
-    [[ "$INSTALLEDVERSION" == 2.3.30 || "$INSTALLEDVERSION" == 2.3.40 ]] && up_2.3.3X_to_2.3.50
+    true
 }
 
 postupgrade_changes() {
     # This function is to add any new pillar items if needed.
     echo "Running post upgrade processes."
     
-    [[ "$POSTVERSION" =~ rc.1 ]] && post_rc1_to_rc2
+    [[ "$POSTVERSION" == 2.3.0 || "$POSTVERSION" == 2.3.1 || "$POSTVERSION" == 2.3.2 || "$POSTVERSION" == 2.3.10 || "$POSTVERSION" == 2.3.20 ]] && post_to_2.3.21
-    [[ "$POSTVERSION" == 2.3.20 || "$POSTVERSION" == 2.3.21 ]] && post_2.3.2X_to_2.3.30
+    [[ "$POSTVERSION" == 2.3.21 || "$POSTVERSION" == 2.3.30 ]] && post_to_2.3.40
-    [[ "$POSTVERSION" == 2.3.30 ]] && post_2.3.30_to_2.3.40
+    [[ "$POSTVERSION" == 2.3.40 || "$POSTVERSION" == 2.3.50 || "$POSTVERSION" == 2.3.51 || "$POSTVERSION" == 2.3.52 ]] && post_to_2.3.60
+    [[ "$POSTVERSION" == 2.3.60 || "$POSTVERSION" == 2.3.61 || "$POSTVERSION" == 2.3.70 || "$POSTVERSION" == 2.3.80 ]] && post_to_2.3.90
+    true
 }
 
-post_rc1_to_2.3.21() {
+post_to_2.3.21() {
   salt-call state.apply playbook.OLD_db_init
   rm -f /opt/so/rules/elastalert/playbook/*.yaml
   so-playbook-ruleupdate >> /root/soup_playbook_rule_update.log 2>&1 &
   POSTVERSION=2.3.21
 }
 
-post_2.3.2X_to_2.3.30() {
+post_to_2.3.40() {
-  so-playbook-sigma-refresh >> /root/soup_playbook_sigma_refresh.log 2>&1 &
-  POSTVERSION=2.3.30
-}
-
-post_2.3.30_to_2.3.40() {
   so-playbook-sigma-refresh >> /root/soup_playbook_sigma_refresh.log 2>&1 &
   so-kibana-space-defaults
   POSTVERSION=2.3.40
 }
 
+post_to_2.3.60() {
+  for table in identity_recovery_addresses selfservice_recovery_flows selfservice_registration_flows selfservice_verification_flows identities identity_verification_tokens identity_credentials selfservice_settings_flows identity_recovery_tokens continuity_containers identity_credential_identifiers identity_verifiable_addresses courier_messages selfservice_errors sessions selfservice_login_flows
+  do
+    echo "Forcing Kratos network migration: $table"
+    sqlite3 /opt/so/conf/kratos/db/db.sqlite "update $table set nid=(select id from networks limit 1);"
+  done
 
-rc1_to_rc2() {
+  POSTVERSION=2.3.60
-
-  # Move the static file to global.sls
-  echo "Migrating static.sls to global.sls"
-  mv -v /opt/so/saltstack/local/pillar/static.sls /opt/so/saltstack/local/pillar/global.sls >> "$SOUP_LOG" 2>&1
-  sed -i '1c\global:' /opt/so/saltstack/local/pillar/global.sls >> "$SOUP_LOG" 2>&1
-
-  # Moving baseurl from minion sls file to inside global.sls
-  local line=$(grep '^  url_base:' /opt/so/saltstack/local/pillar/minions/$MINIONID.sls)
-  sed -i '/^  url_base:/d' /opt/so/saltstack/local/pillar/minions/$MINIONID.sls;
-  sed -i "/^global:/a \\$line" /opt/so/saltstack/local/pillar/global.sls;
-
-  # Adding play values to the global.sls
-  local HIVEPLAYSECRET=$(get_random_value)
-  local CORTEXPLAYSECRET=$(get_random_value)
-  sed -i "/^global:/a \\  hiveplaysecret: $HIVEPLAYSECRET" /opt/so/saltstack/local/pillar/global.sls;
-  sed -i "/^global:/a \\  cortexplaysecret: $CORTEXPLAYSECRET" /opt/so/saltstack/local/pillar/global.sls;
-
-  # Move storage nodes to hostname for SSL
-  # Get a list we can use:
-  grep -A1 searchnode /opt/so/saltstack/local/pillar/data/nodestab.sls | grep -v '\-\-' | sed '$!N;s/\n/ /' | awk '{print $1,$3}' | awk '/_searchnode:/{gsub(/\_searchnode:/, "_searchnode"); print}' >/tmp/nodes.txt
-  # Remove the nodes from cluster settings
-  while read p; do
-  local NAME=$(echo $p | awk '{print $1}')
-  local IP=$(echo $p | awk '{print $2}')
-  echo "Removing the old cross cluster config for $NAME"
-  curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_cluster/settings -d '{"persistent":{"cluster":{"remote":{"'$NAME'":{"skip_unavailable":null,"seeds":null}}}}}'
-  done </tmp/nodes.txt
-  # Add the nodes back using hostname
-  while read p; do
-    local NAME=$(echo $p | awk '{print $1}')
-    local EHOSTNAME=$(echo $p | awk -F"_" '{print $1}')
-    local IP=$(echo $p | awk '{print $2}')
-    echo "Adding the new cross cluster config for $NAME"
-    curl -XPUT http://localhost:9200/_cluster/settings -H'Content-Type: application/json' -d '{"persistent": {"search": {"remote": {"'$NAME'": {"skip_unavailable": "true", "seeds": ["'$EHOSTNAME':9300"]}}}}}'
-  done </tmp/nodes.txt
-
-  INSTALLEDVERSION=rc.2
 }
 
-rc2_to_rc3() {
+post_to_2.3.90() {
+  # Do Kibana dashboard things
+  salt-call state.apply kibana.so_savedobjects_defaults queue=True
 
-  # move location of local.rules
+  # Create FleetDM service account
-  cp /opt/so/saltstack/default/salt/idstools/localrules/local.rules /opt/so/saltstack/local/salt/idstools/local.rules
+    FLEET_MANAGER=$(lookup_pillar fleet_manager)
+    if [[ "$FLEET_MANAGER" == "True" ]]; then
+      FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
+      FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+      MYSQL_PW=$(lookup_pillar_secret mysql)
 
-  if [ -f /opt/so/saltstack/local/salt/idstools/localrules/local.rules ]; then
+      FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_SA_PW'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
-    cat /opt/so/saltstack/local/salt/idstools/localrules/local.rules >> /opt/so/saltstack/local/salt/idstools/local.rules
+      MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PW fleet -e \
+      "INSERT INTO users (password,salt,email,name,global_role) VALUES ('$FLEET_HASH','','$FLEET_SA_EMAIL','$FLEET_SA_EMAIL','admin')" 2>&1)
+
+      if [[ $? -eq 0 ]]; then
+          echo "Successfully added service account to Fleet"
+      else
+          echo "Unable to add service account to Fleet"
+          echo "$MYSQL_OUTPUT"
       fi
-  rm -rf /opt/so/saltstack/local/salt/idstools/localrules
-  rm -rf /opt/so/saltstack/default/salt/idstools/localrules
-
-  # Rename mdengine to MDENGINE
-  sed -i "s/  zeekversion/  mdengine/g" /opt/so/saltstack/local/pillar/global.sls
-  # Enable Strelka Rules
-  sed -i "/  rules:/c\  rules: 1" /opt/so/saltstack/local/pillar/global.sls
-
-  INSTALLEDVERSION=rc.3
-
-}
-
-rc3_to_2.3.0() {
-  # Fix Tab Complete
-  if [ ! -f /etc/profile.d/securityonion.sh ]; then
-    echo "complete -cf sudo" > /etc/profile.d/securityonion.sh
     fi
 
-  {
-    echo "redis_settings:"
-    echo "  redis_maxmemory: 827"
-    echo "playbook:"
-    echo "  api_key: de6639318502476f2fa5aa06f43f51fb389a3d7f" 
-  } >> /opt/so/saltstack/local/pillar/global.sls
   
-  sed -i 's/playbook:/playbook_db:/' /opt/so/saltstack/local/pillar/secrets.sls
-  {
-    echo "playbook_admin: $(get_random_value)"
-    echo "playbook_automation: $(get_random_value)"
-  } >> /opt/so/saltstack/local/pillar/secrets.sls
 
-  INSTALLEDVERSION=2.3.0
+  POSTVERSION=2.3.90
 }
 
-up_2.3.0_to_2.3.20(){
+
+up_to_2.3.20(){
   DOCKERSTUFFBIP=$(echo $DOCKERSTUFF | awk -F'.' '{print $1,$2,$3,1}' OFS='.')/24
   # Remove PCAP from global
   sed '/pcap:/d' /opt/so/saltstack/local/pillar/global.sls
@@ -414,7 +502,7 @@ up_2.3.0_to_2.3.20(){
   INSTALLEDVERSION=2.3.20
 }
 
-up_2.3.2X_to_2.3.30() {
+up_to_2.3.30() {
   # Replace any curly brace scalars with the same scalar in single quotes
   readarray -t minion_pillars <<< "$(find /opt/so/saltstack/local/pillar/minions -type f -name '*.sls')"
   for pillar in "${minion_pillars[@]}"; do
@@ -426,7 +514,7 @@ up_2.3.2X_to_2.3.30() {
   sed -i "/  imagerepo: securityonion/c\  imagerepo: 'security-onion-solutions'" /opt/so/saltstack/local/pillar/global.sls 
 
   # Strelka rule repo pillar addition
-  if [ $is_airgap -eq 0 ]; then
+  if [[ $is_airgap -eq 0 ]]; then
       # Add manager as default Strelka YARA rule repo
       sed -i "/^strelka:/a \\  repos: \n    - https://$HOSTNAME/repo/rules/strelka" /opt/so/saltstack/local/pillar/global.sls;
   else
@@ -437,32 +525,7 @@ up_2.3.2X_to_2.3.30() {
   INSTALLEDVERSION=2.3.30
 }
 
-up_2.3.3X_to_2.3.50_repo() {
+up_to_2.3.50() {
-  echo "Performing 2.3.50 repo actions."
-  if [[ "$OS" == "centos" ]]; then
-    # Import GPG Keys
-    gpg_rpm_import
-    echo "Disabling fastestmirror."
-    disable_fastestmirror
-    echo "Deleting unneeded repo files."
-    DELREPOS=('CentOS-Base' 'CentOS-CR' 'CentOS-Debuginfo' 'docker-ce' 'CentOS-fasttrack' 'CentOS-Media' 'CentOS-Sources' 'CentOS-Vault' 'CentOS-x86_64-kernel' 'epel' 'epel-testing' 'saltstack' 'wazuh')
-
-    for DELREPO in "${DELREPOS[@]}"; do
-      if [[ -f "/etc/yum.repos.d/$DELREPO.repo" ]]; then
-        echo "Deleting $DELREPO.repo"
-        rm -f "/etc/yum.repos.d/$DELREPO.repo"
-      fi
-    done
-    if [ $is_airgap -eq 1 ]; then
-      # Copy the new repo file if not airgap
-      cp $UPDATE_DIR/salt/repo/client/files/centos/securityonion.repo /etc/yum.repos.d/
-      yum clean all
-      yum repolist
-    fi
-  fi
-}
-
-up_2.3.3X_to_2.3.50() {
   
   cat <<EOF > /tmp/supersed.txt
 /so-zeek:/ {
@@ -494,6 +557,89 @@ EOF
   INSTALLEDVERSION=2.3.50
 }
 
+up_to_2.3.80() {
+  
+  # Remove watermark settings from global.sls
+  sed -i '/  cluster_routing_allocation_disk/d' /opt/so/saltstack/local/pillar/global.sls
+
+  # Add new indices to the global
+  sed -i '/  index_settings:/a \\    so-elasticsearch: \n      shards: 1 \n      warm: 7 \n      close: 30 \n      delete: 365' /opt/so/saltstack/local/pillar/global.sls
+  sed -i '/  index_settings:/a \\    so-logstash: \n      shards: 1 \n      warm: 7 \n      close: 30 \n      delete: 365' /opt/so/saltstack/local/pillar/global.sls
+  sed -i '/  index_settings:/a \\    so-kibana: \n      shards: 1 \n      warm: 7 \n      close: 30 \n      delete: 365' /opt/so/saltstack/local/pillar/global.sls
+  sed -i '/  index_settings:/a \\    so-redis: \n      shards: 1 \n      warm: 7 \n      close: 30 \n      delete: 365' /opt/so/saltstack/local/pillar/global.sls
+  
+  # Do some pillar formatting
+  tc=$(grep -w true_cluster /opt/so/saltstack/local/pillar/global.sls | awk -F: {'print tolower($2)'}| xargs)
+
+  if [[ "$tc" == "true" ]]; then
+    tcname=$(grep -w true_cluster_name /opt/so/saltstack/local/pillar/global.sls | awk -F: {'print $2'})
+    sed -i "/^elasticsearch:/a \\  config: \n    cluster: \n      name: $tcname" /opt/so/saltstack/local/pillar/global.sls
+    sed -i '/  true_cluster_name/d' /opt/so/saltstack/local/pillar/global.sls
+    sed -i '/  esclustername/d' /opt/so/saltstack/local/pillar/global.sls
+
+    for file in /opt/so/saltstack/local/pillar/minions/*.sls; do
+      if [[ ${file} != *"manager.sls"* ]]; then
+        noderoutetype=$(grep -w node_route_type $file | awk -F: {'print $2'})
+        if [ -n "$noderoutetype" ]; then
+          sed -i "/^elasticsearch:/a \\  config: \n    node: \n      attr: \n        box_type: $noderoutetype" $file
+          sed -i '/  node_route_type/d' $file
+          noderoutetype=''
+        fi
+      fi
+    done
+  fi
+
+  # check for local es config to inform user that the config in local is now ignored and those options need to be placed in the pillar
+  if [ -f "/opt/so/saltstack/local/salt/elasticsearch/files/elasticsearch.yml" ]; then
+    NOTIFYCUSTOMELASTICCONFIG=true
+  fi
+
+  INSTALLEDVERSION=2.3.80
+}
+
+up_to_2.3.90() {
+  for i in manager managersearch eval standalone; do
+    if compgen -G "/opt/so/saltstack/local/pillar/minions/*_$i.sls" > /dev/null; then
+      echo "soc:" >> /opt/so/saltstack/local/pillar/minions/*_$i.sls
+      sed -i "/^soc:/a \\  es_index_patterns: '*:so-*,*:endgame-*'" /opt/so/saltstack/local/pillar/minions/*_$i.sls
+    fi
+  done
+  
+  # Create Endgame Hostgroup
+  so-firewall addhostgroup endgame
+  
+  # Force influx to generate a new cert
+  mv /etc/pki/influxdb.crt /etc/pki/influxdb.crt.2390upgrade
+  mv /etc/pki/influxdb.key /etc/pki/influxdb.key.2390upgrade
+  
+  # remove old common ingest pipeline in default
+  rm -vf /opt/so/saltstack/default/salt/elasticsearch/files/ingest/common
+  # if custom common, move from local ingest to local ingest-dynamic
+  mkdir -vp /opt/so/saltstack/local/salt/elasticsearch/files/ingest-dynamic
+  if [[ -f "/opt/so/saltstack/local/salt/elasticsearch/files/ingest/common" ]]; then
+    mv -v /opt/so/saltstack/local/salt/elasticsearch/files/ingest/common /opt/so/saltstack/local/salt/elasticsearch/files/ingest-dynamic/common
+    # since json file, we need to wrap with raw
+    sed -i '1s/^/{{'{% raw %}'}}\n/' /opt/so/saltstack/local/salt/elasticsearch/files/ingest-dynamic/common
+    sed -i -e '$a{{'{% endraw %}'}}\n' /opt/so/saltstack/local/salt/elasticsearch/files/ingest-dynamic/common
+  fi
+
+  # Generate FleetDM Service Account creds if they do not exist
+  if grep -q "fleet_sa_email" /opt/so/saltstack/local/pillar/secrets.sls; then
+    echo "FleetDM Service Account credentials already created..."
+  else
+    echo "Generating FleetDM Service Account credentials..."
+    FLEETSAPASS=$(get_random_value)
+    	printf '%s\n'\
+        "  fleet_sa_email: service.account@securityonion.invalid"\
+        "  fleet_sa_password: $FLEETSAPASS"\
+        >> /opt/so/saltstack/local/pillar/secrets.sls
+
+  fi
+  
+  INSTALLEDVERSION=2.3.90
+}
+
+
 verify_upgradespace() {
   CURRENTSPACE=$(df -BG / | grep -v Avail | awk '{print $4}' | sed 's/.$//')
   if [ "$CURRENTSPACE" -lt "10" ]; then
@@ -509,7 +655,7 @@ upgrade_space() {
     clean_dockers
     if ! verify_upgradespace; then
       echo "There is not enough space to perform the upgrade. Please free up space and try again"
-      exit 1
+      exit 0
     fi
   else
       echo "You have enough space for upgrade. Proceeding with soup."
@@ -534,8 +680,8 @@ thehive_maint() {
   done
   if [ "$THEHIVE_CONNECTED" == "yes" ]; then
     echo "Migrating thehive databases if needed."
-    curl -v -k -XPOST -L "https://localhost/thehive/api/maintenance/migrate"
+    curl -v -k -XPOST -L "https://localhost/thehive/api/maintenance/migrate" >> "$SOUP_LOG" 2>&1
-    curl -v -k -XPOST -L "https://localhost/cortex/api/maintenance/migrate"
+    curl -v -k -XPOST -L "https://localhost/cortex/api/maintenance/migrate" >> "$SOUP_LOG" 2>&1
   fi
 }
 
@@ -569,7 +715,7 @@ upgrade_check() {
   # Let's make sure we actually need to update.
   NEWVERSION=$(cat $UPDATE_DIR/VERSION)
   HOTFIXVERSION=$(cat $UPDATE_DIR/HOTFIX)
-  CURRENTHOTFIX=$(cat /etc/sohotfix 2>/dev/null)
+  [[ -f /etc/sohotfix ]] && CURRENTHOTFIX=$(cat /etc/sohotfix)
   if [ "$INSTALLEDVERSION" == "$NEWVERSION" ]; then
     echo "Checking to see if there are hotfixes needed"
     if [ "$HOTFIXVERSION" == "$CURRENTHOTFIX" ]; then
@@ -586,13 +732,14 @@ upgrade_check() {
 }
 
 upgrade_check_salt() {
-  NEWSALTVERSION=$(grep version: $UPDATE_DIR/salt/salt/master.defaults.yaml | awk {'print $2'})
+  NEWSALTVERSION=$(grep version: $UPDATE_DIR/salt/salt/master.defaults.yaml | awk '{print $2}')
   if [ "$INSTALLEDSALTVERSION" == "$NEWSALTVERSION" ]; then
     echo "You are already running the correct version of Salt for Security Onion."
   else
     UPGRADESALT=1
   fi
 }   
+
 upgrade_salt() {
   SALTUPGRADED=True
   echo "Performing upgrade of Salt from $INSTALLEDSALTVERSION to $NEWSALTVERSION."
@@ -604,7 +751,11 @@ upgrade_salt() {
     yum versionlock delete "salt-*"
     echo "Updating Salt packages and restarting services."
     echo ""
-    sh $UPDATE_DIR/salt/salt/scripts/bootstrap-salt.sh -r -F -M -x python3 stable "$NEWSALTVERSION"
+    set +e
+    run_check_net_err \
+    "sh $UPDATE_DIR/salt/salt/scripts/bootstrap-salt.sh -r -F -M -x python3 stable \"$NEWSALTVERSION\"" \
+    "Could not update salt, please check $SOUP_LOG for details."
+    set -e
     echo "Applying yum versionlock for Salt."
     echo ""
     yum versionlock add "salt-*"
@@ -617,7 +768,11 @@ upgrade_salt() {
     apt-mark unhold "salt-minion"
     echo "Updating Salt packages and restarting services."
     echo ""
-    sh $UPDATE_DIR/salt/salt/scripts/bootstrap-salt.sh -F -M -x python3 stable "$NEWSALTVERSION"
+    set +e
+    run_check_net_err \
+    "sh $UPDATE_DIR/salt/salt/scripts/bootstrap-salt.sh -F -M -x python3 stable \"$NEWSALTVERSION\"" \
+    "Could not update salt, please check $SOUP_LOG for details."
+    set -e
     echo "Applying apt hold for Salt."
     echo ""
     apt-mark hold "salt-common"
@@ -626,6 +781,31 @@ upgrade_salt() {
   fi
 }
 
+upgrade_to_2.3.50_repo() {
+  echo "Performing repo changes."
+  if [[ "$OS" == "centos" ]]; then
+    # Import GPG Keys
+    gpg_rpm_import
+    echo "Disabling fastestmirror."
+    disable_fastestmirror
+    echo "Deleting unneeded repo files."
+    DELREPOS=('CentOS-Base' 'CentOS-CR' 'CentOS-Debuginfo' 'docker-ce' 'CentOS-fasttrack' 'CentOS-Media' 'CentOS-Sources' 'CentOS-Vault' 'CentOS-x86_64-kernel' 'epel' 'epel-testing' 'saltstack' 'wazuh')
+
+    for DELREPO in "${DELREPOS[@]}"; do
+      if [[ -f "/etc/yum.repos.d/$DELREPO.repo" ]]; then
+        echo "Deleting $DELREPO.repo"
+        rm -f "/etc/yum.repos.d/$DELREPO.repo"
+      fi
+    done
+    if [[ $is_airgap -eq 1 ]]; then
+      # Copy the new repo file if not airgap
+      cp $UPDATE_DIR/salt/repo/client/files/centos/securityonion.repo /etc/yum.repos.d/
+      yum clean all
+      yum repolist
+    fi
+  fi
+}
+
 verify_latest_update_script() {
   # Check to see if the update scripts match. If not run the new one.
   CURRENTSOUP=$(md5sum /opt/so/saltstack/default/salt/common/tools/sbin/soup | awk '{print $1}')
@@ -642,7 +822,7 @@ verify_latest_update_script() {
     cp $UPDATE_DIR/salt/common/tools/sbin/soup $DEFAULT_SALT_DIR/salt/common/tools/sbin/
     cp $UPDATE_DIR/salt/common/tools/sbin/so-common $DEFAULT_SALT_DIR/salt/common/tools/sbin/
     cp $UPDATE_DIR/salt/common/tools/sbin/so-image-common $DEFAULT_SALT_DIR/salt/common/tools/sbin/
-    salt-call state.apply common queue=True
+    salt-call state.apply -l info common queue=True
     echo ""
     echo "soup has been updated. Please run soup again."
     exit 0
@@ -650,50 +830,41 @@ verify_latest_update_script() {
 }
 
 main() {
-echo "### Preparing soup at `date` ###"
+  trap 'check_err $?' EXIT
-while getopts ":b" opt; do
-  case "$opt" in
-    b ) # process option b
-       shift
-       BATCHSIZE=$1
-       if ! [[ "$BATCHSIZE" =~ ^[0-9]+$ ]]; then
-         echo "Batch size must be a number greater than 0."
-         exit 1
-       fi
-    ;;
-    \? ) 
-      echo "Usage: cmd [-b]"
-    ;;
-  esac
-done
   
+  check_pillar_items
+
+  echo "Checking to see if this is an airgap install."
+  echo ""
+  check_airgap
+  if [[ $is_airgap -eq 0 && $UNATTENDED == true && -z $ISOLOC ]]; then
+    echo "Missing file argument (-f <FILENAME>) for unattended airgap upgrade."
+    exit 0
+  fi
   echo "Checking to see if this is a manager."
   echo ""
   require_manager
   set_minionid
-echo "Checking to see if this is an airgap install"
-echo ""
-check_airgap
   echo "Found that Security Onion $INSTALLEDVERSION is currently installed."
   echo ""
-set_os
+  if [[ $is_airgap -eq 0 ]]; then
-set_palette
-check_elastic_license
-echo ""
-if [ $is_airgap -eq 0 ]; then
     # Let's mount the ISO since this is airgap
     airgap_mounted
   else
     echo "Cloning Security Onion github repo into $UPDATE_DIR."
     echo "Removing previous upgrade sources."
     rm -rf $UPDATE_DIR
+    echo "Cloning the Security Onion Repo."
     clone_to_tmp
   fi
-check_os_updates
-echo ""
   echo "Verifying we have the latest soup script."
   verify_latest_update_script
   echo ""
+  set_os
+  set_palette
+  check_elastic_license
+  echo ""
+  check_os_updates
 
   echo "Generating new repo archive"
   generate_and_clean_tarballs
@@ -709,7 +880,7 @@ upgrade_space
 
   echo "Checking for Salt Master and Minion updates."
   upgrade_check_salt
-
+  set -e
 
   if [ "$is_hotfix" == "true" ]; then
     echo "Applying $HOTFIXVERSION" 
@@ -717,37 +888,40 @@ if [ "$is_hotfix" == "true" ]; then
     echo ""
     update_version
     salt-call state.highstate -l info queue=True
-  
   else
     echo ""
     echo "Performing upgrade from Security Onion $INSTALLEDVERSION to Security Onion $NEWVERSION."
     echo ""
 
     echo "Updating dockers to $NEWVERSION."
-  if [ $is_airgap -eq 0 ]; then
+    if [[ $is_airgap -eq 0 ]]; then
       airgap_update_dockers
       update_centos_repo
       yum clean all
       check_os_updates
     else
       update_registry
+      set +e
       update_docker_containers "soup"
+      set -e
     fi
 
     echo ""
     echo "Stopping Salt Minion service."
     systemctl stop salt-minion
     echo "Killing any remaining Salt Minion processes."
+    set +e
     pkill -9 -ef /usr/bin/salt-minion
+    set -e
     echo ""
     echo "Stopping Salt Master service."
     systemctl stop salt-master
     echo ""
 
-  preupgrade_changes_2.3.50_repo
+    upgrade_to_2.3.50_repo
 
     # Does salt need upgraded. If so update it.
-  if [ "$UPGRADESALT" == "1" ]; then
+    if [[ $UPGRADESALT -eq 1 ]]; then
       echo "Upgrading Salt"
       # Update the repo files so it can actually upgrade
       upgrade_salt
@@ -756,13 +930,13 @@ else
     echo "Checking if Salt was upgraded."
     echo ""
     # Check that Salt was upgraded
-  SALTVERSIONPOSTUPGRADE=$(salt --versions-report | grep Salt: | awk {'print $2'})
+    SALTVERSIONPOSTUPGRADE=$(salt --versions-report | grep Salt: | awk '{print $2}')
     if [[ "$SALTVERSIONPOSTUPGRADE" != "$NEWSALTVERSION" ]]; then
       echo "Salt upgrade failed. Check of indicators of failure in $SOUP_LOG."
       echo "Once the issue is resolved, run soup again."
       echo "Exiting."
       echo ""
-    exit 1
+      exit 0
     else
       echo "Salt upgrade success."
       echo ""
@@ -771,13 +945,13 @@ else
     preupgrade_changes
     echo ""
 
-  if [ $is_airgap -eq 0 ]; then
+    if [[ $is_airgap -eq 0 ]]; then
       echo "Updating Rule Files to the Latest."
       update_airgap_rules
     fi
 
     # Only update the repo if its airgap
-  if [[ $is_airgap -eq 0 ]] && [[ "$UPGRADESALT" != "1" ]]; then
+    if [[ $is_airgap -eq 0 && $UPGRADESALT -ne 1 ]]; then
       update_centos_repo
     fi
 
@@ -795,6 +969,17 @@ else
     echo "Starting Salt Master service."
     systemctl start salt-master
 
+    # Testing that salt-master is up by checking that is it connected to itself
+    set +e
+    echo "Waiting on the Salt Master service to be ready."
+    salt-call state.show_top -l error queue=True || fail "salt-master could not be reached. Check $SOUP_LOG for details."
+    set -e
+
+    echo ""
+    echo "Ensuring python modules for Salt are installed and patched."
+    salt-call state.apply salt.python3-influxdb -l info queue=True
+    echo ""
+
     # Only regenerate osquery packages if Fleet is enabled
     FLEET_MANAGER=$(lookup_pillar fleet_manager)
     FLEET_NODE=$(lookup_pillar fleet_node)
@@ -807,9 +992,9 @@ else
 
     echo ""
     echo "Running a highstate to complete the Security Onion upgrade on this manager. This could take several minutes."
+    set +e
     salt-call state.highstate -l info queue=True
-  echo ""
+    set -e
-  echo "Upgrade from $INSTALLEDVERSION to $NEWVERSION complete."
 
     echo ""
     echo "Stopping Salt Master to remove ACL"
@@ -820,14 +1005,28 @@ else
     echo ""
     echo "Starting Salt Master service."
     systemctl start salt-master
+
+    set +e
+    echo "Waiting on the Salt Master service to be ready."
+    salt-call state.show_top -l error queue=True || fail "salt-master could not be reached. Check $SOUP_LOG for details."
+    set -e
+    
     echo "Running a highstate. This could take several minutes."
     salt-call state.highstate -l info queue=True
     postupgrade_changes
-  unmount_update
+    [[ $is_airgap -eq 0 ]] && unmount_update
     thehive_maint
 
-  if [ "$UPGRADESALT" == "1" ]; then
+    echo ""
-    if [ $is_airgap -eq 0 ]; then
+    echo "Upgrade to $NEWVERSION complete."
+
+    # Everything beyond this is post-upgrade checking, don't fail past this point if something here causes an error
+    set +e
+
+    echo "Checking the number of minions."
+    NUM_MINIONS=$(ls /opt/so/saltstack/local/pillar/minions/*_*.sls | wc -l)
+    if [[ $UPGRADESALT -eq 1 ]] && [[ $NUM_MINIONS -gt 1 ]]; then
+      if [[ $is_airgap -eq 0 ]]; then
         echo ""
         echo "Cleaning repos on remote Security Onion nodes."
         salt -C 'not *_eval and not *_helixsensor and not *_manager and not *_managersearch and not *_standalone and G@os:CentOS' cmd.run "yum clean all"
@@ -835,8 +1034,15 @@ else
       fi
     fi
 
+    echo "Checking for local modifications."
+    check_local_mods
+
+    echo "Checking sudoers file."
     check_sudoers
 
+    echo "Checking for necessary user migrations."
+    so-user migrate
+
     if [[ -n $lsl_msg ]]; then
       case $lsl_msg in
         'distributed')
@@ -853,9 +1059,7 @@ else
       esac
     fi
 
-  NUM_MINIONS=$(ls /opt/so/saltstack/local/pillar/minions/*_*.sls | wc -l)
+    if [[ $NUM_MINIONS -gt 1 ]]; then
-
-  if [ $NUM_MINIONS -gt 1 ]; then
 
       cat << EOF
     
@@ -874,9 +1078,55 @@ EOF
     fi
   fi
 
-echo "### soup has been served at `date` ###"
+  if [ "$NOTIFYCUSTOMELASTICCONFIG" = true ] ; then
+
+    cat << EOF
+
+
+A custom Elasticsearch configuration has been found at /opt/so/saltstack/local/elasticsearch/files/elasticsearch.yml. This file is no longer referenced in Security Onion versions >= 2.3.80.
+
+If you still need those customizations, you'll need to manually migrate them to the new Elasticsearch config as shown at https://docs.securityonion.net/en/2.3/elasticsearch.html.
+
+EOF
+
+  fi
+
+  echo "### soup has been served at $(date) ###"
 }
 
+while getopts ":b:f:y" opt; do
+  case ${opt} in
+    b )
+      BATCHSIZE="$OPTARG"
+      if ! [[ "$BATCHSIZE" =~ ^[1-9][0-9]*$ ]]; then
+        echo "Batch size must be a number greater than 0."
+        exit 1
+      fi
+    ;;
+    y )
+      if [[ ! -f /opt/so/state/yeselastic.txt ]]; then
+        echo "Cannot run soup in unattended mode. You must run soup manually to accept the Elastic License."
+        exit 1
+      else
+        UNATTENDED=true 
+      fi
+    ;;
+    f )
+      ISOLOC="$OPTARG"
+    ;;
+    \? ) 
+      echo "Usage: soup [-b] [-y] [-f <iso location>]"
+      exit 1
+    ;;
+    : )
+      echo "Invalid option: $OPTARG requires an argument"
+      exit 1
+    ;;
+  esac
+done
+shift $((OPTIND - 1))
+
+if [[ -z $UNATTENDED ]]; then
   cat << EOF
 
 SOUP - Security Onion UPdater
@@ -889,6 +1139,9 @@ Press Enter to continue or Ctrl-C to cancel.
 
 EOF
 
-read input
+  read -r input
+fi
 
+echo "### Preparing soup at $(date) ###"
 main "$@" | tee -a $SOUP_LOG
+  

salt/curator/files/action/so-aws-close.yml

@@ -0,0 +1,29 @@
+{%- set cur_close_days = salt['pillar.get']('elasticsearch:index_settings:so-aws:close', 30) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: close
+    description: >-
+      Close aws indices older than {{cur_close_days}} days.
+    options:
+      delete_aliases: False
+      timeout_override:
+      continue_if_exception: False
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex 
+      value: '^(logstash-aws.*|so-aws.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{cur_close_days}}
+      exclude:

salt/curator/files/action/so-aws-delete.yml

@@ -0,0 +1,29 @@
+{%- set DELETE_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-aws:delete', 365) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: delete_indices
+    description: >-
+      Delete aws indices when older than {{ DELETE_DAYS }} days.
+    options:
+      ignore_empty_list: True
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex
+      value: '^(logstash-aws.*|so-aws.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ DELETE_DAYS }}
+      exclude:
+  
+      

salt/curator/files/action/so-aws-warm.yml

@@ -0,0 +1,24 @@
+{%- set WARM_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-aws:warm', 7) -%}
+actions:
+  1:
+    action: allocation
+    description: "Apply shard allocation filtering rules to the specified indices"
+    options:
+      key: box_type
+      value: warm
+      allocation_type: require
+      wait_for_completion: true
+      timeout_override:
+      continue_if_exception: false
+      disable_action: false
+    filters:
+    - filtertype: pattern
+      kind: prefix
+      value: so-aws
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ WARM_DAYS }} 
+

salt/curator/files/action/so-azure-close.yml

@@ -0,0 +1,29 @@
+{%- set cur_close_days = salt['pillar.get']('elasticsearch:index_settings:so-azure:close', 30) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: close
+    description: >-
+      Close azure indices older than {{cur_close_days}} days.
+    options:
+      delete_aliases: False
+      timeout_override:
+      continue_if_exception: False
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex 
+      value: '^(logstash-azure.*|so-azure.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{cur_close_days}}
+      exclude:

salt/curator/files/action/so-azure-delete.yml

@@ -0,0 +1,29 @@
+{%- set DELETE_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-azure:delete', 365) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: delete_indices
+    description: >-
+      Delete azure indices when older than {{ DELETE_DAYS }} days.
+    options:
+      ignore_empty_list: True
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex
+      value: '^(logstash-azure.*|so-azure.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ DELETE_DAYS }}
+      exclude:
+  
+      

salt/curator/files/action/so-azure-warm.yml

@@ -0,0 +1,24 @@
+{%- set WARM_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-azure:warm', 7) -%}
+actions:
+  1:
+    action: allocation
+    description: "Apply shard allocation filtering rules to the specified indices"
+    options:
+      key: box_type
+      value: warm
+      allocation_type: require
+      wait_for_completion: true
+      timeout_override:
+      continue_if_exception: false
+      disable_action: false
+    filters:
+    - filtertype: pattern
+      kind: prefix
+      value: so-azure
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ WARM_DAYS }} 
+

salt/curator/files/action/so-barracuda-close.yml

@@ -0,0 +1,29 @@
+{%- set cur_close_days = salt['pillar.get']('elasticsearch:index_settings:so-barracuda:close', 30) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: close
+    description: >-
+      Close barracuda indices older than {{cur_close_days}} days.
+    options:
+      delete_aliases: False
+      timeout_override:
+      continue_if_exception: False
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex 
+      value: '^(logstash-barracuda.*|so-barracuda.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{cur_close_days}}
+      exclude:

salt/curator/files/action/so-barracuda-delete.yml

@@ -0,0 +1,29 @@
+{%- set DELETE_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-barracuda:delete', 365) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: delete_indices
+    description: >-
+      Delete barracuda indices when older than {{ DELETE_DAYS }} days.
+    options:
+      ignore_empty_list: True
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex
+      value: '^(logstash-barracuda.*|so-barracuda.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ DELETE_DAYS }}
+      exclude:
+  
+      

salt/curator/files/action/so-barracuda-warm.yml

@@ -0,0 +1,24 @@
+{%- set WARM_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-barracuda:warm', 7) -%}
+actions:
+  1:
+    action: allocation
+    description: "Apply shard allocation filtering rules to the specified indices"
+    options:
+      key: box_type
+      value: warm
+      allocation_type: require
+      wait_for_completion: true
+      timeout_override:
+      continue_if_exception: false
+      disable_action: false
+    filters:
+    - filtertype: pattern
+      kind: prefix
+      value: so-barracuda
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ WARM_DAYS }} 
+

salt/curator/files/action/so-beats-delete.yml

@@ -0,0 +1,29 @@
+{%- set DELETE_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-beats:delete', 365) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: delete_indices
+    description: >-
+      Delete beats indices when older than {{ DELETE_DAYS }} days.
+    options:
+      ignore_empty_list: True
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex
+      value: '^(logstash-beats.*|so-beats.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ DELETE_DAYS }}
+      exclude:
+  
+      

salt/curator/files/action/so-beats-warm.yml

@@ -0,0 +1,24 @@
+{%- set WARM_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-beats:warm', 7) -%}
+actions:
+  1:
+    action: allocation
+    description: "Apply shard allocation filtering rules to the specified indices"
+    options:
+      key: box_type
+      value: warm
+      allocation_type: require
+      wait_for_completion: true
+      timeout_override:
+      continue_if_exception: false
+      disable_action: false
+    filters:
+    - filtertype: pattern
+      kind: prefix
+      value: so-beats
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ WARM_DAYS }} 
+

salt/curator/files/action/so-bluecoat-close.yml

@@ -0,0 +1,29 @@
+{%- set cur_close_days = salt['pillar.get']('elasticsearch:index_settings:so-bluecoat:close', 30) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: close
+    description: >-
+      Close bluecoat indices older than {{cur_close_days}} days.
+    options:
+      delete_aliases: False
+      timeout_override:
+      continue_if_exception: False
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex 
+      value: '^(logstash-bluecoat.*|so-bluecoat.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{cur_close_days}}
+      exclude:

salt/curator/files/action/so-bluecoat-delete.yml

@@ -0,0 +1,29 @@
+{%- set DELETE_DAYS = salt['pillar.get']('elasticsearch:index_settings:so-bluecoat:delete', 365) -%}
+---
+# Remember, leave a key empty if there is no value.  None will be a string,
+# not a Python "NoneType"
+#
+# Also remember that all examples have 'disable_action' set to True.  If you
+# want to use this action as a template, be sure to set this to False after
+# copying it.
+actions:
+  1:
+    action: delete_indices
+    description: >-
+      Delete bluecoat indices when older than {{ DELETE_DAYS }} days.
+    options:
+      ignore_empty_list: True
+      disable_action: False
+    filters:
+    - filtertype: pattern
+      kind: regex
+      value: '^(logstash-bluecoat.*|so-bluecoat.*)$'
+    - filtertype: age
+      source: name
+      direction: older
+      timestring: '%Y.%m.%d'
+      unit: days
+      unit_count: {{ DELETE_DAYS }}
+      exclude:
+  
+