Merge pull request #6335 from Security-Onion-Solutions/hotfix/2.3.90

Hotfix/2.3.90

.github/ISSUE_TEMPLATE

@@ -0,0 +1,12 @@
+PLEASE STOP AND READ THIS INFORMATION!
+
+If you are creating an issue just to ask a question, you will likely get faster and better responses by posting to our discussions forum instead:
+https://securityonion.net/discuss
+
+If you think you have found a possible bug or are observing a behavior that you weren't expecting, use the discussion forum to start a conversation about it instead of creating an issue.
+
+If you are very familiar with the latest version of the product and are confident you have found a bug in Security Onion, you can continue with creating an issue here, but please make sure you have done the following:
+- duplicated the issue on a fresh installation of the latest version
+- provide information about your system and how you installed Security Onion
+- include relevant log files
+- include reproduction steps

.github/workflows/leaktest.yml

@@ -0,0 +1,15 @@
+name: leak-test
+
+on: [pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        fetch-depth: '0'
+
+    - name: Gitleaks
+      uses: zricethezav/gitleaks-action@master

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Contributing to Security Onion
+
+### Questions, suggestions, and general comments
+* Security Onion uses GitHub's [Discussions](https://github.com/Security-Onion-Solutions/securityonion/discussions) to provide a forum where the community and developers can interact as well as ask and answer questions.
+
+### Reporting a bug
+* The primary place to report unexpected behavior or possible bugs is the repo's [Discussions forum](https://github.com/Security-Onion-Solutions/securityonion/discussions).
+
+*  **If you are familiar with the current version of Security Onion and are confident you've discovered a bug**, first ensure there is not already an issue present by searching the open [issues](https://github.com/Security-Onion-Solutions/securityonion/issues). If there is, a thumbs up :+1: is a great way to show this bug is affecting you too.
+
+* If an issue doesn't exist, [open a new one](https://github.com/Security-Onion-Solutions/securityonion/issues/new), following the directions in the issue template. This means including:
+  * **System information** and how Security Onion was installed
+  * **Log files** relevant to the bug report
+  * **Reproduction steps** 
+
+### Contributing code
+
+* **All commits must be signed** with a valid key that has been added to your GitHub account. Each commit should have the "**Verified**" tag when viewed on GitHub as shown below:
+  
+  <img src="./assets/images/verified-commit-1.png" width="450">
+
+* If an issue does not already exist for the bug or feature for which you are submitting a pull request, [create one](https://github.com/Security-Onion-Solutions/securityonion/issues/new) with the relevant prefix. (**`FIX:`** for bug fixes, **`FEATURE:`** for new features.)
+
+* Link the PR to the related issue, either using [keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) in the PR description, or [manually](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue).
+
+* **Pull requests should be opened against the `dev` branch of this repo**, and should clearly describe the problem and solution.
+
+* Be sure you have tested your changes and are confident they will not break other parts of the product.
+
+* See this document's [code styling and conventions section](#code-style-and-conventions) below to be sure your PR fits our code requirements prior to submitting.
+
+
+
+### Code style and conventions
+* **Keep code [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)**. For example, Bash code used by multiple scripts will likely best be added to <span style="white-space: nowrap;">[`so-common`](salt/common/tools/sbin/so-common)</span>.
+
+* All new Bash code should pass [ShellCheck](https://www.shellcheck.net/) analysis. Where errors can be *safely* [ignored](https://github.com/koalaman/shellcheck/wiki/Ignore), the relevant disable directive should be accompanied by a brief explanation as to why the error is being ignored.
+
+* **Ensure all YAML (this includes Salt states and pillars) is properly formatted**. The spec for YAML v1.2 can be found [here](https://yaml.org/spec/1.2/spec.html), however there are numerous online resources with simpler descriptions of its formatting rules. 

HOTFIX

@@ -0,0 +1 @@
+WAZUH

KEYS

@@ -1,4 +1,5 @@
 -----BEGIN PGP PUBLIC KEY BLOCK-----
+
 mQINBF7rzwEBEADBg87uJhnC3Ls7s60hbHGaywGrPtbz2WuYA/ev3YS3X7WS75p8
 PGlzTWUCujx0pEHbK2vYfExl3zksZ8ZmLyZ9VB3oSLiWBzJgKAeB7YCFEo8te+eE
 P2Z+8c+kX4eOV+2waxZyewA2TipSkhWgStSI4Ow8SyVUcUWA3hCw7mo2duNVi7KO

README.md

@@ -1,7 +1,14 @@
-## Security Onion 2.3.2
+## Security Onion 2.3.90-WAZUH
 
-Security Onion 2.3.2 is here!
+Security Onion 2.3.90-WAZUH is here!
 
+## Screenshots
+
+Alerts
+![Alerts](./assets/images/screenshots/alerts-1.png)
+
+Hunt
+![Hunt](./assets/images/screenshots/hunt-1.png)
 
 ### Release Notes
 

SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x.x   | :white_check_mark: |
+| 16.04.x | :x:                |
+
+Security Onion 16.04 has reached End Of Life and is no longer supported.
+
+## Reporting a Vulnerability
+
+If you have any security concerns regarding Security Onion or believe you have uncovered a vulnerability, please follow these steps:
+
+- send an email to security@securityonion.net
+- include a description of the issue and steps to reproduce
+- please use plain text format (no Word documents or PDF files)
+- please do not disclose publicly until we have had sufficient time to resolve the issue
+
+This security address should be used only for undisclosed vulnerabilities. Dealing with fixed issues or general questions on how to use Security Onion should be handled via the normal support channels.

VERIFY_ISO.md

@@ -1,16 +1,18 @@
-### 2.3.2 ISO image built on 2020/10/25
+### 2.3.90-WAZUH ISO image built on 2021/11/23
+
+
 
 ### Download and Verify
 
-2.3.2 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-2.3.2.iso
+2.3.90-WAZUH ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-2.3.90-WAZUH.iso
 
-MD5: EF2DEBCCBAE0B0BCCC906552B5FF918A  
-SHA1: 16AFCACB102BD217A038044D64E7A86DA351640E  
-SHA256: 7125F90B6323179D0D29F5745681BE995BD2615E64FA1E0046D94888A72C539E 
+MD5: B7141C8627CDB45F4A8741B2ADE4A9F3  
+SHA1: 16087B385CA651659EC98F139AFDF90922430FB6  
+SHA256: 667AF11BBCFE3248AF59E45043703B55A543E059899AE387FF55EB8077304F04 
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.2.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.90-WAZUH.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS  
@@ -24,22 +26,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/ma
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.2.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.90-WAZUH.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-2.3.2.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-2.3.90-WAZUH.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-2.3.2.iso.sig securityonion-2.3.2.iso
+gpg --verify securityonion-2.3.90-WAZUH.iso.sig securityonion-2.3.90-WAZUH.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Sun 25 Oct 2020 10:44:27 AM EDT using RSA key ID FE507013
+gpg: Signature made Tue 23 Nov 2021 03:19:08 PM EST using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

VERSION

@@ -1 +1 @@
-2.3.2
+2.3.90

files/firewall/hostgroups.local.yaml

@@ -16,6 +16,10 @@ firewall:
       ips:
         delete:
         insert:
+    endgame:
+      ips:
+        delete:
+        insert:
     fleet:
       ips:
         delete:

files/salt/master/master

@@ -13,6 +13,8 @@
 # user: socore
 
 log_file: /opt/so/log/salt/master
+log_level_logfile: info
+log_level: info
 
 #####      File Server settings      #####
 ##########################################
@@ -65,3 +67,7 @@ peer:
 reactor:
   - 'so/fleet':
     - salt://reactor/fleet.sls
+  - 'salt/beacon/*/watch_sqlite_db//opt/so/conf/kratos/db/sqlite.db':
+    - salt://reactor/kratos.sls
+
+

pillar/data/addtotab.sh

@@ -54,7 +54,8 @@ if [ $TYPE == 'evaltab' ] || [ $TYPE == 'standalonetab' ]; then
     salt-call state.apply utility queue=True
   fi
 fi
-#if [ $TYPE == 'nodestab' ]; then
+if [ $TYPE == 'nodestab' ]; then
+  salt-call state.apply elasticsearch queue=True
 #  echo "    nodetype: $NODETYPE" >> $local_salt_dir/pillar/data/$TYPE.sls
 #  echo "    hotname: $HOTNAME" >> $local_salt_dir/pillar/data/$TYPE.sls
-#fi
+fi

pillar/docker/config.sls

@@ -1,208 +0,0 @@
-{%- set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
-{%- set FLEETNODE = salt['pillar.get']('global:fleet_node', False) -%}
-{% set WAZUH = salt['pillar.get']('manager:wazuh', '0') %}
-{% set THEHIVE = salt['pillar.get']('manager:thehive', '0') %}
-{% set PLAYBOOK = salt['pillar.get']('manager:playbook', '0') %}
-{% set FREQSERVER = salt['pillar.get']('manager:freq', '0') %}
-{% set DOMAINSTATS = salt['pillar.get']('manager:domainstats', '0') %}
-{% set ZEEKVER = salt['pillar.get']('global:mdengine', 'COMMUNITY') %}
-{% set GRAFANA = salt['pillar.get']('manager:grafana', '0') %}
-
-eval:
-  containers:
-    - so-nginx
-    - so-telegraf
-    {% if  GRAFANA == '1' %}
-    - so-influxdb
-    - so-grafana
-    {% endif %}
-    - so-dockerregistry
-    - so-soc
-    - so-kratos
-    - so-idstools
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    - so-elasticsearch
-    - so-logstash
-    - so-kibana
-    - so-steno
-    - so-suricata
-    - so-zeek
-    - so-curator
-    - so-elastalert
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-heavy_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-redis
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-steno
-    - so-suricata
-    - so-wazuh
-    - so-filebeat
-    {% if ZEEKVER != 'SURICATA' %}
-    - so-zeek
-    {% endif %}
-helix:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-idstools
-    - so-steno
-    - so-zeek
-    - so-redis
-    - so-logstash
-    - so-filebeat
-hot_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-manager_search:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-soc
-    - so-kratos
-    - so-acng
-    - so-idstools
-    - so-redis
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-kibana
-    - so-elastalert
-    - so-filebeat
-    - so-soctopus
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-manager:
-  containers:
-    - so-dockerregistry
-    - so-nginx
-    - so-telegraf
-    {% if  GRAFANA == '1' %}
-    - so-influxdb
-    - so-grafana
-    {% endif %}
-    - so-soc
-    - so-kratos
-    - so-acng
-    - so-idstools
-    - so-redis
-    - so-elasticsearch
-    - so-logstash
-    - so-kibana
-    - so-elastalert
-    - so-filebeat
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-parser_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-search_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-filebeat
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-sensor:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-steno
-    - so-suricata
-    {% if ZEEKVER != 'SURICATA' %}
-    - so-zeek
-    {% endif %}
-    - so-wazuh
-    - so-filebeat
-warm_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-elasticsearch
-fleet:
-  containers:
-    {% if FLEETNODE %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    - so-filebeat
-    - so-nginx
-    - so-telegraf
-    {% endif %}

pillar/elasticsearch/eval.sls

@@ -1,7 +1,7 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-common-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja
@@ -10,4 +10,4 @@ elasticsearch:
     - so/so-ossec-template.json.jinja
     - so/so-strelka-template.json.jinja
     - so/so-syslog-template.json.jinja
-    - so/so-zeek-template.json.jinja
+    - so/so-zeek-template.json.jinja

pillar/elasticsearch/manager.sls

@@ -1,7 +1,8 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-common-template.json.jinja
+    - so/so-endgame-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/elasticsearch/search.sls

@@ -1,7 +1,8 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-common-template.json.jinja
+    - so/so-endgame-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/firewall/ports.sls

@@ -1,65 +0,0 @@
-firewall:
-  analyst:
-    ports:
-      tcp:
-        - 80
-        - 443
-      udp:
-  beats_endpoint:
-    ports:
-      tcp:
-        - 5044
-  forward_nodes:
-    ports:
-      tcp:
-        - 443
-        - 5044
-        - 5644
-        - 9822
-      udp:
-  manager:
-    ports:
-      tcp:
-        - 1514
-        - 3200
-        - 3306
-        - 4200
-        - 5601
-        - 6379
-        - 7788
-        - 8086
-        - 8090
-        - 9001
-        - 9200
-        - 9300
-        - 9400  
-        - 9500
-        - 9595
-        - 9696
-      udp:
-        - 1514
-  minions:
-    ports:
-      tcp:
-        - 3142
-        - 4505
-        - 4506
-        - 5000
-        - 8080
-        - 8086
-        - 55000      
-  osquery_endpoint:
-    ports:
-      tcp:
-        - 8090
-  search_nodes:
-    ports:
-      tcp:
-        - 6379
-        - 9300
-  wazuh_endpoint:
-    ports:
-      tcp:
-        - 1514
-      udp: 
-        -1514

pillar/logrotate/init.sls

@@ -8,4 +8,6 @@ logrotate:
     create
     extension .log
     dateext
-    dateyesterday
+    dateyesterday
+  group_conf: |
+    su root socore

pillar/logstash/init.sls

@@ -1,6 +1,7 @@
 logstash:
   docker_options:
     port_bindings:
+      - 0.0.0.0:3765:3765
       - 0.0.0.0:5044:5044
       - 0.0.0.0:5644:5644
       - 0.0.0.0:6050:6050

pillar/logstash/manager.sls

@@ -5,5 +5,6 @@ logstash:
       config:
         - so/0009_input_beats.conf      
         - so/0010_input_hhbeats.conf
+        - so/0011_input_endgame.conf
         - so/9999_output_redis.conf.jinja
         

pillar/logstash/search.sls

@@ -7,8 +7,11 @@ logstash:
         - so/9000_output_zeek.conf.jinja
         - so/9002_output_import.conf.jinja
         - so/9034_output_syslog.conf.jinja
-        - so/9100_output_osquery.conf.jinja
+        - so/9050_output_filebeatmodules.conf.jinja
+        - so/9100_output_osquery.conf.jinja  
         - so/9400_output_suricata.conf.jinja
         - so/9500_output_beats.conf.jinja
         - so/9600_output_ossec.conf.jinja
         - so/9700_output_strelka.conf.jinja
+        - so/9800_output_logscan.conf.jinja
+        - so/9900_output_endgame.conf.jinja

pillar/top.sls

@@ -3,7 +3,7 @@ base:
     - patch.needs_restarting
     - logrotate
 
-  '*_eval or *_helix or *_heavynode or *_sensor or *_standalone or *_import':
+  '*_eval or *_helixsensor or *_heavynode or *_sensor or *_standalone or *_import':
     - match: compound
     - zeek
 
@@ -22,6 +22,12 @@ base:
   '*_manager or *_managersearch':
     - match: compound
     - data.*
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - secrets
     - global
     - minions.{{ grains.id }}
@@ -38,6 +44,12 @@ base:
     - secrets
     - healthcheck.eval
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - global
     - minions.{{ grains.id }}
 
@@ -46,6 +58,12 @@ base:
     - logstash.manager
     - logstash.search
     - elasticsearch.search
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - data.*
     - zeeklogs
     - secrets
@@ -59,10 +77,11 @@ base:
 
   '*_heavynode':
     - zeeklogs
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
 
-  '*_helix':
+  '*_helixsensor':
     - fireeye
     - zeeklogs
     - logstash
@@ -80,12 +99,20 @@ base:
     - logstash
     - logstash.search
     - elasticsearch.search
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
+    - data.nodestab
 
   '*_import':
     - zeeklogs
     - secrets
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
+{% endif %}
     - global
-    - minions.{{ grains.id }}
+    - minions.{{ grains.id }}

pillar/zeek/init.sls

@@ -52,5 +52,4 @@ zeek:
       - frameworks/signatures/detect-windows-shells
     redef:
       - LogAscii::use_json = T;
-      - LogAscii::json_timestamps = JSON::TS_ISO8601;
-      - CaptureLoss::watch_interval = 5 mins;
+      - CaptureLoss::watch_interval = 5 mins;

salt/_modules/so.py

@@ -1,4 +1,64 @@
 #!py
 
+import logging
+
 def status():
-  return __salt__['cmd.run']('/usr/sbin/so-status')
+    return __salt__['cmd.run']('/usr/sbin/so-status')
+
+
+def mysql_conn(retry):
+    log = logging.getLogger(__name__)
+
+    from time import sleep
+
+    try:
+        from MySQLdb import _mysql
+    except ImportError as e:
+        log.error(e)
+        return False
+
+    mainint = __salt__['pillar.get']('host:mainint')
+    ip_arr = __salt__['grains.get']('ip4_interfaces').get(mainint)
+
+    mysql_up = False
+
+    if len(ip_arr) == 1:
+        mainip = ip_arr[0]
+
+        if not(retry >= 1):
+            log.debug('`retry` set to value below 1, resetting it to 1 to prevent errors.')
+            retry = 1
+
+        for i in range(0, retry):
+            log.debug(f'Connection attempt {i+1}')
+            try:
+                db = _mysql.connect(
+                    host=mainip,
+                    user='root',
+                    passwd=__salt__['pillar.get']('secrets:mysql')
+                )
+                log.debug(f'Connected to MySQL server on {mainip} after {i+1} attempts.')
+                
+                db.query("""SELECT 1;""")
+                log.debug(f'Successfully completed query against MySQL server on {mainip}')
+                
+                db.close()
+                mysql_up = True
+                break
+            except _mysql.OperationalError as e:
+                log.debug(e)
+            except Exception as e:
+                log.error('Unexpected error occured.')
+                log.error(e)
+                break
+            sleep(1)
+
+        if not mysql_up:
+            log.error(f'Could not connect to MySQL server on {mainip} after {retry} attempts.')
+    else:
+        log.error(f'Main interface {mainint} has more than one IP address assigned to it, which is not supported.')
+        log.debug(f'{mainint}:')
+        for addr in ip_arr:
+            log.debug(f'  - {addr}')
+
+    return mysql_up

salt/airgap/init.sls

@@ -1,60 +0,0 @@
-{% set MANAGER = salt['grains.get']('master') %}
-airgapyum:
-  file.managed:
-    - name: /etc/yum/yum.conf
-    - source: salt://airgap/files/yum.conf
-
-airgap_repo:
-  pkgrepo.managed:
-    - humanname: Airgap Repo
-    - baseurl: https://{{ MANAGER }}/repo
-    - gpgcheck: 0
-    - sslverify: 0
-
-agbase:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Base.repo
-
-agcr:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-CR.repo
-
-agdebug:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Debuginfo.repo
-
-agfasttrack:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-fasttrack.repo
-
-agmedia:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Media.repo
-
-agsources:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Sources.repo
-
-agvault:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Vault.repo
-
-agkernel:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-x86_64-kernel.repo
-
-agepel:
-  file.absent:
-    - name: /etc/yum.repos.d/epel.repo
-
-agtesting:
-  file.absent:
-    - name: /etc/yum.repos.d/epel-testing.repo
-
-agssrepo:
-  file.absent:
-    - name: /etc/yum.repos.d/saltstack.repo
-
-agwazrepo:
-  file.absent:
-    - name: /etc/yum.repos.d/wazuh.repo

salt/allowed_states.map.jinja

@@ -0,0 +1,314 @@
+{% set ZEEKVER = salt['pillar.get']('global:mdengine', '') %}
+{% set WAZUH = salt['pillar.get']('global:wazuh', '0') %}
+{% set THEHIVE = salt['pillar.get']('manager:thehive', '0') %}
+{% set PLAYBOOK = salt['pillar.get']('manager:playbook', '0') %}
+{% set FREQSERVER = salt['pillar.get']('manager:freq', '0') %}
+{% set DOMAINSTATS = salt['pillar.get']('manager:domainstats', '0') %}
+{% set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) %}
+{% set FLEETNODE = salt['pillar.get']('global:fleet_node', False) %}
+{% set ELASTALERT = salt['pillar.get']('elastalert:enabled', True) %}
+{% set ELASTICSEARCH = salt['pillar.get']('elasticsearch:enabled', True) %}
+{% set FILEBEAT = salt['pillar.get']('filebeat:enabled', True) %}
+{% set KIBANA = salt['pillar.get']('kibana:enabled', True) %}
+{% set LOGSTASH = salt['pillar.get']('logstash:enabled', True) %}
+{% set CURATOR = salt['pillar.get']('curator:enabled', True) %}
+{% set REDIS = salt['pillar.get']('redis:enabled', True) %}
+{% set STRELKA = salt['pillar.get']('strelka:enabled', '0') %}
+{% set ISAIRGAP = salt['pillar.get']('global:airgap', False) %}
+{% import_yaml 'salt/minion.defaults.yaml' as saltversion %}
+{% set saltversion = saltversion.salt.minion.version %}
+
+{# this is the list we are returning from this map file, it gets built below #}
+{% set allowed_states= [] %}
+
+{% if grains.saltversion | string == saltversion | string %}
+
+  {% set allowed_states= salt['grains.filter_by']({
+      'so-eval': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'kratos',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'healthcheck',
+          'pcap',
+          'suricata',
+          'utility',
+          'schedule',
+          'soctopus',
+          'tcpreplay',
+          'docker_clean',
+          'learn'
+          ],
+      'so-heavynode': [
+          'ca',
+          'ssl',
+          'nginx',
+          'telegraf',
+          'firewall',
+          'pcap',
+          'suricata',
+          'healthcheck',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-helixsensor': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'telegraf',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'zeek',
+          'redis',
+          'elasticsearch',
+          'logstash',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-fleet': [
+          'ca',
+          'ssl',
+          'nginx',
+          'telegraf',
+          'firewall',
+          'mysql',
+          'redis',
+          'fleet',
+          'fleet.install_package',
+          'filebeat',
+          'schedule',
+          'docker_clean'
+          ],
+      'so-import': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'soc',
+          'kratos',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'pcap',
+          'utility',
+          'suricata',
+          'zeek',
+          'schedule',
+          'tcpreplay',
+          'docker_clean',
+          'learn'
+          ],
+      'so-manager': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'kratos',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'utility',
+          'schedule',
+          'soctopus',
+          'docker_clean',
+          'learn'
+          ],
+      'so-managersearch': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'kratos',
+          'firewall',
+          'manager',
+          'idstools',
+          'suricata.manager',
+          'utility',
+          'schedule',
+          'soctopus',
+          'docker_clean',
+          'learn'
+          ],
+      'so-node': [
+          'ca',
+          'ssl',
+          'nginx',
+          'telegraf',
+          'firewall',
+          'schedule',
+          'docker_clean'
+          ],
+      'so-standalone': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'kratos',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'pcap',
+          'suricata',
+          'healthcheck',
+          'utility',
+          'schedule',
+          'soctopus',
+          'tcpreplay',
+          'docker_clean',
+          'learn'
+          ],
+      'so-sensor': [
+          'ca',
+          'ssl',
+          'telegraf',
+          'firewall',
+          'nginx',
+          'pcap',
+          'suricata',
+          'healthcheck',
+          'wazuh',
+          'filebeat',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+  }, grain='role') %}
+
+  {% if FILEBEAT and grains.role in ['so-helixsensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-import'] %}
+    {% do allowed_states.append('filebeat') %}
+  {% endif %}
+
+  {% if ((FLEETMANAGER or FLEETNODE) or PLAYBOOK != 0) and grains.role in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone'] %}
+    {% do allowed_states.append('mysql') %}
+  {% endif %}
+
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-sensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('fleet.install_package') %}
+  {% endif %}
+
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('fleet') %}
+  {% endif %}
+
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-eval'] %}
+    {% do allowed_states.append('redis') %}
+  {% endif %}
+
+  {%- if ZEEKVER != 'SURICATA' and grains.role in ['so-sensor', 'so-eval', 'so-standalone', 'so-heavynode'] %}
+    {% do allowed_states.append('zeek') %}
+  {%- endif %}
+
+  {% if STRELKA and grains.role in ['so-sensor', 'so-eval', 'so-standalone', 'so-heavynode'] %}
+    {% do allowed_states.append('strelka') %}
+  {% endif %}
+
+  {% if WAZUH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode']%}
+    {% do allowed_states.append('wazuh') %}
+  {% endif %}
+
+  {% if ELASTICSEARCH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-import'] %}
+    {% do allowed_states.append('elasticsearch') %}
+  {% endif %}
+
+  {% if ELASTICSEARCH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    {% do allowed_states.append('elasticsearch.auth') %}
+  {% endif %}
+
+  {% if KIBANA and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    {% do allowed_states.append('kibana') %}
+    {% do allowed_states.append('kibana.secrets') %}
+  {% endif %}
+
+  {% if grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-manager'] %}
+    {% do allowed_states.append('curator') %}
+  {% endif %}
+
+  {% if ELASTALERT and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('elastalert') %}
+  {% endif %}
+
+  {% if (THEHIVE != 0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('thehive') %}
+  {% endif %}
+
+  {% if (PLAYBOOK !=0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('playbook') %}
+  {% endif %}
+
+  {% if (PLAYBOOK !=0) and grains.role in ['so-eval'] %}
+    {% do allowed_states.append('redis') %}
+  {% endif %}
+
+  {% if (FREQSERVER !=0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('freqserver') %}
+  {% endif %}
+
+  {% if (DOMAINSTATS !=0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('domainstats') %}
+  {% endif %}
+
+  {% if LOGSTASH and grains.role in ['so-helixsensor', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('logstash') %}
+  {% endif %}
+
+  {% if REDIS and grains.role in ['so-manager', 'so-standalone', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('redis') %}
+  {% endif %}
+
+  {% if grains.os == 'CentOS' %}
+    {% if not ISAIRGAP %}
+      {% do allowed_states.append('yum') %}
+    {% endif %}
+    {% do allowed_states.append('yum.packages') %}
+  {% endif %}
+
+  {# all nodes on the right salt version can run the following states #}
+  {% do allowed_states.append('common') %}
+  {% do allowed_states.append('patch.os.schedule') %}
+  {% do allowed_states.append('motd') %}
+  {% do allowed_states.append('salt.minion-check') %}
+  {% do allowed_states.append('sensoroni') %}
+  {% do allowed_states.append('salt.lasthighstate') %}
+
+{% endif %}
+
+
+{% if ISAIRGAP %}
+  {% do allowed_states.append('airgap') %}
+{% endif %}
+
+{# all nodes can always run salt.minion state #}
+{% do allowed_states.append('salt.minion') %}

salt/ca/init.sls

@@ -1,7 +1,5 @@
-{% set show_top = salt['state.show_top']() %}
-{% set top_states = show_top.values() | join(', ') %}
-
-{% if 'ca' in top_states %}
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls in allowed_states %}
 
 {% set manager = salt['grains.get']('master') %}
 /etc/salt/minion.d/signing_policies.conf:
@@ -26,8 +24,9 @@ pki_private_key:
       - x509: /etc/pki/ca.crt
     {%- endif %}
 
-/etc/pki/ca.crt:
+pki_public_ca_crt:
   x509.certificate_managed:
+    - name: /etc/pki/ca.crt
     - signing_private_key: /etc/pki/ca.key
     - CN: {{ manager }}
     - C: US
@@ -44,6 +43,10 @@ pki_private_key:
     - replace: False
     - require:
       - file: /etc/pki
+    - timeout: 30
+    - retry:
+        attempts: 5
+        interval: 30
 
 x509_pem_entries:
   module.run:
@@ -60,8 +63,8 @@ cakeyperms:
 
 {% else %}
 
-ca_state_not_allowed:
+{{sls}}_state_not_allowed:
   test.fail_without_changes:
-    - name: ca_state_not_allowed
+    - name: {{sls}}_state_not_allowed
 
-{% endif %}
+{% endif %}

salt/common/cron/common-rotate

@@ -1,2 +1,2 @@
 #!/bin/bash
-logrotate -f /opt/so/conf/log-rotate.conf >/dev/null 2>&1
+/usr/sbin/logrotate -f /opt/so/conf/log-rotate.conf > /dev/null 2>&1

salt/common/files/99-reserved-ports.conf

@@ -0,0 +1 @@
+net.ipv4.ip_local_reserved_ports=55000,57314,47760-47860

salt/common/files/daemon.json

@@ -0,0 +1,12 @@
+{%- set DOCKERRANGE = salt['pillar.get']('docker:range', '172.17.0.0/24') %}
+{%- set DOCKERBIND = salt['pillar.get']('docker:bip', '172.17.0.1/24') %}
+{
+    "registry-mirrors": [ "https://:5000" ],
+    "bip": "{{ DOCKERBIND }}",
+    "default-address-pools": [
+      {
+        "base" : "{{ DOCKERRANGE }}",
+        "size" : 24
+      }
+    ]
+}

salt/common/files/log-rotate.conf

@@ -1,4 +1,6 @@
 {%- set logrotate_conf = salt['pillar.get']('logrotate:conf') %}
+{%- set group_conf = salt['pillar.get']('logrotate:group_conf') %}
+
 
 /opt/so/log/aptcacher-ng/*.log
 /opt/so/log/idstools/*.log
@@ -13,11 +15,22 @@
 /opt/so/log/fleet/*.log
 /opt/so/log/suricata/*.log
 /opt/so/log/mysql/*.log
-/opt/so/log/playbook/*.log
-/opt/so/log/logstash/*.log
-/opt/so/log/filebeat/*.log
 /opt/so/log/telegraf/*.log
 /opt/so/log/redis/*.log
+/opt/so/log/sensoroni/*.log
+/opt/so/log/stenographer/*.log
+/opt/so/log/salt/so-salt-minion-check
+/opt/so/log/salt/minion
+/opt/so/log/salt/master
+/opt/so/log/logscan/*.log
 {
     {{ logrotate_conf | indent(width=4) }}
 }
+
+# Playbook's log directory needs additional configuration
+# because Playbook requires a more permissive directory
+/opt/so/log/playbook/*.log
+{
+    {{ logrotate_conf | indent(width=4) }}
+    {{ group_conf | indent(width=4) }}
+}

salt/common/files/sensor-rotate.conf

@@ -6,5 +6,17 @@
     nocompress
     create
     sharedscripts
-    endscript
 }
+
+/nsm/strelka/log/strelka.log
+{
+    daily
+    rotate 14
+    missingok
+    copytruncate
+    compress
+    create
+    extension .log
+    dateext
+    dateyesterday
+}

salt/common/files/soversion

@@ -0,0 +1,2 @@
+{%- set VERSION = salt['pillar.get']('global:soversion') -%}
+{{ VERSION }}

salt/common/files/vimrc

@@ -0,0 +1,6 @@
+" Activates filetype detection
+filetype plugin indent on
+
+" Sets .sls files to use YAML syntax highlighting
+autocmd BufNewFile,BufRead *.sls set syntax=yaml
+set number

salt/common/init.sls

@@ -1,15 +1,19 @@
-{% set show_top = salt['state.show_top']() %}
-{% set top_states = show_top.values() | join(', ') %}
-
-{% if 'common' in top_states %}
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls in allowed_states %}
 
 {% set role = grains.id.split('_') | last %}
+{% from 'elasticsearch/auth.map.jinja' import ELASTICAUTH with context %}
 
 # Remove variables.txt from /tmp - This is temp
 rmvariablesfile:
   file.absent:
     - name: /tmp/variables.txt
 
+dockergroup:
+  group.present:
+    - name: docker
+    - gid: 920
+
 # Add socore Group
 socoregroup:
   group.present:
@@ -32,6 +36,18 @@ soconfperms:
     - gid: 939
     - dir_mode: 770
 
+sostatusconf:
+  file.directory:
+    - name: /opt/so/conf/so-status
+    - uid: 939
+    - gid: 939
+    - dir_mode: 770
+
+so-status.conf:
+  file.touch:
+    - name: /opt/so/conf/so-status/so-status.conf
+    - unless: ls /opt/so/conf/so-status/so-status.conf
+
 sosaltstackperms:
   file.directory:
     - name: /opt/so/saltstack
@@ -39,6 +55,11 @@ sosaltstackperms:
     - gid: 939
     - dir_mode: 770
 
+so_log_perms:
+  file.directory:
+    - name: /opt/so/log
+    - dir_mode: 755
+
 # Create a state directory
 statedir:
   file.directory:
@@ -54,20 +75,12 @@ salttmp:
     - group: 939
     - makedirs: True
 
-# Install epel
-{% if grains['os'] == 'CentOS' %}
-repair_yumdb:
-  cmd.run:
-    - name: 'mv -f /var/lib/rpm/__db* /tmp && yum clean all'
-    - onlyif:
-      - 'yum check-update 2>&1 | grep "Error: rpmdb open failed"'
-
-epel:
-  pkg.installed:
-    - skip_suggestions: True
-    - pkgs:
-      - epel-release
-{% endif %}
+# VIM config
+vimconfig:
+  file.managed:
+    - name: /root/.vimrc
+    - source: salt://common/files/vimrc
+    - replace: False
 
 # Install common packages
 {% if grains['os'] != 'CentOS' %}     
@@ -80,7 +93,6 @@ commonpkgs:
       - ntpdate
       - jq
       - python3-docker
-      - docker-ce
       - curl
       - ca-certificates
       - software-properties-common
@@ -89,17 +101,29 @@ commonpkgs:
       - netcat
       - python3-mysqldb
       - sqlite3
-      - argon2
       - libssl-dev
       - python3-dateutil
       - python3-m2crypto
       - python3-mysqldb
+      - python3-packaging
+      - python3-lxml
       - git
+      - vim
+
 heldpackages:
   pkg.installed:
     - pkgs:
-      - containerd.io: 1.2.13-2
-      - docker-ce: 5:19.03.12~3-0~ubuntu-bionic
+    {% if grains['oscodename'] == 'bionic' %}
+      - containerd.io: 1.4.4-1
+      - docker-ce: 5:20.10.5~3-0~ubuntu-bionic
+      - docker-ce-cli: 5:20.10.5~3-0~ubuntu-bionic
+      - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-bionic
+    {% elif grains['oscodename'] == 'focal' %}
+      - containerd.io: 1.4.9-1
+      - docker-ce: 5:20.10.8~3-0~ubuntu-focal
+      - docker-ce-cli: 5:20.10.5~3-0~ubuntu-focal
+      - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-focal
+    {% endif %}
     - hold: True
     - update_holds: True
 
@@ -117,7 +141,6 @@ commonpkgs:
       - net-tools
       - curl
       - sqlite
-      - argon2
       - mariadb-devel
       - nmap-ncat
       - python3
@@ -125,17 +148,22 @@ commonpkgs:
       - python36-dateutil
       - python36-m2crypto
       - python36-mysql
+      - python36-packaging
+      - python36-lxml
       - yum-utils
       - device-mapper-persistent-data
       - lvm2
       - openssl
       - git
+      - vim-enhanced
 
 heldpackages:
   pkg.installed:
     - pkgs:
-      - containerd.io: 1.2.13-3.2.el7
-      - docker-ce: 3:19.03.12-3.el7
+      - containerd.io: 1.4.4-3.1.el7
+      - docker-ce: 3:20.10.5-3.el7
+      - docker-ce-cli: 1:20.10.5-3.el7
+      - docker-ce-rootless-extras: 20.10.5-3.el7
     - hold: True
     - update_holds: True
 {% endif %}
@@ -154,15 +182,27 @@ alwaysupdated:
 Etc/UTC:
   timezone.system
 
+elastic_curl_config:
+  file.managed:
+    - name: /opt/so/conf/elasticsearch/curl.config
+    - source: salt://elasticsearch/curl.config
+    - mode: 600
+    - show_changes: False
+    - makedirs: True
+
 # Sync some Utilities
 utilsyncscripts:
   file.recurse:
     - name: /usr/sbin
-    - user: 0
-    - group: 0
+    - user: root
+    - group: root
     - file_mode: 755
     - template: jinja
     - source: salt://common/tools/sbin
+    - defaults:
+        ELASTICCURL: 'curl'
+    - context:
+        ELASTICCURL: {{ ELASTICAUTH.elasticcurl }}
 
 {% if role in ['eval', 'standalone', 'sensor', 'heavynode'] %}
 # Add sensor cleanup
@@ -220,7 +260,40 @@ commonlogrotateconf:
     - month: '*'
     - dayweek: '*'
 
+# Create the status directory
+sostatusdir:
+  file.directory:
+    - name: /opt/so/log/sostatus
+    - user: 0
+    - group: 0
+    - makedirs: True
+
+sostatus_log:
+  file.managed:
+    - name: /opt/so/log/sostatus/status.log
+    - mode: 644
+    
+# Install sostatus check cron
+'/usr/sbin/so-status -q; echo $? > /opt/so/log/sostatus/status.log 2>&1':
+  cron.present:
+    - user: root
+    - minute: '*/1'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
+
 {% if role in ['eval', 'manager', 'managersearch', 'standalone'] %}
+# Lock permissions on the backup directory
+backupdir:
+  file.directory:
+    - name: /nsm/backup
+    - user: 0
+    - group: 0
+    - makedirs: True
+    - mode: 700
+  
 # Add config backup
 /usr/sbin/so-config-backup > /dev/null 2>&1:
   cron.present:
@@ -230,17 +303,69 @@ commonlogrotateconf:
     - daymonth: '*'
     - month: '*'
     - dayweek: '*'
+{% else %}
+soversionfile:
+  file.managed:
+    - name: /etc/soversion
+    - source: salt://common/files/soversion
+    - mode: 644
+    - template: jinja
+    
 {% endif %}
 
+# Manager daemon.json
+docker_daemon:
+  file.managed:
+    - source: salt://common/files/daemon.json
+    - name: /etc/docker/daemon.json
+    - template: jinja 
+
 # Make sure Docker is always running
 docker:
   service.running:
     - enable: True
+    - watch:
+      - file: docker_daemon
+
+# Reserve OS ports for Docker proxy in case boot settings are not already applied/present
+# 55000 = Wazuh, 57314 = Strelka, 47760-47860 = Zeek
+dockerapplyports:
+    cmd.run:
+      - name: if [ ! -s /etc/sysctl.d/99-reserved-ports.conf ]; then sysctl -w net.ipv4.ip_local_reserved_ports="55000,57314,47760-47860"; fi
+
+# Reserve OS ports for Docker proxy
+dockerreserveports:
+  file.managed:
+    - source: salt://common/files/99-reserved-ports.conf
+    - name: /etc/sysctl.d/99-reserved-ports.conf
+
+{% if salt['grains.get']('sosmodel', '') %}
+  {% if grains['os'] == 'CentOS' %}     
+# Install Raid tools
+raidpkgs:
+  pkg.installed:
+    - skip_suggestions: True
+    - pkgs:
+      - securityonion-raidtools
+      - securityonion-megactl
+  {% endif %}
+
+# Install raid check cron
+/usr/sbin/so-raid-status > /dev/null 2>&1:
+  cron.present:
+    - user: root
+    - minute: '*/15'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
+{% endif %}
 
 {% else %}
 
-common_state_not_allowed:
+{{sls}}_state_not_allowed:
   test.fail_without_changes:
-    - name: common_state_not_allowed
+    - name: {{sls}}_state_not_allowed
 
 {% endif %}

salt/common/maps/domainstats.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-domainstats'
-  ]
-} %}

salt/common/maps/eval.map.jinja

@@ -1,20 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-filebeat',
-    'so-nginx',
-    'so-telegraf',
-    'so-dockerregistry',
-    'so-soc',
-    'so-kratos',
-    'so-idstools',
-    'so-elasticsearch',
-    'so-kibana',
-    'so-steno',
-    'so-suricata',
-    'so-zeek',
-    'so-curator',
-    'so-elastalert',
-    'so-soctopus',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/fleet.map.jinja

@@ -1,10 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-mysql',
-    'so-fleet',
-    'so-redis',
-    'so-filebeat',
-    'so-nginx',
-    'so-telegraf'
-  ]
-} %}

salt/common/maps/fleet_manager.map.jinja

@@ -1,7 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-mysql',
-    'so-fleet',
-    'so-redis'
-  ]
-} %}

salt/common/maps/freq.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-freqserver'
-  ]
-} %}

salt/common/maps/grafana.map.jinja

@@ -1,6 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-influxdb',
-    'so-grafana'
-  ]
-} %}

salt/common/maps/heavynode.map.jinja

@@ -1,15 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-redis',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-steno',
-    'so-suricata',
-    'so-wazuh',
-    'so-filebeat',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/helixsensor.map.jinja

@@ -1,12 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-idstools',
-    'so-steno',
-    'so-zeek',
-    'so-redis',
-    'so-logstash',
-    'so-filebeat
-  ]
-} %}

salt/common/maps/hotnode.map.jinja

@@ -1,9 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-     'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-   ]
-} %}

salt/common/maps/import.map.jinja

@@ -1,10 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-filebeat',
-    'so-nginx',
-    'so-soc',
-    'so-kratos',
-    'so-elasticsearch',
-    'so-kibana'
-  ]
-} %}

salt/common/maps/manager.map.jinja

@@ -1,18 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-dockerregistry',
-    'so-nginx',
-    'so-telegraf',
-    'so-soc',
-    'so-kratos',
-    'so-aptcacherng',
-    'so-idstools',
-    'so-redis',
-    'so-elasticsearch',
-    'so-logstash',
-    'so-kibana',
-    'so-elastalert',
-    'so-filebeat',
-    'so-soctopus'
-  ]
-} %}

salt/common/maps/managersearch.map.jinja

@@ -1,18 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-soc',
-    'so-kratos',
-    'so-aptcacherng',
-    'so-idstools',
-    'so-redis',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-kibana',
-    'so-elastalert',
-    'so-filebeat',
-    'so-soctopus'
-  ]
-} %}

salt/common/maps/mdengine.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-zeek'
-  ]
-} %}

salt/common/maps/playbook.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-playbook'
-  ]
-} %}

salt/common/maps/searchnode.map.jinja

@@ -1,10 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-filebeat'
-  ]
-} %}

salt/common/maps/sensor.map.jinja

@@ -1,9 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-telegraf',
-    'so-steno',
-    'so-suricata',
-    'so-filebeat',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/so-status.map.jinja

@@ -1,48 +0,0 @@
-{% set role = grains.id.split('_') | last %}
-{% from 'common/maps/'~ role ~'.map.jinja' import docker with context %}
-
-# Check if the service is enabled and append it's required containers
-# to the list predefined by the role / minion id affix
-{% macro append_containers(pillar_name, k, compare )%}
-  {% if salt['pillar.get'](pillar_name~':'~k, {}) != compare %}
-    {% if k == 'enabled' %}
-      {% set k = pillar_name %}
-    {% endif %}
-    {% from 'common/maps/'~k~'.map.jinja' import docker as d with context %}
-    {% for li in d['containers'] %}
-      {{ docker['containers'].append(li) }}
-    {% endfor %}
-  {% endif %}
-{% endmacro %}
-
-{% set docker = salt['grains.filter_by']({
-  '*_'~role: {
-    'containers': docker['containers']
-  } 
-},grain='id', merge=salt['pillar.get']('docker')) %}
-
-{% if role in ['eval', 'managersearch', 'manager', 'standalone'] %}
-  {{ append_containers('manager', 'grafana', 0) }}
-  {{ append_containers('global', 'fleet_manager', 0) }}
-  {{ append_containers('global', 'wazuh', 0) }}
-  {{ append_containers('manager', 'thehive', 0) }}
-  {{ append_containers('manager', 'playbook', 0) }}
-  {{ append_containers('manager', 'freq', 0) }}
-  {{ append_containers('manager', 'domainstats', 0) }}
-{% endif %}
-
-{% if role in ['eval', 'heavynode', 'sensor', 'standalone'] %}
-  {{ append_containers('strelka', 'enabled', 0) }}
-{% endif %}
-
-{% if role in ['heavynode', 'standalone'] %}
-  {{ append_containers('global', 'mdengine', 'SURICATA') }}
-{% endif %}
-
-{% if role == 'searchnode' %}
-  {{ append_containers('manager', 'wazuh', 0) }}
-{% endif %}
-
-{% if role == 'sensor' %}
-  {{ append_containers('global', 'mdengine', 'SURICATA') }}
-{% endif %}

salt/common/maps/standalone.map.jinja

@@ -1,22 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-soc',
-    'so-kratos',
-    'so-aptcacherng',
-    'so-idstools',
-    'so-redis',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-kibana',
-    'so-elastalert',
-    'so-filebeat',
-    'so-suricata',
-    'so-steno',
-    'so-dockerregistry',
-    'so-soctopus',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/strelka.map.jinja

@@ -1,9 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-strelka-coordinator',
-    'so-strelka-gatekeeper',
-    'so-strelka-manager',
-    'so-strelka-frontend',
-    'so-strelka-filestream'
-  ]
-} %}

salt/common/maps/thehive.map.jinja

@@ -1,7 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-thehive',
-    'so-thehive-es',
-    'so-cortex'
-  ]
-} %}

salt/common/maps/warmnode.map.jinja

@@ -1,7 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-elasticsearch'
-  ]
-} %}

salt/common/maps/wazuh.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-wazuh'
-  ]
-} %}

salt/common/scripts/dockernet.sh

@@ -1,8 +0,0 @@
-#!/bin/bash
-
-if [ ! -f /opt/so/state/dockernet.state ]; then
-    docker network create -d bridge so-elastic-net
-    touch /opt/so/state/dockernet.state
-else
-    exit
-fi

salt/common/tools/sbin/so-allow

@@ -1,6 +1,6 @@
-#!/bin/bash
+#!/usr/bin/env python3
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,152 +15,193 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-. /usr/sbin/so-common
+import ipaddress
+import textwrap
+import os
+import subprocess
+import sys
+import argparse
+import re
+from lxml import etree as ET
+from datetime import datetime as dt
+from datetime import timezone as tz
 
-local_salt_dir=/opt/so/saltstack/local
-
-SKIP=0
-
-function usage {
-
-cat << EOF
-
-Usage: $0 [-abefhoprsw] [ -i IP ]
-
-This program allows you to add a firewall rule to allow connections from a new IP address or CIDR range.
-
-If you run this program with no arguments, it will present a menu for you to choose your options.
-
-If you want to automate and skip the menu, you can pass the desired options as command line arguments.
-
-EXAMPLES
-
-To add 10.1.2.3 to the analyst role:
-so-allow -a -i 10.1.2.3
-
-To add 10.1.2.0/24 to the osquery role:
-so-allow -o -i 10.1.2.0/24
-
-EOF
 
+LOCAL_SALT_DIR='/opt/so/saltstack/local'
+WAZUH_CONF='/nsm/wazuh/etc/ossec.conf'
+VALID_ROLES = {
+  'a': { 'role': 'analyst','desc': 'Analyst - 80/tcp, 443/tcp' },
+  'b': { 'role': 'beats_endpoint', 'desc': 'Logstash Beat - 5044/tcp' },
+  'e': { 'role': 'elasticsearch_rest', 'desc': 'Elasticsearch REST API - 9200/tcp' },
+  'f': { 'role': 'strelka_frontend', 'desc': 'Strelka frontend - 57314/tcp' },
+  'o': { 'role': 'osquery_endpoint', 'desc': 'Osquery endpoint - 8090/tcp' },
+  's': { 'role': 'syslog', 'desc': 'Syslog device - 514/tcp/udp' },
+  'w': { 'role': 'wazuh_agent', 'desc': 'Wazuh agent - 1514/tcp/udp' },
+  'p': { 'role': 'wazuh_api', 'desc': 'Wazuh API - 55000/tcp' },
+  'r': { 'role': 'wazuh_authd', 'desc': 'Wazuh registration service - 1515/tcp' }
 }
 
-while getopts "ahfesprbowi:" OPTION
-do
-    case $OPTION in
-        h)
-            usage
-            exit 0
-            ;;
-        a)
-            FULLROLE="analyst"
-            SKIP=1
-            ;;
-        b)
-            FULLROLE="beats_endpoint"
-            SKIP=1
-            ;;
-	e)
-            FULLROLE="elasticsearch_rest"
-            SKIP=1
-            ;;
-        f)
-	    FULLROLE="strelka_frontend"
-            SKIP=1
-            ;;
-        i)  IP=$OPTARG
-            ;;
-        o)
-            FULLROLE="osquery_endpoint"
-            SKIP=1
-            ;;
-        w)
-            FULLROLE="wazuh_agent"
-            SKIP=1
-            ;;
-        s)
-            FULLROLE="syslog"
-            SKIP=1
-            ;;
-        p)
-            FULLROLE="wazuh_api"
-            SKIP=1
-            ;;
-        r)
-            FULLROLE="wazuh_authd"
-            SKIP=1
-            ;;
-        *)
-            usage
-            exit 0
-            ;;
-    esac
-done
 
-if [ "$SKIP" -eq 0 ]; then
+def validate_ip_cidr(ip_cidr: str) -> bool:
+  try:
+    ipaddress.ip_address(ip_cidr)
+  except ValueError:
+    try:
+      ipaddress.ip_network(ip_cidr)
+    except ValueError:
+      return False
+  return True
 
-    echo "This program allows you to add a firewall rule to allow connections from a new IP address."
-    echo ""
-    echo "Choose the role for the IP or Range you would like to add"
-    echo ""
-    echo "[a] - Analyst - ports 80/tcp and 443/tcp"
-    echo "[b] - Logstash Beat - port 5044/tcp"
-    echo "[e] - Elasticsearch REST API - port 9200/tcp"
-    echo "[f] - Strelka frontend - port 57314/tcp"
-    echo "[o] - Osquery endpoint - port 8090/tcp"
-    echo "[s] - Syslog device - 514/tcp/udp"
-    echo "[w] - Wazuh agent - port 1514/tcp/udp"
-    echo "[p] - Wazuh API - port 55000/tcp"
-    echo "[r] - Wazuh registration service - 1515/tcp"
-    echo ""
-    echo "Please enter your selection:"
-    read -r ROLE
-    echo "Enter a single ip address or range to allow (example: 10.10.10.10 or 10.10.0.0/16):"
-    read -r IP
 
-    if [ "$ROLE" == "a" ]; then
-        FULLROLE=analyst
-    elif [ "$ROLE" == "b" ]; then
-        FULLROLE=beats_endpoint
-    elif [ "$ROLE" == "e" ]; then
-        FULLROLE=elasticsearch_rest
-    elif [ "$ROLE" == "f" ]; then
-        FULLROLE=strelka_frontend
-    elif [ "$ROLE" == "o" ]; then
-        FULLROLE=osquery_endpoint
-    elif [ "$ROLE" == "w" ]; then
-        FULLROLE=wazuh_agent
-    elif [ "$ROLE" == "s" ]; then
-        FULLROLE=syslog
-    elif [ "$ROLE" == "p" ]; then
-        FULLROLE=wazuh_api
-    elif [ "$ROLE" == "r" ]; then
-        FULLROLE=wazuh_authd
-    else
-        echo "I don't recognize that role"
-        exit 1
-    fi
+def role_prompt() -> str:
+  print()
+  print('Choose the role for the IP or Range you would like to allow')
+  print()
+  for role in VALID_ROLES:
+    print(f'[{role}] - {VALID_ROLES[role]["desc"]}')
+  print()
+  role = input('Please enter your selection: ')
+  if role in VALID_ROLES.keys():
+    return VALID_ROLES[role]['role']
+  else:
+    print(f'Invalid role \'{role}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+  
 
-fi
+def ip_prompt() -> str:
+  ip = input('Enter a single ip address or range to allow (ex: 10.10.10.10 or 10.10.0.0/16): ')
+  if validate_ip_cidr(ip):
+    return ip
+  else:
+    print(f'Invalid IP address or CIDR block \'{ip}\', please try again.', file=sys.stderr)
+    sys.exit(1)
 
-echo "Adding $IP to the $FULLROLE role. This can take a few seconds"
-/usr/sbin/so-firewall includehost $FULLROLE $IP
-salt-call state.apply firewall queue=True
 
-# Check if Wazuh enabled
-if grep -q -R "wazuh: 1" $local_salt_dir/pillar/*; then
-    # If analyst, add to Wazuh AR whitelist
-    if [ "$FULLROLE" == "analyst" ]; then
-        WAZUH_MGR_CFG="/nsm/wazuh/etc/ossec.conf"
-        if ! grep -q "<white_list>$IP</white_list>" $WAZUH_MGR_CFG ; then
-            DATE=$(date)
-            sed -i 's/<\/ossec_config>//' $WAZUH_MGR_CFG
-            sed -i '/^$/N;/^\n$/D' $WAZUH_MGR_CFG
-            echo -e "<!--Address $IP added by /usr/sbin/so-allow on \"$DATE\"-->\n  <global>\n    <white_list>$IP</white_list>\n  </global>\n</ossec_config>" >> $WAZUH_MGR_CFG
-            echo "Added whitelist entry for $IP in $WAZUH_MGR_CFG."
-        echo
-            echo "Restarting OSSEC Server..."
-            /usr/sbin/so-wazuh-restart
-        fi
-    fi
-fi
+def wazuh_enabled() -> bool:
+  file = f'{LOCAL_SALT_DIR}/pillar/global.sls'
+  with open(file, 'r') as pillar:
+    if 'wazuh: 1' in pillar.read():
+      return True
+  return False
+
+
+def root_to_str(root: ET.ElementTree) -> str:
+    return ET.tostring(root, encoding='unicode', method='xml', xml_declaration=False, pretty_print=True)
+
+
+def add_wl(ip):
+    parser = ET.XMLParser(remove_blank_text=True)
+    with open(WAZUH_CONF, 'rb') as wazuh_conf:
+        tree = ET.parse(wazuh_conf, parser)
+    root = tree.getroot()
+
+    source_comment = ET.Comment(f'Address {ip} added by /usr/sbin/so-allow on {dt.utcnow().replace(tzinfo=tz.utc).strftime("%a %b %e %H:%M:%S %Z %Y")}')
+    new_global = ET.Element("global")
+    new_wl = ET.SubElement(new_global, 'white_list')
+    new_wl.text = ip
+
+    root.append(source_comment)
+    root.append(new_global)
+
+    with open(WAZUH_CONF, 'w') as add_out:
+        add_out.write(root_to_str(root))
+
+
+def apply(role: str, ip: str) -> int:
+  firewall_cmd = ['so-firewall', 'includehost', role, ip]
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'firewall', 'queue=True']
+  restart_wazuh_cmd = ['so-wazuh-restart']
+  print(f'Adding {ip} to the {role} role. This can take a few seconds...')
+  cmd = subprocess.run(firewall_cmd)
+  if cmd.returncode == 0:
+    cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  else:
+    return cmd.returncode
+  if cmd.returncode == 0:
+    if wazuh_enabled() and role=='analyst':
+      try:
+        add_wl(ip)
+        print(f'Added whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+      except Exception as e:
+        print(f'Failed to add whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+        print(e)
+        return 1
+      print('Restarting OSSEC Server...')
+      cmd = subprocess.run(restart_wazuh_cmd)
+    else:
+      return cmd.returncode
+  else:
+    print(f'Commmand \'{" ".join(salt_cmd)}\' failed.', file=sys.stderr)
+    return cmd.returncode
+  if cmd.returncode != 0:
+    print('Failed to restart OSSEC server.')
+  return cmd.returncode
+
+
+def main():
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=textwrap.dedent(f'''\
+        additional information:
+                      To use this script in interactive mode call it with no arguments
+        '''
+  ))
+
+  group = main_parser.add_argument_group(title='roles')
+  group.add_argument('-a', dest='roles', action='append_const', const=VALID_ROLES['a']['role'], help="Analyst - 80/tcp, 443/tcp")
+  group.add_argument('-b', dest='roles', action='append_const', const=VALID_ROLES['b']['role'], help="Logstash Beat - 5044/tcp")
+  group.add_argument('-e', dest='roles', action='append_const', const=VALID_ROLES['e']['role'], help="Elasticsearch REST API - 9200/tcp")
+  group.add_argument('-f', dest='roles', action='append_const', const=VALID_ROLES['f']['role'], help="Strelka frontend - 57314/tcp")
+  group.add_argument('-o', dest='roles', action='append_const', const=VALID_ROLES['o']['role'], help="Osquery endpoint - 8090/tcp")
+  group.add_argument('-s', dest='roles', action='append_const', const=VALID_ROLES['s']['role'], help="Syslog device - 514/tcp/udp")
+  group.add_argument('-w', dest='roles', action='append_const', const=VALID_ROLES['w']['role'], help="Wazuh agent - 1514/tcp/udp")
+  group.add_argument('-p', dest='roles', action='append_const', const=VALID_ROLES['p']['role'], help="Wazuh API - 55000/tcp")
+  group.add_argument('-r', dest='roles', action='append_const', const=VALID_ROLES['r']['role'], help="Wazuh registration service - 1515/tcp")
+  
+  ip_g = main_parser.add_argument_group(title='allow')
+  ip_g.add_argument('-i', help="IP or CIDR block to disallow connections from, requires at least one role argument", metavar='', dest='ip')
+
+  args = main_parser.parse_args(sys.argv[1:])
+
+  if args.roles is None:
+    role = role_prompt()
+    ip = ip_prompt()
+    try:
+      return_code = apply(role, ip)
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+    sys.exit(return_code)
+  elif args.roles is not None and args.ip is None:
+    if os.environ.get('IP') is None:
+      main_parser.print_help()
+      sys.exit(1)
+    else:
+      args.ip = os.environ['IP']
+  
+  if validate_ip_cidr(args.ip):
+    try:
+      for role in args.roles:
+        return_code = apply(role, args.ip)
+        if return_code > 0:
+          break
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+  else:
+    print(f'Invalid IP address or CIDR block \'{args.ip}\', please try again.', file=sys.stderr)
+    return_code = 1
+    
+  sys.exit(return_code)
+
+
+if __name__ == '__main__':
+  try:
+    main()
+  except KeyboardInterrupt:
+    sys.exit(1)
+

salt/common/tools/sbin/so-allow-view

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-analyst-install

@@ -84,7 +84,7 @@ while [[ $INSTALL != "yes" ]] && [[ $INSTALL != "no" ]]; do
     echo "##                                       ##"
     echo "##    Installing the Security Onion      ##"
     echo "##   analyst node on this device will    ##"
-    echo "##      make permanenet changes to       ##"
+    echo "##       make permanent changes to       ##"
     echo "##              the system.              ##"
     echo "##                                       ##"
     echo "###########################################"

salt/common/tools/sbin/so-checkin

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-salt-call state.highstate
+salt-call state.highstate -l info 

salt/common/tools/sbin/so-common

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,30 +15,77 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-IMAGEREPO=securityonion
+DEFAULT_SALT_DIR=/opt/so/saltstack/default
 
 # Check for prerequisites
 if [ "$(id -u)" -ne 0 ]; then
-    echo "This script must be run using sudo!"
-    exit 1
+	echo "This script must be run using sudo!"
+	exit 1
 fi
 
 # Define a banner to separate sections
 banner="========================================================================="
 
-header() {
-	echo
-    printf '%s\n' "$banner" "$*" "$banner"
-}
+add_interface_bond0() {
+	local BNIC=$1
+	if [[ -z $MTU ]]; then
+		local MTU
+		MTU=$(lookup_pillar "mtu" "sensor")
+	fi
+	local nic_error=0
 
-lookup_pillar() {
-    key=$1
-    salt-call --no-color  pillar.get global:${key} --out=newline_values_only
-}
+	# Check if specific offload features are able to be disabled
+	for string in "generic-segmentation-offload" "generic-receive-offload" "tcp-segmentation-offload"; do
+		if ethtool -k "$BNIC" | grep $string | grep -q "on [fixed]"; then
+			echo "The hardware or driver for interface ${BNIC} is not supported, packet capture may not work as expected."
+			((nic_error++))
+			break
+		fi
+	done
 
-lookup_pillar_secret() {
-    key=$1
-    salt-call --no-color  pillar.get secrets:${key} --out=newline_values_only
+	case "$2" in
+		-v|--verbose)
+			local verbose=true
+		;;
+	esac
+
+	for i in rx tx sg tso ufo gso gro lro; do
+		if [[ $verbose == true ]]; then
+			ethtool -K "$BNIC" $i off
+		else
+			ethtool -K "$BNIC" $i off &>/dev/null
+		fi
+	done
+	# Check if the bond slave connection has already been created
+	nmcli -f name,uuid -p con | grep -q "bond0-slave-$BNIC"
+	local found_int=$?
+
+	if [[ $found_int != 0 ]]; then
+		# Create the slave interface and assign it to the bond
+		nmcli con add type ethernet ifname "$BNIC" con-name "bond0-slave-$BNIC" master bond0 -- \
+			ethernet.mtu "$MTU" \
+			connection.autoconnect "yes"
+	else
+		local int_uuid
+		int_uuid=$(nmcli -f name,uuid -p con | sed -n "s/bond0-slave-$BNIC //p" | tr -d ' ')
+
+		nmcli con mod "$int_uuid" \
+			ethernet.mtu "$MTU" \
+			connection.autoconnect "yes"
+	fi
+
+	ip link set dev "$BNIC" arp off multicast off allmulticast off promisc on
+			
+	# Bring the slave interface up
+	if [[ $verbose == true ]]; then
+		nmcli con up "bond0-slave-$BNIC"
+	else
+		nmcli con up "bond0-slave-$BNIC" &>/dev/null
+	fi
+	 
+	if [ "$nic_error" != 0 ]; then
+		return "$nic_error"
+	fi
 }
 
 check_container() {
@@ -47,7 +94,469 @@ check_container() {
 }
 
 check_password() {
-    local password=$1
-    echo "$password" | egrep -v "'|\"|\\\\" > /dev/null 2>&1
-    return $?
-}
+	local password=$1
+	echo "$password" | egrep -v "'|\"|\\$|\\\\" > /dev/null 2>&1
+	return $?
+}
+
+check_password_and_exit() {
+	local password=$1
+	if ! check_password "$password"; then
+		echo "Password is invalid. Do not include single quotes, double quotes, dollar signs, and backslashes in the password."
+		exit 2
+	fi
+	return 0
+}
+
+check_elastic_license() {
+
+	[ -n "$TESTING" ] && return
+
+	# See if the user has already accepted the license
+	if [ ! -f /opt/so/state/yeselastic.txt ]; then
+	  elastic_license
+	else
+	  echo "Elastic License has already been accepted"
+	fi  
+}
+
+copy_new_files() {
+  # Copy new files over to the salt dir
+  cd $UPDATE_DIR
+  rsync -a salt $DEFAULT_SALT_DIR/
+  rsync -a pillar $DEFAULT_SALT_DIR/
+  chown -R socore:socore $DEFAULT_SALT_DIR/
+  chmod 755 $DEFAULT_SALT_DIR/pillar/firewall/addfirewall.sh
+  cd /tmp
+}
+
+disable_fastestmirror() {
+	sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
+}
+
+elastic_license() {
+
+read -r -d '' message <<- EOM
+\n
+Starting in Elastic Stack version 7.11, the Elastic Stack binaries are only available under the Elastic License:
+https://securityonion.net/elastic-license
+
+Please review the Elastic License:
+https://www.elastic.co/licensing/elastic-license
+
+Do you agree to the terms of the Elastic License?
+
+If so, type AGREE to accept the Elastic License and continue.  Otherwise, press Enter to exit this program without making any changes.
+EOM
+
+	AGREED=$(whiptail --title "$whiptail_title" --inputbox \
+	"$message" 20 75 3>&1 1>&2 2>&3)
+
+	if [ "${AGREED^^}" = 'AGREE' ]; then
+		mkdir -p /opt/so/state
+		touch /opt/so/state/yeselastic.txt 
+	else
+		echo "Starting in 2.3.40 you must accept the Elastic license if you want to run Security Onion."
+		exit 1
+	fi
+
+}
+
+fail() {
+	msg=$1
+	echo "ERROR: $msg"
+	echo "Exiting."
+	exit 1
+}
+
+get_random_value() {
+	length=${1:-20}
+	head -c 5000 /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w $length | head -n 1
+}
+
+gpg_rpm_import() {
+	if [[ "$OS" == "centos" ]]; then
+	    if [[ "$WHATWOULDYOUSAYYAHDOHERE" == "setup" ]]; then
+	        local RPMKEYSLOC="../salt/repo/client/files/centos/keys"
+	    else
+	        local RPMKEYSLOC="$UPDATE_DIR/salt/repo/client/files/centos/keys"
+	    fi
+	
+	    RPMKEYS=('RPM-GPG-KEY-EPEL-7' 'GPG-KEY-WAZUH' 'docker.pub' 'SALTSTACK-GPG-KEY.pub' 'securityonion.pub')
+
+	    for RPMKEY in "${RPMKEYS[@]}"; do
+    	        rpm --import $RPMKEYSLOC/$RPMKEY
+	        echo "Imported $RPMKEY"
+	    done
+	fi
+}
+
+header() {
+	printf '%s\n' "" "$banner" " $*" "$banner" 
+}
+
+init_monitor() {
+	MONITORNIC=$1
+	
+	if [[ $MONITORNIC == "bond0" ]]; then 
+	  BIFACES=$(lookup_bond_interfaces)
+	else
+	  BIFACES=$MONITORNIC
+	fi
+
+	for DEVICE_IFACE in $BIFACES; do
+	  for i in rx tx sg tso ufo gso gro lro; do
+	    ethtool -K "$DEVICE_IFACE" "$i" off;
+  	  done
+	  ip link set dev "$DEVICE_IFACE" arp off multicast off allmulticast off promisc on
+	done
+}
+
+is_manager_node() {
+	# Check to see if this is a manager node
+	role=$(lookup_role)
+	is_single_node_grid && return 0
+	[ $role == 'manager' ] && return 0
+	[ $role == 'managersearch' ] && return 0
+	[ $role == 'helix' ] && return 0
+	return 1
+}
+
+is_sensor_node() {
+	# Check to see if this is a sensor (forward) node
+	role=$(lookup_role)
+	is_single_node_grid && return 0
+	[ $role == 'sensor' ] && return 0
+	[ $role == 'heavynode' ] && return 0
+	[ $role == 'helix' ] && return 0
+	return 1
+}
+
+is_single_node_grid() {
+	role=$(lookup_role)
+	[ $role == 'eval' ] && return 0
+	[ $role == 'standalone' ] && return 0
+	[ $role == 'import' ] && return 0
+	return 1
+}
+
+lookup_bond_interfaces() {
+	cat /proc/net/bonding/bond0 | grep "Slave Interface:" | sed -e "s/Slave Interface: //g"
+}
+
+lookup_salt_value() {
+	key=$1
+	group=$2
+	kind=$3
+	output=${4:-newline_values_only}
+
+	if [ -z "$kind" ]; then
+		kind=pillar
+	fi
+
+	if [ -n "$group" ]; then
+		group=${group}:
+	fi
+
+	salt-call --no-color  ${kind}.get ${group}${key} --out=${output}
+}
+
+lookup_pillar() {
+	key=$1
+	pillar=$2
+	if [ -z "$pillar" ]; then
+		pillar=global
+	fi
+	lookup_salt_value "$key" "$pillar" "pillar"
+}
+
+lookup_pillar_secret() {
+	lookup_pillar "$1" "secrets"
+}
+
+lookup_grain() {
+	lookup_salt_value "$1" "" "grains"
+}
+
+lookup_role() {
+	id=$(lookup_grain id)
+	pieces=($(echo $id | tr '_' ' '))
+	echo ${pieces[1]}
+}
+
+require_manager() {
+	if is_manager_node; then
+		echo "This is a manager, so we can proceed."
+	else
+		echo "Please run this command on the manager; the manager controls the grid."
+		exit 1
+	fi
+}
+
+retry() {
+	maxAttempts=$1
+	sleepDelay=$2
+	cmd=$3
+	expectedOutput=$4
+	attempt=0
+	local exitcode=0
+	while [[ $attempt -lt $maxAttempts ]]; do			
+		attempt=$((attempt+1))
+		echo "Executing command with retry support: $cmd"
+		output=$(eval "$cmd")
+		exitcode=$?
+		echo "Results: $output ($exitcode)"
+		if [ -n "$expectedOutput" ]; then
+			if [[ "$output" =~ "$expectedOutput" ]]; then
+				return $exitCode
+			else
+				echo "Expected '$expectedOutput' but got '$output'"
+			fi
+		elif [[ $exitcode -eq 0 ]]; then
+			return $exitCode
+		fi
+		echo "Command failed with exit code $exitcode; will retry in $sleepDelay seconds ($attempt / $maxAttempts)..."
+		sleep $sleepDelay
+	done
+	echo "Command continues to fail; giving up."
+	return $exitcode
+}
+
+run_check_net_err() {
+	local cmd=$1
+	local err_msg=${2:-"Unknown error occured, please check /root/$WHATWOULDYOUSAYYAHDOHERE.log for details."} # Really need to rename that variable
+	local no_retry=$3
+
+	local exit_code
+	if [[ -z $no_retry ]]; then
+		retry 5 60 "$cmd"
+		exit_code=$?
+	else
+		eval "$cmd"
+		exit_code=$?
+	fi
+	
+	if [[ $exit_code -ne 0 ]]; then
+		ERR_HANDLED=true
+		[[ -z $no_retry ]] || echo "Command failed with error $exit_code"
+		echo "$err_msg"
+		exit $exit_code
+	fi
+}
+
+set_os() {
+	if [ -f /etc/redhat-release ]; then
+		OS=centos
+	else
+		OS=ubuntu
+	fi
+}
+
+set_minionid() {
+	MINIONID=$(lookup_grain id)
+}
+
+set_palette() {
+	if [ "$OS" == ubuntu ]; then
+	    update-alternatives --set newt-palette /etc/newt/palette.original
+    fi
+}
+
+set_version() {
+	CURRENTVERSION=0.0.0
+	if [ -f /etc/soversion ]; then
+		CURRENTVERSION=$(cat /etc/soversion)
+	fi
+	if [ -z "$VERSION" ]; then
+		if [ -z "$NEWVERSION" ]; then
+			if [ "$CURRENTVERSION" == "0.0.0" ]; then
+				echo "ERROR: Unable to detect Security Onion version; terminating script."
+				exit 1
+			else
+				VERSION=$CURRENTVERSION
+			fi
+		else
+			VERSION="$NEWVERSION"
+		fi
+	fi
+}
+
+has_uppercase() {
+	local string=$1
+
+	echo "$string" | grep -qP '[A-Z]' \
+		&& return 0 \
+		|| return 1
+}
+
+valid_cidr() {
+	# Verify there is a backslash in the string
+	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
+
+	valid_ip4_cidr_mask "$1" && return 0 || return 1
+	
+	local cidr="$1"
+	local ip
+	ip=$(echo "$cidr" | sed 's/\/.*//' )
+	
+	if valid_ip4 "$ip"; then
+		local ip1 ip2 ip3 ip4 N
+		IFS="./" read -r ip1 ip2 ip3 ip4 N <<< "$cidr"
+		ip_total=$((ip1 * 256 ** 3 + ip2 * 256 ** 2 + ip3 * 256 + ip4))
+		[[ $((ip_total % 2**(32-N))) == 0 ]] && return 0 || return 1
+	else
+		return 1
+	fi
+}
+
+valid_cidr_list() {
+	local all_valid=0
+
+	IFS="," read -r -a net_arr <<< "$1"
+
+	for net in "${net_arr[@]}"; do
+		valid_cidr "$net" || all_valid=1
+	done
+
+	return $all_valid
+}
+
+valid_dns_list() {
+	local all_valid=0
+
+	IFS="," read -r -a dns_arr <<< "$1"
+
+	for addr in "${dns_arr[@]}"; do
+		valid_ip4 "$addr" || all_valid=1
+	done
+
+	return $all_valid
+}
+
+valid_fqdn() {
+	local fqdn=$1
+
+	echo "$fqdn" | grep -qP '(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{0,62}[a-zA-Z0-9]\.)+[a-zA-Z]{2,63}$)' \
+		&& return 0 \
+		|| return 1
+}
+
+valid_hostname() {
+	local hostname=$1
+
+	[[ $hostname =~ ^[a-zA-Z0-9\-]+$ ]] && [[ $hostname != 'localhost' ]] && return 0 || return 1
+}
+
+valid_ip4() {
+	local ip=$1
+
+	echo "$ip" | grep -qP '^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$' && return 0 || return 1
+}
+
+valid_ip4_cidr_mask() {
+	# Verify there is a backslash in the string
+	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
+
+	local cidr
+	local ip
+
+	cidr=$(echo "$1" | sed 's/.*\///')
+	ip=$(echo "$1" | sed 's/\/.*//' )
+
+	if valid_ip4 "$ip"; then
+		[[ $cidr =~ ^([0-9]|[1-2][0-9]|3[0-2])$ ]] && return 0 || return 1
+	else
+		return 1
+	fi
+}
+
+valid_int() {
+	local num=$1
+	local min=${2:-1}
+	local max=${3:-1000000000}
+
+	[[ $num =~ ^[0-9]*$ ]] && [[ $num -ge $min ]] && [[ $num -le $max ]] && return 0 || return 1
+}
+
+# {% raw %}
+
+valid_proxy() {
+	local proxy=$1
+	local url_prefixes=( 'http://' 'https://' )
+
+	local has_prefix=false
+	for prefix in "${url_prefixes[@]}"; do
+		echo "$proxy" | grep -q "$prefix" && has_prefix=true && proxy=${proxy#"$prefix"} && break
+	done
+	
+	local url_arr
+	mapfile -t url_arr <<< "$(echo "$proxy" | tr ":" "\n")"
+
+	local valid_url=true
+	if ! valid_ip4 "${url_arr[0]}" && ! valid_fqdn "${url_arr[0]}" && ! valid_hostname "${url_arr[0]}"; then
+		valid_url=false
+	fi
+
+	[[ $has_prefix == true ]] && [[ $valid_url == true ]] && return 0 || return 1
+}
+
+valid_ntp_list() {
+	local string=$1
+	local ntp_arr
+	IFS="," read -r -a ntp_arr <<< "$string"
+
+	for ntp in "${ntp_arr[@]}"; do
+		if ! valid_ip4 "$ntp" && ! valid_hostname "$ntp" && ! valid_fqdn "$ntp"; then
+			return 1
+		fi
+	done
+
+	return 0
+}
+
+valid_string() {
+	local str=$1
+	local min_length=${2:-1}
+	local max_length=${3:-64}
+
+	echo "$str" | grep -qP '^\S+$' && [[ ${#str} -ge $min_length ]] && [[ ${#str} -le $max_length ]] && return 0 || return 1
+}
+
+# {% endraw %}
+
+valid_username() {
+	local user=$1
+
+	echo "$user" | grep -qP '^[a-z_]([a-z0-9_-]{0,31}|[a-z0-9_-]{0,30}\$)$' && return 0 || return 1
+}
+
+wait_for_web_response() {
+	url=$1
+	expected=$2
+	maxAttempts=${3:-300}
+	curlcmd=${4:-curl}
+	logfile=/root/wait_for_web_response.log
+	truncate -s 0 "$logfile"
+	attempt=0
+	while [[ $attempt -lt $maxAttempts ]]; do
+		attempt=$((attempt+1))
+		echo "Waiting for value '$expected' at '$url' ($attempt/$maxAttempts)"
+		result=$($curlcmd -ks -L $url)
+		exitcode=$?
+
+		echo "--------------------------------------------------" >> $logfile
+		echo "$(date) - Checking web URL: $url ($attempt/$maxAttempts)" >> $logfile
+		echo "$result" >> $logfile
+		echo "exit code=$exitcode" >> $logfile
+		echo "" >> $logfile
+
+		if [[ $exitcode -eq 0 && "$result" =~ $expected ]]; then
+			echo "Received expected response; proceeding."
+			return 0
+		fi
+		echo "Server is not ready"
+		sleep 1
+	done
+	echo "Server still not ready after $maxAttempts attempts; giving up."
+	return 1
+}

salt/common/tools/sbin/so-config-backup

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -33,12 +33,16 @@ if [ ! -f $BACKUPFILE ]; then
   {%- for LOCATION in BACKUPLOCATIONS %}
   tar -rf $BACKUPFILE {{ LOCATION }}
   {%- endfor %}
+  tar -rf $BACKUPFILE /etc/pki
+  tar -rf $BACKUPFILE /etc/salt
+  tar -rf $BACKUPFILE /opt/so/conf/kratos
 
 fi
 
-# Find oldest backup file and remove it
+# Find oldest backup files and remove them
 NUMBACKUPS=$(find /nsm/backup/ -type f -name "so-config-backup*" | wc -l)
-OLDESTBACKUP=$(find /nsm/backup/ -type f -name "so-config-backup*" | ls -1t | tail -1)
-if [ "$NUMBACKUPS" -gt "$MAXBACKUPS" ]; then
-  rm -f /nsm/backup/$OLDESTBACKUP
-fi
+while [ "$NUMBACKUPS" -gt "$MAXBACKUPS" ]; do
+  OLDESTBACKUP=$(find /nsm/backup/ -type f -name "so-config-backup*" -type f -printf '%T+ %p\n' | sort | head -n 1 | awk -F" " '{print $2}')
+  rm -f $OLDESTBACKUP
+  NUMBACKUPS=$(find /nsm/backup/ -type f -name "so-config-backup*" | wc -l)
+done

salt/common/tools/sbin/so-cortex-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-user-add

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -30,8 +30,8 @@ fi
 
 USER=$1
 
-CORTEX_KEY=$(lookup_pillar cortexkey)
-CORTEX_IP=$(lookup_pillar managerip)
+CORTEX_KEY=$(lookup_pillar cortexorguserkey)
+CORTEX_API_URL="$(lookup_pillar url_base)/cortex/api"
 CORTEX_ORG_NAME=$(lookup_pillar cortexorgname)
 CORTEX_USER=$USER
 
@@ -43,7 +43,7 @@ fi
 read -rs CORTEX_PASS
 
 # Create new user in Cortex
-resp=$(curl -sk -XPOST -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" "https://$CORTEX_IP/cortex/api/user" -d "{\"name\": \"$CORTEX_USER\",\"roles\": [\"read\",\"analyze\",\"orgadmin\"],\"organization\": \"$CORTEX_ORG_NAME\",\"login\": \"$CORTEX_USER\",\"password\" : \"$CORTEX_PASS\" }")
+resp=$(curl -sk -XPOST -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" -L "https://$CORTEX_API_URL/user" -d "{\"name\": \"$CORTEX_USER\",\"roles\": [\"read\",\"analyze\",\"orgadmin\"],\"organization\": \"$CORTEX_ORG_NAME\",\"login\": \"$CORTEX_USER\",\"password\" : \"$CORTEX_PASS\" }")
 if [[ "$resp" =~ \"status\":\"Ok\" ]]; then
     echo "Successfully added user to Cortex."
 else

salt/common/tools/sbin/so-cortex-user-enable

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -30,8 +30,8 @@ fi
 
 USER=$1
 
-CORTEX_KEY=$(lookup_pillar cortexkey)
-CORTEX_IP=$(lookup_pillar managerip)
+CORTEX_KEY=$(lookup_pillar cortexorguserkey)
+CORTEX_API_URL="$(lookup_pillar url_base)/cortex/api"
 CORTEX_USER=$USER
 
 case "${2^^}" in
@@ -46,7 +46,7 @@ case "${2^^}" in
 		;;
 esac
 
-resp=$(curl -sk -XPATCH -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" "https://$CORTEX_IP/cortex/api/user/${CORTEX_USER}" -d "{\"status\":\"${CORTEX_STATUS}\" }")
+resp=$(curl -sk -XPATCH -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" -L "https://$CORTEX_API_URL/user/${CORTEX_USER}" -d "{\"status\":\"${CORTEX_STATUS}\" }")
 if [[ "$resp" =~ \"status\":\"Locked\" || "$resp" =~ \"status\":\"Ok\" ]]; then
     echo "Successfully updated user in Cortex."
 else

salt/common/tools/sbin/so-curator-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-curator-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-curator-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-deny

@@ -0,0 +1,213 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import ipaddress
+import textwrap
+import os
+import subprocess
+import sys
+import argparse
+import re
+from lxml import etree as ET
+from xml.dom import minidom
+
+
+LOCAL_SALT_DIR='/opt/so/saltstack/local'
+WAZUH_CONF='/nsm/wazuh/etc/ossec.conf'
+VALID_ROLES = {
+  'a': { 'role': 'analyst','desc': 'Analyst - 80/tcp, 443/tcp' },
+  'b': { 'role': 'beats_endpoint', 'desc': 'Logstash Beat - 5044/tcp' },
+  'e': { 'role': 'elasticsearch_rest', 'desc': 'Elasticsearch REST API - 9200/tcp' },
+  'f': { 'role': 'strelka_frontend', 'desc': 'Strelka frontend - 57314/tcp' },
+  'o': { 'role': 'osquery_endpoint', 'desc': 'Osquery endpoint - 8090/tcp' },
+  's': { 'role': 'syslog', 'desc': 'Syslog device - 514/tcp/udp' },
+  'w': { 'role': 'wazuh_agent', 'desc': 'Wazuh agent - 1514/tcp/udp' },
+  'p': { 'role': 'wazuh_api', 'desc': 'Wazuh API - 55000/tcp' },
+  'r': { 'role': 'wazuh_authd', 'desc': 'Wazuh registration service - 1515/tcp' }
+}
+
+
+def validate_ip_cidr(ip_cidr: str) -> bool:
+  try:
+    ipaddress.ip_address(ip_cidr)
+  except ValueError:
+    try:
+      ipaddress.ip_network(ip_cidr)
+    except ValueError:
+      return False
+  return True
+
+
+def role_prompt() -> str:
+  print()
+  print('Choose the role for the IP or Range you would like to deny')
+  print()
+  for role in VALID_ROLES:
+    print(f'[{role}] - {VALID_ROLES[role]["desc"]}')
+  print()
+  role = input('Please enter your selection: ')
+  if role in VALID_ROLES.keys():
+    return VALID_ROLES[role]['role']
+  else:
+    print(f'Invalid role \'{role}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+  
+
+def ip_prompt() -> str:
+  ip = input('Enter a single ip address or range to deny (ex: 10.10.10.10 or 10.10.0.0/16): ')
+  if validate_ip_cidr(ip):
+    return ip
+  else:
+    print(f'Invalid IP address or CIDR block \'{ip}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+
+
+def wazuh_enabled() -> bool:
+  for file in os.listdir(f'{LOCAL_SALT_DIR}/pillar'):
+    with open(file, 'r') as pillar:
+      if 'wazuh: 1' in pillar.read():
+        return True
+  return False
+
+
+def root_to_str(root: ET.ElementTree) -> str:
+    xml_str = ET.tostring(root, encoding='unicode', method='xml').replace('\n', '')
+    xml_str = re.sub(r'(?:(?<=>) *)', '', xml_str)
+
+    # Remove specific substrings to better format comments on intial parse/write
+    xml_str = re.sub(r'       -', '', xml_str)
+    xml_str = re.sub(r'      -->', ' -->', xml_str)
+    
+    dom = minidom.parseString(xml_str)
+    return dom.toprettyxml(indent="  ")
+
+
+def rem_wl(ip):
+    parser = ET.XMLParser(remove_blank_text=True)
+    with open(WAZUH_CONF, 'rb') as wazuh_conf:
+        tree = ET.parse(wazuh_conf, parser)
+    root = tree.getroot()
+
+    global_elems = root.findall(f"global/white_list[. = '{ip}']/..")
+    if len(global_elems) > 0:
+      for g_elem in global_elems:
+          ge_index = list(root).index(g_elem)
+          if ge_index > 0 and root[list(root).index(g_elem) - 1].tag == ET.Comment:
+              root.remove(root[ge_index - 1])
+          root.remove(g_elem)
+
+      with open(WAZUH_CONF, 'w') as out:
+          out.write(root_to_str(root))
+
+
+def apply(role: str, ip: str) -> int:
+  firewall_cmd = ['so-firewall', 'excludehost', role, ip]
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'firewall', 'queue=True']
+  restart_wazuh_cmd = ['so-wazuh-restart']
+  print(f'Removing {ip} from the {role} role. This can take a few seconds...')
+  cmd = subprocess.run(firewall_cmd)
+  if cmd.returncode == 0:
+    cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  else:
+    return cmd.returncode
+  if cmd.returncode == 0:
+    if wazuh_enabled and role=='analyst':
+      try:
+        rem_wl(ip)
+        print(f'Removed whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+      except Exception as e:
+        print(f'Failed to remove whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+        print(e)
+        return 1
+      print('Restarting OSSEC Server...')
+      cmd = subprocess.run(restart_wazuh_cmd)
+    else:
+      return cmd.returncode
+  else:
+    print(f'Commmand \'{" ".join(salt_cmd)}\' failed.', file=sys.stderr)
+    return cmd.returncode
+  if cmd.returncode != 0:
+    print('Failed to restart OSSEC server.')
+  return cmd.returncode
+
+
+def main():
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=textwrap.dedent(f'''\
+        additional information:
+                      To use this script in interactive mode call it with no arguments
+        '''
+  ))
+
+  group = main_parser.add_argument_group(title='roles')
+  group.add_argument('-a', dest='roles', action='append_const', const=VALID_ROLES['a']['role'], help="Analyst - 80/tcp, 443/tcp")
+  group.add_argument('-b', dest='roles', action='append_const', const=VALID_ROLES['b']['role'], help="Logstash Beat - 5044/tcp")
+  group.add_argument('-e', dest='roles', action='append_const', const=VALID_ROLES['e']['role'], help="Elasticsearch REST API - 9200/tcp")
+  group.add_argument('-f', dest='roles', action='append_const', const=VALID_ROLES['f']['role'], help="Strelka frontend - 57314/tcp")
+  group.add_argument('-o', dest='roles', action='append_const', const=VALID_ROLES['o']['role'], help="Osquery endpoint - 8090/tcp")
+  group.add_argument('-s', dest='roles', action='append_const', const=VALID_ROLES['s']['role'], help="Syslog device - 514/tcp/udp")
+  group.add_argument('-w', dest='roles', action='append_const', const=VALID_ROLES['w']['role'], help="Wazuh agent - 1514/tcp/udp")
+  group.add_argument('-p', dest='roles', action='append_const', const=VALID_ROLES['p']['role'], help="Wazuh API - 55000/tcp")
+  group.add_argument('-r', dest='roles', action='append_const', const=VALID_ROLES['r']['role'], help="Wazuh registration service - 1515/tcp")
+  
+  ip_g = main_parser.add_argument_group(title='allow')
+  ip_g.add_argument('-i', help="IP or CIDR block to disallow connections from, requires at least one role argument", metavar='', dest='ip')
+
+  args = main_parser.parse_args(sys.argv[1:])
+
+  if args.roles is None:
+    role = role_prompt()
+    ip = ip_prompt()
+    try:
+      return_code = apply(role, ip)
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+    sys.exit(return_code)
+  elif args.roles is not None and args.ip is None:
+    if os.environ.get('IP') is None:
+      main_parser.print_help()
+      sys.exit(1)
+    else:
+      args.ip = os.environ['IP']
+  
+  if validate_ip_cidr(args.ip):
+    try:
+      for role in args.roles:
+        return_code = apply(role, args.ip)
+        if return_code > 0:
+          break
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+  else:
+    print(f'Invalid IP address or CIDR block \'{args.ip}\', please try again.', file=sys.stderr)
+    return_code = 1
+    
+  sys.exit(return_code)
+
+
+if __name__ == '__main__':
+  try:
+    main()
+  except KeyboardInterrupt:
+    sys.exit(1)

salt/common/tools/sbin/so-docker-prune

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys, argparse, re, docker
+from packaging.version import Version, InvalidVersion
+from itertools import groupby, chain
+
+
+def get_image_name(string) -> str:
+  return ':'.join(string.split(':')[:-1])
+
+
+def get_so_image_basename(string) -> str:
+  return get_image_name(string).split('/so-')[-1]
+
+
+def get_image_version(string) -> str:
+  ver = string.split(':')[-1]
+  if ver == 'latest':
+    # Version doesn't like "latest", so use a high semver
+    return '99999.9.9'
+  else:
+    try:
+      Version(ver)
+    except InvalidVersion:
+      # Also return a very high semver for any version 
+      # with a dash in it since it will likely be a dev version of some kind
+      if '-' in ver:
+        return '999999.9.9'
+    return ver
+
+
+def main(quiet):
+  client = docker.from_env()
+
+  # Prune old/stopped containers
+  if not quiet: print('Pruning old containers')
+  client.containers.prune()
+
+  image_list = client.images.list(filters={ 'dangling': False })
+
+  # Map list of image objects to flattened list of tags (format: "name:version")
+  tag_list = list(chain.from_iterable(list(map(lambda x: x.attrs.get('RepoTags'), image_list))))
+
+  # Filter to only SO images (base name begins with "so-")
+  tag_list = list(filter(lambda x: re.match(r'^.*\/so-[^\/]*$', get_image_name(x)), tag_list))
+
+  # Group tags into lists by base name (sort by same projection first)
+  tag_list.sort(key=lambda x: get_so_image_basename(x))
+  grouped_tag_lists = [ list(it) for _, it in groupby(tag_list, lambda x: get_so_image_basename(x)) ]
+
+  no_prunable = True
+  for t_list in grouped_tag_lists:
+    try:
+      # Group tags by version, in case multiple images exist with the same version string
+      t_list.sort(key=lambda x: Version(get_image_version(x)), reverse=True)
+      grouped_t_list = [ list(it) for _,it in groupby(t_list, lambda x: get_image_version(x)) ]
+
+      # Keep the 2 most current version groups
+      if len(grouped_t_list) <= 2:
+        continue
+      else:
+        no_prunable = False
+        for group in grouped_t_list[2:]:
+          for tag in group:
+            if not quiet: print(f'Removing image {tag}')
+            try:
+              client.images.remove(tag, force=True)
+            except docker.errors.ClientError as e:
+              print(f'Could not remove image {tag}, continuing...')
+    except (docker.errors.APIError, InvalidVersion) as e:
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+    except Exception as e:
+      print('Unhandled exception occurred:')
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+
+  if no_prunable and not quiet:
+    print('No Security Onion images to prune')
+
+
+if __name__ == "__main__":
+  main_parser = argparse.ArgumentParser(add_help=False)
+  main_parser.add_argument('-q', '--quiet', action='store_const', const=True, required=False)
+  args = main_parser.parse_args(sys.argv[1:])
+ 
+  main(args.quiet)

salt/common/tools/sbin/so-docker-refresh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -16,96 +16,7 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 . /usr/sbin/so-common
+. /usr/sbin/so-image-common
 
-manager_check() {
-  # Check to see if this is a manager
-  MANAGERCHECK=$(cat /etc/salt/grains | grep role | awk '{print $2}')
-  if [ $MANAGERCHECK == 'so-eval' ] || [ $MANAGERCHECK == 'so-manager' ] || [ $MANAGERCHECK == 'so-managersearch' ] || [ $MANAGERCHECK == 'so-standalone' ] || [ $MANAGERCHECK == 'so-helix' ]; then
-    echo "This is a manager. We can proceed"
-  else
-    echo "Please run soup on the manager. The manager controls all updates."
-    exit 1
-  fi
-}
-
-update_docker_containers() {
-  
-  # Download the containers from the interwebs
-  for i in "${TRUSTED_CONTAINERS[@]}"
-  do
-    # Pull down the trusted docker image
-    echo "Downloading $i"
-    docker pull --disable-content-trust=false docker.io/$IMAGEREPO/$i
-    # Tag it with the new registry destination
-    docker tag $IMAGEREPO/$i $HOSTNAME:5000/$IMAGEREPO/$i
-    docker push $HOSTNAME:5000/$IMAGEREPO/$i
-  done
-  
-}
-
-version_check() {
-  if [ -f /etc/soversion ]; then
-    VERSION=$(cat /etc/soversion)
-  else
-    echo "Unable to detect version. I will now terminate."
-    exit 1
-  fi
-}
-
-manager_check
-version_check
-
-# Use the hostname
-HOSTNAME=$(hostname)
-# List all the containers
-if [ $MANAGERCHECK != 'so-helix' ]; then
-    TRUSTED_CONTAINERS=( \
-    "so-acng:$VERSION" \
-    "so-thehive-cortex:$VERSION" \
-    "so-curator:$VERSION" \
-    "so-domainstats:$VERSION" \
-    "so-elastalert:$VERSION" \
-    "so-elasticsearch:$VERSION" \
-    "so-filebeat:$VERSION" \
-    "so-fleet:$VERSION" \
-    "so-fleet-launcher:$VERSION" \
-    "so-freqserver:$VERSION" \
-    "so-grafana:$VERSION" \
-    "so-idstools:$VERSION" \
-    "so-influxdb:$VERSION" \
-    "so-kibana:$VERSION" \
-    "so-kratos:$VERSION" \
-    "so-logstash:$VERSION" \
-    "so-minio:$VERSION" \
-    "so-mysql:$VERSION" \
-    "so-nginx:$VERSION" \
-    "so-pcaptools:$VERSION" \
-    "so-playbook:$VERSION" \
-    "so-redis:$VERSION" \
-    "so-soc:$VERSION" \
-    "so-soctopus:$VERSION" \
-    "so-steno:$VERSION" \
-    "so-strelka-frontend:$VERSION" \
-		"so-strelka-manager:$VERSION" \
-		"so-strelka-backend:$VERSION" \
-		"so-strelka-filestream:$VERSION" \
-    "so-suricata:$VERSION" \
-    "so-telegraf:$VERSION" \
-    "so-thehive:$VERSION" \
-    "so-thehive-es:$VERSION" \
-    "so-wazuh:$VERSION" \
-    "so-zeek:$VERSION" )
-  else
-    TRUSTED_CONTAINERS=( \
-    "so-filebeat:$VERSION" \
-    "so-idstools:$VERSION" \
-    "so-logstash:$VERSION" \
-    "so-nginx:$VERSION" \
-    "so-redis:$VERSION" \
-    "so-steno:$VERSION" \
-    "so-suricata:$VERSION" \
-    "so-telegraf:$VERSION" \
-    "so-zeek:$VERSION" )
-  fi
-
-update_docker_containers
+require_manager
+update_docker_containers "refresh"

salt/common/tools/sbin/so-elastalert-create

@@ -145,9 +145,9 @@ EOF
    rulename=$(echo ${raw_rulename,,} | sed 's/ /_/g')
 
 cat << EOF >> "$rulename.yaml"
-# Elasticsearch Host
-es_host: elasticsearch
-es_port: 9200
+# Elasticsearch Host Override (optional)
+# es_host: elasticsearch
+# es_port: 9200
 
 # (Required)
 # Rule name, must be unique

salt/common/tools/sbin/so-elastalert-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-test

@@ -19,8 +19,7 @@
 #
 # Purpose:  This script will allow you to test your elastalert rule without entering the Docker container.
 
-. /usr/sbin/so-elastic-common
-
+HOST_RULE_DIR=/opt/so/rules/elastalert
 OPTIONS=""
 SKIP=0
 RESULTS_TO_LOG="n"
@@ -29,114 +28,109 @@ FILE_SAVE_LOCATION=""
 
 usage()
 {
-cat <<EOF
+	cat <<EOF
 
 Test Elastalert Rule
   Options:
-  -h         		This message
-  -a         		Trigger real alerts instead of the debug alert
-  -l <path_to_file>     Write results to specified log file
-  -o '<options>'	Specify Elastalert options ( Ex. --schema-only , --count-only, --days N ) 
-  -r <rule_name>        Specify path/name of rule to test
+  -h                This message
+  -a                Trigger real alerts instead of the debug alert
+  -l <path_to_file> Write results to specified log file
+  -o '<options>'    Specify Elastalert options ( Ex. --schema-only , --count-only, --days N ) 
+  -r <rule_name>    Specify filename of rule to test (must exist in $HOST_RULE_DIR; do not include path)
 
 EOF
 }
 
 while getopts "hal:o:r:" OPTION
 do
-        case $OPTION in
-                h)
-                        usage
-                        exit 0
-                        ;;
-                a)
-                        OPTIONS="--alert"
-                        ;;
-                l)
-                        RESULTS_TO_LOG="y"
-                        FILE_SAVE_LOCATION=$OPTARG
-                        ;;
-                
-                o)
-			OPTIONS=$OPTARG
-                        ;;  
-                
-                r)
-                        RULE_NAME=$OPTARG
-                        SKIP=1
-                        ;;
-                *)
-                        usage
-                        exit 0
-                        ;;
-        esac
+	case $OPTION in
+	h)
+		usage
+		exit 0
+		;;
+	a)
+		OPTIONS="--alert"
+		;;
+	l)
+		RESULTS_TO_LOG="y"
+		FILE_SAVE_LOCATION=$OPTARG
+		;;		
+	o)
+		OPTIONS=$OPTARG
+		;;  	
+	r)
+		RULE_NAME=$OPTARG
+		SKIP=1
+		;;
+	*)
+		usage
+		exit 0
+		;;
+	esac
 done
 
 docker_exec(){
-	if [ ${RESULTS_TO_LOG,,} = "y" ] ; then
-        	docker exec -it so-elastalert bash -c "elastalert-test-rule $RULE_NAME $OPTIONS" > $FILE_SAVE_LOCATION
+	CMD="docker exec -it so-elastalert elastalert-test-rule /opt/elastalert/rules/$RULE_NAME --config /opt/elastalert/config.yaml $OPTIONS"
+	if [ "${RESULTS_TO_LOG,,}" = "y" ] ; then
+		$CMD > "$FILE_SAVE_LOCATION"
 	else
-        	docker exec -it so-elastalert bash -c "elastalert-test-rule $RULE_NAME $OPTIONS"
+		$CMD
 	fi	
 }
 
 rule_prompt(){
-	CURRENT_RULES=$(find /opt/so/rules/elastalert -name "*.yaml")
-        echo
-        echo "This script will allow you to test an Elastalert rule."
-        echo
-        echo "Below is a list of active Elastalert rules:"
-        echo
+	CURRENT_RULES=$(cd "$HOST_RULE_DIR" && find . -type f \( -name "*.yaml" -o -name "*.yml" \) | sed -e 's/^\.\///')
+	if [ -z "$CURRENT_RULES" ]; then
+		echo "There are no rules available to test. Rule files must be placed in the $HOST_RULE_DIR directory."
+		exit 1
+	fi
+	echo
+	echo "This script will allow you to test an Elastalert rule."
+	echo
+	echo "Below is a list of available Elastalert rules:"
+	echo
 	echo "-----------------------------------"
-        echo
-        echo "$CURRENT_RULES"
-        echo
+	echo
+	echo "$CURRENT_RULES"
+	echo
 	echo "-----------------------------------"
-	echo 
-        echo "Note: To test a rule it must be accessible by the Elastalert Docker container."
-        echo
-        echo "Make sure to swap the local path (/opt/so/rules/elastalert/) for the docker path (/etc/elastalert/rules/)"
-        echo "Example: /opt/so/rules/elastalert/nids2hive.yaml would be /etc/elastalert/rules/nids2hive.yaml"
-        echo
-        while [ -z $RULE_NAME ]; do
-		echo "Please enter the file path and rule name you want to test."
-		read -e RULE_NAME
+	echo
+	while [ -z "$RULE_NAME" ]; do
+		read -p "Choose a rule to test from the list above (must be typed exactly as shown above): " -e RULE_NAME
 	done
 }
 
 log_save_prompt(){
 	RESULTS_TO_LOG=""
-        while [ -z $RESULTS_TO_LOG ]; do
-		echo "The results can be rather long.  Would you like to write the results to a file? (Y/N)"
-		read RESULTS_TO_LOG
-	done
+	read -p "The results can be rather long.  Would you like to write the results to a file? (y/N) " -e RESULTS_TO_LOG
 }
 
 log_path_prompt(){
-        while [ -z $FILE_SAVE_LOCATION ]; do
-		echo "Please enter the file path and file name."
-		read -e FILE_SAVE_LOCATION
-        done
+		while [ -z "$FILE_SAVE_LOCATION" ]; do
+			read -p "Please enter the log file path and file name: " -e FILE_SAVE_LOCATION
+		done
 		echo "Depending on the rule this may take a while."
 }
 
 if [ $SKIP -eq 0 ]; then
 	rule_prompt
 	log_save_prompt
-        if [ ${RESULTS_TO_LOG,,} = "y" ] ; then
-        	log_path_prompt
-        fi
+	if [ "${RESULTS_TO_LOG,,}" = "y" ] ; then
+		log_path_prompt
+	fi
 fi
 
+echo 
+
 docker_exec
-
-if [ $? -eq 0 ]; then
-	echo "Test completed successfully!"
-else
-	echo "Something went wrong..."
-fi
+RESULT=$?
 
 echo
 
+if [ $RESULT -eq 0 ]; then
+	echo "Test completed successfully!"
+else
+	echo "Test failed."
+fi
 
-
+echo

salt/common/tools/sbin/so-elastic-auth

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if [ -f "/usr/sbin/so-common" ]; then
+  . /usr/sbin/so-common
+fi
+
+ES_AUTH_PILLAR=${ELASTIC_AUTH_PILLAR:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+ES_USERS_FILE=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+
+authEnable=$1
+
+if ! grep -q "enabled: " "$ES_AUTH_PILLAR"; then
+  echo "Elastic auth pillar file is invalid. Unable to proceed."
+  exit 1
+fi
+
+function restart() {
+  if [[ -z "$ELASTIC_AUTH_SKIP_HIGHSTATE" ]]; then
+    echo "Elasticsearch on all affected minions will now be stopped and then restarted..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' cmd.run so-elastic-stop queue=True
+    echo "Applying highstate to all affected minions..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.highstate queue=True
+  fi
+}
+
+if [[ "$authEnable" == "true" ]]; then
+  if grep -q "enabled: False" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: False/enabled: True/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now enabled."
+    if grep -q "argon" "$ES_USERS_FILE"; then
+      echo ""
+      echo "IMPORTANT: The following users will need to change their password, after logging into SOC, in order to access Kibana:"
+      grep argon "$ES_USERS_FILE" | cut -d ":" -f 1
+    fi
+  else
+    echo "Auth is already enabled."
+  fi
+elif [[ "$authEnable" == "false" ]]; then
+  if grep -q "enabled: True" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: True/enabled: False/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now disabled."
+  else
+    echo "Auth is already disabled."
+  fi
+else
+  echo "Usage: $0 <true|false>"
+  echo ""
+  echo "Toggles Elastic authentication. Elasticsearch will be restarted on each affected minion."
+  echo ""
+fi

salt/common/tools/sbin/so-elastic-auth-password-reset

@@ -0,0 +1,155 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+source $(dirname $0)/so-common
+require_manager
+
+user=$1
+elasticUsersFile=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+elasticAuthPillarFile=${ELASTIC_AUTH_PILLAR_FILE:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+
+if [[ $# -ne 1 ]]; then
+  echo "Usage: $0 <user>"
+  echo ""
+  echo " where <user> is one of the following:"
+  echo ""
+  echo "         all: Reset the password for the so_elastic, so_kibana, so_logstash, so_beats, and so_monitor users"
+  echo "  so_elastic: Reset the password for the so_elastic user"
+  echo "   so_kibana: Reset the password for the so_kibana user"
+  echo " so_logstash: Reset the password for the so_logstash user"
+  echo "    so_beats: Reset the password for the so_beats user"
+  echo "  so_monitor: Reset the password for the so_monitor user"
+  echo ""
+  exit 1
+fi
+
+# function to create a lock so that the so-user sync cronjob can't run while this is running
+function lock() {
+  # Obtain file descriptor lock
+  exec 99>/var/tmp/so-user.lock || fail "Unable to create lock descriptor; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  flock -w 10 99 || fail "Another process is using so-user; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  trap 'rm -f /var/tmp/so-user.lock' EXIT
+}
+
+function unlock() {
+  rm -f /var/tmp/so-user.lock
+}
+
+function fail() {
+  msg=$1
+  echo "$1"
+  exit 1
+}
+
+function removeSingleUserPass() {
+  local user=$1
+  sed -i '/user: '"${user}"'/{N;/pass: /d}' "${elasticAuthPillarFile}"
+}
+
+function removeAllUserPass() {
+  local userList=("so_elastic" "so_kibana" "so_logstash" "so_beats" "so_monitor")
+
+  for u in ${userList[@]}; do
+    removeSingleUserPass "$u"
+  done
+}
+
+function removeElasticUsersFile() {
+  rm -f "$elasticUsersFile"
+}
+
+function createElasticAuthPillar() {
+  salt-call state.apply elasticsearch.auth queue=True
+}
+
+# this will disable highstate to prevent a highstate from starting while the script is running
+# will also disable salt.minion-state-apply-test allow so-salt-minion-check cronjob to restart salt-minion service incase
+function disableSaltStates() {
+  printf "\nDisabling salt.minion-state-apply-test and highstate from running.\n\n"
+  salt-call state.disable salt.minion-state-apply-test
+  salt-call state.disable highstate
+}
+
+function enableSaltStates() {
+  printf "\nEnabling salt.minion-state-apply-test and highstate.\n\n"
+  salt-call state.enable salt.minion-state-apply-test
+  salt-call state.enable highstate
+}
+
+function killAllSaltJobs() {
+  printf "\nKilling all running salt jobs.\n\n"
+  salt-call saltutil.kill_all_jobs
+}
+
+function soUserSync() {
+  # apply this state to update /opt/so/saltstack/local/salt/elasticsearch/curl.config on the manager
+  salt-call state.sls_id elastic_curl_config_distributed manager queue=True
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' saltutil.kill_all_jobs
+  # apply this state to get the curl.config
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.sls_id elastic_curl_config common queue=True
+  $(dirname $0)/so-user sync
+  printf "\nApplying logstash state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply logstash queue=True
+  printf "\nApplying filebeat state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode or G@role:so-sensor or G@role:so-fleet' state.apply filebeat queue=True
+  printf "\nApplying kibana state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch' state.apply kibana queue=True
+  printf "\nApplying curator state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply curator queue=True
+}
+
+function highstateManager() {
+  killAllSaltJobs
+  printf "\nRunning highstate on the manager to finalize password reset.\n\n"
+  salt-call state.highstate -linfo queue=True
+}
+
+case "${user}" in
+
+  so_elastic | so_kibana | so_logstash | so_beats | so_monitor)
+    lock
+    killAllSaltJobs
+    disableSaltStates
+    removeSingleUserPass "$user"
+    createElasticAuthPillar
+    removeElasticUsersFile
+    unlock
+    soUserSync
+    enableSaltStates
+    highstateManager
+    ;;
+
+  all)
+    lock
+    killAllSaltJobs
+    disableSaltStates
+    removeAllUserPass
+    createElasticAuthPillar
+    removeElasticUsersFile
+    unlock
+    soUserSync
+    enableSaltStates
+    highstateManager
+    ;;
+
+  *)
+    fail "Unsupported user: $user"
+    ;;
+
+esac
+
+exit 0

salt/common/tools/sbin/so-elastic-clear

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -50,11 +50,7 @@ done
 if [ $SKIP -ne 1 ]; then
         # List indices
         echo 
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -k https://{{ NODEIP }}:9200/_cat/indices?v
-        {% else %}
-        curl {{ NODEIP }}:9200/_cat/indices?v
-        {% endif %}
+        {{ ELASTICCURL }} -k -L https://{{ NODEIP }}:9200/_cat/indices?v
         echo
         # Inform user we are about to delete all data
         echo
@@ -93,18 +89,10 @@ fi
 # Delete data
 echo "Deleting data..."
 
-{% if grains['role'] in ['so-node','so-heavynode'] %}
-INDXS=$(curl -s -XGET -k https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
-{% else %}
-INDXS=$(curl -s -XGET {{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
-{% endif %}
+INDXS=$({{ ELASTICCURL }} -s -XGET -k -L https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
 for INDX in ${INDXS}
 do
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -XDELETE -k https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
-    {% else %}
-    curl -XDELETE "{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
-    {% endif %}
+    {{ ELASTICCURL }} -XDELETE -k -L https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
 done
 
 #Start Logstash/Filebeat

salt/common/tools/sbin/so-elastic-diagnose

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-restart

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-import']%}
+/usr/sbin/so-restart elasticsearch $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone', 'so-import']%}
+/usr/sbin/so-restart kibana $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-restart logstash $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-sensor']%}
+/usr/sbin/so-restart filebeat $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-restart curator $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone']%}
+/usr/sbin/so-restart elastalert $1
+{%- endif %}

salt/common/tools/sbin/so-elastic-start

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-import']%}
+/usr/sbin/so-start elasticsearch $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone', 'so-import']%}
+/usr/sbin/so-start kibana $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-start logstash $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-sensor']%}
+/usr/sbin/so-start filebeat $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-start curator $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone']%}
+/usr/sbin/so-start elastalert $1
+{%- endif %}

salt/common/tools/sbin/so-elastic-stop

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-import']%}
+/usr/sbin/so-stop elasticsearch $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone', 'so-import']%}
+/usr/sbin/so-stop kibana $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-stop logstash $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-sensor']%}
+/usr/sbin/so-stop filebeat $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-stop curator $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone']%}
+/usr/sbin/so-stop elastalert $1
+{%- endif %}

salt/common/tools/sbin/so-elasticsearch-indices-list

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/indices?pretty

salt/common/tools/sbin/so-elasticsearch-indices-rw

@@ -1,7 +1,7 @@
 #!/bin/bash
 #
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,6 +21,5 @@ THEHIVEESPORT=9400
 
 echo "Removing read only attributes for indices..."
 echo
-for p in $ESPORT $THEHIVEESPORT; do
-   curl -XPUT -H "Content-Type: application/json" http://$IP:$p/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
-done
+{{ ELASTICCURL }} -s -k -XPUT -H "Content-Type: application/json" -L https://$IP:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
+{{ ELASTICCURL }} -XPUT -H "Content-Type: application/json" -L http://$IP:9400/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;

salt/common/tools/sbin/so-elasticsearch-pipeline-stats

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -19,15 +19,7 @@
 . /usr/sbin/so-common
 
 if [ "$1" == "" ]; then
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -s -k https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
-        {% else %}
-        curl -s {{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
-        {% endif %}
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
 else
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -s -k https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
-        {% else %}
-        curl -s {{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
-        {% endif %}
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
 fi

salt/common/tools/sbin/so-elasticsearch-pipeline-view

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq .
+else
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq .[]
+fi

salt/common/tools/sbin/so-elasticsearch-pipelines-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,15 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
-    {% else %}
-    curl -s {{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
 else
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
-    {% else %}
-    curl -s {{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-query

@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+. /usr/sbin/so-common
+
+if [[ $# -lt 1 ]]; then
+	echo "Submit a cURL request to the local Security Onion Elasticsearch host."
+	echo ""
+	echo "Usage: $0 <PATH> [ARGS,...]"
+	echo ""
+  echo "Where "
+  echo "   PATH represents the elastic function being requested."
+  echo "   ARGS is used to specify additional, optional curl parameters."
+  echo ""
+  echo "Examples:"
+  echo "   $0 /"
+  echo "   $0 '*:so-*/_search' -d '{\"query\": {\"match_all\": {}},\"size\": 1}' | jq"
+  exit 1
+fi
+
+QUERYPATH=$1
+shift
+
+{{ ELASTICCURL }} -s -k -L -H "Content-Type: application/json" "https://localhost:9200/${QUERYPATH}" "$@"

salt/common/tools/sbin/so-elasticsearch-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-roles-load

@@ -0,0 +1,57 @@
+#!/bin/bash
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
+default_conf_dir=/opt/so/conf
+ELASTICSEARCH_HOST="{{ MYIP }}"
+ELASTICSEARCH_PORT=9200
+
+# Define a default directory to load roles from
+ELASTICSEARCH_ROLES="$default_conf_dir/elasticsearch/roles/"
+
+# Wait for ElasticSearch to initialize
+echo -n "Waiting for ElasticSearch..."
+COUNT=0
+ELASTICSEARCH_CONNECTED="no"
+while [[ "$COUNT" -le 240 ]]; do
+    {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+	if [ $? -eq 0 ]; then
+		ELASTICSEARCH_CONNECTED="yes"
+		echo "connected!"
+		break
+	else
+		((COUNT+=1))
+		sleep 1
+		echo -n "."
+	fi
+done
+if [ "$ELASTICSEARCH_CONNECTED" == "no" ]; then
+	echo
+	echo -e "Connection attempt timed out.  Unable to connect to ElasticSearch.  \nPlease try: \n  -checking log(s) in /var/log/elasticsearch/\n  -running 'sudo docker ps' \n  -running 'sudo so-elastic-restart'"
+	echo
+fi
+
+cd ${ELASTICSEARCH_ROLES}
+
+echo "Loading templates..."
+for role in *; do
+	name=$(echo "$role" | cut -d. -f1)
+	so-elasticsearch-query _security/role/$name -XPUT -d @"$role"
+done
+
+cd - >/dev/null

salt/common/tools/sbin/so-elasticsearch-shards-list

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/shards?pretty

salt/common/tools/sbin/so-elasticsearch-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by