Merge pull request #4863 from Security-Onion-Solutions/hotfix/2.3.60

Hotfix/2.3.60 CuratorFix

.github/ISSUE_TEMPLATE

@@ -0,0 +1,12 @@
+PLEASE STOP AND READ THIS INFORMATION!
+
+If you are creating an issue just to ask a question, you will likely get faster and better responses by posting to our discussions forum instead:
+https://securityonion.net/discuss
+
+If you think you have found a possible bug or are observing a behavior that you weren't expecting, use the discussion forum to start a conversation about it instead of creating an issue.
+
+If you are very familiar with the latest version of the product and are confident you have found a bug in Security Onion, you can continue with creating an issue here, but please make sure you have done the following:
+- duplicated the issue on a fresh installation of the latest version
+- provide information about your system and how you installed Security Onion
+- include relevant log files
+- include reproduction steps

.github/workflows/leaktest.yml

@@ -0,0 +1,15 @@
+name: leak-test
+
+on: [pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        fetch-depth: '0'
+
+    - name: Gitleaks
+      uses: zricethezav/gitleaks-action@master

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Contributing to Security Onion
+
+### Questions, suggestions, and general comments
+* Security Onion uses GitHub's [Discussions](https://github.com/Security-Onion-Solutions/securityonion/discussions) to provide a forum where the community and developers can interact as well as ask and answer questions.
+
+### Reporting a bug
+* The primary place to report unexpected behavior or possible bugs is the repo's [Discussions forum](https://github.com/Security-Onion-Solutions/securityonion/discussions).
+
+*  **If you are familiar with the current version of Security Onion and are confident you've discovered a bug**, first ensure there is not already an issue present by searching the open [issues](https://github.com/Security-Onion-Solutions/securityonion/issues). If there is, a thumbs up :+1: is a great way to show this bug is affecting you too.
+
+* If an issue doesn't exist, [open a new one](https://github.com/Security-Onion-Solutions/securityonion/issues/new), following the directions in the issue template. This means including:
+  * **System information** and how Security Onion was installed
+  * **Log files** relevant to the bug report
+  * **Reproduction steps** 
+
+### Contributing code
+
+* **All commits must be signed** with a valid key that has been added to your GitHub account. The commits should have all the "**Verified**" tag when viewed on GitHub as shown below:
+  
+  <img src="./assets/images/verified-commit-1.png" width="450">
+
+* If an issue does not already exist for the bug or feature for which you are submitting a pull request, [create one](https://github.com/Security-Onion-Solutions/securityonion/issues/new) with the relevant prefix. (**`FIX:`** for bug fixes, **`FEATURE:`** for new features.)
+
+* Link the PR to the related issue, either using [keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) in the PR description, or [manually](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue).
+
+* **Pull requests should be opened against the `dev` branch of this repo**, and should clearly describe the problem and solution.
+
+* Be sure you have tested your changes and are confident they will not break other parts of the product.
+
+* See this document's [code styling and conventions section](#code-style-and-conventions) below to be sure your PR fits our code requirements prior to submitting.
+
+
+
+### Code style and conventions
+* **Keep code [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)**. For example, Bash code used by multiple scripts will likely best be added to <span style="white-space: nowrap;">[`so-common`](salt/common/tools/sbin/so-common)</span>.
+
+* All new Bash code should pass [ShellCheck](https://www.shellcheck.net/) analysis. Where errors can be *safely* [ignored](https://github.com/koalaman/shellcheck/wiki/Ignore), the relevant disable directive should be accompanied by a brief explanation as to why the error is being ignored.
+
+* **Ensure all YAML (this includes Salt states and pillars) is properly formatted**. The spec for YAML v1.2 can be found [here](https://yaml.org/spec/1.2/spec.html), however there are numerous online resources with simpler descriptions of its formatting rules. 

HOTFIX

@@ -0,0 +1 @@
+ECSFIX HEAVYNODE_SSL_LOGSTASH_REDIS_PIPELINES FBPIPELINE CURATORAUTH

KEYS

@@ -1,4 +1,5 @@
 -----BEGIN PGP PUBLIC KEY BLOCK-----
+
 mQINBF7rzwEBEADBg87uJhnC3Ls7s60hbHGaywGrPtbz2WuYA/ev3YS3X7WS75p8
 PGlzTWUCujx0pEHbK2vYfExl3zksZ8ZmLyZ9VB3oSLiWBzJgKAeB7YCFEo8te+eE
 P2Z+8c+kX4eOV+2waxZyewA2TipSkhWgStSI4Ow8SyVUcUWA3hCw7mo2duNVi7KO

README.md

@@ -1,7 +1,14 @@
-## Security Onion 2.3.2
+## Security Onion 2.3.60
 
-Security Onion 2.3.2 is here!
+Security Onion 2.3.60 is here!
 
+## Screenshots
+
+Alerts
+![Alerts](./assets/images/screenshots/alerts-1.png)
+
+Hunt
+![Hunt](./assets/images/screenshots/hunt-1.png)
 
 ### Release Notes
 

SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x.x   | :white_check_mark: |
+| 16.04.x | :x:                |
+
+Security Onion 16.04 has reached End Of Life and is no longer supported.
+
+## Reporting a Vulnerability
+
+If you have any security concerns regarding Security Onion or believe you have uncovered a vulnerability, please follow these steps:
+
+- send an email to security@securityonion.net
+- include a description of the issue and steps to reproduce
+- please use plain text format (no Word documents or PDF files)
+- please do not disclose publicly until we have had sufficient time to resolve the issue
+
+This security address should be used only for undisclosed vulnerabilities. Dealing with fixed issues or general questions on how to use Security Onion should be handled via the normal support channels.

VERIFY_ISO.md

@@ -1,16 +1,18 @@
-### 2.3.2 ISO image built on 2020/10/25
+### 2.3.60-CURATORAUTH ISO image built on 2021/07/19
+
+
 
 ### Download and Verify
 
-2.3.2 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-2.3.2.iso
+2.3.60-CURATORAUTH ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-2.3.60-CURATORAUTH.iso
 
-MD5: EF2DEBCCBAE0B0BCCC906552B5FF918A  
-SHA1: 16AFCACB102BD217A038044D64E7A86DA351640E  
-SHA256: 7125F90B6323179D0D29F5745681BE995BD2615E64FA1E0046D94888A72C539E 
+MD5: 953DD42AB3A3560BB35F4E9F69212AE3  
+SHA1: 5D18B98B19FD7F8C799E88FC28ABC46990FC6B9B  
+SHA256: E26F43F969241985DC74915842492F876EC7B8CBAF5F2F52405554E7C92408C2 
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.2.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.60-CURATORAUTH.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS  
@@ -24,22 +26,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/ma
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.2.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.60-CURATORAUTH.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-2.3.2.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-2.3.60-CURATORAUTH.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-2.3.2.iso.sig securityonion-2.3.2.iso
+gpg --verify securityonion-2.3.60-CURATORAUTH.iso.sig securityonion-2.3.60-CURATORAUTH.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Sun 25 Oct 2020 10:44:27 AM EDT using RSA key ID FE507013
+gpg: Signature made Mon 19 Jul 2021 01:25:34 PM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

VERSION

@@ -1 +1 @@
-2.3.2
+2.3.60

files/salt/master/master

@@ -13,6 +13,8 @@
 # user: socore
 
 log_file: /opt/so/log/salt/master
+log_level_logfile: info
+log_level: info
 
 #####      File Server settings      #####
 ##########################################
@@ -65,3 +67,7 @@ peer:
 reactor:
   - 'so/fleet':
     - salt://reactor/fleet.sls
+  - 'salt/beacon/*/watch_sqlite_db//opt/so/conf/kratos/db/sqlite.db':
+    - salt://reactor/kratos.sls
+
+

pillar/data/addtotab.sh

@@ -54,7 +54,8 @@ if [ $TYPE == 'evaltab' ] || [ $TYPE == 'standalonetab' ]; then
     salt-call state.apply utility queue=True
   fi
 fi
-#if [ $TYPE == 'nodestab' ]; then
+if [ $TYPE == 'nodestab' ]; then
+  salt-call state.apply elasticsearch queue=True
 #  echo "    nodetype: $NODETYPE" >> $local_salt_dir/pillar/data/$TYPE.sls
 #  echo "    hotname: $HOTNAME" >> $local_salt_dir/pillar/data/$TYPE.sls
-#fi
+fi

pillar/docker/config.sls

@@ -1,208 +0,0 @@
-{%- set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
-{%- set FLEETNODE = salt['pillar.get']('global:fleet_node', False) -%}
-{% set WAZUH = salt['pillar.get']('manager:wazuh', '0') %}
-{% set THEHIVE = salt['pillar.get']('manager:thehive', '0') %}
-{% set PLAYBOOK = salt['pillar.get']('manager:playbook', '0') %}
-{% set FREQSERVER = salt['pillar.get']('manager:freq', '0') %}
-{% set DOMAINSTATS = salt['pillar.get']('manager:domainstats', '0') %}
-{% set ZEEKVER = salt['pillar.get']('global:mdengine', 'COMMUNITY') %}
-{% set GRAFANA = salt['pillar.get']('manager:grafana', '0') %}
-
-eval:
-  containers:
-    - so-nginx
-    - so-telegraf
-    {% if  GRAFANA == '1' %}
-    - so-influxdb
-    - so-grafana
-    {% endif %}
-    - so-dockerregistry
-    - so-soc
-    - so-kratos
-    - so-idstools
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    - so-elasticsearch
-    - so-logstash
-    - so-kibana
-    - so-steno
-    - so-suricata
-    - so-zeek
-    - so-curator
-    - so-elastalert
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-heavy_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-redis
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-steno
-    - so-suricata
-    - so-wazuh
-    - so-filebeat
-    {% if ZEEKVER != 'SURICATA' %}
-    - so-zeek
-    {% endif %}
-helix:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-idstools
-    - so-steno
-    - so-zeek
-    - so-redis
-    - so-logstash
-    - so-filebeat
-hot_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-manager_search:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-soc
-    - so-kratos
-    - so-acng
-    - so-idstools
-    - so-redis
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-kibana
-    - so-elastalert
-    - so-filebeat
-    - so-soctopus
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-manager:
-  containers:
-    - so-dockerregistry
-    - so-nginx
-    - so-telegraf
-    {% if  GRAFANA == '1' %}
-    - so-influxdb
-    - so-grafana
-    {% endif %}
-    - so-soc
-    - so-kratos
-    - so-acng
-    - so-idstools
-    - so-redis
-    - so-elasticsearch
-    - so-logstash
-    - so-kibana
-    - so-elastalert
-    - so-filebeat
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-parser_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-search_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-filebeat
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-sensor:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-steno
-    - so-suricata
-    {% if ZEEKVER != 'SURICATA' %}
-    - so-zeek
-    {% endif %}
-    - so-wazuh
-    - so-filebeat
-warm_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-elasticsearch
-fleet:
-  containers:
-    {% if FLEETNODE %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    - so-filebeat
-    - so-nginx
-    - so-telegraf
-    {% endif %}

pillar/firewall/ports.sls

@@ -1,65 +0,0 @@
-firewall:
-  analyst:
-    ports:
-      tcp:
-        - 80
-        - 443
-      udp:
-  beats_endpoint:
-    ports:
-      tcp:
-        - 5044
-  forward_nodes:
-    ports:
-      tcp:
-        - 443
-        - 5044
-        - 5644
-        - 9822
-      udp:
-  manager:
-    ports:
-      tcp:
-        - 1514
-        - 3200
-        - 3306
-        - 4200
-        - 5601
-        - 6379
-        - 7788
-        - 8086
-        - 8090
-        - 9001
-        - 9200
-        - 9300
-        - 9400  
-        - 9500
-        - 9595
-        - 9696
-      udp:
-        - 1514
-  minions:
-    ports:
-      tcp:
-        - 3142
-        - 4505
-        - 4506
-        - 5000
-        - 8080
-        - 8086
-        - 55000      
-  osquery_endpoint:
-    ports:
-      tcp:
-        - 8090
-  search_nodes:
-    ports:
-      tcp:
-        - 6379
-        - 9300
-  wazuh_endpoint:
-    ports:
-      tcp:
-        - 1514
-      udp: 
-        -1514

pillar/logrotate/init.sls

@@ -8,4 +8,6 @@ logrotate:
     create
     extension .log
     dateext
-    dateyesterday
+    dateyesterday
+  group_conf: |
+    su root socore

pillar/logstash/search.sls

@@ -7,7 +7,8 @@ logstash:
         - so/9000_output_zeek.conf.jinja
         - so/9002_output_import.conf.jinja
         - so/9034_output_syslog.conf.jinja
-        - so/9100_output_osquery.conf.jinja
+        - so/9050_output_filebeatmodules.conf.jinja
+        - so/9100_output_osquery.conf.jinja  
         - so/9400_output_suricata.conf.jinja
         - so/9500_output_beats.conf.jinja
         - so/9600_output_ossec.conf.jinja

pillar/top.sls

@@ -3,7 +3,7 @@ base:
     - patch.needs_restarting
     - logrotate
 
-  '*_eval or *_helix or *_heavynode or *_sensor or *_standalone or *_import':
+  '*_eval or *_helixsensor or *_heavynode or *_sensor or *_standalone or *_import':
     - match: compound
     - zeek
 
@@ -22,6 +22,9 @@ base:
   '*_manager or *_managersearch':
     - match: compound
     - data.*
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - secrets
     - global
     - minions.{{ grains.id }}
@@ -38,6 +41,9 @@ base:
     - secrets
     - healthcheck.eval
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - global
     - minions.{{ grains.id }}
 
@@ -46,6 +52,9 @@ base:
     - logstash.manager
     - logstash.search
     - elasticsearch.search
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - data.*
     - zeeklogs
     - secrets
@@ -59,10 +68,11 @@ base:
 
   '*_heavynode':
     - zeeklogs
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
 
-  '*_helix':
+  '*_helixsensor':
     - fireeye
     - zeeklogs
     - logstash
@@ -80,12 +90,17 @@ base:
     - logstash
     - logstash.search
     - elasticsearch.search
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
+    - data.nodestab
 
   '*_import':
     - zeeklogs
     - secrets
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - global
-    - minions.{{ grains.id }}
+    - minions.{{ grains.id }}

pillar/zeek/init.sls

@@ -52,5 +52,4 @@ zeek:
       - frameworks/signatures/detect-windows-shells
     redef:
       - LogAscii::use_json = T;
-      - LogAscii::json_timestamps = JSON::TS_ISO8601;
-      - CaptureLoss::watch_interval = 5 mins;
+      - CaptureLoss::watch_interval = 5 mins;

salt/_modules/so.py

@@ -1,4 +1,64 @@
 #!py
 
+import logging
+
 def status():
-  return __salt__['cmd.run']('/usr/sbin/so-status')
+    return __salt__['cmd.run']('/usr/sbin/so-status')
+
+
+def mysql_conn(retry):
+    log = logging.getLogger(__name__)
+
+    from time import sleep
+
+    try:
+        from MySQLdb import _mysql
+    except ImportError as e:
+        log.error(e)
+        return False
+
+    mainint = __salt__['pillar.get']('host:mainint')
+    ip_arr = __salt__['grains.get']('ip4_interfaces').get(mainint)
+
+    mysql_up = False
+
+    if len(ip_arr) == 1:
+        mainip = ip_arr[0]
+
+        if not(retry >= 1):
+            log.debug('`retry` set to value below 1, resetting it to 1 to prevent errors.')
+            retry = 1
+
+        for i in range(0, retry):
+            log.debug(f'Connection attempt {i+1}')
+            try:
+                db = _mysql.connect(
+                    host=mainip,
+                    user='root',
+                    passwd=__salt__['pillar.get']('secrets:mysql')
+                )
+                log.debug(f'Connected to MySQL server on {mainip} after {i+1} attempts.')
+                
+                db.query("""SELECT 1;""")
+                log.debug(f'Successfully completed query against MySQL server on {mainip}')
+                
+                db.close()
+                mysql_up = True
+                break
+            except _mysql.OperationalError as e:
+                log.debug(e)
+            except Exception as e:
+                log.error('Unexpected error occured.')
+                log.error(e)
+                break
+            sleep(1)
+
+        if not mysql_up:
+            log.error(f'Could not connect to MySQL server on {mainip} after {retry} attempts.')
+    else:
+        log.error(f'Main interface {mainint} has more than one IP address assigned to it, which is not supported.')
+        log.debug(f'{mainint}:')
+        for addr in ip_arr:
+            log.debug(f'  - {addr}')
+
+    return mysql_up

salt/airgap/init.sls

@@ -1,60 +0,0 @@
-{% set MANAGER = salt['grains.get']('master') %}
-airgapyum:
-  file.managed:
-    - name: /etc/yum/yum.conf
-    - source: salt://airgap/files/yum.conf
-
-airgap_repo:
-  pkgrepo.managed:
-    - humanname: Airgap Repo
-    - baseurl: https://{{ MANAGER }}/repo
-    - gpgcheck: 0
-    - sslverify: 0
-
-agbase:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Base.repo
-
-agcr:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-CR.repo
-
-agdebug:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Debuginfo.repo
-
-agfasttrack:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-fasttrack.repo
-
-agmedia:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Media.repo
-
-agsources:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Sources.repo
-
-agvault:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Vault.repo
-
-agkernel:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-x86_64-kernel.repo
-
-agepel:
-  file.absent:
-    - name: /etc/yum.repos.d/epel.repo
-
-agtesting:
-  file.absent:
-    - name: /etc/yum.repos.d/epel-testing.repo
-
-agssrepo:
-  file.absent:
-    - name: /etc/yum.repos.d/saltstack.repo
-
-agwazrepo:
-  file.absent:
-    - name: /etc/yum.repos.d/wazuh.repo

salt/allowed_states.map.jinja

@@ -0,0 +1,299 @@
+{% set ZEEKVER = salt['pillar.get']('global:mdengine', '') %}
+{% set WAZUH = salt['pillar.get']('global:wazuh', '0') %}
+{% set THEHIVE = salt['pillar.get']('manager:thehive', '0') %}
+{% set PLAYBOOK = salt['pillar.get']('manager:playbook', '0') %}
+{% set FREQSERVER = salt['pillar.get']('manager:freq', '0') %}
+{% set DOMAINSTATS = salt['pillar.get']('manager:domainstats', '0') %}
+{% set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) %}
+{% set FLEETNODE = salt['pillar.get']('global:fleet_node', False) %}
+{% set ELASTALERT = salt['pillar.get']('elastalert:enabled', True) %}
+{% set ELASTICSEARCH = salt['pillar.get']('elasticsearch:enabled', True) %}
+{% set FILEBEAT = salt['pillar.get']('filebeat:enabled', True) %}
+{% set KIBANA = salt['pillar.get']('kibana:enabled', True) %}
+{% set LOGSTASH = salt['pillar.get']('logstash:enabled', True) %}
+{% set CURATOR = salt['pillar.get']('curator:enabled', True) %}
+{% set REDIS = salt['pillar.get']('redis:enabled', True) %}
+{% set STRELKA = salt['pillar.get']('strelka:enabled', '0') %}
+{% set ISAIRGAP = salt['pillar.get']('global:airgap', False) %}
+{% import_yaml 'salt/minion.defaults.yaml' as saltversion %}
+{% set saltversion = saltversion.salt.minion.version %}
+
+{# this is the list we are returning from this map file, it gets built below #}
+{% set allowed_states= [] %}
+
+{% if grains.saltversion | string == saltversion | string %}
+
+  {% set allowed_states= salt['grains.filter_by']({
+      'so-eval': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'healthcheck',
+          'pcap',
+          'suricata',
+          'utility',
+          'schedule',
+          'soctopus',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-heavynode': [
+          'ca',
+          'ssl',
+          'nginx',
+          'telegraf',
+          'firewall',
+          'pcap',
+          'suricata',
+          'healthcheck',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-helixsensor': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'telegraf',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'zeek',
+          'redis',
+          'elasticsearch',
+          'logstash',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-fleet': [
+          'ca',
+          'ssl',
+          'nginx',
+          'telegraf',
+          'firewall',
+          'mysql',
+          'redis',
+          'fleet',
+          'fleet.install_package',
+          'filebeat',
+          'schedule',
+          'docker_clean'
+          ],
+      'so-import': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'soc',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'pcap',
+          'utility',
+          'suricata',
+          'zeek',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-manager': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'utility',
+          'schedule',
+          'soctopus',
+          'docker_clean'
+          ],
+      'so-managersearch': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'firewall',
+          'manager',
+          'idstools',
+          'suricata.manager',
+          'utility',
+          'schedule',
+          'soctopus',
+          'docker_clean'
+          ],
+      'so-node': [
+          'ca',
+          'ssl',
+          'nginx',
+          'telegraf',
+          'firewall',
+          'schedule',
+          'docker_clean'
+          ],
+      'so-standalone': [
+          'salt.master',
+          'ca',
+          'ssl',
+          'registry',
+          'manager',
+          'nginx',
+          'telegraf',
+          'influxdb',
+          'grafana',
+          'soc',
+          'firewall',
+          'idstools',
+          'suricata.manager',
+          'pcap',
+          'suricata',
+          'healthcheck',
+          'utility',
+          'schedule',
+          'soctopus',
+          'tcpreplay',
+          'docker_clean'
+          ],
+      'so-sensor': [
+          'ca',
+          'ssl',
+          'telegraf',
+          'firewall',
+          'nginx',
+          'pcap',
+          'suricata',
+          'healthcheck',
+          'wazuh',
+          'filebeat',
+          'schedule',
+          'tcpreplay',
+          'docker_clean'
+          ],
+  }, grain='role') %}
+
+  {% if FILEBEAT and grains.role in ['so-helixsensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-import'] %}
+    {% do allowed_states.append('filebeat') %}
+  {% endif %}
+
+  {% if ((FLEETMANAGER or FLEETNODE) or PLAYBOOK != 0) and grains.role in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone'] %}
+    {% do allowed_states.append('mysql') %}
+  {% endif %}
+
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-sensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('fleet.install_package') %}
+  {% endif %}
+
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('fleet') %}
+  {% endif %}
+
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-eval'] %}
+    {% do allowed_states.append('redis') %}
+  {% endif %}
+
+  {%- if ZEEKVER != 'SURICATA' and grains.role in ['so-sensor', 'so-eval', 'so-standalone', 'so-heavynode'] %}
+    {% do allowed_states.append('zeek') %}
+  {%- endif %}
+
+  {% if STRELKA and grains.role in ['so-sensor', 'so-eval', 'so-standalone', 'so-heavynode'] %}
+    {% do allowed_states.append('strelka') %}
+  {% endif %}
+
+  {% if WAZUH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode']%}
+    {% do allowed_states.append('wazuh') %}
+  {% endif %}
+
+  {% if ELASTICSEARCH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-import'] %}
+    {% do allowed_states.append('elasticsearch') %}
+  {% endif %}
+
+  {% if KIBANA and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    {% do allowed_states.append('kibana') %}
+  {% endif %}
+
+  {% if CURATOR and grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('curator') %}
+  {% endif %}
+
+  {% if ELASTALERT and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('elastalert') %}
+  {% endif %}
+
+  {% if (THEHIVE != 0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('thehive') %}
+  {% endif %}
+
+  {% if (PLAYBOOK !=0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('playbook') %}
+  {% endif %}
+
+  {% if (PLAYBOOK !=0) and grains.role in ['so-eval'] %}
+    {% do allowed_states.append('redis') %}
+  {% endif %}
+
+  {% if (FREQSERVER !=0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('freqserver') %}
+  {% endif %}
+
+  {% if (DOMAINSTATS !=0) and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch'] %}
+    {% do allowed_states.append('domainstats') %}
+  {% endif %}
+
+  {% if LOGSTASH and grains.role in ['so-helixsensor', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('logstash') %}
+  {% endif %}
+
+  {% if REDIS and grains.role in ['so-manager', 'so-standalone', 'so-managersearch', 'so-heavynode'] %}
+    {% do allowed_states.append('redis') %}
+  {% endif %}
+
+  {% if grains.os == 'CentOS' %}
+    {% if not ISAIRGAP %}
+      {% do allowed_states.append('yum') %}
+    {% endif %}
+    {% do allowed_states.append('yum.packages') %}
+  {% endif %}
+
+  {# all nodes on the right salt version can run the following states #}
+  {% do allowed_states.append('common') %}
+  {% do allowed_states.append('patch.os.schedule') %}
+  {% do allowed_states.append('motd') %}
+  {% do allowed_states.append('salt.minion-check') %}
+  {% do allowed_states.append('sensoroni') %}
+  {% do allowed_states.append('salt.lasthighstate') %}
+
+{% endif %}
+
+
+{% if ISAIRGAP %}
+  {% do allowed_states.append('airgap') %}
+{% endif %}
+
+{# all nodes can always run salt.minion state #}
+{% do allowed_states.append('salt.minion') %}

salt/ca/init.sls

@@ -1,7 +1,5 @@
-{% set show_top = salt['state.show_top']() %}
-{% set top_states = show_top.values() | join(', ') %}
-
-{% if 'ca' in top_states %}
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls in allowed_states %}
 
 {% set manager = salt['grains.get']('master') %}
 /etc/salt/minion.d/signing_policies.conf:
@@ -44,6 +42,10 @@ pki_private_key:
     - replace: False
     - require:
       - file: /etc/pki
+    - timeout: 30
+    - retry:
+        attempts: 5
+        interval: 30
 
 x509_pem_entries:
   module.run:
@@ -60,8 +62,8 @@ cakeyperms:
 
 {% else %}
 
-ca_state_not_allowed:
+{{sls}}_state_not_allowed:
   test.fail_without_changes:
-    - name: ca_state_not_allowed
+    - name: {{sls}}_state_not_allowed
 
 {% endif %}

salt/common/cron/common-rotate

@@ -1,2 +1,2 @@
 #!/bin/bash
-logrotate -f /opt/so/conf/log-rotate.conf >/dev/null 2>&1
+/usr/sbin/logrotate -f /opt/so/conf/log-rotate.conf > /dev/null 2>&1

salt/common/files/99-reserved-ports.conf

@@ -0,0 +1 @@
+net.ipv4.ip_local_reserved_ports=55000,57314,47760-47860

salt/common/files/daemon.json

@@ -0,0 +1,12 @@
+{%- set DOCKERRANGE = salt['pillar.get']('docker:range', '172.17.0.0/24') %}
+{%- set DOCKERBIND = salt['pillar.get']('docker:bip', '172.17.0.1/24') %}
+{
+    "registry-mirrors": [ "https://:5000" ],
+    "bip": "{{ DOCKERBIND }}",
+    "default-address-pools": [
+      {
+        "base" : "{{ DOCKERRANGE }}",
+        "size" : 24
+      }
+    ]
+}

salt/common/files/log-rotate.conf

@@ -1,4 +1,6 @@
 {%- set logrotate_conf = salt['pillar.get']('logrotate:conf') %}
+{%- set group_conf = salt['pillar.get']('logrotate:group_conf') %}
+
 
 /opt/so/log/aptcacher-ng/*.log
 /opt/so/log/idstools/*.log
@@ -13,11 +15,21 @@
 /opt/so/log/fleet/*.log
 /opt/so/log/suricata/*.log
 /opt/so/log/mysql/*.log
-/opt/so/log/playbook/*.log
-/opt/so/log/logstash/*.log
-/opt/so/log/filebeat/*.log
 /opt/so/log/telegraf/*.log
 /opt/so/log/redis/*.log
+/opt/so/log/sensoroni/*.log
+/opt/so/log/stenographer/*.log
+/opt/so/log/salt/so-salt-minion-check
+/opt/so/log/salt/minion
+/opt/so/log/salt/master
 {
     {{ logrotate_conf | indent(width=4) }}
 }
+
+# Playbook's log directory needs additional configuration
+# because Playbook requires a more permissive directory
+/opt/so/log/playbook/*.log
+{
+    {{ logrotate_conf | indent(width=4) }}
+    {{ group_conf | indent(width=4) }}
+}

salt/common/files/sensor-rotate.conf

@@ -6,5 +6,17 @@
     nocompress
     create
     sharedscripts
-    endscript
 }
+
+/nsm/strelka/log/strelka.log
+{
+    daily
+    rotate 14
+    missingok
+    copytruncate
+    compress
+    create
+    extension .log
+    dateext
+    dateyesterday
+}

salt/common/files/soversion

@@ -0,0 +1,2 @@
+{%- set VERSION = salt['pillar.get']('global:soversion') -%}
+{{ VERSION }}

salt/common/files/vimrc

@@ -0,0 +1,6 @@
+" Activates filetype detection
+filetype plugin indent on
+
+" Sets .sls files to use YAML syntax highlighting
+autocmd BufNewFile,BufRead *.sls set syntax=yaml
+set number

salt/common/init.sls

@@ -1,9 +1,8 @@
-{% set show_top = salt['state.show_top']() %}
-{% set top_states = show_top.values() | join(', ') %}
-
-{% if 'common' in top_states %}
+{% from 'allowed_states.map.jinja' import allowed_states %}
+{% if sls in allowed_states %}
 
 {% set role = grains.id.split('_') | last %}
+{% from 'elasticsearch/auth.map.jinja' import ELASTICAUTH with context %}
 
 # Remove variables.txt from /tmp - This is temp
 rmvariablesfile:
@@ -32,6 +31,18 @@ soconfperms:
     - gid: 939
     - dir_mode: 770
 
+sostatusconf:
+  file.directory:
+    - name: /opt/so/conf/so-status
+    - uid: 939
+    - gid: 939
+    - dir_mode: 770
+
+so-status.conf:
+  file.touch:
+    - name: /opt/so/conf/so-status/so-status.conf
+    - unless: ls /opt/so/conf/so-status/so-status.conf
+
 sosaltstackperms:
   file.directory:
     - name: /opt/so/saltstack
@@ -39,6 +50,11 @@ sosaltstackperms:
     - gid: 939
     - dir_mode: 770
 
+so_log_perms:
+  file.directory:
+    - name: /opt/so/log
+    - dir_mode: 755
+
 # Create a state directory
 statedir:
   file.directory:
@@ -54,20 +70,12 @@ salttmp:
     - group: 939
     - makedirs: True
 
-# Install epel
-{% if grains['os'] == 'CentOS' %}
-repair_yumdb:
-  cmd.run:
-    - name: 'mv -f /var/lib/rpm/__db* /tmp && yum clean all'
-    - onlyif:
-      - 'yum check-update 2>&1 | grep "Error: rpmdb open failed"'
-
-epel:
-  pkg.installed:
-    - skip_suggestions: True
-    - pkgs:
-      - epel-release
-{% endif %}
+# VIM config
+vimconfig:
+  file.managed:
+    - name: /root/.vimrc
+    - source: salt://common/files/vimrc
+    - replace: False
 
 # Install common packages
 {% if grains['os'] != 'CentOS' %}     
@@ -80,7 +88,6 @@ commonpkgs:
       - ntpdate
       - jq
       - python3-docker
-      - docker-ce
       - curl
       - ca-certificates
       - software-properties-common
@@ -89,17 +96,21 @@ commonpkgs:
       - netcat
       - python3-mysqldb
       - sqlite3
-      - argon2
       - libssl-dev
       - python3-dateutil
       - python3-m2crypto
       - python3-mysqldb
+      - python3-packaging
       - git
+      - vim
+
 heldpackages:
   pkg.installed:
     - pkgs:
-      - containerd.io: 1.2.13-2
-      - docker-ce: 5:19.03.12~3-0~ubuntu-bionic
+      - containerd.io: 1.4.4-1
+      - docker-ce: 5:20.10.5~3-0~ubuntu-bionic
+      - docker-ce-cli: 5:20.10.5~3-0~ubuntu-bionic
+      - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-bionic
     - hold: True
     - update_holds: True
 
@@ -117,7 +128,6 @@ commonpkgs:
       - net-tools
       - curl
       - sqlite
-      - argon2
       - mariadb-devel
       - nmap-ncat
       - python3
@@ -125,17 +135,21 @@ commonpkgs:
       - python36-dateutil
       - python36-m2crypto
       - python36-mysql
+      - python36-packaging
       - yum-utils
       - device-mapper-persistent-data
       - lvm2
       - openssl
       - git
+      - vim-enhanced
 
 heldpackages:
   pkg.installed:
     - pkgs:
-      - containerd.io: 1.2.13-3.2.el7
-      - docker-ce: 3:19.03.12-3.el7
+      - containerd.io: 1.4.4-3.1.el7
+      - docker-ce: 3:20.10.5-3.el7
+      - docker-ce-cli: 1:20.10.5-3.el7
+      - docker-ce-rootless-extras: 20.10.5-3.el7
     - hold: True
     - update_holds: True
 {% endif %}
@@ -154,15 +168,27 @@ alwaysupdated:
 Etc/UTC:
   timezone.system
 
+elastic_curl_config:
+  file.managed:
+    - name: /opt/so/conf/elasticsearch/curl.config
+    - source: salt://elasticsearch/curl.config
+    - mode: 600
+    - show_changes: False
+    - makedirs: True
+
 # Sync some Utilities
 utilsyncscripts:
   file.recurse:
     - name: /usr/sbin
-    - user: 0
-    - group: 0
+    - user: root
+    - group: root
     - file_mode: 755
     - template: jinja
     - source: salt://common/tools/sbin
+    - defaults:
+        ELASTICCURL: 'curl'
+    - context:
+        ELASTICCURL: {{ ELASTICAUTH.elasticcurl }}
 
 {% if role in ['eval', 'standalone', 'sensor', 'heavynode'] %}
 # Add sensor cleanup
@@ -220,7 +246,40 @@ commonlogrotateconf:
     - month: '*'
     - dayweek: '*'
 
+# Create the status directory
+sostatusdir:
+  file.directory:
+    - name: /opt/so/log/sostatus
+    - user: 0
+    - group: 0
+    - makedirs: True
+
+sostatus_log:
+  file.managed:
+    - name: /opt/so/log/sostatus/status.log
+    - mode: 644
+    
+# Install sostatus check cron
+'/usr/sbin/so-status -q; echo $? > /opt/so/log/sostatus/status.log 2>&1':
+  cron.present:
+    - user: root
+    - minute: '*/1'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
+
 {% if role in ['eval', 'manager', 'managersearch', 'standalone'] %}
+# Lock permissions on the backup directory
+backupdir:
+  file.directory:
+    - name: /nsm/backup
+    - user: 0
+    - group: 0
+    - makedirs: True
+    - mode: 700
+  
 # Add config backup
 /usr/sbin/so-config-backup > /dev/null 2>&1:
   cron.present:
@@ -230,17 +289,59 @@ commonlogrotateconf:
     - daymonth: '*'
     - month: '*'
     - dayweek: '*'
+{% else %}
+soversionfile:
+  file.managed:
+    - name: /etc/soversion
+    - source: salt://common/files/soversion
+    - mode: 644
+    - template: jinja
+    
 {% endif %}
 
+# Manager daemon.json
+docker_daemon:
+  file.managed:
+    - source: salt://common/files/daemon.json
+    - name: /etc/docker/daemon.json
+    - template: jinja 
+
 # Make sure Docker is always running
 docker:
   service.running:
     - enable: True
+    - watch:
+      - file: docker_daemon
+
+# Reserve OS ports for Docker proxy in case boot settings are not already applied/present
+# 55000 = Wazuh, 57314 = Strelka, 47760-47860 = Zeek
+dockerapplyports:
+    cmd.run:
+      - name: if [ ! -s /etc/sysctl.d/99-reserved-ports.conf ]; then sysctl -w net.ipv4.ip_local_reserved_ports="55000,57314,47760-47860"; fi
+
+# Reserve OS ports for Docker proxy
+dockerreserveports:
+  file.managed:
+    - source: salt://common/files/99-reserved-ports.conf
+    - name: /etc/sysctl.d/99-reserved-ports.conf
+
+{% if salt['grains.get']('sosmodel', '') %}
+# Install raid check cron
+/usr/sbin/so-raid-status > /dev/null 2>&1:
+  cron.present:
+    - user: root
+    - minute: '*/15'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
+{% endif %}
 
 {% else %}
 
-common_state_not_allowed:
+{{sls}}_state_not_allowed:
   test.fail_without_changes:
-    - name: common_state_not_allowed
+    - name: {{sls}}_state_not_allowed
 
 {% endif %}

salt/common/maps/domainstats.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-domainstats'
-  ]
-} %}

salt/common/maps/eval.map.jinja

@@ -1,20 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-filebeat',
-    'so-nginx',
-    'so-telegraf',
-    'so-dockerregistry',
-    'so-soc',
-    'so-kratos',
-    'so-idstools',
-    'so-elasticsearch',
-    'so-kibana',
-    'so-steno',
-    'so-suricata',
-    'so-zeek',
-    'so-curator',
-    'so-elastalert',
-    'so-soctopus',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/fleet.map.jinja

@@ -1,10 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-mysql',
-    'so-fleet',
-    'so-redis',
-    'so-filebeat',
-    'so-nginx',
-    'so-telegraf'
-  ]
-} %}

salt/common/maps/fleet_manager.map.jinja

@@ -1,7 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-mysql',
-    'so-fleet',
-    'so-redis'
-  ]
-} %}

salt/common/maps/freq.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-freqserver'
-  ]
-} %}

salt/common/maps/grafana.map.jinja

@@ -1,6 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-influxdb',
-    'so-grafana'
-  ]
-} %}

salt/common/maps/heavynode.map.jinja

@@ -1,15 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-redis',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-steno',
-    'so-suricata',
-    'so-wazuh',
-    'so-filebeat',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/helixsensor.map.jinja

@@ -1,12 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-idstools',
-    'so-steno',
-    'so-zeek',
-    'so-redis',
-    'so-logstash',
-    'so-filebeat
-  ]
-} %}

salt/common/maps/hotnode.map.jinja

@@ -1,9 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-     'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-   ]
-} %}

salt/common/maps/import.map.jinja

@@ -1,10 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-filebeat',
-    'so-nginx',
-    'so-soc',
-    'so-kratos',
-    'so-elasticsearch',
-    'so-kibana'
-  ]
-} %}

salt/common/maps/manager.map.jinja

@@ -1,18 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-dockerregistry',
-    'so-nginx',
-    'so-telegraf',
-    'so-soc',
-    'so-kratos',
-    'so-aptcacherng',
-    'so-idstools',
-    'so-redis',
-    'so-elasticsearch',
-    'so-logstash',
-    'so-kibana',
-    'so-elastalert',
-    'so-filebeat',
-    'so-soctopus'
-  ]
-} %}

salt/common/maps/managersearch.map.jinja

@@ -1,18 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-soc',
-    'so-kratos',
-    'so-aptcacherng',
-    'so-idstools',
-    'so-redis',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-kibana',
-    'so-elastalert',
-    'so-filebeat',
-    'so-soctopus'
-  ]
-} %}

salt/common/maps/mdengine.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-zeek'
-  ]
-} %}

salt/common/maps/playbook.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-playbook'
-  ]
-} %}

salt/common/maps/searchnode.map.jinja

@@ -1,10 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-filebeat'
-  ]
-} %}

salt/common/maps/sensor.map.jinja

@@ -1,9 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-telegraf',
-    'so-steno',
-    'so-suricata',
-    'so-filebeat',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/so-status.map.jinja

@@ -1,48 +0,0 @@
-{% set role = grains.id.split('_') | last %}
-{% from 'common/maps/'~ role ~'.map.jinja' import docker with context %}
-
-# Check if the service is enabled and append it's required containers
-# to the list predefined by the role / minion id affix
-{% macro append_containers(pillar_name, k, compare )%}
-  {% if salt['pillar.get'](pillar_name~':'~k, {}) != compare %}
-    {% if k == 'enabled' %}
-      {% set k = pillar_name %}
-    {% endif %}
-    {% from 'common/maps/'~k~'.map.jinja' import docker as d with context %}
-    {% for li in d['containers'] %}
-      {{ docker['containers'].append(li) }}
-    {% endfor %}
-  {% endif %}
-{% endmacro %}
-
-{% set docker = salt['grains.filter_by']({
-  '*_'~role: {
-    'containers': docker['containers']
-  } 
-},grain='id', merge=salt['pillar.get']('docker')) %}
-
-{% if role in ['eval', 'managersearch', 'manager', 'standalone'] %}
-  {{ append_containers('manager', 'grafana', 0) }}
-  {{ append_containers('global', 'fleet_manager', 0) }}
-  {{ append_containers('global', 'wazuh', 0) }}
-  {{ append_containers('manager', 'thehive', 0) }}
-  {{ append_containers('manager', 'playbook', 0) }}
-  {{ append_containers('manager', 'freq', 0) }}
-  {{ append_containers('manager', 'domainstats', 0) }}
-{% endif %}
-
-{% if role in ['eval', 'heavynode', 'sensor', 'standalone'] %}
-  {{ append_containers('strelka', 'enabled', 0) }}
-{% endif %}
-
-{% if role in ['heavynode', 'standalone'] %}
-  {{ append_containers('global', 'mdengine', 'SURICATA') }}
-{% endif %}
-
-{% if role == 'searchnode' %}
-  {{ append_containers('manager', 'wazuh', 0) }}
-{% endif %}
-
-{% if role == 'sensor' %}
-  {{ append_containers('global', 'mdengine', 'SURICATA') }}
-{% endif %}

salt/common/maps/standalone.map.jinja

@@ -1,22 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-soc',
-    'so-kratos',
-    'so-aptcacherng',
-    'so-idstools',
-    'so-redis',
-    'so-logstash',
-    'so-elasticsearch',
-    'so-curator',
-    'so-kibana',
-    'so-elastalert',
-    'so-filebeat',
-    'so-suricata',
-    'so-steno',
-    'so-dockerregistry',
-    'so-soctopus',
-    'so-sensoroni'
-  ]
-} %}

salt/common/maps/strelka.map.jinja

@@ -1,9 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-strelka-coordinator',
-    'so-strelka-gatekeeper',
-    'so-strelka-manager',
-    'so-strelka-frontend',
-    'so-strelka-filestream'
-  ]
-} %}

salt/common/maps/thehive.map.jinja

@@ -1,7 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-thehive',
-    'so-thehive-es',
-    'so-cortex'
-  ]
-} %}

salt/common/maps/warmnode.map.jinja

@@ -1,7 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-nginx',
-    'so-telegraf',
-    'so-elasticsearch'
-  ]
-} %}

salt/common/maps/wazuh.map.jinja

@@ -1,5 +0,0 @@
-{% set docker = {
-  'containers': [
-    'so-wazuh'
-  ]
-} %}

salt/common/scripts/dockernet.sh

@@ -1,8 +0,0 @@
-#!/bin/bash
-
-if [ ! -f /opt/so/state/dockernet.state ]; then
-    docker network create -d bridge so-elastic-net
-    touch /opt/so/state/dockernet.state
-else
-    exit
-fi

salt/common/tools/sbin/so-allow

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-allow-view

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-analyst-install

@@ -84,7 +84,7 @@ while [[ $INSTALL != "yes" ]] && [[ $INSTALL != "no" ]]; do
     echo "##                                       ##"
     echo "##    Installing the Security Onion      ##"
     echo "##   analyst node on this device will    ##"
-    echo "##      make permanenet changes to       ##"
+    echo "##       make permanent changes to       ##"
     echo "##              the system.              ##"
     echo "##                                       ##"
     echo "###########################################"

salt/common/tools/sbin/so-checkin

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-salt-call state.highstate
+salt-call state.highstate -linfo 

salt/common/tools/sbin/so-common

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,30 +15,77 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-IMAGEREPO=securityonion
+DEFAULT_SALT_DIR=/opt/so/saltstack/default
 
 # Check for prerequisites
 if [ "$(id -u)" -ne 0 ]; then
-    echo "This script must be run using sudo!"
-    exit 1
+	echo "This script must be run using sudo!"
+	exit 1
 fi
 
 # Define a banner to separate sections
 banner="========================================================================="
 
-header() {
-	echo
-    printf '%s\n' "$banner" "$*" "$banner"
-}
+add_interface_bond0() {
+	local BNIC=$1
+	if [[ -z $MTU ]]; then
+		local MTU
+		MTU=$(lookup_pillar "mtu" "sensor")
+	fi
+	local nic_error=0
 
-lookup_pillar() {
-    key=$1
-    salt-call --no-color  pillar.get global:${key} --out=newline_values_only
-}
+	# Check if specific offload features are able to be disabled
+	for string in "generic-segmentation-offload" "generic-receive-offload" "tcp-segmentation-offload"; do
+		if ethtool -k "$BNIC" | grep $string | grep -q "on [fixed]"; then
+			echo "The hardware or driver for interface ${BNIC} is not supported, packet capture may not work as expected."
+			((nic_error++))
+			break
+		fi
+	done
 
-lookup_pillar_secret() {
-    key=$1
-    salt-call --no-color  pillar.get secrets:${key} --out=newline_values_only
+	case "$2" in
+		-v|--verbose)
+			local verbose=true
+		;;
+	esac
+
+	for i in rx tx sg tso ufo gso gro lro; do
+		if [[ $verbose == true ]]; then
+			ethtool -K "$BNIC" $i off
+		else
+			ethtool -K "$BNIC" $i off &>/dev/null
+		fi
+	done
+	# Check if the bond slave connection has already been created
+	nmcli -f name,uuid -p con | grep -q "bond0-slave-$BNIC"
+	local found_int=$?
+
+	if [[ $found_int != 0 ]]; then
+		# Create the slave interface and assign it to the bond
+		nmcli con add type ethernet ifname "$BNIC" con-name "bond0-slave-$BNIC" master bond0 -- \
+			ethernet.mtu "$MTU" \
+			connection.autoconnect "yes"
+	else
+		local int_uuid
+		int_uuid=$(nmcli -f name,uuid -p con | sed -n "s/bond0-slave-$BNIC //p" | tr -d ' ')
+
+		nmcli con mod "$int_uuid" \
+			ethernet.mtu "$MTU" \
+			connection.autoconnect "yes"
+	fi
+
+	ip link set dev "$BNIC" arp off multicast off allmulticast off promisc on
+			
+	# Bring the slave interface up
+	if [[ $verbose == true ]]; then
+		nmcli con up "bond0-slave-$BNIC"
+	else
+		nmcli con up "bond0-slave-$BNIC" &>/dev/null
+	fi
+	 
+	if [ "$nic_error" != 0 ]; then
+		return "$nic_error"
+	fi
 }
 
 check_container() {
@@ -47,7 +94,432 @@ check_container() {
 }
 
 check_password() {
-    local password=$1
-    echo "$password" | egrep -v "'|\"|\\\\" > /dev/null 2>&1
-    return $?
-}
+	local password=$1
+	echo "$password" | egrep -v "'|\"|\\$|\\\\" > /dev/null 2>&1
+	return $?
+}
+
+check_elastic_license() {
+
+	[ -n "$TESTING" ] && return
+
+	# See if the user has already accepted the license
+	if [ ! -f /opt/so/state/yeselastic.txt ]; then
+	  elastic_license
+	else
+	  echo "Elastic License has already been accepted"
+	fi  
+}
+
+copy_new_files() {
+  # Copy new files over to the salt dir
+  cd $UPDATE_DIR
+  rsync -a salt $DEFAULT_SALT_DIR/
+  rsync -a pillar $DEFAULT_SALT_DIR/
+  chown -R socore:socore $DEFAULT_SALT_DIR/
+  chmod 755 $DEFAULT_SALT_DIR/pillar/firewall/addfirewall.sh
+  cd /tmp
+}
+
+disable_fastestmirror() {
+	sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
+}
+
+elastic_license() {
+
+read -r -d '' message <<- EOM
+\n
+Starting in Elastic Stack version 7.11, the Elastic Stack binaries are only available under the Elastic License:
+https://securityonion.net/elastic-license
+
+Please review the Elastic License:
+https://www.elastic.co/licensing/elastic-license
+
+Do you agree to the terms of the Elastic License?
+
+If so, type AGREE to accept the Elastic License and continue.  Otherwise, press Enter to exit this program without making any changes.
+EOM
+
+	AGREED=$(whiptail --title "$whiptail_title" --inputbox \
+	"$message" 20 75 3>&1 1>&2 2>&3)
+
+	if [ "${AGREED^^}" = 'AGREE' ]; then
+		mkdir -p /opt/so/state
+		touch /opt/so/state/yeselastic.txt 
+	else
+		echo "Starting in 2.3.40 you must accept the Elastic license if you want to run Security Onion."
+		exit 1
+	fi
+
+}
+
+fail() {
+	msg=$1
+	echo "ERROR: $msg"
+	echo "Exiting."
+	exit 1
+}
+
+get_random_value() {
+	length=${1:-20}
+	head -c 5000 /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w $length | head -n 1
+}
+
+gpg_rpm_import() {
+	if [[ "$OS" == "centos" ]]; then
+	    if [[ "$WHATWOULDYOUSAYYAHDOHERE" == "setup" ]]; then
+	        local RPMKEYSLOC="../salt/repo/client/files/centos/keys"
+	    else
+	        local RPMKEYSLOC="$UPDATE_DIR/salt/repo/client/files/centos/keys"
+	    fi
+	
+	    RPMKEYS=('RPM-GPG-KEY-EPEL-7' 'GPG-KEY-WAZUH' 'docker.pub' 'SALTSTACK-GPG-KEY.pub' 'securityonion.pub')
+
+	    for RPMKEY in "${RPMKEYS[@]}"; do
+    	        rpm --import $RPMKEYSLOC/$RPMKEY
+	        echo "Imported $RPMKEY"
+	    done
+	fi
+}
+
+header() {
+	printf '%s\n' "" "$banner" " $*" "$banner" 
+}
+
+init_monitor() {
+	MONITORNIC=$1
+	
+	if [[ $MONITORNIC == "bond0" ]]; then 
+	  BIFACES=$(lookup_bond_interfaces)
+	else
+	  BIFACES=$MONITORNIC
+	fi
+
+	for DEVICE_IFACE in $BIFACES; do
+	  for i in rx tx sg tso ufo gso gro lro; do
+	    ethtool -K "$DEVICE_IFACE" "$i" off;
+  	  done
+	  ip link set dev "$DEVICE_IFACE" arp off multicast off allmulticast off promisc on
+	done
+}
+
+is_manager_node() {
+	# Check to see if this is a manager node
+	role=$(lookup_role)
+	is_single_node_grid && return 0
+	[ $role == 'manager' ] && return 0
+	[ $role == 'managersearch' ] && return 0
+	[ $role == 'helix' ] && return 0
+	return 1
+}
+
+is_sensor_node() {
+	# Check to see if this is a sensor (forward) node
+	role=$(lookup_role)
+	is_single_node_grid && return 0
+	[ $role == 'sensor' ] && return 0
+	[ $role == 'heavynode' ] && return 0
+	[ $role == 'helix' ] && return 0
+	return 1
+}
+
+is_single_node_grid() {
+	role=$(lookup_role)
+	[ $role == 'eval' ] && return 0
+	[ $role == 'standalone' ] && return 0
+	[ $role == 'import' ] && return 0
+	return 1
+}
+
+lookup_bond_interfaces() {
+	cat /proc/net/bonding/bond0 | grep "Slave Interface:" | sed -e "s/Slave Interface: //g"
+}
+
+lookup_salt_value() {
+	key=$1
+	group=$2
+	kind=$3
+	output=${4:-newline_values_only}
+
+	if [ -z "$kind" ]; then
+		kind=pillar
+	fi
+
+	if [ -n "$group" ]; then
+		group=${group}:
+	fi
+
+	salt-call --no-color  ${kind}.get ${group}${key} --out=${output}
+}
+
+lookup_pillar() {
+	key=$1
+	pillar=$2
+	if [ -z "$pillar" ]; then
+		pillar=global
+	fi
+	lookup_salt_value "$key" "$pillar" "pillar"
+}
+
+lookup_pillar_secret() {
+	lookup_pillar "$1" "secrets"
+}
+
+lookup_grain() {
+	lookup_salt_value "$1" "" "grains"
+}
+
+lookup_role() {
+	id=$(lookup_grain id)
+	pieces=($(echo $id | tr '_' ' '))
+	echo ${pieces[1]}
+}
+
+require_manager() {
+	if is_manager_node; then
+		echo "This is a manager, so we can proceed."
+	else
+		echo "Please run this command on the manager; the manager controls the grid."
+		exit 1
+	fi
+}
+
+retry() {
+	maxAttempts=$1
+	sleepDelay=$2
+	cmd=$3
+	expectedOutput=$4
+	attempt=0
+	local exitcode=0
+	while [[ $attempt -lt $maxAttempts ]]; do			
+		attempt=$((attempt+1))
+		echo "Executing command with retry support: $cmd"
+		output=$(eval "$cmd")
+		exitcode=$?
+		echo "Results: $output ($exitcode)"
+		if [ -n "$expectedOutput" ]; then
+			if [[ "$output" =~ "$expectedOutput" ]]; then
+				return $exitCode
+			else
+				echo "Expected '$expectedOutput' but got '$output'"
+			fi
+		elif [[ $exitcode -eq 0 ]]; then
+			return $exitCode
+		fi
+		echo "Command failed with exit code $exitcode; will retry in $sleepDelay seconds ($attempt / $maxAttempts)..."
+		sleep $sleepDelay
+	done
+	echo "Command continues to fail; giving up."
+	return $exitcode
+}
+
+run_check_net_err() {
+	local cmd=$1
+	local err_msg=${2:-"Unknown error occured, please check /root/$WHATWOULDYOUSAYYAHDOHERE.log for details."} # Really need to rename that variable
+	local no_retry=$3
+
+	local exit_code
+	if [[ -z $no_retry ]]; then
+		retry 5 60 "$cmd"
+		exit_code=$?
+	else
+		eval "$cmd"
+		exit_code=$?
+	fi
+	
+	if [[ $exit_code -ne 0 ]]; then
+		ERR_HANDLED=true
+		[[ -z $no_retry ]] || echo "Command failed with error $exit_code"
+		echo "$err_msg"
+		exit $exit_code
+	fi
+}
+
+set_os() {
+	if [ -f /etc/redhat-release ]; then
+		OS=centos
+	else
+		OS=ubuntu
+	fi
+}
+
+set_minionid() {
+	MINIONID=$(lookup_grain id)
+}
+
+set_palette() {
+	if [ "$OS" == ubuntu ]; then
+	    update-alternatives --set newt-palette /etc/newt/palette.original
+    fi
+}
+
+set_version() {
+	CURRENTVERSION=0.0.0
+	if [ -f /etc/soversion ]; then
+		CURRENTVERSION=$(cat /etc/soversion)
+	fi
+	if [ -z "$VERSION" ]; then
+		if [ -z "$NEWVERSION" ]; then
+			if [ "$CURRENTVERSION" == "0.0.0" ]; then
+				echo "ERROR: Unable to detect Security Onion version; terminating script."
+				exit 1
+			else
+				VERSION=$CURRENTVERSION
+			fi
+		else
+			VERSION="$NEWVERSION"
+		fi
+	fi
+}
+
+valid_cidr() {
+	# Verify there is a backslash in the string
+	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
+	
+	local cidr
+	local ip
+
+	cidr=$(echo "$1" | sed 's/.*\///')
+	ip=$(echo "$1" | sed 's/\/.*//' )
+	
+	if valid_ip4 "$ip"; then
+		[[ $cidr =~ ([0-9]|[1-2][0-9]|3[0-2]) ]] && return 0 || return 1
+	else
+		return 1
+	fi
+}
+
+valid_cidr_list() {
+	local all_valid=0
+
+	IFS="," read -r -a net_arr <<< "$1"
+
+	for net in "${net_arr[@]}"; do
+		valid_cidr "$net" || all_valid=1
+	done
+
+	return $all_valid
+}
+
+valid_dns_list() {
+	local all_valid=0
+
+	IFS="," read -r -a dns_arr <<< "$1"
+
+	for addr in "${dns_arr[@]}"; do
+		valid_ip4 "$addr" || all_valid=1
+	done
+
+	return $all_valid
+}
+
+valid_fqdn() {
+	local fqdn=$1
+
+	echo "$fqdn" | grep -qP '(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{0,62}[a-zA-Z0-9]\.)+[a-zA-Z]{2,63}$)' \
+		&& return 0 \
+		|| return 1
+}
+
+valid_hostname() {
+	local hostname=$1
+
+	[[ $hostname =~ ^[a-zA-Z0-9\-]+$ ]] && [[ $hostname != 'localhost' ]] && return 0 || return 1
+}
+
+valid_ip4() {
+	local ip=$1
+
+	echo "$ip" | grep -qP '^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$' && return 0 || return 1
+}
+
+valid_int() {
+	local num=$1
+	local min=${2:-1}
+	local max=${3:-1000000000}
+
+	[[ $num =~ ^[0-9]*$ ]] && [[ $num -ge $min ]] && [[ $num -le $max ]] && return 0 || return 1
+}
+
+# {% raw %}
+
+valid_proxy() {
+	local proxy=$1
+	local url_prefixes=( 'http://' 'https://' )
+
+	local has_prefix=false
+	for prefix in "${url_prefixes[@]}"; do
+		echo "$proxy" | grep -q "$prefix" && has_prefix=true && proxy=${proxy#"$prefix"} && break
+	done
+	
+	local url_arr
+	mapfile -t url_arr <<< "$(echo "$proxy" | tr ":" "\n")"
+
+	local valid_url=true
+	if ! valid_ip4 "${url_arr[0]}" && ! valid_fqdn "${url_arr[0]}" && ! valid_hostname "${url_arr[0]}"; then
+		valid_url=false
+	fi
+
+	[[ $has_prefix == true ]] && [[ $valid_url == true ]] && return 0 || return 1
+}
+
+valid_ntp_list() {
+	local string=$1
+	local ntp_arr
+	IFS="," read -r -a ntp_arr <<< "$string"
+
+	for ntp in "${ntp_arr[@]}"; do
+		if ! valid_ip4 "$ntp" && ! valid_hostname "$ntp" && ! valid_fqdn "$ntp"; then
+			return 1
+		fi
+	done
+
+	return 0
+}
+
+valid_string() {
+	local str=$1
+	local min_length=${2:-1}
+	local max_length=${3:-64}
+
+	echo "$str" | grep -qP '^\S+$' && [[ ${#str} -ge $min_length ]] && [[ ${#str} -le $max_length ]] && return 0 || return 1
+}
+
+# {% endraw %}
+
+valid_username() {
+	local user=$1
+
+	echo "$user" | grep -qP '^[a-z_]([a-z0-9_-]{0,31}|[a-z0-9_-]{0,30}\$)$' && return 0 || return 1
+}
+
+wait_for_web_response() {
+	url=$1
+	expected=$2
+	maxAttempts=${3:-300}
+	curlcmd=${4:-curl}
+	logfile=/root/wait_for_web_response.log
+	truncate -s 0 "$logfile"
+	attempt=0
+	while [[ $attempt -lt $maxAttempts ]]; do
+		attempt=$((attempt+1))
+		echo "Waiting for value '$expected' at '$url' ($attempt/$maxAttempts)"
+		result=$($curlcmd -ks -L $url)
+		exitcode=$?
+
+		echo "--------------------------------------------------" >> $logfile
+		echo "$(date) - Checking web URL: $url ($attempt/$maxAttempts)" >> $logfile
+		echo "$result" >> $logfile
+		echo "exit code=$exitcode" >> $logfile
+		echo "" >> $logfile
+
+		if [[ $exitcode -eq 0 && "$result" =~ $expected ]]; then
+			echo "Received expected response; proceeding."
+			return 0
+		fi
+		echo "Server is not ready"
+		sleep 1
+	done
+	echo "Server still not ready after $maxAttempts attempts; giving up."
+	return 1
+}

salt/common/tools/sbin/so-config-backup

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -33,12 +33,16 @@ if [ ! -f $BACKUPFILE ]; then
   {%- for LOCATION in BACKUPLOCATIONS %}
   tar -rf $BACKUPFILE {{ LOCATION }}
   {%- endfor %}
+  tar -rf $BACKUPFILE /etc/pki
+  tar -rf $BACKUPFILE /etc/salt
+  tar -rf $BACKUPFILE /opt/so/conf/kratos
 
 fi
 
-# Find oldest backup file and remove it
+# Find oldest backup files and remove them
 NUMBACKUPS=$(find /nsm/backup/ -type f -name "so-config-backup*" | wc -l)
-OLDESTBACKUP=$(find /nsm/backup/ -type f -name "so-config-backup*" | ls -1t | tail -1)
-if [ "$NUMBACKUPS" -gt "$MAXBACKUPS" ]; then
-  rm -f /nsm/backup/$OLDESTBACKUP
-fi
+while [ "$NUMBACKUPS" -gt "$MAXBACKUPS" ]; do
+  OLDESTBACKUP=$(find /nsm/backup/ -type f -name "so-config-backup*" -type f -printf '%T+ %p\n' | sort | head -n 1 | awk -F" " '{print $2}')
+  rm -f $OLDESTBACKUP
+  NUMBACKUPS=$(find /nsm/backup/ -type f -name "so-config-backup*" | wc -l)
+done

salt/common/tools/sbin/so-cortex-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-user-add

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -30,8 +30,8 @@ fi
 
 USER=$1
 
-CORTEX_KEY=$(lookup_pillar cortexkey)
-CORTEX_IP=$(lookup_pillar managerip)
+CORTEX_KEY=$(lookup_pillar cortexorguserkey)
+CORTEX_API_URL="$(lookup_pillar url_base)/cortex/api"
 CORTEX_ORG_NAME=$(lookup_pillar cortexorgname)
 CORTEX_USER=$USER
 
@@ -43,7 +43,7 @@ fi
 read -rs CORTEX_PASS
 
 # Create new user in Cortex
-resp=$(curl -sk -XPOST -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" "https://$CORTEX_IP/cortex/api/user" -d "{\"name\": \"$CORTEX_USER\",\"roles\": [\"read\",\"analyze\",\"orgadmin\"],\"organization\": \"$CORTEX_ORG_NAME\",\"login\": \"$CORTEX_USER\",\"password\" : \"$CORTEX_PASS\" }")
+resp=$(curl -sk -XPOST -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" -L "https://$CORTEX_API_URL/user" -d "{\"name\": \"$CORTEX_USER\",\"roles\": [\"read\",\"analyze\",\"orgadmin\"],\"organization\": \"$CORTEX_ORG_NAME\",\"login\": \"$CORTEX_USER\",\"password\" : \"$CORTEX_PASS\" }")
 if [[ "$resp" =~ \"status\":\"Ok\" ]]; then
     echo "Successfully added user to Cortex."
 else

salt/common/tools/sbin/so-cortex-user-enable

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -30,8 +30,8 @@ fi
 
 USER=$1
 
-CORTEX_KEY=$(lookup_pillar cortexkey)
-CORTEX_IP=$(lookup_pillar managerip)
+CORTEX_KEY=$(lookup_pillar cortexorguserkey)
+CORTEX_API_URL="$(lookup_pillar url_base)/cortex/api"
 CORTEX_USER=$USER
 
 case "${2^^}" in
@@ -46,7 +46,7 @@ case "${2^^}" in
 		;;
 esac
 
-resp=$(curl -sk -XPATCH -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" "https://$CORTEX_IP/cortex/api/user/${CORTEX_USER}" -d "{\"status\":\"${CORTEX_STATUS}\" }")
+resp=$(curl -sk -XPATCH -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" -L "https://$CORTEX_API_URL/user/${CORTEX_USER}" -d "{\"status\":\"${CORTEX_STATUS}\" }")
 if [[ "$resp" =~ \"status\":\"Locked\" || "$resp" =~ \"status\":\"Ok\" ]]; then
     echo "Successfully updated user in Cortex."
 else

salt/common/tools/sbin/so-curator-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-curator-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-curator-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-docker-prune

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys, argparse, re, docker
+from packaging.version import Version, InvalidVersion
+from itertools import groupby, chain
+
+
+def get_image_name(string) -> str:
+  return ':'.join(string.split(':')[:-1])
+
+
+def get_so_image_basename(string) -> str:
+  return get_image_name(string).split('/so-')[-1]
+
+
+def get_image_version(string) -> str:
+  ver = string.split(':')[-1]
+  if ver == 'latest':
+    # Version doesn't like "latest", so use a high semver
+    return '99999.9.9'
+  else:
+    try:
+      Version(ver)
+    except InvalidVersion:
+      # Also return a very high semver for any version 
+      # with a dash in it since it will likely be a dev version of some kind
+      if '-' in ver:
+        return '999999.9.9'
+    return ver
+
+
+def main(quiet):
+  client = docker.from_env()
+
+  # Prune old/stopped containers
+  if not quiet: print('Pruning old containers')
+  client.containers.prune()
+
+  image_list = client.images.list(filters={ 'dangling': False })
+
+  # Map list of image objects to flattened list of tags (format: "name:version")
+  tag_list = list(chain.from_iterable(list(map(lambda x: x.attrs.get('RepoTags'), image_list))))
+
+  # Filter to only SO images (base name begins with "so-")
+  tag_list = list(filter(lambda x: re.match(r'^.*\/so-[^\/]*$', get_image_name(x)), tag_list))
+
+  # Group tags into lists by base name (sort by same projection first)
+  tag_list.sort(key=lambda x: get_so_image_basename(x))
+  grouped_tag_lists = [ list(it) for _, it in groupby(tag_list, lambda x: get_so_image_basename(x)) ]
+
+  no_prunable = True
+  for t_list in grouped_tag_lists:
+    try:
+      # Group tags by version, in case multiple images exist with the same version string
+      t_list.sort(key=lambda x: Version(get_image_version(x)), reverse=True)
+      grouped_t_list = [ list(it) for _,it in groupby(t_list, lambda x: get_image_version(x)) ]
+
+      # Keep the 2 most current version groups
+      if len(grouped_t_list) <= 2:
+        continue
+      else:
+        no_prunable = False
+        for group in grouped_t_list[2:]:
+          for tag in group:
+            if not quiet: print(f'Removing image {tag}')
+            try:
+              client.images.remove(tag, force=True)
+            except docker.errors.ClientError as e:
+              print(f'Could not remove image {tag}, continuing...')
+    except (docker.errors.APIError, InvalidVersion) as e:
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+    except Exception as e:
+      print('Unhandled exception occurred:')
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+
+  if no_prunable and not quiet:
+    print('No Security Onion images to prune')
+
+
+if __name__ == "__main__":
+  main_parser = argparse.ArgumentParser(add_help=False)
+  main_parser.add_argument('-q', '--quiet', action='store_const', const=True, required=False)
+  args = main_parser.parse_args(sys.argv[1:])
+ 
+  main(args.quiet)

salt/common/tools/sbin/so-docker-refresh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -16,96 +16,7 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 . /usr/sbin/so-common
+. /usr/sbin/so-image-common
 
-manager_check() {
-  # Check to see if this is a manager
-  MANAGERCHECK=$(cat /etc/salt/grains | grep role | awk '{print $2}')
-  if [ $MANAGERCHECK == 'so-eval' ] || [ $MANAGERCHECK == 'so-manager' ] || [ $MANAGERCHECK == 'so-managersearch' ] || [ $MANAGERCHECK == 'so-standalone' ] || [ $MANAGERCHECK == 'so-helix' ]; then
-    echo "This is a manager. We can proceed"
-  else
-    echo "Please run soup on the manager. The manager controls all updates."
-    exit 1
-  fi
-}
-
-update_docker_containers() {
-  
-  # Download the containers from the interwebs
-  for i in "${TRUSTED_CONTAINERS[@]}"
-  do
-    # Pull down the trusted docker image
-    echo "Downloading $i"
-    docker pull --disable-content-trust=false docker.io/$IMAGEREPO/$i
-    # Tag it with the new registry destination
-    docker tag $IMAGEREPO/$i $HOSTNAME:5000/$IMAGEREPO/$i
-    docker push $HOSTNAME:5000/$IMAGEREPO/$i
-  done
-  
-}
-
-version_check() {
-  if [ -f /etc/soversion ]; then
-    VERSION=$(cat /etc/soversion)
-  else
-    echo "Unable to detect version. I will now terminate."
-    exit 1
-  fi
-}
-
-manager_check
-version_check
-
-# Use the hostname
-HOSTNAME=$(hostname)
-# List all the containers
-if [ $MANAGERCHECK != 'so-helix' ]; then
-    TRUSTED_CONTAINERS=( \
-    "so-acng:$VERSION" \
-    "so-thehive-cortex:$VERSION" \
-    "so-curator:$VERSION" \
-    "so-domainstats:$VERSION" \
-    "so-elastalert:$VERSION" \
-    "so-elasticsearch:$VERSION" \
-    "so-filebeat:$VERSION" \
-    "so-fleet:$VERSION" \
-    "so-fleet-launcher:$VERSION" \
-    "so-freqserver:$VERSION" \
-    "so-grafana:$VERSION" \
-    "so-idstools:$VERSION" \
-    "so-influxdb:$VERSION" \
-    "so-kibana:$VERSION" \
-    "so-kratos:$VERSION" \
-    "so-logstash:$VERSION" \
-    "so-minio:$VERSION" \
-    "so-mysql:$VERSION" \
-    "so-nginx:$VERSION" \
-    "so-pcaptools:$VERSION" \
-    "so-playbook:$VERSION" \
-    "so-redis:$VERSION" \
-    "so-soc:$VERSION" \
-    "so-soctopus:$VERSION" \
-    "so-steno:$VERSION" \
-    "so-strelka-frontend:$VERSION" \
-		"so-strelka-manager:$VERSION" \
-		"so-strelka-backend:$VERSION" \
-		"so-strelka-filestream:$VERSION" \
-    "so-suricata:$VERSION" \
-    "so-telegraf:$VERSION" \
-    "so-thehive:$VERSION" \
-    "so-thehive-es:$VERSION" \
-    "so-wazuh:$VERSION" \
-    "so-zeek:$VERSION" )
-  else
-    TRUSTED_CONTAINERS=( \
-    "so-filebeat:$VERSION" \
-    "so-idstools:$VERSION" \
-    "so-logstash:$VERSION" \
-    "so-nginx:$VERSION" \
-    "so-redis:$VERSION" \
-    "so-steno:$VERSION" \
-    "so-suricata:$VERSION" \
-    "so-telegraf:$VERSION" \
-    "so-zeek:$VERSION" )
-  fi
-
-update_docker_containers
+require_manager
+update_docker_containers "refresh"

salt/common/tools/sbin/so-elastalert-create

@@ -145,9 +145,9 @@ EOF
    rulename=$(echo ${raw_rulename,,} | sed 's/ /_/g')
 
 cat << EOF >> "$rulename.yaml"
-# Elasticsearch Host
-es_host: elasticsearch
-es_port: 9200
+# Elasticsearch Host Override (optional)
+# es_host: elasticsearch
+# es_port: 9200
 
 # (Required)
 # Rule name, must be unique

salt/common/tools/sbin/so-elastalert-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-test

@@ -19,8 +19,7 @@
 #
 # Purpose:  This script will allow you to test your elastalert rule without entering the Docker container.
 
-. /usr/sbin/so-elastic-common
-
+HOST_RULE_DIR=/opt/so/rules/elastalert
 OPTIONS=""
 SKIP=0
 RESULTS_TO_LOG="n"
@@ -29,114 +28,109 @@ FILE_SAVE_LOCATION=""
 
 usage()
 {
-cat <<EOF
+	cat <<EOF
 
 Test Elastalert Rule
   Options:
-  -h         		This message
-  -a         		Trigger real alerts instead of the debug alert
-  -l <path_to_file>     Write results to specified log file
-  -o '<options>'	Specify Elastalert options ( Ex. --schema-only , --count-only, --days N ) 
-  -r <rule_name>        Specify path/name of rule to test
+  -h                This message
+  -a                Trigger real alerts instead of the debug alert
+  -l <path_to_file> Write results to specified log file
+  -o '<options>'    Specify Elastalert options ( Ex. --schema-only , --count-only, --days N ) 
+  -r <rule_name>    Specify filename of rule to test (must exist in $HOST_RULE_DIR; do not include path)
 
 EOF
 }
 
 while getopts "hal:o:r:" OPTION
 do
-        case $OPTION in
-                h)
-                        usage
-                        exit 0
-                        ;;
-                a)
-                        OPTIONS="--alert"
-                        ;;
-                l)
-                        RESULTS_TO_LOG="y"
-                        FILE_SAVE_LOCATION=$OPTARG
-                        ;;
-                
-                o)
-			OPTIONS=$OPTARG
-                        ;;  
-                
-                r)
-                        RULE_NAME=$OPTARG
-                        SKIP=1
-                        ;;
-                *)
-                        usage
-                        exit 0
-                        ;;
-        esac
+	case $OPTION in
+	h)
+		usage
+		exit 0
+		;;
+	a)
+		OPTIONS="--alert"
+		;;
+	l)
+		RESULTS_TO_LOG="y"
+		FILE_SAVE_LOCATION=$OPTARG
+		;;		
+	o)
+		OPTIONS=$OPTARG
+		;;  	
+	r)
+		RULE_NAME=$OPTARG
+		SKIP=1
+		;;
+	*)
+		usage
+		exit 0
+		;;
+	esac
 done
 
 docker_exec(){
-	if [ ${RESULTS_TO_LOG,,} = "y" ] ; then
-        	docker exec -it so-elastalert bash -c "elastalert-test-rule $RULE_NAME $OPTIONS" > $FILE_SAVE_LOCATION
+	CMD="docker exec -it so-elastalert elastalert-test-rule /opt/elastalert/rules/$RULE_NAME --config /opt/config/elastalert_config.yaml $OPTIONS"
+	if [ "${RESULTS_TO_LOG,,}" = "y" ] ; then
+		$CMD > "$FILE_SAVE_LOCATION"
 	else
-        	docker exec -it so-elastalert bash -c "elastalert-test-rule $RULE_NAME $OPTIONS"
+		$CMD
 	fi	
 }
 
 rule_prompt(){
-	CURRENT_RULES=$(find /opt/so/rules/elastalert -name "*.yaml")
-        echo
-        echo "This script will allow you to test an Elastalert rule."
-        echo
-        echo "Below is a list of active Elastalert rules:"
-        echo
+	CURRENT_RULES=$(cd "$HOST_RULE_DIR" && find . -type f \( -name "*.yaml" -o -name "*.yml" \) | sed -e 's/^\.\///')
+	if [ -z "$CURRENT_RULES" ]; then
+		echo "There are no rules available to test. Rule files must be placed in the $HOST_RULE_DIR directory."
+		exit 1
+	fi
+	echo
+	echo "This script will allow you to test an Elastalert rule."
+	echo
+	echo "Below is a list of available Elastalert rules:"
+	echo
 	echo "-----------------------------------"
-        echo
-        echo "$CURRENT_RULES"
-        echo
+	echo
+	echo "$CURRENT_RULES"
+	echo
 	echo "-----------------------------------"
-	echo 
-        echo "Note: To test a rule it must be accessible by the Elastalert Docker container."
-        echo
-        echo "Make sure to swap the local path (/opt/so/rules/elastalert/) for the docker path (/etc/elastalert/rules/)"
-        echo "Example: /opt/so/rules/elastalert/nids2hive.yaml would be /etc/elastalert/rules/nids2hive.yaml"
-        echo
-        while [ -z $RULE_NAME ]; do
-		echo "Please enter the file path and rule name you want to test."
-		read -e RULE_NAME
+	echo
+	while [ -z "$RULE_NAME" ]; do
+		read -p "Choose a rule to test from the list above (must be typed exactly as shown above): " -e RULE_NAME
 	done
 }
 
 log_save_prompt(){
 	RESULTS_TO_LOG=""
-        while [ -z $RESULTS_TO_LOG ]; do
-		echo "The results can be rather long.  Would you like to write the results to a file? (Y/N)"
-		read RESULTS_TO_LOG
-	done
+	read -p "The results can be rather long.  Would you like to write the results to a file? (y/N) " -e RESULTS_TO_LOG
 }
 
 log_path_prompt(){
-        while [ -z $FILE_SAVE_LOCATION ]; do
-		echo "Please enter the file path and file name."
-		read -e FILE_SAVE_LOCATION
-        done
+		while [ -z "$FILE_SAVE_LOCATION" ]; do
+			read -p "Please enter the log file path and file name: " -e FILE_SAVE_LOCATION
+		done
 		echo "Depending on the rule this may take a while."
 }
 
 if [ $SKIP -eq 0 ]; then
 	rule_prompt
 	log_save_prompt
-        if [ ${RESULTS_TO_LOG,,} = "y" ] ; then
-        	log_path_prompt
-        fi
+	if [ "${RESULTS_TO_LOG,,}" = "y" ] ; then
+		log_path_prompt
+	fi
 fi
 
+echo 
+
 docker_exec
-
-if [ $? -eq 0 ]; then
-	echo "Test completed successfully!"
-else
-	echo "Something went wrong..."
-fi
+RESULT=$?
 
 echo
 
+if [ $RESULT -eq 0 ]; then
+	echo "Test completed successfully!"
+else
+	echo "Test failed."
+fi
 
-
+echo

salt/common/tools/sbin/so-elastic-auth

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if [ -f "/usr/sbin/so-common" ]; then
+  . /usr/sbin/so-common
+fi
+
+ES_AUTH_PILLAR=${ELASTIC_AUTH_PILLAR:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+ES_USERS_FILE=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+
+authEnable=$1
+
+if ! grep -q "enabled: " "$ES_AUTH_PILLAR"; then
+  echo "Elastic auth pillar file is invalid. Unable to proceed."
+  exit 1
+fi
+
+function restart() {
+  if [[ -z "$ELASTIC_AUTH_SKIP_HIGHSTATE" ]]; then
+    echo "Elasticsearch on all affected minions will now be stopped and then restarted..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' cmd.run so-elastic-stop queue=True
+    echo "Applying highstate to all affected minions..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.highstate queue=True
+  fi
+}
+
+if [[ "$authEnable" == "true" ]]; then
+  if grep -q "enabled: False" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: False/enabled: True/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now enabled."
+    if grep -q "argon" "$ES_USERS_FILE"; then
+      echo ""
+      echo "IMPORTANT: The following users will need to change their password, after logging into SOC, in order to access Kibana:"
+      grep argon "$ES_USERS_FILE" | cut -d ":" -f 1
+    fi
+  else
+    echo "Auth is already enabled."
+  fi
+elif [[ "$authEnable" == "false" ]]; then
+  if grep -q "enabled: True" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: True/enabled: False/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now disabled."
+  else
+    echo "Auth is already disabled."
+  fi
+else
+  echo "Usage: $0 <true|false>"
+  echo ""
+  echo "Toggles Elastic authentication. Elasticsearch will be restarted on each affected minion."
+  echo ""
+fi

salt/common/tools/sbin/so-elastic-clear

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -50,11 +50,7 @@ done
 if [ $SKIP -ne 1 ]; then
         # List indices
         echo 
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -k https://{{ NODEIP }}:9200/_cat/indices?v
-        {% else %}
-        curl {{ NODEIP }}:9200/_cat/indices?v
-        {% endif %}
+        {{ ELASTICCURL }} -k -L https://{{ NODEIP }}:9200/_cat/indices?v
         echo
         # Inform user we are about to delete all data
         echo
@@ -93,18 +89,10 @@ fi
 # Delete data
 echo "Deleting data..."
 
-{% if grains['role'] in ['so-node','so-heavynode'] %}
-INDXS=$(curl -s -XGET -k https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
-{% else %}
-INDXS=$(curl -s -XGET {{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
-{% endif %}
+INDXS=$({{ ELASTICCURL }} -s -XGET -k -L https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
 for INDX in ${INDXS}
 do
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -XDELETE -k https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
-    {% else %}
-    curl -XDELETE "{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
-    {% endif %}
+    {{ ELASTICCURL }} -XDELETE -k -L https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
 done
 
 #Start Logstash/Filebeat

salt/common/tools/sbin/so-elastic-diagnose

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-restart

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-import']%}
+/usr/sbin/so-restart elasticsearch $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone', 'so-import']%}
+/usr/sbin/so-restart kibana $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-restart logstash $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-sensor']%}
+/usr/sbin/so-restart filebeat $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-restart curator $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone']%}
+/usr/sbin/so-restart elastalert $1
+{%- endif %}

salt/common/tools/sbin/so-elastic-start

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-import']%}
+/usr/sbin/so-start elasticsearch $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone', 'so-import']%}
+/usr/sbin/so-start kibana $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-start logstash $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-sensor']%}
+/usr/sbin/so-start filebeat $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-start curator $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone']%}
+/usr/sbin/so-start elastalert $1
+{%- endif %}

salt/common/tools/sbin/so-elastic-stop

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-import']%}
+/usr/sbin/so-stop elasticsearch $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval', 'so-manager', 'so-managersearch', 'so-standalone', 'so-import']%}
+/usr/sbin/so-stop kibana $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-stop logstash $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node', 'so-sensor']%}
+/usr/sbin/so-stop filebeat $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-manager', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-node']%}
+/usr/sbin/so-stop curator $1
+{%- endif %}
+
+{%- if grains['role'] in ['so-eval','so-manager', 'so-managersearch', 'so-standalone']%}
+/usr/sbin/so-stop elastalert $1
+{%- endif %}

salt/common/tools/sbin/so-elasticsearch-indices-list

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/indices?pretty

salt/common/tools/sbin/so-elasticsearch-indices-rw

@@ -1,7 +1,7 @@
 #!/bin/bash
 #
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,6 +21,5 @@ THEHIVEESPORT=9400
 
 echo "Removing read only attributes for indices..."
 echo
-for p in $ESPORT $THEHIVEESPORT; do
-   curl -XPUT -H "Content-Type: application/json" http://$IP:$p/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
-done
+{{ ELASTICCURL }} -s -k -XPUT -H "Content-Type: application/json" -L https://$IP:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
+{{ ELASTICCURL }} -XPUT -H "Content-Type: application/json" -L http://$IP:9400/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;

salt/common/tools/sbin/so-elasticsearch-pipeline-stats

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -19,15 +19,7 @@
 . /usr/sbin/so-common
 
 if [ "$1" == "" ]; then
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -s -k https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
-        {% else %}
-        curl -s {{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
-        {% endif %}
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
 else
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -s -k https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
-        {% else %}
-        curl -s {{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
-        {% endif %}
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
 fi

salt/common/tools/sbin/so-elasticsearch-pipeline-view

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq .
+else
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq .[]
+fi

salt/common/tools/sbin/so-elasticsearch-pipelines-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,15 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
-    {% else %}
-    curl -s {{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
 else
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
-    {% else %}
-    curl -s {{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-query

@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+. /usr/sbin/so-common
+
+if [[ $# -lt 1 ]]; then
+	echo "Submit a cURL request to the local Security Onion Elasticsearch host."
+	echo ""
+	echo "Usage: $0 <PATH> [ARGS,...]"
+	echo ""
+  echo "Where "
+  echo "   PATH represents the elastic function being requested."
+  echo "   ARGS is used to specify additional, optional curl parameters."
+  echo ""
+  echo "Examples:"
+  echo "   $0 /"
+  echo "   $0 '*:so-*/_search' -d '{\"query\": {\"match_all\": {}},\"size\": 1}' | jq"
+  exit 1
+fi
+
+QUERYPATH=$1
+shift
+
+{{ ELASTICCURL }} -s -k -L -H "Content-Type: application/json" "https://localhost:9200/${QUERYPATH}" "$@"

salt/common/tools/sbin/so-elasticsearch-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-shards-list

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/shards?pretty

salt/common/tools/sbin/so-elasticsearch-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-template-remove

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L -XDELETE https://{{ NODEIP }}:9200/_template/$1

salt/common/tools/sbin/so-elasticsearch-template-view

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/* | jq .
+else
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq .
+fi

salt/common/tools/sbin/so-elasticsearch-templates-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,15 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k https://{{ NODEIP }}:9200/_template/* | jq 'keys'
-    {% else %}
-    curl -s {{ NODEIP }}:9200/_template/* | jq 'keys'
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/* | jq 'keys'
 else
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k https://{{ NODEIP }}:9200/_template/$1 | jq
-    {% else %}
-    curl -s {{ NODEIP }}:9200/_template/$1 | jq
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-templates-load

@@ -1,8 +1,5 @@
-{%- set mainint = salt['pillar.get']('host:mainint') %}
-{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
-
 #!/bin/bash
-# Copyright 2014,2015,2016,2017,2018,2019 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,6 +14,9 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
 default_conf_dir=/opt/so/conf
 ELASTICSEARCH_HOST="{{ MYIP }}"
 ELASTICSEARCH_PORT=9200
@@ -30,11 +30,7 @@ echo -n "Waiting for ElasticSearch..."
 COUNT=0
 ELASTICSEARCH_CONNECTED="no"
 while [[ "$COUNT" -le 240 ]]; do
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -k --output /dev/null --silent --head --fail https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
-    {% else %}
-	curl --output /dev/null --silent --head --fail http://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
-	{% endif %}
+    {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
 	if [ $? -eq 0 ]; then
 		ELASTICSEARCH_CONNECTED="yes"
 		echo "connected!"
@@ -55,11 +51,7 @@ cd ${ELASTICSEARCH_TEMPLATES}
 
 
 echo "Loading templates..."
-{% if grains['role'] in ['so-node','so-heavynode'] %}
-for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; curl -k ${ELASTICSEARCH_AUTH} -s -XPUT https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
-{% else %}
-for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; curl ${ELASTICSEARCH_AUTH} -s -XPUT http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
-{% endif %}
+for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; {{ ELASTICCURL }} -k ${ELASTICSEARCH_AUTH} -s -XPUT -L https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
 echo
 
 cd - >/dev/null

salt/common/tools/sbin/so-elasticsearch-wait

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+. /usr/sbin/so-common
+
+wait_for_web_response "https://localhost:9200/_cat/indices/.kibana*" "green open" 300 "{{ ELASTICCURL }}"

salt/common/tools/sbin/so-features-enable

@@ -1,72 +0,0 @@
-#!/bin/bash
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
-
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License as published by
-#    the Free Software Foundation, either version 3 of the License, or
-#    (at your option) any later version.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-. /usr/sbin/so-common
-local_salt_dir=/opt/so/saltstack/local
-
-cat << EOF
-This program will switch from the open source version of the Elastic Stack to the Features version licensed under the Elastic license.
-If you proceed, then we will download new Docker images and restart services.
-
-Please review the Elastic license:
-https://raw.githubusercontent.com/elastic/elasticsearch/master/licenses/ELASTIC-LICENSE.txt
-
-Please also note that, if you have a distributed deployment and continue with this change, Elastic traffic between nodes will change from encrypted to cleartext!
-(We expect to support Elastic Features Security at some point in the future.)
-
-Do you agree to the terms of the Elastic license and understand the note about encryption?
-
-If so, type AGREE to accept the Elastic license and continue.  Otherwise, just press Enter to exit this program without making any changes.
-EOF
-
-read INPUT
-if [ "$INPUT" != "AGREE" ]; then
-        exit
-fi
-
-echo "Please wait while switching to Elastic Features."
-
-manager_check() {
-  # Check to see if this is a manager
-  MANAGERCHECK=$(cat /etc/salt/grains | grep role | awk '{print $2}')
-  if [[ "$MANAGERCHECK" =~ ^('so-eval'|'so-manager'|'so-standalone'|'so-managersearch')$ ]]; then
-    echo "This is a manager. We can proceed"
-  else
-    echo "Please run so-features-enable on the manager."
-    exit 0
-  fi
-}
-
-manager_check
-VERSION=$(lookup_pillar soversion) 
-# Modify global.sls to enable Features
-sed -i 's/features: False/features: True/' $local_salt_dir/pillar/global.sls
-SUFFIX="-features"
-TRUSTED_CONTAINERS=( \
-  "so-elasticsearch:$VERSION$SUFFIX" \
-  "so-filebeat:$VERSION$SUFFIX" \
-  "so-kibana:$VERSION$SUFFIX" \
-  "so-logstash:$VERSION$SUFFIX" )
-
-for i in "${TRUSTED_CONTAINERS[@]}"
-do
-    # Pull down the trusted docker image
-    echo "Downloading $i"
-    docker pull --disable-content-trust=false docker.io/$IMAGEREPO/$i
-    # Tag it with the new registry destination
-    docker tag $IMAGEREPO/$i $HOSTNAME:5000/$IMAGEREPO/$i
-    docker push $HOSTNAME:5000/$IMAGEREPO/$i
-done

salt/common/tools/sbin/so-filebeat-module-setup

@@ -0,0 +1,67 @@
+#!/bin/bash
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
+default_conf_dir=/opt/so/conf
+ELASTICSEARCH_HOST="{{ MYIP }}"
+ELASTICSEARCH_PORT=9200
+#ELASTICSEARCH_AUTH=""
+
+# Define a default directory to load pipelines from
+FB_MODULE_YML="/usr/share/filebeat/module-setup.yml"
+
+
+# Wait for ElasticSearch to initialize
+echo -n "Waiting for ElasticSearch..."
+COUNT=0
+ELASTICSEARCH_CONNECTED="no"
+while [[ "$COUNT" -le 240 ]]; do
+        {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+        if [ $? -eq 0 ]; then
+                ELASTICSEARCH_CONNECTED="yes"
+                echo "connected!"
+                break
+        else
+                ((COUNT+=1))
+                sleep 1
+                echo -n "."
+        fi
+done
+if [ "$ELASTICSEARCH_CONNECTED" == "no" ]; then
+        echo
+        echo -e "Connection attempt timed out.  Unable to connect to ElasticSearch.  \nPlease try: \n  -checking log(s) in /var/log/elasticsearch/\n  -running 'sudo docker ps' \n  -running 'sudo so-elastic-restart'"
+        echo
+fi
+echo "Testing to see if the pipelines are already applied"
+ESVER=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT" |jq .version.number |tr -d \")
+PIPELINES=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"/_ingest/pipeline/filebeat-$ESVER-suricata-eve-pipeline | jq . | wc -c)
+
+if [[ "$PIPELINES" -lt 5 ]]; then
+  echo "Setting up ingest pipeline(s)"
+
+  for MODULE in activemq apache auditd aws azure barracuda bluecoat cef checkpoint cisco coredns crowdstrike cyberark cylance elasticsearch envoyproxy f5 fortinet gcp google_workspace googlecloud gsuite haproxy ibmmq icinga iis imperva infoblox iptables juniper kafka kibana logstash microsoft misp mongodb mssql mysql nats netscout nginx o365 okta osquery panw postgresql rabbitmq radware redis santa snort snyk sonicwall sophos squid suricata system tomcat traefik zeek zscaler
+  do
+    echo "Loading $MODULE"
+    docker exec -i so-filebeat filebeat setup modules -pipelines -modules $MODULE -c $FB_MODULE_YML
+    sleep 2
+  done
+else
+  exit 0
+fi
+
+

salt/common/tools/sbin/so-filebeat-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by